Hero Image

Let's get Apache Kafka running!

Aloha people,

In this post, I will explain what you need to do to set up Apache Kafka locally with Docker so that you can list some topics. The project I am currently working on requires me to learn about Apache Kafka, and so I went to Udemy, picked the first high-ranking course “Apache Kafka Series - Learn Apache Kafka for Beginners v2” and started learning. The way the course teaches me to set up Kafka locally seems to be outdated. I don't understand why anyone would not go for Docker to teach people on how to set up stuff. The first of many problems arose when I tried to start Apache Kafka, as the port it tried to connect to was already taken. I changed the port number it tries to use just to get another exception that it couldn't connect to zookeeper. Truth to be told, I am not going to figure out what I need to do so that these applications work properly.

That's the moment I switch to a docker container, but that also had some bumps along the way, but there the issue sat in front of the keyboard. I went for the https://github.com/bitnami/bitnami-docker-kafka image, which I found out some minutes later, was the right decision. What you want to do is to download the docker-compose.yaml as advertised:

$ curl -sSL https://raw.githubusercontent.com/bitnami/bitnami-docker-kafka/master/docker-compose.yml > docker-compose.yml

Now before you start the containers with docker-compose up -d we need to configure the docker-compose.yaml. Otherwise, the call kafka-topics --bootstrap-server localhost:9092 --list does not work.
The difficulty is that the Kafka CLI tool cannot find the Kafka standard service endpoint 9092 on your local machine. I found the solution for the network issues on stackoverflow which is much better explained by the same author on his blog post.

Under macOS install the Apache Kafka CLI tool with brew install kafka

The docker-compose.yaml needs to configure as described in this section. Your yaml file should look like this:

version: "2"

    image: docker.io/bitnami/zookeeper:3.7
      - "2181:2181"
      - "zookeeper_data:/bitnami"
    image: docker.io/bitnami/kafka:3
      - "9093:9093"
      - "kafka_data:/bitnami"
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093
      - zookeeper

    driver: local
    driver: local

Now you can access the running instance under macOS with the CLI like this kafka-topics --bootstrap-server localhost:9093 --list.

I had some fun figuring this out, and I hope this post will save you some time.