Let's get Apache Kafka running!

Aloha people,

In this post, I will explain what you need to do to set up Apache Kafka locally with Docker so that you can list some topics. The project I am currently working on requires me to learn about Apache Kafka, and so I went to Udemy, picked the first high-ranking course “Apache Kafka Series - Learn Apache Kafka for Beginners v2” and started learning. The way the course teaches me to set up Kafka locally seems to be outdated. I don't understand why anyone would not go for Docker to teach people on how to set up stuff. The first of many problems arose when I tried to start Apache Kafka, as the port it tried to connect to was already taken. I changed the port number it tries to use just to get another exception that it couldn't connect to zookeeper. Truth to be told, I am not going to figure out what I need to do so that these applications work properly.

That's the moment I switch to a docker container, but that also had some bumps along the way, but there the issue sat in front of the keyboard. I went for the https://github.com/bitnami/bitnami-docker-kafka image, which I found out some minutes later, was the right decision. What you want to do is to download the docker-compose.yaml as advertised:

$ curl -sSL https://raw.githubusercontent.com/bitnami/bitnami-docker-kafka/master/docker-compose.yml > docker-compose.yml

Now before you start the containers with docker-compose up -d we need to configure the docker-compose.yaml. Otherwise, the call kafka-topics --bootstrap-server localhost:9092 --list does not work.
The difficulty is that the Kafka CLI tool cannot find the Kafka standard service endpoint 9092 on your local machine. I found the solution for the network issues on stackoverflow which is much better explained by the same author on his blog post.

The docker-compose.yaml needs to configure as described in this section. Your yaml file should look like this:

version: "2"

services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.7
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: docker.io/bitnami/kafka:3
    ports:
      - "9093:9093"
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

Now you can access the running instance under macOS with the CLI like this kafka-topics --bootstrap-server localhost:9093 --list.

I had some fun figuring this out, and I hope this post will save you some time.