How to set up multi-broker Kafka cluster?

In my previous post, I explained how to set up Apache Kafka locally using a single broker. In this post, I’ll tell you how to add more brokers (nodes) to the Kafka cluster. It is very easy to start several Kafka server instances and by doing that we can add broker(s) to the clusters.

If you don’t have Kafka running, please refer to my previous post as this post assumes that you have Kafka, ZooKeeper up and running. Let’s add two more brokers to the Kafka cluster but all running locally.

Step 1: Setting up a multi-broker cluster

Starting a new Kafka server is very easy by using the `server.properties` file. Based on the previous article, one broker is already running that listens to the request on `localhost:9092` based on default configuration values.

To add more broker(s), let’s duplicate the properties file and modify some key configurations like broker-id, listener and log directory to make the new broker(s) unique in the cluster. As such these brokers will be running locally so we don’t want to overwrite logs logged by an individual broker.

Duplicate the existing properties file for each new broker:

cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties

Now edit these new files and set the following properties:

//config/server-1.properties:
broker.id=1
listeners=PLAINTEXT://:9093
log.dir=/tmp/kafka-logs-1

//config/server-2.properties:
broker.id=2
listeners=PLAINTEXT://:9094
log.dir=/tmp/kafka-logs-2

Step 2: Add 2 new Kafka brokers to the cluster

We already have a ZooKeeper instance running, we can now add two more Kafka brokers:

bin/kafka-server-start.sh config/server-1.properties

bin/kafka-server-start.sh config/server-2.properties

Step 3: Create a new topic with a replication factor of 3

As such we now have 3 brokers running in the cluster, it is good time to try replication for a topic so that the messages are copied to other brokers as well. This basically helps in case of fault tolerance when a broker goes down, we will still have the data written somewhere.

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

The command above will create a topic named “my-replicated-topic” with replication factor as 3 and 1 partition.

Step 4: Monitor a topic using describe flag

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

// output is below
Topic:my-replicated-topic   PartitionCount:1	ReplicationFactor:3 Configs:
    Topic: my-replicated-topic  Partition: 0	Leader: 1   Replicas: 1,2,0 Isr: 1,2,0

Here is an explanation of the output above. The first line gives a summary of all the partitions, each additional line gives information about one partition. Since we have only one partition for this topic there is only one line.

• “leader” is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions.
• “replicas” is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
• “isr” is the set of “in-sync” replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.

I’ve got an exercise for you: Try to monitor the previous topic where Replicas will be 0.

Step 5: Send messages to a new replicated topic

Sending messages to our new replicated topic is easy and same as shown in the previous article:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic

Enter messages and hit enter to send messages to the broker which will then be replicated across other registered brokers in the cluster.

Step 6: Read messages from the new replicated topic

Reading messages from the replicated topic is also the same as shown in the previous article:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic

As you can get from the post, adding new brokers to the cluster is very easy. The concepts of sending/reading messages to/from the topic respectively, are same no matter what the replication factor or partitions configuration is registered while creating the topic.

If you would like to learn more about Apache Kafka, please subscribe to my blog as I’ll be writing more how-to articles very soon.