Each broker stores one or more partitions of one or more topics and replicates these partitions to other brokers to provide data redundancy. The brokers also handle the routing of incoming messages from the producers to the appropriate partitions and forwarding the messages from the partitions to the consumers.
The broker is responsible for maintaining the order of messages within a partition and for providing the offsets (i.e., the position in a partition) for each message, which is used by the consumers to track their progress through the partition.
The broker also manages the cluster metadata such as the list of topics, the list of partitions for each topic, and the location of the replicas for each partition. The metadata is stored in Zookeeper, and the brokers communicate with Zookeeper to obtain an information about the cluster and coordinate their activities.
This short tutorial teaches us how to create a new Kafka cluster and start the available brokers.
Requirements:
To follow this tutorial, you must ensure that you have an installed Apache Kafka on your target machines. You can also have multiple instances of Apache Kafka that run on the same machine under different addresses.
You can check our tutorial on how to install the Apache Kafka on various operating systems to learn more.
Setup the Brokers
The first step is to assign a unique broker ID to each instance of Apache Kafka that you wish to form the Kafka cluster.
You can do this by editing the “server.properties” file for each Kafka instance. Locate the “broker.id” entry and assign a unique ID to the broker.
For example, you can assign the first Kafka instance an ID of 0, the second one with an ID of 1, and so on.
Do this for each of the instances that you wish to connect.
Configure the Advertised Listeners
The next step is to specify the advertised listeners for each broker in the “server.properties” configuration file so that the clients can connect to the broker.
Locate the entries and add the addresses for each broker in the “advertised.listeners” entry. An example is shown in the following:
In this case, the setting specifies two listeners using the PLAINTEXT protocol (i.e., unencrypted communication) and listening on IP addresses 192.168.0.100 and 192.168.0.101 on port 9092, respectively.
This setting is essential because the clients, such as producers and consumers, need to know the addresses of the brokers to communicate with the cluster. The “advertised.listeners” setting allows a broker to advertise the different addresses than the ones that it is listening on, which can be helpful in situations where the broker is behind a firewall or a load balancer.
You can do this for each broker in the cluster.
Start the Brokers
Finally, start the brokers with the following command:
Do this for each broker in the cluster.
Once the brokers are up and running, verify that each broker instance has started by checking the log files or using the following command:
There you have it!
Conclusion
We discussed how we can configure the multiple brokers in a Kafka cluster using the unique broker IDs and advertised addresses.
