A Kafka topic refers to a category or a stream name under which specific publishers write the messages. Each record that is written to a Kafka cluster belongs to a specific category or topic in a partitioned log structure.
The partitioned log structure is then divided into segments, each containing a sequence of records with a unique offset. Each partition is then stored on a separate broker within the cluster. A topic can then have multiple partitions that are distributed across multiple brokers.
Kafka topics can have multiple consumers which are known as a consumer group. The consumers within a group can then work in parallel to consume the messages from the brokers.
This allows us to scale the consumption of messages from a topic horizontally by adding more consumers to the group. Additionally, Kafka topics support different levels of durability and retention based on the application requirements.
Therefore, understanding how to set up a Kafka topic quickly is fundamental as it allows the publishers and subscribers to write and read the messages.
In this post, we will learn how you can use the Kafka CLI tools to create a Kafka topic in very simple steps.
What Are Kafka CLI Tools?
Kafka CLI tools are a set of command-line tools that allow us to manage and monitor the Kafka clusters from the terminal.
Kafka CLI tools are included in every Kafka installation. They are built on top of the Kafka API which allows us to perform administrative tasks such as creating and deleting Kafka topics, modifying cluster configurations, and more.
Kafka Topics CLI Tool
For our case, we are interested in the Kafka-topics CLI tool. This tool allows us to perform tasks on Kafka topics such as creating, deleting, and modifying the Kafka topics.
By default, the CLI tools including the kafka-topics utility are stored in the INSTALL_DIR/bin directory.
For example, if you installed Kafka in the /opt/kafka directory, the CLI tools are located in the /opt/kafka/bin directory.
Create a Topic Using CLI
To create a Kafka topic using the CLI, we can use the “kafka-topics.sh” utility. The command syntax is as shown in the following:
The command syntax is explained in the following:
–create – This parameter specifies the action. In this case, we want to use the create action.
–topic – The topic parameter sets the topic name that we wish to create.
–partitions – This parameter determines the number of partitions for the topic.
–replication-factor – This sets the replication factor for the topic. This means that the number of copies of each partition is created across the Kafka cluster.
–bootstrap-server – This parameter specifies the broker address in the hostname and port number format.
We can adjust the values of the partitions and replication factor to fit our required needs.
For example, suppose we want to create a new topic called “coordinates”. Then, we can run the command as shown in the following:
In the previous example, we use the –create option to set up a new topic called “coordinates.” We then specify the replication factor of 3. This means that each topic partition is replicated across 3 Kafka brokers.
Finally, we specify the number of partitions as 3. This means that the topic is divided into three partitions.
Produce Messages
Once we created the topic, we can write to it using the “kafka-console-producer” command. An example is as shown in the following:
> {"latitude": 37.7749, "longitude": -122.4194}
> {"latitude": 40.7128, "longitude": -74.0060}
> {"latitude": 51.5074, "longitude": -0.1278}
> {"latitude": 35.6895, "longitude": 139.6917}
> {"latitude": 55.7558, "longitude": 37.6173}
> {"latitude": -33.8651, "longitude": 151.2093}
> {"latitude": 48.8566, "longitude": 2.3522}
> {"latitude": 37.5665, "longitude": 126.9780}
> {"latitude": 35.6762, "longitude": 139.6503}
> {"latitude": 43.6532, "longitude": -79.3832}
> EOF
In the previous command, we use the “kafka-console-producer.sh” utility to produce the messages to the coordinates topic. The –topic parameter lets us specify which topic we wish to write.
We also use a heredoc to write multiple messages to the topic where each message is a JSON object that represents the latitude and longitude.
The <<EOF and >EOF lines indicate the start and end of the heredoc, respectively.
Consume Messages
As you guessed, we can consume the messages from the coordinates topic using the kafka-console-consumer application as shown in the following:
In this case, the command should read the messages from the coordinates from the beginning as determined by the –from-beginning parameter.
Conclusion
We discussed about working with a Kafka topic using the Kafka CLI tools. We learned how we can create a topic, how to write to a topic, and how to read from a topic.
