How to Add Messages to an Apache Kafka Topic

In Kafka, messages refer to the units of data which are produced by the producers and are consumed by the consumers in the Apache Kafka cluster. They consist of a key, a value, and a timestamp and are organized into topics for efficient processing and storage.

In this tutorial, we will learn how to write messages to a Kafka topic using the provided CLI tools.

NOTE: Keep in mind that this article assumes you have the Apache Kafka cluster installed and configured on your system. We will use the commands on localhost and port 9092, the default port for the Kafka broker.

Create a Topic in Kafka

Before we can write a message, we need a topic to store the created data. We do this by creating a topic with the kafka-topics.sh utility. You can check our tutorial on the topic to discover more.

For now, the following command creates a new topic called “users”:

The previous command creates a topic called “users” with default properties such as a replication factor of 1 and a single partition.

Write Messages to a Topic in Kafka

Once created, we can write messages about the topic using the kakfa-console-producer.sh utility.

An example command is shown in the following:

This starts the console producer and connects it to the Kafka broker at the localhost:9092. We can then produce the messages by typing each message on each line in the provided cursor.

An example is shown in the following:

This should write the messages to the users’ topic ready for consumption by the subscribed consumers.

NOTE: The written messages do not contain a key or timestamp. We need to write a producer client application in a specific programming language to specify a key or timestamp. Check our Kafka series to learn more about that.

We can consume the messages using the following command:

This starts the console consumer and connects it to your Kafka broker at the localhost:9092. It consumes all the messages from the beginning of the topic’s log, which means that it retrieves all messages that have been sent to the topic so far.

As the new messages are produced and sent to the topic, they are displayed in the console consumer’s output in real time.

Output:

Keep in mind that the console consumer tool is used for demonstration purposes. You can define your consumer applications more robustly and extensively in large-scale environments.

Conclusion

You can write messages about a Kafka topic using the kafka-console-producer.sh utility, and consume messages from the same topic using the kafka-console-consumer.sh tool. This can be useful to test and demonstrate the basic functionality of Apache Kafka.