Apache Kafka Producer Example from CLI

The kafka-console-producer CLI utility is a part of Apache Kafka CLI tools. It allows us to produce the messages to a Kafka topic from the command line. This in turn provides a simple testing interface that can allow you to quickly write the messages to an existing topic straight from the terminal. Once you run the command, it opens an input buffer which allows you to type the messages sequentially or read the messages from a file.

The messages are serialized using the specified serialization format which can be one of several supported formats such as plain text or Avro.

In this tutorial, we will learn how we can use the Kafka producer command line utility to write the messages to a Kafka topics. We also explore some common command-line arguments such as the address of the Kafka broker, the name of the topic to produce to, the serialization format, etc.

We will also discuss about partitioning and replication factors, among others.

Create a Test Kafka Topic

Before proceeding to write the messages to a topic, we need to ensure that the topic exists on the server. We can do this using the “kafka-topics.sh” command line utility.

An example command is as shown in the following:

The previous command uses the “kafka-topics.sh” utility to create a new a new topic called “users” using the –bootstrap-server command to specify the address to the Kafka broker. We also use the –replication-factor argument to set the replication of the topic. Finally, the –partition argument allows us to specify the number of partitions that are assigned to the Kafka topic.

Start Producing the Messages

Once we verify that the topic exists on the server, we can start writing the messages to the topic using the kafka-producer utility.

The command syntax is as follows:

If you are running the Apache Kafka 2.4 and less, you need to substitute the –bootstrap-server option with the –broker-list option as shown in the following syntax:

An example demonstration command is as shown in the following:

This command should open an input in the console which allows us to write the messages to the “users” topic.

We can write a new message on each line as shown in the following:

To stop writing messages to the topic, use the CTRL + C key combination to send a SIGKILL signal to the producer process.

You can also write the messages from a file using the input operator as shown in the following command syntax:

In this case, the input operator (<) allows you to read the contents of the file and write them to the topic.

Conclusion

You learned how to create a new Kafka topic using the “kafka-topics.sh” utility. You also learned how to write the messages to a Kafka topic using the “kafka-console-producer.sh” utility.