How to Start and Stop the Apache Kafka Consumer

A Kafka refers to an application that can subscribe to one or more topics in a Kafka cluster. Once subscribed, the consumer application can retrieve the messages that are produced to the subscribed topics and perform transformative actions on the messages.

Kafka consumers can also perform in a pull or push-based manner where the consumer explicitly requests the messages from the broker by issuing a poll request. The broker then pushes the messages to the consumer as soon as they are available.

Kafka consumers can run as standalone or as part of a consumer group. In which case, the consumer group splits the partitions among the consumers in the group and balances the load. The consumer group is also responsible for providing a fault tolerance mechanism which determines the action that is taken when a consumer in the group fails. For example, the consumer group can allow the partitions of the failed consumer to be assigned to the other consumers, ensuring the data availability across the cluster.

Therefore, you must ensure that the consumer application is running if you wish to read the messages that are published on a given Kafka topic.

Starting and Stopping the Kafka Consumer

It is good to understand that starting or stopping the consumer application heavily depends on your application configuration and the programming language of your choice.

This post covers the fundamentals by discussing how to start and stop a Kafka consumer from the CLI.

Start by creating a topic on which we wish to read the messages. For demonstration purposes, we use the “Kafka-console-producer.sh” utility. An example command is as follows:

This creates a topic called “sample_topic”.

Once we create the topic, we can publish the messages to the topic as follows:

Input Buffer:

Once we are done writing the messages to the topic, we can press CTRL + C to end the write.

Finally, to start the console consumer, we can use the following command:

The previous command should subscribe to the sample_topic and start reading the messages from the first offset.

The command should launch the consumer application and read the messages that are published on the specified topic.

Once the consumer has finished reading the messages, it opens the read buffer which allows all the incoming messages to be read.

To stop the consumer application, you can send a stop signal to the consumer process, such as a SIGTERM or SIGINT signal in Unix-based systems. This involves pressing the CTRL + C key combination on your device.

This allows you to kill the consumer process.

Conclusion

You now discovered how to start and stop the Kafka consumer application using the Kafka consumer CLI.

NOTE: If you are not using the Kafka consumer CLI app, you can stop the application by defining the logic to start or stop the consumer on a given condition. For example, you can define a condition that stops the consumer at the end of the message offset.