Get the Set of Partitions that Are Currently Assigned to this Consumer in Apache Kafka Consumer

We can define a Kafka partition as a single unit of parallelism and scalability. Think of it as a way to divide a topic into multiple, smaller parts that can be spread across multiple servers. Each Kafka partition is an ordered, immutable sequence of messages that are appended over time.

Once you manually write or a producer application writes the messages to a Kafka topic, the Kafka broker chooses the partition on which to assign the message using a default partitioning technique, mainly using the round-robin algorithm. In some cases, Kafka allows you to manually specify the target partition using the key parameter in the message.

During message consumption, the consumer application subscribes to the topic. The consumer can choose to read from any available partitions in the topic. Multiple consumers can also consume from different partitions in parallel which allows for efficient parallel processing of the messages.

Get the Set Partition

It is good to remember that the method and technique of fetching the partition that is currently assigned to a given consumer depends on your target language. This tutorial demonstrates how we can use the assignment property in the confluent-kafka package in the Python programming language.

You can reference the documentation for your target language to learn more about this feature.

In Python, we can use the source code as demonstrated in the following example:

Let us break down how the previous code sample works:

Importing the required modules – The first step is importing the required classes. In this case, we need the KafkaConsumer and the TopicPartition.

Setup a Kafka Consumer – In the second section, we configure the Kafka Consumer instance using the KafkaConsumer class and passing the parameters to the constructor. Although the parameters depend on your Consumer configuration, we define these parameters in our example:

• bootstrap_servers – This parameter allows us to define the list of Kafka brokers to connect to. In this case, we set it to “localhost:9092” which means that the consumer will connect to a single Kafka broker on the local machine on port 9092.
• auto_offset_reset – This parameter specifies what to do when there is no initial offset in Kafka or the existing offset no longer exists on the server. In this case, “earliest” is specified, which means that the consumer will start consuming from the earliest message in the partition.
• enable_auto_commit – Finally, the enable_auto_commit parameter defines whether to enable the auto-commit of offsets. In our example, we set it to “True” which allows the consumer to automatically commit the offsets that it processed to the Kafka broker.

Setup Partitions – Once the consumer is ready, we use the assign method to assign the partitions to the consumer. In our example, we pass the value as a list of TopicPartition objects. Note that we need to specify the topic name in the parameter.

Fetch Partitions – Last but not the least, we use the assignment method from the consumer to retrieve the current partitions that are assigned to the consumer.

Print the Assigned Partitions – The last step is simply printing the assigned partitions to the console.

We can run the previous code as follows:

This should return the assigned partitions as follows:

Conclusion

You learned how you can use the confluent-kafka Python package to get the list of partitions that is assigned to a specific consumer.