The metadata includes information such as the partition’s leader broker, the replicas for the partition, and the offset of the latest message in the partition.
Kafka Leader Broker
The leader broker for a partition is responsible for all read and write requests for that partition.
Partition Replica
The replicas for a partition are the brokers that store the duplicates of the partition data which allows for failover if the leader broker becomes unavailable.
Partition Offset
The offset of the latest message in a partition is used to track the consumer’s position and determine which messages have been consumed.
Having such metadata about the partition allows the Kafka broker to manage the distribution and replication of the messages that are produced to a given topic within the cluster. It also ensures the data consistency and durability across the cluster. In addition, Kafka clients can use it to determine the location of the target partitions before consuming or producing the messages.
This post discusses how we can use the Kafka API to gather the metadata information about a partition topic. For this post, we use Python and the confluent-kafka package. However, you can check the client library for your target language to learn more.
Source Code:
# Create a Kafka consumer instance
consumer = Consumer({
'bootstrap.servers': 'localhost:9092',
'group.id': '',
'auto.offset.reset': 'earliest'
})
# Retrieve metadata for the specified topic
topic = 'users'
metadata = consumer.list_topics(topic).topics[topic].partitions.values()
# Iterate over the topic partitions and retrieve their metadata
for partition in metadata:
partition_metadata = partition.leader
if partition_metadata is None:
print(f'Partition {partition.partition}: Leader unknown')
else:
print(f'Partition id {partition.id} replicas {partition.replicas}')
# Close the consumer
consumer.close()
This prints the partition metadata for the specified topic. Check the documentation to learn more.
Conclusion
Using the confluent_kafka library, you can get the metadata information about the partition of a topic.
