Once you manually write or a producer application writes the messages to a Kafka topic, the Kafka broker chooses the partition on which to assign the message using a default partitioning technique, mainly using the round-robin algorithm. In some cases, Kafka allows you to manually specify the target partition using the key parameter in the message.
During message consumption, the consumer application subscribes to the topic. The consumer can choose to read from any available partitions in the topic. Multiple consumers can also consume from different partitions in parallel which allows for efficient parallel processing of the messages.
It is good to remember that the order of messages within a partition is guaranteed. However, the order of messages across multiple partitions is not guaranteed. This means that the consumer cannot “hop” from one topic to another and expect to read the messages in the same order.
If you need to process the messages in a specific order, you must send them to the same partition.
What Is an Offset Value in a Partition?
Next, let us talk about the offset value.
In a Kafka partition, each message is assigned with a unique identifier value which is known as a message offset. This identifier is a 64-bit integer which is allocated to each message by the broker when it is produced.
The broker then uses it to keep track of the consumer’s position within a given partition. This offset value allows the consumer to read the messages in a given order, keep track of where it left off to resume the operations, and such. This is necessary for the reliability of the consumer since it allows it to pick up from where it left off in case of failures or rebalances.
A consumer can also explicitly state from which offset it wishes to start reading using the seek method. A consumer can also start at the latest offset which is known as the partition’s end. Similarly, the consumer can start reading from the earliest offset in the partition which is also known as the beginning of the partition.
In addition to providing reliability, the offset allows for flexible consumption patterns. For example, a consumer can rewind to an earlier offset in the partition and start consuming from there. Alternatively, it can skip over several offsets to consume only specific messages.
Get the Last Offset for a Given Partition
In this tutorial, we will build a simple Python script that fetches the latest offset from a Kafka partition using the confluent-kafka package.
# configure consumer
consumer = KafkaConsumer(
bootstrap_servers=["localhost:9092"],
auto_offset_reset="latest"
)
# fetch the first partition
partition = TopicPartition("users", 0)
consumer.assign([partition])
# # seek earliest offset
consumer.seek_to_end()
offset = consumer.position(partition)
print(f"The latest offset for partition {partition}: {offset}")
In this case, the method uses the auto offset reset method as the latest and the seek_to_end() method to jump to the latest offset value of the first partition. This should print the latest offset as shown in the following:
Conclusion
This tutorial demonstrates how to use the Python confluent-kafka package to fetch the latest offset in a given Kafka partition.
