Ruby supports multiple programming paradigms including procedural, functional, and object-oriented and has a large, supportive community and extensive libraries known as gems that can be easily installed and used in projects.
Ruby mainly focuses on simplicity and functionality while retaining powerful features for various development types. It is a popular programming language which is used to build massive enterprise applications such as Shopify and GitHub.
In this tutorial, we will learn how to create a Kafka producer and consumer using the Ruby programming language and the rdkafka gem.
Requirements
Before proceeding with this tutorial, it is good to remember that this article does not cover the basic features such as installing Ruby or Apache Kafka on your target system. Make sure that you have all the required tools to run Kafka and Ruby.
Installing the Target Gem
The first step is to install the required Ruby gem. For this tutorial, we use the rdkafka gem.
This gem provides a Ruby interface to interact with the Apache Kafka platform. The rdkafka gem is a Ruby client library for the Apache Kafka protocol and provides a high-level, idiomatic Ruby API to work with Apache Kafka.
The rdkafka gem enables us to publish and subscribe to Kafka topics, produce and consume messages, handle the metadata, and use various Kafka configuration options.
Once installed, we can proceed and build a consumer and producer applications for Kafka.
Setting up the Project
Start by creating a directory to store your projects:
Navigate into the created directory:
Create a producer file:
Edit the file and add the source code as shown in the following:
producer = Rdkafka::Config.new(config).producer
delivery_handles = []
10.times do |i|
puts "Writing message #{i}"
delivery_handles << producer.produce(
topic: "new_topic",
payload: "Payload #{i}",
key: "Key #{i}"
)
end
delivery_handles.each(&:wait)
The previous code creates a hash config that sets the value of the "bootstrap.servers" configuration option to "localhost:9092". The Kafka client uses the bootstrap server to discover the location of the Kafka brokers in the cluster.
Next, we create a new Rdkafka::Config object using the config hash and call the producer method to get a Kafka producer instance.
The delivery_handles array is used to store the handles of the produced messages.
Using a loop with ten iterations, we produce a message to the "new_topic" topic with a payload and a key, which include the iteration count. The producer's “produce” method returns a delivery handle which is stored in the delivery_handles array.
Finally, the code calls to wait on each delivery handle in the delivery_handles array for the messages to be sent and confirmed by the broker.
Consumer Application
Next, create a file to store the consumer code and add the source code as shown in the following:
consumer = Rdkafka::Config.new(config).consumer
consumer.subscribe("new_topic")
consumer.each do |message|
puts "#{message}"
end
The previous code subscribes to the topic “new_topic” and consumes the messages that are written to the topic.
Conclusion
This short post discusses how to use the rdkafka gem to write a consumer and producer application in Ruby for Kafka.
