Apache Kafka is a free and open-source, distributed, scalable, and high-throughput message broker that enables the real-time data streaming and event-driven architecture. In this tutorial, we will learn how to create a Kafka producer in Ruby which allows us to write the messages to an existing Kafka topic.
By following this tutorial, we assume that you have a basic understanding of Kafka concepts such as topics, partitions, and brokers. We also assume that you are not new to Ruby and are able to install and configure the Ruby gems in easy steps.
We use the rdkakfa gem, a modern and well-optimized client library to interact with Apache Kafka using Ruby. It is based on the librdkafka implementation which is created in C. It also supports Apache Kafka 1.0 and above, making it extensible in a wide array of Kafka versions.
At the end of this tutorial, you will have a working Kafka producer in Ruby that can write the data to a Kafka topic. It is good to note that this is a basic tutorial and is not intended as a full-fledged producer application.
Project Setup
We need to start by setting up the project structure. For this demonstration, we use the combine Kakfa CLI tools to setup the various blocks such as topics.
Start by creating the directory to store your source code:
Navigate into the project directory and add the file for your source code:
The next step is to create a Kafka topic to store the messages. We can do this using the Kafka Topic CLI utility.
The command is as shown in the following:
The previous command should create a new Kafka topic called “kafka_rb” with a replication factor of 1 and one partition.
Once successful, we can proceed and setup the producer applications to write the messages to the topic.
Start by installing the rdkafka gem with the following command:
Once installed, edit the source file and add the code as follows:
Source code:
config = {:"bootstrap.servers" => "localhost:9092"}
producer = Rdkafka::Config.new(config).producer
delivery_handles = []
index = 0
File.open("user_data.txt").each do |line|
index += 1
delivery_handles << producer.produce(
topic: "kafka_rb",
payload: "#{line}",
key: "#index}"
)
end
delivery_handles.each(&:wait)
The previous code uses the Rdkafka module to create a new Producer instance on the server which is specified in the config property.
We then define the delivery_handles as an empty array which is used to store the messages that are read from the file.
The contents of the user_data.txt file is as follows:
Clareta,[email protected],55.177.226.116
Clem,[email protected],245.235.202.52
Alene,[email protected],70.240.174.255
Elysha,[email protected],133.220.136.158
Finally, we use the producer.produce() method to write the messages to the kafka_rb topic in the Kafka instance. Once completed, we can run the code as follows:
This should read the file and allow you to write the messages from the file to the topic. We can verify that the messages are produced to the topic by reading from the topic with the following command:
The previous command should allow you to read the messages in the topic as shown in the following output:
Conclusion
You now learned how to create a Kafka producer application in Ruby that reads the messages from a file and writes them to an existing Kafka topic.
