This is where the tools such as Apache Kafka comes into play. Apache Kafka, commonly known as Kafka, is a free and open-source distributed event streaming platform. Kafka uses the pub-sub model that allows the publishers to write the events. At the same time, the subscribers can read the messages that are written by the publishers.
Although Kafka supports a wide range of programming languages, we will learn how to create a basic Kafka consumer application using Node.JS in this tutorial.
Since its launch, Node.JS quickly became one of the most widely adopted JavaScript runtime environments. Thanks to the external packages, it has continued to grow and power the small to large enterprise applications.
Requirements
For you to follow along with this tutorial and ensure a maximum compatibility, we recommend the following:
- Installed latest version of Apache Kafka on your local machine
- Node.JS version 18.50
- Basic JavaScript development knowledge
Get your IDEs ready, and let us dive in!
Project Setup
Start by creating a directory to store the source code for your project.
Navigate into the created directory:
Install the Required Dependencies
The next step is to install the required libraries. For this tutorial, we use the node-rdkafka library.
The node-rdkafka is a free, open-source, high-performance Node.JS client library to interact with Apache Kafka. The library leverages the power of the librdkafka C/C++ library. This allows us to build the efficient, robust consumer and producer applications in Node.JS with low latency and high throughput from the librdkafka implementation.
To install this package, run the npm command as shown in the following:
Initialize a New Kafka Topic
The next step is to create a new Kafka topic to which we read and write the messages. For this, we can use the Kafka CLI tools as shown in the following command:
st:9092 --replication-factor 1 --partitions 1
The previous command should create a topic called “users” with a replication factor of 1 and one partitions.
Setup the Kafka Producer
In the kafka-node directory, create a file to store the source code for the producer application.
Edit the file and add the code as shown in the following:
// Kafka broker config
const kafkaConfig = {
'metadata.broker.list': 'localhost:9092',
};
// Create a Kafka producer instance
const producer = new kafka.Producer(kafkaConfig);
// Connect the producer to the broker
producer.connect();
// Wait for the 'ready' event to be emitted before sending messages
producer.on('ready', () => {
console.log('Kafka producer is ready');
// Send 5 messages to the 'users' topic
for (let i = 1; i <= 5; i++) {
const message = `User ${i}`;
producer.produce('users', null, Buffer.from(message));
console.log(`Sent message: ${message}`);
}
// Disconnect the producer from the broker after sending messages
producer.disconnect();
});
// Handle errors
producer.on('error', (err) => {
console.error('Error in Kafka producer:', err);
});
Setup the Kafka Consumer
Next, create a file to store the source code for the consumer application.
Add the source code as follows:
// Kafka broker configuration
const kafkaConfig = {
'metadata.broker.list': 'localhost:9092',
'group.id': 'node-app',
};
// Create a Kafka consumer instance
const consumer = new kafka.KafkaConsumer(kafkaConfig);
// Subscribe to the 'users' topic
consumer.subscribe(['users']);
// Consume messages from the 'users' topic
consumer.on('data', (message) => {
console.log(`Received message: ${message.value.toString()}`);
});
// Connect the consumer to the broker
consumer.connect();
// Handle errors
consumer.on('error', (err) => {
console.error('Error in Kafka consumer:', err);
});
// on SIGINT
process.on('SIGINT', () => {
consumer.disconnect(() => {
console.log('Kafka consumer disconnected');
process.exit();
});
});
Save the source code and use the node.js to run both the producer and consumer applications.
Output:
Kafka producer is ready
Sent message: User 2
Sent message: User 3
Sent message: User 4
Sent message: User 5
To read the sent messages, run the consumer application:
Output:
User 2
User 3
User 4
User 5
Conclusion
This article teaches us how to set up a basic consumer and producer application for Apache Kafka using Node.JS. Feel free to explore the node-rdkafka documentation to learn more about how you can expand your application.
