Apache Kafka Producer Example in NodeJS

In this tutorial, we will learn how we can setup a basic Kafka producer and consumer applications using the latest Apache Kafka version and the node-rdkafka package.

Requirements

For you to follow along with this tutorial and ensure a maximum compatibility, we recommend the following:

The latest version of Apache Kafka installed on your local machine.
Node.JS version 18.50
Basic JavaScript development knowledge

Get your IDEs ready, and let us dive in!

Project Setup

Start by creating a directory to store the source code for your project.

$ mkdir kafka-node

Navigate into the created directory:

$ cd kafka-node

Install the Required Dependencies

The next step is to install the required libraries. For this tutorial, we use the node-rdkafka library.

The node-rdkafka is a free, open-source, high-performance Node.JS client library to interact with Apache Kafka. The library leverages the power of the librdkafka C/C++ library. This allows us to build efficient, robust consumer and producer applications in Node.JS with low latency and high throughput from the librdkafka implementation.

To install this package, run the npm command which is shown in the following:

$ npm install node-rdkafka

Initialize a New Kafka Topic

The next step is to create a new Kafka topic to which we read and write the messages. For this, we can use the Kafka CLI tools as shown in the following command:

$ ./bin/kafka-topics.sh --create --topic users --bootstrap-server localho
st:9092 --replication-factor 1 --partitions 1

The previous command should create a topic called “users” with a replication factor of 1 and one partitions.

Setup the Kafka Producer

Create a file to store the source code for the producer application in the kafka-node directory.

$ touch producer.js

Edit the file and add the code as shown in the following:

constkafka = require('node-rdkafka');

// Kafka broker config
constkafkaConfig = {
'metadata.broker.list': 'localhost:9092',
};

// Create a Kafka producer instance
const producer = newkafka.Producer(kafkaConfig);

// Connect the producer to the broker
producer.connect();

// Wait for the 'ready' event to be emitted before sending messages
producer.on('ready', () => {
console.log('Kafka producer is ready');

// Send 5 messages to the 'users' topic
for (leti = 1; i {
console.error('Error in Kafka producer:', err);
});

Setup the Kafka Consumer

Next, create a file to store the source code for the consumer application.

$ touch consumer.js

Add the source code as follows:

constkafka = require('node-rdkafka');

// Kafka broker configuration
constkafkaConfig = {
'metadata.broker.list': 'localhost:9092',
'group.id': 'node-app',
};

// Create a Kafka consumer instance
const consumer = newkafka.KafkaConsumer(kafkaConfig);

// Subscribe to the 'users' topic
consumer.subscribe(['users']);

// Consume messages from the 'users' topic
consumer.on('data', (message) => {
console.log(`Received message: ${message.value.toString()}`);
});

// Connect the consumer to the broker
consumer.connect();

// Handle errors
consumer.on('error', (err) => {
console.error('Error in Kafka consumer:', err);
});

// on SIGINT
process.on('SIGINT', () => {
consumer.disconnect(() => {
console.log('Kafka consumer disconnected');
process.exit();
});
});

Save the source code and use the “node.js” to run both the producer and consumer applications.

$ node producer.js

Output:

Kafka producer is ready
Sent message: User 1
Sent message: User 2
Sent message: User 3
Sent message: User 4
Sent message: User 5

To read the sent messages, run the following consumer application:

$ node consumer.js

Output:

User 1
User 2
User 3
User 4
User 5

Conclusion

This article taught us how to set up a basic consumer and producer application for Apache Kafka using Node.JS. Feel free to explore the node-rdkafka documentation to learn more about how you can expand your application.