In this tutorial, we will guide you through the process of running Logstash in a Docker container with basic configuration.
Requirements:
Before we get started with the tutorial, ensure that you have the following:
- Installed Docker on your host machine (version 23 and above is recommended)
- Installed Docker Compose on your machine
With the given requirements met, we can proceed with the tutorial.
Setup the Logstash Configuration File
Logstash uses the configuration files to define how the data is ingested, filtered, and sent to the output. There are variety of options that you can configure as you can reference on the official documentation.
For our example, we focus on the basic configuration that ingests the data from a log file, filters it for the matching records, and outputs the data to a file.
Create a file called “logstash.conf” and add the configuration as follows:
file {
path => "/var/log/apache/access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
ignore_older => 0
}
}
filter {
if [message] =~ "ERROR" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
}
output {
file {
path => "/var/log/apache/error_logs.log"
}
}
The previous file defines the configuration as shown in the following:
- Input section – The input section uses the file input plugin to read the Apache log file located in /var/log/apache/access.log.
- We then set the start position at the beginning which allows Logstash to read the entire file from the beginning.
- Sincedb_path – This parameter allows us to disable Logstash’s sincedb tracking by setting the value to /dev/null. This ensures that Logstash always reads from the beginning of the file.
- Ignore_older – Setting the value of this parameter to 0 allows Logstash to process all entries of the log file.
- Filter section – In the filter section, we define the filter pattern to check if the log message contains the word ERROR. You can adjust the conditions for the filter blocks to filter for more precise matches in the file.
- If the condition is met, we use the grok filter to parse the Apache log line using the COMBINEDAPACHELOG pattern which is a built-in pattern in Logstash to parse the Apache logs.
- Output section – This section allows us to define the output format for the matching entries.
- In our case, we write them to the /var/log/apache/error_logs.log file using the path parameter.
This should provide us with a basic Logstash configuration that allows us to demonstrate some basic Logstash workings.
Please reference more about creating and configuring the Logstash pipelines in the following provided documentation resource:
https://www.elastic.co/guide/en/logstash/current/configuration.html
Create the Dockerfile
Once we define the Logstash configurations, we can proceed and learn how to run the container. Inside the same directory as the “logstash.conf” file, create a new file called “Dockerfile”.
Edit this file and add the entries as follows:
COPY logstash.conf /usr/share/logstash/pipeline/logstash.conf
In the given example, we define the base image as the official Logstash image using the version 8.9.2.
We then copy the “logstash.conf” file that we created to the /usr/share/logstash/pipeline/logstash.conf in the image.
Build the Docker Image
Next, navigate to the directory that contains the Dockerfile and Logstash configuration file. Run the following command to build the image:
Run the Logstash Container
Now that we built the Docker image, we can run a Logstash container using the Docker “run” command as follows:
This should run the Logstash container using the image that we built in the previous step.
Verify the Logstash Container Logs
To verify that Logstash is running correctly, you can check the container logs using the following command:
Output:
Conclusion
In this tutorial, you learned how you can quickly setup a Logstash server that runs as Docker using a custom configuration file. We recommend checking the documentation on the image parameters and configuration options.
