- Logstash: It is an Open Source tool which is used to collect, parse and store logs for an application or a system which can be used later for metrics of APIs, errors faced in a system and many more use-cases which are beyond the scope of this lesson
- Kibana: This is a dashboard interface on the web which is an excellent dashboard used to search and view the logs that Logstash has indexed into the Elasticsearch index
- Filebeat: This is installed on the client-server who want to send their logs to Logstash. Filebeat acts as a log shipping agent and communicates with Logstash.
Let us also visualise how things will work:
For this lesson and all installations it needs, you should have root access to the machine. We will be using a machine with this configuration:
- Ubuntu 16.04
- RAM: 4 GB
- CPU: 2
Few application servers from where you want to gather data from would also be a good to have.
To install Elasticsearch on Ubuntu, we must install Java first. Java might not be installed by default. We can verify it by using this command:
Checking Java version
Here is what we get back with this command:
We will now install Java on our system. Use this command to do so:
sudo apt-get update
sudo apt-get install oracle-java8-installer
Once these commands are done running, we can again verify that Java is now installed by using the same version command.
Next step for the ELK Stack setup is installing Elasticsearch on Ubuntu Machine which will store the logs generated by systems and applications. Before we can install Elasticsearch, we need to import its public GPG keys to the rpm package manager:
Now, insert the mentioned lines to the configuration file for the repository ‘elasticsearch.repo’:
Now, read the lesson Install ElasticSearch on Ubuntu for installation process. Once ES is up and running, make sure it responds normally to this curl command:
The normal output will be:
Installing Logstash is very easy using the apt package manager and is available with the same repository and public key as Elasticsearch, so we don’t have to do that again. Let’s create the source list to start:
Create Source list
Update the apt package list:
Install Logstash with a single command:
Logstash is installed but it is not configured yet. We will configure Logstash in coming sections.
Kibana is very easy to install. We can start by creating the Kibana source list:
Create Kibana source list
Now, we will update the apt package list:
We are ready to install Kibana now:
Once Kibana is installed, we can run it:
Start Kibana Service
sudo service kibana start
Before we show you the Kibana Dashboard, we need to setup the Filebeat Log shipping agent as well.
We are ready to install Filebeat now:
Before we can start the Filebeat service, we need to configure it for the input type and document type. Because we’re using system logs only as of now, let’s mention this in the configuration file in ‘/etc/filebeat/filebeat.yml’:
We can also start filebeat now:
Start Filebeat Service
sudo service filebeat start
Once filebeat is up and running, we can check that it is OK by issuing the following curl command:
We should receive a similar result as we got in the ES installation.
Connecting to Kibana
We are now ready to connect to Kibana. As we already started the Kibana service, its dashboard should be visible at:
Kibana Dashoboard URL
Once you’re up on Kibana, create an index on Kibana with name ‘filebeat-*’. Now based on the logs available, you can see the metrics and logs in your Kibana Dashboard:
In this lesson, we looked at how we can install and start using the ELK Stack for log visualisation and support an excellent Dashboard for business teams.