Elastic Search Ubuntu

Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu

In this lesson, we will see how we can get our ELK Stack up and running on our Ubuntu machines. ELK consist of Elasticsearch 2.2.x, Logstash 2.2.x, and Kibana 4.5.x. We will show how we can configure this stack and use Kibana to visualise the logs which our applications and systems create in a centralized location, using Filebeat 1.1.x. To understand these tools, let’s look at a basic definition for each of them:

  • Logstash: It is an Open Source tool which is used to collect, parse and store logs for an application or a system which can be used later for metrics of APIs, errors faced in a system and many more use-cases which are beyond the scope of this lesson
  • Kibana: This is a dashboard interface on the web which is an excellent dashboard used to search and view the logs that Logstash has indexed into the Elasticsearch index
  • Filebeat: This is installed on the client-server who want to send their logs to Logstash. Filebeat acts as a log shipping agent and communicates with Logstash.

Let us also visualise how things will work:

ELK Setup on Ubuntu

Prerequisites

For this lesson and all installations it needs, you should have root access to the machine. We will be using a machine with this configuration:

  • Ubuntu 16.04
  • RAM: 4 GB
  • CPU: 2

Few application servers from where you want to gather data from would also be a good to have.

Install Java

To install Elasticsearch on Ubuntu, we must install Java first. Java might not be installed by default. We can verify it by using this command:

Checking Java version

java -version

Here is what we get back with this command:

Checking Java version


We will now install Java on our system. Use this command to do so:

Installing Java

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

Once these commands are done running, we can again verify that Java is now installed by using the same version command.

Install Elasticsearch

Next step for the ELK Stack setup is installing Elasticsearch on Ubuntu Machine which will store the logs generated by systems and applications. Before we can install Elasticsearch, we need to import its public GPG keys to the rpm package manager:

GPG Keys

rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch

Now, insert the mentioned lines to the configuration file for the repository ‘elasticsearch.repo’:
Repository Config

[elasticsearch]
name=Elasticsearch repository
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Now, read the lesson Install ElasticSearch on Ubuntu for installation process. Once ES is up and running, make sure it responds normally to this curl command:

ES Status

curl -X GET 'http://localhost:9200'

The normal output will be:

Install Logstash

Installing Logstash is very easy using the apt package manager and is available with the same repository and public key as Elasticsearch, so we don’t have to do that again. Let’s create the source list to start:

Create Source list

echo 'deb http://packages.elastic.co/logstash/2.2/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list

Update the apt package list:

Updating Packages

sudo apt-get update

Install Logstash with a single command:

Install Logstash

sudo apt-get install logstash

Logstash is installed but it is not configured yet. We will configure Logstash in coming sections.

Install Kibana

Kibana is very easy to install. We can start by creating the Kibana source list:

Create Kibana source list

echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | sudo tee -a /etc/apt/sources.list.d/kibana-4.5.x.list

Now, we will update the apt package list:

Updating Packages

sudo apt-get update

We are ready to install Kibana now:

Install Kibana

sudo apt-get -y install kibana

Once Kibana is installed, we can run it:

Start Kibana Service

sudo update-rc.d kibana defaults 96 9
sudo service kibana start

Before we show you the Kibana Dashboard, we need to setup the Filebeat Log shipping agent as well.

Setup Filebeat

We are ready to install Filebeat now:

Install Filebeat

sudo apt-get -y install filebeat

Before we can start the Filebeat service, we need to configure it for the input type and document type. Because we’re using system logs only as of now, let’s mention this in the configuration file in ‘/etc/filebeat/filebeat.yml’:

Configure Filebeat

...
input_type: log
document_type: syslog
...

We can also start filebeat now:

Start Filebeat Service

sudo update-rc.d kibana defaults 96 9
sudo service filebeat start

Once filebeat is up and running, we can check that it is OK by issuing the following curl command:

Testing Filebeat

curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

We should receive a similar result as we got in the ES installation.

Connecting to Kibana

We are now ready to connect to Kibana. As we already started the Kibana service, its dashboard should be visible at:

Kibana Dashoboard URL

http://localhost:5601

Once you’re up on Kibana, create an index on Kibana with name ‘filebeat-*’. Now based on the logs available, you can see the metrics and logs in your Kibana Dashboard:

Conclusion

In this lesson, we looked at how we can install and start using the ELK Stack for log visualisation and support an excellent Dashboard for business teams.

About the author

Shubham Aggarwal

Shubham Aggarwal

I’m a Java EE Engineer with about 4 years of experience in building quality products. I have excellent problem-solving skills in Spring Boot, Hibernate ORM, AWS, Git, Python and I am an emerging Data Scientist.