Linux, Windows, Virtualization, OpenSource & Blogging

Updated: Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 / RHEL 7

3

ELK stack is also known as the Elastic stack, consists of Elasticsearch, Logstash, and Kibana. It helps you to have all of your logs stored in one place and analyze the issues by correlating the events on a particular time.

This guide helps you to install ELK stack on CentOS 7 / RHEL 7.

Components:

Logstash – It does the processing (Collect, enrich and send it to Elasticsearch) of incoming logs sent by beats (forwarder).

Elasticsearch – It stores incoming logs from Logstash and provides an ability to search the logs/data in a real time

Kibana – Provides visualization of logs.

Beats – Installed on client machines, send logs to Logstash through beats protocol.

Environment

To have a full featured ELK stack, we would need two machines to test the collection of logs.

ELK Stack

Operating system : CentOS 7 Minimal
IP Address       : 192.168.12.10
HostName         : server.itzgeek.local

Filebeat

Operating System : CentOS 7 Minimal
IP Address       : 192.168.12.20
HostName         : client.itzgeek.local

Prerequisites

Since Elasticsearch is based on Java, make sure you have either OpenJDK or Oracle JDK is installed on your machine.

READ: How to Install Oracle Java 8 on CentOS 7 / RHEL 7

Here, I am using OpenJDK 1.8.

yum -y install java-1.8.0 wget

Verify the Java version.

java -version

Output:

openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)

Install Elasticsearch

Elasticsearch is an open source search engine, offers a real-time distributed search and analytics with the RESTful web interface. Elasticsearch stores all the data are sent by the logstash and displays through the web interface (kibana) on users request.

Import the Elastic signing key.

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Setup the Elasticsearch repository and install it.

vi /etc/yum.repos.d/elk.repo

Add the below content to the elk.repo file.

[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Install Elasticsearch.

yum install -y elasticsearch

Configure Elasticsearch to start during system startup.

systemctl daemon-reload
systemctl enable elasticsearch
systemctl start elasticsearch

Use CURL to check whether the Elasticsearch is responding to the queries or not.

curl -X GET http://localhost:9200

Output:

 {
  "name" : "it83YoZ",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "Z9Od_XdcTfWxoDL7149Rhg",
  "version" : {
    "number" : "5.5.2",
    "build_hash" : "b2f0c09",
    "build_date" : "2017-08-14T12:33:14.154Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

Install Logstash

Logstash is an open source tool for managing events and logs, it collects the logs, parse them and store them on Elasticsearch for searching. Over 160+ plugins are available for Logstash which provides the capability of processing thea different type of events with no extra work.

Install the Logstash package.

yum -y install logstash

Create SSL certificate

Filebeat (Logstash Forwarder) are normally installed on client servers, and they use SSL certificate to validate the identity of Logstash server for secure communication.

Create SSL certificate either with the hostname or IP SAN.

Option 1: (Hostname FQDN)

If you use the logstash server hostname in the beats (forwarder) configuration, make sure you have A record for logstash server and also ensure that client machine can resolve the hostname of the logstash server.

Go to the OpenSSL directory.

cd /etc/pki/tls/

Now, create the SSL certificate. Replace “green” one in with the hostname of your real logstash server.

openssl req -x509 -nodes -newkey rsa:2048 -days 365 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt -subj /CN=server.itzgeek.local

Option 2: (IP Address)

If you are planning to use an IP address instead of hostname, please follow the steps to create an SSL certificate for IP SAN.

To create an IP SAN certificate, you would need to add an IP address of logstash server to the SubjectAltName in the OpenSSL config file.

vi /etc/pki/tls/openssl.cnf

Look for “[ v3_ca ]” section and replace “green” one with the IP of your logstash server.

subjectAltName = IP:192.168.12.10

Goto OpenSSL directory.

cd /etc/pki/tls/

Now, create an SSL certificate by running following command.

openssl req -x509 -days 365 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

This logstash-forwarder.crt should be copied to all client servers those who send logs to logstash server.

Configure Logstash

Logstash configuration can be found in /etc/logstash/conf.d/. If the files don’t exist, create a new one. logstash configuration file consists of three sections input, filter, and the output; all three sections can be found either in a single file or each section will have separate files end with .conf.

I recommend you to use a single file to placing input, filter and output sections.

vi /etc/logstash/conf.d/logstash.conf

In the first section, we will put an entry for input configuration. The following configuration sets Logstash to listen on port 5044 for incoming logs from the beats (forwarder) that sit on client machines. Also, add the SSL certificate details in the input section for secure communication.

input {
 beats {
   port => 5044
   ssl => true
   ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
   ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
   congestion_threshold => "40"
  }
}

In the filter section. We will use Grok to parse the logs ahead of sending it to Elasticsearch. The following grok filter will look for the “syslog” labeled logs and tries to parse them to make a structured index.

filter {
if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGLINE}" }
    }

    date {
match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
  }

}

for more filter patterns, take a look at grokdebugger page.

In the output section, we will define the location where the logs to get stored; obviously, it should be Elasticsearch.

output {
 elasticsearch {
  hosts => localhost
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
       }
stdout {
    codec => rubydebug
       }
}

Now start and enable the logstash.

systemctl start logstash
systemctl enable logstash

You can troubleshoot any issues by looking at below log.

cat /var/log/logstash/logstash-plain.log

Next, we will configure beats to ship the logs to logstash server.

Install Filebeat

There are four beats clients available

  1. Packetbeat – Analyze network packet data.
  2. Filebeat – Real-time insight into log data.
  3. Topbeat – Get insights from infrastructure data.
  4. Metricbeat – Ship metrics to Elasticsearch.

To analyze the system logs of the client machines (Ex. client.itzgeek.local), we need to install filebeat. You can download filebeat from the official website, or you can use the following command to install it.

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Create beats.repo file.

vi /etc/yum.repos.d/beats.repo

Add the below content to the above repo file.

[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Now, install Filebeat using the following command.

yum -y install filebeat

Configure Filebeat

Now, it’s the time to connect filebeat with Logstash; follow up the below steps to get filebeat configured with ELK stack.

Before that, setup a host entry on the client machine in case your environment does not have a name server.

# vi /etc/hosts

192.168.12.10 server.itzgeek.local server

Filebeat (beats) uses SSL certificate for validating logstash server identity, so copy the logstash-forwarder.crt from the logstash server to the client.

scp -pr root@server.itzgeek.local:/etc/pki/tls/certs/logstash-forwarder.crt /etc/ssl/certs/

Filebeat configuration file is in YAML format, which means indentation is very important. Make sure you use the same number of spaces used in the guide.

Open up the filebeat configuration file.

vi /etc/filebeat/filebeat.yml

On top, you would see the prospectors section; specifies which log files should be sent to logstash and how they should be handled. Each prospector starts with character.

For testing purpose, we will configure filebeat to send /var/log/messages to Logstash server. To do that, modify the existing prospector under paths section. Comment out the – /var/log/*.log to avoid sending all .log files present in that directory to Logstash.

.  .  .

      paths:

        - /var/log/messages
        # - /var/log/*.log

.  .  .

Comment out the section “output.elasticsearch:” as we are not going to store logs directly to Elasticsearch.

Now, find the line “output.logstash” and modify the entries like below. This section defines filebeat to send logs to logstash server “server.itzgeek.local” on port “5044” and mention the path where the copied SSL certificate is placed

Replace “server.itzgeek.local” with IP address in case if you are using IP SAN.

.   .   .

output.logstash:
  # The Logstash hosts
  hosts: ["server.itzgeek.local:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/ssl/certs/logstash-forwarder.crt"]

.   .   .

Restart the service.

systemctl restart filebeat

Beats logs are typically found syslog file.

cat /var/log/filebeat/filebeat

Firewall

Configure a firewall on the ELK stack node to receive the logs from the client machines.

5044 – For Logstash to receive the logs

5061 – To access the Kibana Interface from the external machine.

firewall-cmd --permanent --zone=public --add-port=5044/tcp
firewall-cmd --permanent --zone=public --add-port=5601/tcp
firewall-cmd --reload

Install & Configure Kibana

Kibana provides visualization of logs stored on the elasticsearch. Install the Kibana using the following command.

yum -y install kibana

Edit the kibana.yml file.

vi /etc/kibana/kibana.yml

By default, Kibana listens on localhost which means you can not access Kibana interface from outside machines. To enable it, edit the below line with your machine IP.

server.host: "192.168.12.10"

Uncomment the following line and update it with the Elasticsearch instance URL. In my case, it is localhost.

elasticsearch.url: "http://localhost:9200"

Start and enable kibana on system startup.

systemctl start kibana
systemctl enable kibana

Access Kibana

Access the Kibana using the following URL.

http://your-ip-address:5601/

On your first login, you have to map the filebeat index.

Type the following in the Index name or pattern box.

filebeat-*

Select @timestamp and then click on create.

@timestamp
Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Mapping Index
Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Mapping Index

Verify your index patterns and its mappings.

Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Index Patterns
Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Index Patterns

Now, click Discover to view the incoming logs and perform search queries.

Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Search Kibana
Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Search Kibana

That’s All. You now have ELK stack running on CentOS 7 / RHEL 7.

POSTS YOU MAY LIKE -:)

You might also like
  • meto

    Everything is fine until go to http://serverIP:5601
    Unable to fetch mapping. Do you have indices matching the pattern…I couldn’t create index pattern bcoz it shows this area as transparent in the start page

  • Becky Jones

    Great guide but I get stuck installing the beats repo. Looks like its maybe out of date. Is there an alternative? I have Googled it but everyone seems to be using the same.

  • Vinod

    HI

    i tried installing ELK and configured the Filebeat, its working good for one of the client1.

    when i install it on Client2, and start Filebeat. The only one of the Client1 or Client2 logs can be see, but not both..
    any1 who has solution for this, let me know, Thanks in Advance

Updated: Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 / RHEL 7

3