How to Add Remote Linux Host into Sensu Monitoring Tool

0

Sensu clients are monitoring agents, installed and run on every server that are being monitored by Sensu tool. The work of a Sensu client is to register a system with Sensu, sending Keepalive messages, and executing monitoring checks.

Each client can be a member of one or more subscriptions – you can simply say; a list of roles assigned to them (Ex, web server, database, etc. ).

When a client subscribes to the check requests published by the Sensu server (via RabbitMQ – Sensu transport), it executes them locally, and publish the result of the check back to Sensu transport, and then a Sensu server will process the results.

In this guide, we will configure client machines to send results of service checks to the Sensu server. Assuming that you have followed our previous articles on Sensu core.

READ: Install Sensu on Ubuntu 16.04 / Debian 8

Sensu core does not come with the graphical dashboard, so you would need to install Uchiwa – an open source dashboard for Sensu monitoring tool.

READ: How to Install Uchiwa – Open Source Dashboard for Sensu Monitoring Tool

Add Sensu Repositories:

Depending upon your operating system, configure the Sensu repository to fetch the package from the official website.

### CentOS 7 / RHEL 7 ###

echo '[sensu]
name=sensu
baseurl=https://sensu.global.ssl.fastly.net/yum/7/x86_64/
gpgcheck=0
enabled=1' | sudo tee /etc/yum.repos.d/sensu.repo

### Ubuntu 16.04 / Debian 8 ###

sudo apt-get update && sudo apt-get -y install apt-transport-https
wget -q https://sensu.global.ssl.fastly.net/apt/pubkey.gpg -O- | sudo apt-key add -
echo "deb     https://sensu.global.ssl.fastly.net/apt sensu main" | sudo tee /etc/apt/sources.list.d/sensu.list
sudo apt-get update

Install Sensu using the following command.

### CentOS 7 / RHEL 7 ### 

yum -y install sensu

### Ubuntu 16.04 / Debian 8 ###

sudo apt-get -y install sensu

Register a System:

The default configuration directory of Sensu is /etc/sensu/config.d, some cases, you may need to create this directory manually if not available.

Configuration files should be in JSON format. Create client.json file.

vi /etc/sensu/conf.d/client.json

Place the following content into the above file.

{
"client": {
  "name": "debian.itzgeek.local",
  "address": "10.128.0.4",
  "environment": "production",
  "subscriptions": [
    "web",
    "debian"
  ]
 }
}

Where,

client – Name for the client, should be a unique name. My machine is Debian 8.

address –  IP Address or hostname of the client.

environment –  A custom attribute, not part of Sensu client specification. These custom attributes by check by Sensu filters which help you to do specific actions based on environment (Ex, only alert on events in the production environment)

subscriptions –  List of roles or responsibilities assigned to the client.

Configure Sensu client to connect with Sensu transport. Create transport.json file.

vi /etc/sensu/conf.d/transport.json

Place the following content into the above file.

{
 "transport": {
   "name": "rabbitmq",
   "reconnect_on_error": true
  }
 }

Configure Sensu transport details by creating rabbitmq.json.

vi /etc/sensu/conf.d/rabbitmq.json

Copy and paste the below contents into the above file. Replace greened values depending upon your Sensu setup.

{
  "rabbitmq": {
    "host": "10.128.0.3",
    "port": 5672,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "password"
  }
}

Where,

host – RabbitMQ server IP (Sensu server)

port – RabbitMQ server port

vhost, user, and password – RabbitMQ virtual host, user, and password which we created in during the installation of Sensu server.

Restart the Sensu client service using the following command.

service sensu-client restart

Open up the Uchiwa dashboard and see whether the new client is successfully registered on Sensu server.

Add Remote Linux Host into Sensu Monitoring Tool - Sensu Clients List
Add Remote Linux Host into Sensu Monitoring Tool – Sensu Clients List

Below screenshot shows you the detail of a newly added client.

Add Remote Linux Host into Sensu Monitoring Tool - Overview of a Client
Add Remote Linux Host into Sensu Monitoring Tool – Overview of a Client

Now, it’s time to configure Sensu to monitor your infrastructure.

Monitor a Service:

Sensu uses checks (commands) to monitor an infrastructure, it is executed by Sensu client to monitor a condition (ex, checking the status of the service) or collect measurements (ex, how much memory is used).

Result data of standard checks are typically in a human-readable message, and metric checks would typically contain measurements gathered by check. Also, exit code status indicates state; 0 – OK, 1 – WARNING, and 2 – CRITICAL, similar to Nagios; means you can use Nagios plugins with Sensu without modification.

These checks are scheduled either by Sensu server or Sensu client.

Subscription checks:

These checks are centrally defined and scheduled by Sensu server, and they have a defined set of subscribers ( you could say – a role) and sensu clients become subscribers to these roles (i.e. subscription) via “subscription” attribute.

Let’s monitor the Apache service using the “Subscription checks” with the interval of 60 seconds. On sensu server, create a configuration file for “Subscription checks” on Sensu server.

raj@server:~$ sudo vi /etc/sensu/conf.d/apache_check.json

Place the following content into above .json file.

If the client is Ubuntu 16.04 / Debian 8:

{
  "checks": {
    "apache-status": {
      "command": "check-process.rb -p apache2",
      "subscribers": [
        "web"
      ],
      "interval": 60    
    }
  }
}

If the client is CentOS 7 / RHEL 7:

{
  "checks": {
    "apache-status": {
      "command": "check-process.rb -p httpd",
      "subscribers": [
        "web"
      ],
      "interval": 60    
    }
  }
}

Restart the “sensu-server” using the following command.

raj@server:~$ sudo service sensu-server restart

Make sure the client machine (ex, debian.itzgeek.local) is subscribed to “web“, if not, update the client.json file.

vi /etc/sensu/conf.d/client.json

The file should look like below.

{
"client": {
  "name": "debian.itzgeek.local",
  "address": "10.128.0.4",
  "environment": "production",
  "subscriptions": [
    "web",
    "debian"
  ]
 }
}

Sensu plugins provide an executable script that can be used as sensu checks to monitor system resources, services, and application health. Install Sensu plugin for process checks “sensu-plugins-process-checks” using gem command on the client machine (debian.itzgeek.local).

/opt/sensu/embedded/bin/gem install sensu-plugins-process-checks

Open up Uchiwa dashboard and wait for some time, new check “apache-status” will show up in a minute. If the Apache process is not running or not found, you may get a critical event message like below in the dashboard.

Add Remote Linux Host into Sensu Monitoring Tool - Subscription Checks - Service in Critical Condition
Add Remote Linux Host into Sensu Monitoring Tool – Subscription Checks – Service in Critical Condition

If the Apache process is running, then the dashboard might look like below.

Add Remote Linux Host into Sensu Monitoring Tool - Subscription Checks - Service in OK Condition
Add Remote Linux Host into Sensu Monitoring Tool – Subscription Checks – Service in OK Condition

I hope you now understood how “Subscription checks” work in Sensu. We will see “Standalone checks” in the next section.

Standalone checks:

These checks are defined on a Sensu client (debian.itzgeek.local) with the check attribute “standalone” set to “true“. Here, Sensu client uses its own scheduler for scheduling standalone checks and sends the result of checks to Sensu server.

Let’s monitor the Disk Space using the “Standalone checks” with the interval of 60 seconds. On the client machine (debian.itzgeek.local), install Sensu disk space plugin using the following command.

/opt/sensu/embedded/bin/gem install sensu-plugins-disk-checks

Create a configuration file for disk check under “/etc/sensu/conf.d” directory.

vi /etc/sensu/conf.d/disk_check.json

Use the below content for this check.

{
  "checks": {
    "disk-status": {
      "command": "check-disk-usage.rb -w 70 -c 90",
      "standalone": true,
      "interval": 60
   }
 }
}

Restart the sensu-client service.

service sensu-client restart

Open up Uchiwa dashboard and see whether the new check is visible on the dashboard.

Add Remote Linux Host into Sensu Monitoring Tool - Standalone checks - Disk Usage Under Threshold
Add Remote Linux Host into Sensu Monitoring Tool – Standalone checks – Disk Usage Under Threshold

Below image shows the disk usage is in warning condition.

Add Remote Linux Host into Sensu Monitoring Tool - Standalone checks - Disk Usage Above Warning
Add Remote Linux Host into Sensu Monitoring Tool – Standalone checks – Disk Usage Above Warning

Interested in Server Monitoring:

READ: Install and Configure Nagios on CentOS 7 / Ubuntu 16.04 / Debian 8

READ: Install Icinga2 on Ubuntu 16.04 / 14.04 , Debian, and Mint

READ: How To Install Zabbix Server 3.2 on CentOS 7 / Ubuntu 16.04 / Debian 8

READ: Install Monitorix on CentOS 7 / Ubuntu 16.04 / Fedora 24 / Debian 8 – OpenSource Lightweight System Monitoring Tool

READ: Netdata – Real Time Performance Monitoring Tool for Linux

That’s All. Thank you for reading.

You might also like