Install Prometheus and Grafana

31 07 2017

These are the steps to install Prometheus and Grafana in a standalone Linux Debian box for monitor their system with a nice dashboard.

Install the package server and exporter from the repo:

apt-get install prometheus prometheus-node-exporter

Change Prometheus to listen only in localhost and the sync strategy (if you want to save battery) /etc/default/prometheus:

ARGS="-web.listen-address=localhost:9090 -storage.local.series-sync-strategy 'never'"

Assure that there is a target in /etc/prometheus/prometheus.yml:

- job_name: node
# If prometheus-node-exporter is installed, grab stats about the local
# machine by default.
static_configs:
- targets: ['localhost:9100']

Restart it and enable in systemd:

systemctl restart prometheus
systemctl enable prometheus

Change the listen address too in the node exporter /etc/default/prometheus-node-exporter:

ARGS="-collector.diskstats.ignored-devices=^(ram|loop|fd)\d+$ \
-collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/) \
-collector.textfile.directory=/var/lib/prometheus/node-exporter -web.listen-address=127.0.0.1:9100"

Restart it and enable in systemd:

systemctl restart prometheus-node-exporter
systemctl enable prometheus-node-exporter

Download Grafana from their website and install it:

mkdir /opt/grafana
cd /opt/grafana
wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_4.4.1_amd64.deb
dpkg -i grafana_4.4.1_amd64.deb

Enable and start it in systemd:

systemctl daemon-reload
systemctl enable grafana-server
systemctl start grafana-server

Use the browser and enter this address (user/password: admin/admin):

http://localhost:3000

Add the Prometheus datasource as in the image:

Add a new dashboard following this path in the menu:

Type the id “1860” inside “Grafana.com Dashboard”. It automatically get it:

Confirm the import fetching the values from localhost:


Enjoy.

Advertisements




Activar plugins de Varnish en Munin

19 03 2014

Si ya se tiene Munin instalado, queda sólo hacer los enlaces simbólicos siguientes dentro de “/etc/munin/plugins/”:

cd /etc/munin/plugins/
ln -sf /usr/share/munin/plugins/varnish_ varnish_expunge
ln -sf /usr/share/munin/plugins/varnish_ varnish_transfer_rates
ln -sf /usr/share/munin/plugins/varnish_ varnish_bad
ln -sf /usr/share/munin/plugins/varnish_ varnish_objects
ln -sf /usr/share/munin/plugins/varnish_ varnish_uptime
ln -sf /usr/share/munin/plugins/varnish_ varnish_request_rate
ln -sf /usr/share/munin/plugins/varnish_ varnish_memory_usage
ln -sf /usr/share/munin/plugins/varnish_ varnish_hit_rate
ln -sf /usr/share/munin/plugins/varnish_ varnish_threads
ln -sf /usr/share/munin/plugins/varnish_ varnish_backend_traffic

Y comprobar que se tienen las líneas siguientes en “/etc/munin/plugin-conf.d/munin-node”, sino, añadirlas:

[varnish_*]
env.varnishstat varnishstat
user root

Instalar las dependencias, si es que no las tiene:
# apt-get install libxml-parser-perl

Probar con:
# munin-run varnish_uptime

Reiniciar munin y esperar a ver las gráficas.





Install munin-beanstalkd in Debian

5 02 2013

For install munin plugin beanstalkd:

First, get from: https://github.com/jasiek/munin-beanstalkd/blob/master/beanstalkd.py

An run this commands:

# apt-get update
# apt-get install python python-yaml python-pip
# easy_install pip
# pip install --upgrade virtualenv
# cd /opt/
# mkdir pybeanstalk
# cd pybeanstalk
# wget http://pybeanstalk.googlecode.com/files/pybeanstalk-0.11.1.tar.gz
# tar -xzvf pybeanstalk-0.11.1.tar.gz
# cd 0.11.1/
# python setup.py install





Monitorizar Apache con Munin

1 10 2012

He creado varios plugins para monitorizar más el funcionamiento que tiene el servidor web Apache. A parte de los que trae munin por defecto, éstos añaden un control aún más concreto, son los siguientes:

Tamaño medio de los procesos hijos de Apache. Son los que gestionan las peticiones y, por tanto, los que debemos monitoirzar. A menor numero de módulos cargados, menor será el tamaño de éstos y el consumo de memoria:

Consumo de memoria de Apache

Media de peticiones al servidor web. Marca la media de peticiones que recibe el servidor web desde que se inició:

Peticiones medias de Apache

Numero de procesos hijos del servidor web. Monitoriza el número de procesos de Apache existentes en la máquina.

Numero de procesos de Apache

Consumo de memoria de Apache. Se registra el consumo de memoria del conjunto de todos los procesos de Apache:

consumo de memoria de Apache

Numero medio de hilos por proceso hijo. Cada proceso hijo crea un numero determinado de procesos hijo, según la configuración que tengamos aplicada:

Para que todos ellos funcionen correctamente, hay que configurar Munin, sustituyendo en cada caso los valores correspondientes, añadir a “/etc/munin//plugin-conf.d/munin-node” las siguientes líneas:
[apache_*]
env.url http://user:pass@127.0.0.1/server-status?auto
env.apuser usuario_sobre_el_que_corren_los_procesos_hijos_de_apache
env.binname nombre_del_binario_de_apache (normalmente httpd o apache)
env.ports 80
Y habilitar en Apache las estadísticas, para ello, definir un nuevo sitio, tal que así:
<VirtualHost 127.0.0.1:80>
SetHandler server-status
AuthUserFile /etc/apache/mipass.txt
AuthName Login
AuthType Basic
require valid-user
# User: user
# Pass: pass
<IfModule mod_status.c>
ExtendedStatus On
<IfModule mod_status.c>
</IfModule>
creando también el archivo de contraseñas para Apache:
htpasswd -c /etc/apache/mipass.txt userQueda, por último, reiniciar Apache y Munin y todo empezará a funcionar…

Para descargar los plugins, bajarlos del repositorio de Munin, son los siguientes:
apache_average_requests
apache_memmory
apache_servers
apache_threads
apache_tmemmory

Y se pueden descargar de:
https://github.com/munin-monitoring/contrib/tree/master/plugins/apache

Más info:
http://httpd.apache.org/
http://munin-monitoring.org/





Monitor NFS mount points with iostat and Munin

20 09 2012

I wrote several plugins to monitor the NFS mount points of a client server.
They use iostat for generate the stats, giving the following information:

– Name of the mount point.

– rBlk_nor/s (rkB_nor/s, rMB_nor) Number of blocks (kilobytes, megabytes) read by applications via the read(2) system call interface.

– wBlk_nor/s (wkB_nor/s, wMB_nor/s) Number of blocks (kilobytes, megabytes) written by applications via the write(2) system call interface.

– rBlk_dir/s (rkB_dir/s, rMB_dir/s) Number of blocks (kilobytes, megabytes) read from files opened with the O_DIRECT flag.

– wBlk_dir/s (wkB_dir/s, wMB_dir/s) Number of blocks (kilobytes, megabytes) written to files opened with the O_DIRECT flag.

– rBlk_svr/s (rkB_svr/s, rMB_svr/s) Number of blocks (kilobytes, megabytes) read from the server by the NFS client via an NFS READ request.

– wBlk_svr/s (wkB_svr/s, wMB_svr/s) Number of blocks (kilobytes, megabytes) written to the server by the NFS client via an NFS WRITE request.

– ops/s Number of operations that were issued to the filesystem per second.

– rops/s – Number of “read” operations that were issued to the filesystem per second.

– wops/s – Number of “write” operations that were issued to the filesystem per second.

Installation:

1.- Download the plugins:
https://github.com/rfrail3/munin-plugins/tree/master/nfs

2.- Put them in the Munin directory, usually “/etc/munin/plugins/”

3.- Configure munin, edit “/etc/munin/plugin-conf.d/munin-node” and add:
[nfs_iostat*]
user root
env.log /var/log/nfs-iostat.log

4.- Install systat for get iostat software:
# apt-get install systat

5.- Download the init.d script and put it in “/etc/init.d”
https://github.com/rfrail3/munin-plugins/tree/master/nfs/init.d

6.- Install and start the new script, a new log file will be created at /var/log/nfs-iostat.log:
# update-rc.d nfs-iostat defaults
/etc/init.d/nfs-iostat start

7.- Download the logrotate conf, for the new log file. Put it in “/etc/init.d/logrotate.d/”
https://github.com/rfrail3/munin-plugins/tree/master/nfs/init.d

8.- Restart Munin and enjoy:
# /etc/init.d/muin-node restart

Examples:





Monitor Haproxy with Munin

15 09 2012

I wrote several plugins to monitor haproxy with munin. Basically, they draw the ​values in the ​Haproxy csv  file.

The installation is very simple:

– Install dependencies:
# apt-get install curl socat coreutils

– Download plugins from Munin Contrib repository:

haproxy_abort_backend
haproxy_active_backend
haproxy_bytes_backend
haproxy_bytes_frontend
haproxy_denied_backend
haproxy_denied_frontend
haproxy_errors_backend
haproxy_errors_frontend
haproxy_queue_backend
haproxy_rate_backend
haproxy_rate_frontend
haproxy_reqrate_frontend
haproxy_responses_backend
haproxy_responses_frontend
haproxy_sessions_backend
haproxy_sessions_frontend
haproxy_sessions_total_backend
haproxy_sessions_total_frontend
haproxy_warnings_backend

https://github.com/munin-monitoring/contrib/tree/master/plugins/haproxy

– Copy the files to “/etc/munin/plugins/”

– Edit the file “/etc/munin/plugin-conf.d/munin-node” and add the following content. You can monitor Haproxy trought http or unix sockets, this is http method:
[haproxy*]
user root
env.backend back-name1 back-name2 back-name3 back-back-name4
env.frontend front-name1 front-name2 front-name3
env.url http://USER:PASSWD@IP:PORT/admin?stats;csv

and this if for unix sockets:
[haproxy*]
user root
env.backend back-name1 back-name2 back-name3 back-back-name4
env.frontend front-name1 front-name2 front-name3
env.socket /var/lib/haproxy/stats.socket

In Haproxy, you can configure the request of the csv trought unix sockets or trought http:

Trought unix socket, add this to the global section (change user and group with the correc user and group):
stats socket /var/run/haproxy.sock mode 0644 user group

and ensure that this works with:
# echo 'show stat' | socat unix-connect:/var/lib/haproxy/stats.socket stdio

Trought http, add this to the conf, basically create a listen: (change ip, port, user, pass):
listen proxy-stats IP:PORT
mode http
option httplog
stats enable
stats show-legends
stats uri /admin?stats
stats realm Haproxy\ Statistics
stats auth user:pass
stats refresh 5s

and ensure that this works with:
# curl 'http://USER:PASSWD@IP:PORT/admin?stats;csv'

– Restart munin-node and wait a few minutes until the graphs will be painted.
# /etc/init.d/munin-node restart

Here are several examples of the graphs:

haproxy_abort_backend-day.png

haproxy_active_backend-day.png

haproxy_bytes_backend-day.png

haproxy_bytes_frontend-day.png

haproxy_denied_backend-day.png

haproxy_denied_frontend-day.png

haproxy_errors_backend-day.png

haproxy_errors_frontend-day.png

haproxy_queue_backend-day.png

haproxy_rate_backend-day.png

haproxy_rate_frontend-day.png

haproxy_reqrate_frontend-day.png

haproxy_responses_backend-day.png

haproxy_responses_frontend-day.png

haproxy_sessions_backend-day.png

haproxy_sessions_frontend-day.png

haproxy_sessions_total_backend-day.png

haproxy_sessions_total_frontend-day.png

haproxy_warnings_backend-day.png

More info:
http://haproxy.1wt.eu/
http://munin-monitoring.org/





Instalar munin-node en RedHat / CentOS 5

10 09 2012

Para instalar munin-node en una máquina RedHat o CentOS 5 hay que utilizar los repositorios dag y forzar algunos paquetes, ya que no se encuentran todos disponibles para ésta release del sistema.

Aquí van los pasos, básicamente, crear un repo nuevo, bajar varios rpm e instalarlos a mano y finalmente instalar munin-node:

cat > /etc/yum.repos.d/dag.repo << FIN
[dag]
name=Dag RPM Repository for Red Hat Enterprise Linux
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag
gpgcheck=1
gpgkey=http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt
enabled=1
FIN
cd /var/cache/yum/dag/packages/
wget 'http://pkgs.repoforge.org/perl-Net-Server/perl-Net-Server-0.90-2.el5.rf.noarch.rpm&#039;
yum install perl-Net-Server-0.90-2.el5.rf.noarch.rpm -y
wget 'http://pkgs.repoforge.org/perl-Socket6/perl-Socket6-0.23-1.el5.rfx.x86_64.rpm&#039;
yum install perl-Socket6-0.23-1.el5.rfx.x86_64.rpm -y
wget 'http://pkgs.repoforge.org/perl-Crypt-DES/perl-Crypt-DES-2.05-3.2.el5.rf.x86_64.rpm&#039;
yum install perl-Crypt-DES-2.05-3.2.el5.rf.x86_64.rpm -y
wget 'http://pkgs.repoforge.org/perl-Net-SNMP/perl-Net-SNMP-5.2.0-1.2.el5.rf.noarch.rpm&#039;
rpm -ivh --force perl-Net-SNMP-5.2.0-1.2.el5.rf.noarch.rpm --nodeps
yum -y install munin-node
/etc/init.d/munin-node restart