Install Prometheus and Grafana

31 07 2017

These are the steps to install Prometheus and Grafana in a standalone Linux Debian box for monitor their system with a nice dashboard.

Install the package server and exporter from the repo:

apt-get install prometheus prometheus-node-exporter

Change Prometheus to listen only in localhost and the sync strategy (if you want to save battery) /etc/default/prometheus:

ARGS="-web.listen-address=localhost:9090 -storage.local.series-sync-strategy 'never'"

Assure that there is a target in /etc/prometheus/prometheus.yml:

- job_name: node
# If prometheus-node-exporter is installed, grab stats about the local
# machine by default.
static_configs:
- targets: ['localhost:9100']

Restart it and enable in systemd:

systemctl restart prometheus
systemctl enable prometheus

Change the listen address too in the node exporter /etc/default/prometheus-node-exporter:

ARGS="-collector.diskstats.ignored-devices=^(ram|loop|fd)\d+$ \
-collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/) \
-collector.textfile.directory=/var/lib/prometheus/node-exporter -web.listen-address=127.0.0.1:9100"

Restart it and enable in systemd:

systemctl restart prometheus-node-exporter
systemctl enable prometheus-node-exporter

Download Grafana from their website and install it:

mkdir /opt/grafana
cd /opt/grafana
wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_4.4.1_amd64.deb
dpkg -i grafana_4.4.1_amd64.deb

Enable and start it in systemd:

systemctl daemon-reload
systemctl enable grafana-server
systemctl start grafana-server

Use the browser and enter this address (user/password: admin/admin):

http://localhost:3000

Add the Prometheus datasource as in the image:

Add a new dashboard following this path in the menu:

Type the id “1860” inside “Grafana.com Dashboard”. It automatically get it:

Confirm the import fetching the values from localhost:


Enjoy.

Advertisements




Delete information from one host in Prometheus

30 05 2017

If we have information in Prometheus from one node and we need to remove it, the following api request solve it:

curl -X DELETE -g 'http://IP_prometheus:9090/api/v1/series?match[]={instance="IP_node_to_remove:9100"}'





Monitorizar Apache con Munin

1 10 2012

He creado varios plugins para monitorizar más el funcionamiento que tiene el servidor web Apache. A parte de los que trae munin por defecto, éstos añaden un control aún más concreto, son los siguientes:

Tamaño medio de los procesos hijos de Apache. Son los que gestionan las peticiones y, por tanto, los que debemos monitoirzar. A menor numero de módulos cargados, menor será el tamaño de éstos y el consumo de memoria:

Consumo de memoria de Apache

Media de peticiones al servidor web. Marca la media de peticiones que recibe el servidor web desde que se inició:

Peticiones medias de Apache

Numero de procesos hijos del servidor web. Monitoriza el número de procesos de Apache existentes en la máquina.

Numero de procesos de Apache

Consumo de memoria de Apache. Se registra el consumo de memoria del conjunto de todos los procesos de Apache:

consumo de memoria de Apache

Numero medio de hilos por proceso hijo. Cada proceso hijo crea un numero determinado de procesos hijo, según la configuración que tengamos aplicada:

Para que todos ellos funcionen correctamente, hay que configurar Munin, sustituyendo en cada caso los valores correspondientes, añadir a “/etc/munin//plugin-conf.d/munin-node” las siguientes líneas:
[apache_*]
env.url http://user:pass@127.0.0.1/server-status?auto
env.apuser usuario_sobre_el_que_corren_los_procesos_hijos_de_apache
env.binname nombre_del_binario_de_apache (normalmente httpd o apache)
env.ports 80
Y habilitar en Apache las estadísticas, para ello, definir un nuevo sitio, tal que así:
<VirtualHost 127.0.0.1:80>
SetHandler server-status
AuthUserFile /etc/apache/mipass.txt
AuthName Login
AuthType Basic
require valid-user
# User: user
# Pass: pass
<IfModule mod_status.c>
ExtendedStatus On
<IfModule mod_status.c>
</IfModule>
creando también el archivo de contraseñas para Apache:
htpasswd -c /etc/apache/mipass.txt userQueda, por último, reiniciar Apache y Munin y todo empezará a funcionar…

Para descargar los plugins, bajarlos del repositorio de Munin, son los siguientes:
apache_average_requests
apache_memmory
apache_servers
apache_threads
apache_tmemmory

Y se pueden descargar de:
https://github.com/munin-monitoring/contrib/tree/master/plugins/apache

Más info:
http://httpd.apache.org/
http://munin-monitoring.org/





Monitor NFS mount points with iostat and Munin

20 09 2012

I wrote several plugins to monitor the NFS mount points of a client server.
They use iostat for generate the stats, giving the following information:

– Name of the mount point.

– rBlk_nor/s (rkB_nor/s, rMB_nor) Number of blocks (kilobytes, megabytes) read by applications via the read(2) system call interface.

– wBlk_nor/s (wkB_nor/s, wMB_nor/s) Number of blocks (kilobytes, megabytes) written by applications via the write(2) system call interface.

– rBlk_dir/s (rkB_dir/s, rMB_dir/s) Number of blocks (kilobytes, megabytes) read from files opened with the O_DIRECT flag.

– wBlk_dir/s (wkB_dir/s, wMB_dir/s) Number of blocks (kilobytes, megabytes) written to files opened with the O_DIRECT flag.

– rBlk_svr/s (rkB_svr/s, rMB_svr/s) Number of blocks (kilobytes, megabytes) read from the server by the NFS client via an NFS READ request.

– wBlk_svr/s (wkB_svr/s, wMB_svr/s) Number of blocks (kilobytes, megabytes) written to the server by the NFS client via an NFS WRITE request.

– ops/s Number of operations that were issued to the filesystem per second.

– rops/s – Number of “read” operations that were issued to the filesystem per second.

– wops/s – Number of “write” operations that were issued to the filesystem per second.

Installation:

1.- Download the plugins:
https://github.com/rfrail3/munin-plugins/tree/master/nfs

2.- Put them in the Munin directory, usually “/etc/munin/plugins/”

3.- Configure munin, edit “/etc/munin/plugin-conf.d/munin-node” and add:
[nfs_iostat*]
user root
env.log /var/log/nfs-iostat.log

4.- Install systat for get iostat software:
# apt-get install systat

5.- Download the init.d script and put it in “/etc/init.d”
https://github.com/rfrail3/munin-plugins/tree/master/nfs/init.d

6.- Install and start the new script, a new log file will be created at /var/log/nfs-iostat.log:
# update-rc.d nfs-iostat defaults
/etc/init.d/nfs-iostat start

7.- Download the logrotate conf, for the new log file. Put it in “/etc/init.d/logrotate.d/”
https://github.com/rfrail3/munin-plugins/tree/master/nfs/init.d

8.- Restart Munin and enjoy:
# /etc/init.d/muin-node restart

Examples:





Instalar Nagios en Linux Debian 6 con snmp

19 01 2012

Para instalar Nagios, lo primero que tenemos que hacer es seguir la guía que se nos presenta en la documentación oficial:

http://nagios.sourceforge.net/docs/3_0/quickstart-ubuntu.html

Aunque es válida para Ubuntu, es exactamente igual para Debian, básicamente es copiar y pegar las instrucciones de la página. Posteriormente, debemos crear algún directorio y cambiar los permisos de ellos para evitar errores que aparecen después de la instalación:

# mkdir -p /usr/local/nagios/var/spool/checkresults
# chown -R nagios:nagcmd /usr/local/nagios/var/spool/checkresults
# chown -R nagios:nagcmd /usr/local/nagios/var/

Por último, instalaremos snmp y seguimos las siguientes guías para hacerlo funcionar:
# apt-get install snmpd snmp
# apt-get install libsnmp-base libsnmp-dev libsnmp-perl

http://www.debuntu.org/how-to-monitor-your-servers-with-snmp-and-cacti
http://forums.meulie.net/viewtopic.php?t=4632
http://www.pacnog.org/pacnog7/track2/configs/etc/nagios-plugins/config/snmp.cfg

En los clientes, habilitar el acceso snmp, dejando una línea del archivo /etc/snmp/snmpd.conf tal que así:
agentAddress udp::161 e instalar los mibs:
# apt-get install snmp snmpd snmp-mibs-downloader

Recompilamos los plugins de nagios y listo.

Otras referencias:
http://nagioswiki.com/wiki/index.php/Installing_Nagios_on_Debian_with_SNMP_trap_handling





Monitorizar puntos de montaje nfs con munin

18 01 2012

Par monitorizar los punto de montaje nfs en un servidor podemos utilizar los siguientes plugins que he desarrollado:

Primero colocamos un script para el init llamado /etc/init.d/nfs-iostat :

#!/bin/sh
### BEGIN INIT INFO
# Provides: nfs-iostat
# Required-Start: $local_fs $remote_fs $network $syslog $named
# Required-Stop: $local_fs $remote_fs $network $syslog $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# X-Interactive: true
# Short-Description: Start/stop nfs iostat monitor
### END INIT INFO
CLT_BIN=/usr/local/bin/iostat.sh
case "$1" in
start)
echo "Iniciando nfs iostat monitor..."
$CLT_BIN start
;;
stop)
echo "Parando nfs iostat monitor..."
killall iostat
;;
restart)
$0 stop
sleep 5
$0 start
;;
status)
if pidof -o %PPID iostat > /dev/null; then
echo "Nfs monitor iniciado."
exit 0
else
echo "Nfs monitor Parado."
exit 1
fi
;;
*)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
esac
Instalándolo con “# update-rc.d nfs-iostat defaults”

Ponemos una línea en el cron para limpiar el log:
00 00 * * * echo "" > /var/log/iostat.log; chmod ugo+r /var/log/iostat.log

Creamos el archivo /usr/local/bin/iostat.sh :
#!/bin/bash
nohup iostat -nk 300 > /var/log/iostat.log 2> /dev/null &

Creamos los archivos de los plugins en /etc/munin/plugins/:
nfs_iostat_ops:
#!/bin/sh
# -*- sh -*-
: <> /dev/null); then
NAME=`echo "$Line" | awk '{print $1} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.label $NAME"
echo "$HASH.type GAUGE"
echo "$HASH.min 0"
echo "$HASH.info I/O on nfs"
# fi
done
exit 0
fi
tac $LOG | grep -m1 Filesystem -B 50 | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
VALUE=`echo "$Line" | awk '{print $8} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.value $VALUE"
done

nfs_iostat_rkB_dir:
#!/bin/sh
# -*- sh -*-
: <> /dev/null); then
NAME=`echo "$Line" | awk '{print $1} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.label $NAME"
echo "$HASH.type GAUGE"
echo "$HASH.min 0"
echo "$HASH.info I/O on nfs"
# fi
done
exit 0
fi
tac $LOG | grep -m1 Filesystem -B 50 | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
VALUE=`echo "$Line" | awk '{print $4} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.value $VALUE"
done

nfs_iostat_rkB_nor:
#!/bin/sh
# -*- sh -*-
: <> /dev/null); then
NAME=`echo "$Line" | awk '{print $1} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.label $NAME"
echo "$HASH.type GAUGE"
echo "$HASH.min 0"
echo "$HASH.info I/O on nfs"
# fi
done
exit 0
fi
tac $LOG | grep -m1 Filesystem -B 50 | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
VALUE=`echo "$Line" | awk '{print $2} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.value $VALUE"
done

nfs_iostat_rkB_svr:
#!/bin/sh
# -*- sh -*-
: <> /dev/null); then
NAME=`echo "$Line" | awk '{print $1} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.label $NAME"
echo "$HASH.type GAUGE"
echo "$HASH.min 0"
echo "$HASH.info I/O on nfs"
# fi
done
exit 0
fi
tac $LOG | grep -m1 Filesystem -B 50 | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
VALUE=`echo "$Line" | awk '{print $6} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.value $VALUE"
done

nfs_iostat_rops:
#!/bin/sh
# -*- sh -*-
: <> /dev/null); then
NAME=`echo "$Line" | awk '{print $1} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.label $NAME"
echo "$HASH.type GAUGE"
echo "$HASH.min 0"
echo "$HASH.info I/O on nfs"
# fi
done
exit 0
fi
tac $LOG | grep -m1 Filesystem -B 50 | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
VALUE=`echo "$Line" | awk '{print $9} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.value $VALUE"
done

nfs_iostat_wkB_dir:
#!/bin/sh
# -*- sh -*-
: <> /dev/null); then
NAME=`echo "$Line" | awk '{print $1} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.label $NAME"
echo "$HASH.type GAUGE"
echo "$HASH.min 0"
echo "$HASH.info I/O on nfs"
# fi
done
exit 0
fi
tac $LOG | grep -m1 Filesystem -B 50 | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
VALUE=`echo "$Line" | awk '{print $5} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.value $VALUE"
done

nfs_iostat_wkB_nor:
#!/bin/sh
# -*- sh -*-
: <> /dev/null); then
NAME=`echo "$Line" | awk '{print $1} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.label $NAME"
echo "$HASH.type GAUGE"
echo "$HASH.min 0"
echo "$HASH.info I/O on nfs"
# fi
done
exit 0
fi
tac $LOG | grep -m1 Filesystem -B 50 | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
VALUE=`echo "$Line" | awk '{print $3} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.value $VALUE"
done

nfs_iostat_wkB_svr:

echo "no (/proc/stat not readable)"
exit 0
fi
fi
if [ "$1" = "config" ]; then
echo 'graph_title kB written by the NFS client via NFS WRITE request.'
echo 'graph_args --base 1000 -l 0 '
echo 'graph_vlabel kB / sec'
echo 'graph_scale no'
echo 'graph_category disk'
echo 'graph_info Indicate the number of kilobytes written from the server by the NFS client via an NFS WRITE request.'
# cat a | grep -A 500 Filesystem a | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
tac $LOG | grep -m1 Filesystem -B 50 | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
# if (echo $Line | grep "/" >> /dev/null); then
NAME=`echo "$Line" | awk '{print $1} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.label $NAME"
echo "$HASH.type GAUGE"
echo "$HASH.min 0"
echo "$HASH.info I/O on nfs"
# fi
done
exit 0
fi
tac $LOG | grep -m1 Filesystem -B 50 | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
VALUE=`echo "$Line" | awk '{print $7} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.value $VALUE"
done

nfs_iostat_wops:
#!/bin/sh
# -*- sh -*-
: <> /dev/null); then
NAME=`echo "$Line" | awk '{print $1} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.label $NAME"
echo "$HASH.type GAUGE"
echo "$HASH.min 0"
echo "$HASH.info I/O on nfs"
# fi
done
exit 0
fi
tac $LOG | grep -m1 Filesystem -B 50 | grep -v Filesystem | sed '/^ *$/d' | while read Line; do
VALUE=`echo "$Line" | awk '{print $10} '`
HASH=`echo "$Line" | awk '{print $1} ' | md5sum | cut -d - -f1 | sed 's/ //g'`
echo "$HASH.value $VALUE"
done

Por último, añadir al final de “/etc/munin/plugin-conf.d/munin-node”:
[nfs_iostat*]
user root

Reiniciar munin y ya debería funcionar, tal que así:

munin nfs monitor