Munin performance boost

Munin is a great resource monitoring tool. However, it can easily max out the server resource depending on number of monitored devices.

I’ve ran into some disk I/O contention issue on my Munin server. I don’t even have that many monitored devices… only eighteen.

 

There are couple ways to speed up Munin. With the combination of both greatly reduces server load… especially disk I/O.

  • rrdcached
  • munin-async

rrdcached will queue RRD changes and flush changes periodically controlled by cron job. This prevents lots of small random writes to disks.

munin-async basically process munin-node data on client itself instead of on server. Server connects to the client and fetch data.

 

rrdcached:

Install rrdcached on Debian 9.

apt-get install rrdcached

 

The easiest way is run rrdcached is as the munin user. This requires no files/folders permission modification for existing Munin data. For my use I only utilize rrdcached for Munin use so this is how I set it up.

Edit /etc/default/rrdcached and make it look like following.

BASE_PATH=/var/lib/munin/

SOCKGROUP=munin

# -w: Data written every 20mins. -z Set random delay in data write up to 15mins. -f garbage collection every 30mins.
BASE_OPTIONS="-B -w 1200 -z 900 -f 1800"

 

Edit /usr/bin/munin-cron and comment out following two lines.

#nice /usr/share/munin/munin-html $@ || exit 1

#nice /usr/share/munin/munin-graph --cron $@ || exit 1

 

Create a new /etc/bin/munin-graph file with following content. Make sure it’s executable.

#!/bin/bash
nice /usr/share/munin/munin-html $@ || exit 1
nice /usr/share/munin/munin-graph --cron $@ || exit 1

 

Edit /etc/cron.d/munin and make it look like following. Comment out the original munin-cron line. Set the nice priority on Munin updates and graphs.

#*/5 * * * * munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi
*/5 * * * * munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/ionice -c 3 /usr/bin/nice -n 19 /usr/bin/munin-cron; fi
*/20 * * * * munin if [ -x /usr/bin/munin-graph ]; then /usr/bin/ionice -c 3 /usr/bin/nice -n 19 /usr/bin/munin-graph; fi

 

Restart rrdcached and check /var/run/rrdcached.sock file. This socket file should have group munin and not root.

 

Ensure rrdcached_socket in /etc/munin/munin.conf point to correct rrdcached sock file.

rrdcached_socket /var/run/rrdcached.sock

 

munin-async:

Install munin-async on Munin client (Debian 9). Once installed data in /var/lib/munin-async/ will start to populate.

apt-get install munin-async

 

munin-async communicates over SSH so we’ll need to generate SSH keys for user “munin” on the server and paste into ~munin-async/.ssh/authorized_keys on the client.

Change the default shell /bin/false for munin user on server to /bin/bash.

chsh -s /bin/bash munin

su into munin user and generate SSH keys.
(When generating the SSH keys just press ENTER bunch times to accept the default.)

su - munin
ssh-keygen -t rsa

 

Now copy content of server’s ~munin/.ssh/id_rsa.pub file and paste into ~munin-async/.ssh/authorized_keys on the client. Prefix the line before “ssh-rsa” so looks something like following. The prefix sets security on SSH logins that only allows running that munin-async command.

no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,no-user-rc,command="/usr/share/munin/munin-async --spooldir /var/lib/munin-async --spoolfetch" ssh-rsa AAAAB3N............

 

While you are still su-ed as the munin user on server you want to SSH to the client at least once, accept the initial connection handshake, and make sure you can get in password-less. You should see some output like following when you do this.

PTY allocation request failed on channel 0
# munin node at testserver.testnet.net

 

Now modify client part your /etc/munin/munin.conf file on server to read something like following.

[testserver.testnet.net]
address ssh://munin-async@testserver.testnet.net /usr/share/munin/munin-async --spooldir /var/lib/munin-async --spoolfetch
use_node_name yes

 

The disk I/O latency drastically improved. The amount of time takes for munin to update on server is almost instant.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.