Munin is a great resource monitoring tool. However, it can easily max out the server resource depending on number of monitored devices.
I’ve ran into some disk I/O contention issue on my Munin server. I don’t even have that many monitored devices… only eighteen.
There are couple ways to speed up Munin. With the combination of both greatly reduces server load… especially disk I/O.
rrdcached will queue RRD changes and flush changes periodically controlled by cron job. This prevents lots of small random writes to disks.
munin-async basically process munin-node data on client itself instead of on server. Server connects to the client and fetch data.
Install rrdcached on Debian 9.
apt-get install rrdcached
The easiest way is run rrdcached is as the munin user. This requires no files/folders permission modification for existing Munin data. For my use I only utilize rrdcached for Munin use so this is how I set it up.
Edit /etc/default/rrdcached and make it look like following.
BASE_PATH=/var/lib/munin/ SOCKGROUP=munin # -w: Data written every 20mins. -z Set random delay in data write up to 15mins. -f garbage collection every 30mins. BASE_OPTIONS="-B -w 1200 -z 900 -f 1800"
Edit /usr/bin/munin-cron and comment out following two lines.
#nice /usr/share/munin/munin-html $@ || exit 1 #nice /usr/share/munin/munin-graph --cron $@ || exit 1
Create a new /etc/bin/munin-graph file with following content. Make sure it’s executable.
#!/bin/bash nice /usr/share/munin/munin-html $@ || exit 1 nice /usr/share/munin/munin-graph --cron $@ || exit 1
Edit /etc/cron.d/munin and make it look like following. Comment out the original munin-cron line. Set the nice priority on Munin updates and graphs.
#*/5 * * * * munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi */5 * * * * munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/ionice -c 3 /usr/bin/nice -n 19 /usr/bin/munin-cron; fi */20 * * * * munin if [ -x /usr/bin/munin-graph ]; then /usr/bin/ionice -c 3 /usr/bin/nice -n 19 /usr/bin/munin-graph; fi
Restart rrdcached and check /var/run/rrdcached.sock file. This socket file should have group munin and not root.
Ensure rrdcached_socket in /etc/munin/munin.conf point to correct rrdcached sock file.
Install munin-async on Munin client (Debian 9). Once installed data in /var/lib/munin-async/ will start to populate.
apt-get install munin-async
munin-async communicates over SSH so we’ll need to generate SSH keys for user “munin” on the server and paste into ~munin-async/.ssh/authorized_keys on the client.
Change the default shell /bin/false for munin user on server to /bin/bash.
chsh -s /bin/bash munin
su into munin user and generate SSH keys.
(When generating the SSH keys just press ENTER bunch times to accept the default.)
su - munin ssh-keygen -t rsa
Now copy content of server’s ~munin/.ssh/id_rsa.pub file and paste into ~munin-async/.ssh/authorized_keys on the client. Prefix the line before “ssh-rsa” so looks something like following. The prefix sets security on SSH logins that only allows running that munin-async command.
no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,no-user-rc,command="/usr/share/munin/munin-async --spooldir /var/lib/munin-async --spoolfetch" ssh-rsa AAAAB3N............
While you are still su-ed as the munin user on server you want to SSH to the client at least once, accept the initial connection handshake, and make sure you can get in password-less. You should see some output like following when you do this.
PTY allocation request failed on channel 0 # munin node at testserver.testnet.net
Now modify client part your /etc/munin/munin.conf file on server to read something like following.
[testserver.testnet.net] address ssh://email@example.com /usr/share/munin/munin-async --spooldir /var/lib/munin-async --spoolfetch use_node_name yes
The disk I/O latency drastically improved. The amount of time takes for munin to update on server is almost instant.