Speed up Munin

    • Speed up Munin

      Munin is a great resource monitoring tool but can be resource taxing on the server if there are a decent number of devices to monitor.

      For my personal setup I have eighteen devices under Munin monitoring within a VM on a six disks RAID-5 spindle setup. And there is a disk I/O contention issue from time to time.


      There are two ways you can speed up Munin:
      • rrdcached
      • munin-async
      rrdcached will queue RRD changes and flush changes periodically controlled by cron job. This prevents lots of small random writes to disks.
      munin-async basically process munin-node data on the node itself instead on the server. Server connects to the node and fetch the data.

      With the combination of both above greatly reduces the server load. Especially on disk I/O.

      rrdcached:
      Install it first.

      Source Code

      1. apt-get install rrdcached

      The easiest way is run rrdcached as munin user. This requires no files/folders permission modification for existing Munin data. For my use I only utilize rrdcached for Munin use so this is how I set it up.

      Edit /etc/init.d/rrdcached and make it look like following. I commented out the original lines.

      Source Code

      1. OPTS="-s munin -l unix:/var/run/rrdcached.sock"
      2. #OPTS="-l unix:/var/run/rrdcached.sock"
      3. OPTS="$OPTS -j /var/lib/rrdcached/journal/ -F"
      4. #OPTS="$OPTS -b /var/lib/rrdcached/db/ -B"
      5. OPTS="$OPTS -b /var/lib/munin/ -B"
      6. # -w: Data written every 20mins. -z Set random delay in data write up to 15mins. -f garbage collection every 30mins.
      7. OPTS="$OPTS -w 1200 -z 900 -f 1800"

      Edit /usr/bin/munin-cron and comment out following two lines.

      Source Code

      1. #nice /usr/share/munin/munin-html $@ || exit 1
      2. #nice /usr/share/munin/munin-graph --cron $@ || exit 1

      Create a new /etc/bin/munin-graph file with following content.

      Shell-Script

      1. #!/bin/bash
      2. nice /usr/share/munin/munin-html $@ || exit 1
      3. nice /usr/share/munin/munin-graph --cron $@ || exit 1

      Edit /etc/cron.d/munin and make it look like following. Comment out the original munin-cron line. Set the nice priority on Munin updates and graphs.

      Source Code

      1. #*/5 * * * * munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi
      2. */5 * * * * munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/ionice -c 3 /usr/bin/nice -n 19 /usr/bin/munin-cron; fi
      3. */20 * * * * munin if [ -x /usr/bin/munin-graph ]; then /usr/bin/ionice -c 3 /usr/bin/nice -n 19 /usr/bin/munin-graph; fi

      Restart rrdcached and check /var/run/rrdcached.sock file. This socket file should have group munin and not root.

      Comment following line in /etc/munin/munin.conf.

      Source Code

      1. rrdcached_socket /run/munin/rrdcached.sock


      munin-async:
      Install munin-async on both the Munin server and nodes.

      Source Code

      1. apt-get install munin-async

      Once installed files in /var/lib/munin-async/ will start to populate.

      munin-async communicates over SSH so we'll need to generate SSH keys for user "munin" on the server and paste into ~munin-async/.ssh/authorized_keys on the client node.

      Change the default shell /bin/false for munin user on server to /bin/bash.

      Source Code

      1. chsh -s /bin/bash munin

      su into munin user and generate SSH keys.
      (When generating the SSH keys just press ENTER bunch times to accept the default.)

      Source Code

      1. su - munin
      2. ssh-keygen -t rsa

      Now cat server's ~munin/.ssh/id_rsa.pub file and paste into ~munin-async/.ssh/authorized_keys on the client. Prefix the line with following so looks something like... the prefixes sets security on SSH logins that only allows running that munin-async command.

      Brainfuck Source Code

      1. no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,no-user-rc,command="/usr/share/munin/munin-async --spooldir /var/lib/munin-async --spoolfetch" ssh-rsa AAAAB3N............

      While you are still su-ed as the munin user on the server you want to make a SSH connection to the client, accept the initial connection handshake, and make sure you can get in password-less. You should see some output like following when you do this.
      PTY allocation request failed on channel 0
      # munin node at testserver.testnet.net

      Now modify your /etc/munin/munin.conf file on server for the client to read something like following.

      You can see the result of doing these two tweaks. The disk I/O latency drastically improved and time takes for munin to update on server is almost instant.
      I like cars and taking pictures! 8)