Remote Monitoring with RRDTOOL
Problem: I wanted to monitor HDD temperatures and network statistics of machines on my network and feed all the data back to a central point for storage and charting. This page outlines a prototype for remote monitoring of a single machines HDD. I'll leave it as an excerise for the reader to extend to networking.
Reference
RRDTOOLS Server
rrdtools in server mode require setting up the following:
/etc/xinet.d/rrdsrv
# default: on # description: RRDTool as a service service rrdsrv { disable = no socket_type = stream protocol = tcp wait = no user = apache server = /usr/bin/rrdtool server_args = - /var/lib/rrd }
We will run the SERVER as a non-privileged user for protection. We choose the apache user as it's going to need access to the files later to graph them anyway. Make sure the rrdtool logging directory we specified for the server exists.
# mkdir /var/lib/rrd # chmod 755 /var/lib/rrd
Append the following to the /etc/services file. This is the well known port that rrdtools will use by default.
# Local services rrdsrv 13900/tcp # RRD server
Reload XINETD to start this service.
[root@elmo xinetd.d]# service xinetd reload Reloading configuration: [ OK ] [root@elmo xinetd.d]#
Verify that its up and running
[root@elmo xinetd.d]# lsof -i:13900 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME xinetd 29549 root 5u IPv4 184497 TCP *:rrdsrv (LISTEN)
Remote Server
Database on elmo
Before we create a cron job and start monitoring on bingo we need an initialized .RRD database for bingo to log to. We setup the server to log all data into /var/lib/rrd so we need to create an initialize an RRD database in this location.
Run the following script once on ELMO as root:
#!/usr/bin/perl use RRDs; my $rrd = '/var/lib/rrd'; &ProcessHDD("bingo", "hda"); sub ProcessHDD { my($server,$hdd) = @_; # if rrdtool database doesn't exist, create it if (! -e "$rrd/$server-$hdd.rrd") { print "creating rrd database for /dev/$hdd...\n"; RRDs::create "$rrd/$server-$hdd.rrd", "-s 300", "DS:temp:GAUGE:600:0:100", "RRA:AVERAGE:0.5:1:576", "RRA:AVERAGE:0.5:6:672", "RRA:AVERAGE:0.5:24:732", "RRA:AVERAGE:0.5:144:1460"; } }
This will create the file /var/lib/rrd/bingo-hda.rrd however it will be own by root, now remember that the RRDTOOL server is running as apache and will need write access to this file to record data so adjust the owner.
# chown apache /var/lib/rrd/bingo_hda.rrd
Remote logging from bingo
This perl fragment is ran on BINGO it will read the temperature of the HDD and make and push the data at an RRD database stored on ELMO.
#!/usr/bin/perl # # rrd_hddtemp.pl use IO::Socket; # Remote RRD server and port my $host = "elmo"; my $port = 13900; my $socket = IO::Socket::INET->new(PeerAddr=> $host, PeerPort=> $port, Proto=> 'tcp', Type=> SOCK_STREAM) or die "Can't talk to $host at $port"; &ProcessHDD($socket, "bingo", "hda"); close $socket; sub ProcessHDD { my($socket,$server,$hdd) = @_; my $temp=`/usr/sbin/hddtemp -n /dev/$hdd`; $temp =~ s/[\n ]//g; print "/dev/$hdd : $temp degrees C\n"; print $socket "update $server-$hdd.rrd -t temp N:$temp\n" ; $answer = <$socket>; print $answer; }
Crontab entry on BINGO to push data to ELMO every 5 mins.
*/5 * * * * /usr/local/bin/rrd_hddtemp.pl >/dev/null
Graphing
It wouldn't be much of a solution without being able to chart and graph that data that is being collected. To do this we create a CGI script that will dynamically generate a PNG file of the data.
This scripts will accept two arguments as part of the URL - the server and harddrive we are charting. So a query URL will look like this
http://localhost/hdd_temp.cgi?server=bingo&drive=hda
hddtemp.cgi - Place this script in /var/www/html
#!/usr/bin/perl use RRDs; use CGI qw/:standard/; my $rrd = '/var/lib/rrd'; my $img = '/var/www/html'; print header; $server = param('server'); $drive = param('drive'); if ("$server" && "$drive" ) { &ProcessHDD($server, $drive); } else { print "Invalid query parameters"; } sub ProcessHDD { my($server, $hdd) = @_; &CreateGraph($server, $hdd, "day" ); &CreateGraph($server, $hdd, "week"); &CreateGraph($server, $hdd, "month"); &CreateGraph($server, $hdd, "year"); &HTML_Page($server, $hdd); } sub HTML_Page { my ($server, $name) = @_; print start_html(-title=>"$server HDD temps", -meta=>{'refresh'=>'200', 'cache-control'=>'no-cache', 'pragma'=>'no-cache'}, ), h1("$server HDD temps"), h2('Daily Graph (5 minute averages)'), img{src=>"$server-$name-day.png"}, h2('Weekly Graph (30 minute averages)'), img{src=>"$server-$name-week.png"}, h2('Monthly Graph (2 hour averages)'), img{src=>"$server-$name-month.png"}, h2('Yearly Graph (12 hour averages)'), img{src=>"$server-$name-year.png"}, end_html; } # creates graph # inputs: $hdd: hdd name (ie, hda, etc) # $interval: interval (ie, day, week, month, year) sub CreateGraph { my ($server, $hdd, $interval) = @_; RRDs::graph "$img/$server-$hdd-$interval.png", "--lazy", "-s -1$interval", "-t hdd temperature (/dev/$hdd)", "-h", "180", "-w", "600", "-a", "PNG", "-v degrees C", "DEF:temp=$rrd/$server-$hdd.rrd:temp:AVERAGE", "LINE2:temp#0000FF: (/dev/$hdd)", "GPRINT:temp:MIN: Min\\: %2.lf", "GPRINT:temp:MAX: Max\\: %2.lf", "GPRINT:temp:AVERAGE: Avg\\: %4.1lf", "GPRINT:temp:LAST: Current\\: %2.lf degrees C\\n"; if ($ERROR = RRDs::error) { print "$0: unable to generate $hdd graph: $ERROR\n"; } }