Remote Monitoring with RRDTOOL

Problem: I wanted to monitor HDD temperatures and network statistics of machines on my network and feed all the data back to a central point for storage and charting. This page outlines a prototype for remote monitoring of a single machines HDD. I'll leave it as an excerise for the reader to extend to networking.

Reference

• http://martybugs.net/linux/hddtemp.cgi

rrdtools in server mode require setting up the following:

/etc/xinet.d/rrdsrv

# default: on
# description: RRDTool as a service
service rrdsrv
{
        disable         = no
        socket_type     = stream
        protocol        = tcp
        wait            = no
        user            = apache
        server          = /usr/bin/rrdtool
        server_args     = - /var/lib/rrd
}

We will run the SERVER as a non-privileged user for protection. We choose the apache user as it's going to need access to the files later to graph them anyway. Make sure the rrdtool logging directory we specified for the server exists.

# mkdir /var/lib/rrd
# chmod 755 /var/lib/rrd

Append the following to the /etc/services file. This is the well known port that rrdtools will use by default.

# Local services
rrdsrv         13900/tcp                       # RRD server

Reload XINETD to start this service.

[root@elmo xinetd.d]# service xinetd reload
Reloading configuration:                                   [  OK  ]
[root@elmo xinetd.d]#

Verify that its up and running

[root@elmo xinetd.d]# lsof -i:13900
COMMAND   PID USER   FD   TYPE DEVICE SIZE NODE NAME
xinetd  29549 root    5u  IPv4 184497       TCP *:rrdsrv (LISTEN)

Before we create a cron job and start monitoring on bingo we need an initialized .RRD database for bingo to log to. We setup the server to log all data into /var/lib/rrd so we need to create an initialize an RRD database in this location.

Run the following script once on ELMO as root:

#!/usr/bin/perl
use RRDs;
 
my $rrd = '/var/lib/rrd';
 
&ProcessHDD("bingo", "hda");
 
sub ProcessHDD
{
    my($server,$hdd) = @_;
 
    # if rrdtool database doesn't exist, create it
    if (! -e "$rrd/$server-$hdd.rrd")
    {
        print "creating rrd database for /dev/$hdd...\n";
        RRDs::create "$rrd/$server-$hdd.rrd",
                        "-s 300",
                        "DS:temp:GAUGE:600:0:100",
                        "RRA:AVERAGE:0.5:1:576",
                        "RRA:AVERAGE:0.5:6:672",
                        "RRA:AVERAGE:0.5:24:732",
                        "RRA:AVERAGE:0.5:144:1460";
    }
}

This will create the file /var/lib/rrd/bingo-hda.rrd however it will be own by root, now remember that the RRDTOOL server is running as apache and will need write access to this file to record data so adjust the owner.

# chown apache /var/lib/rrd/bingo_hda.rrd

This perl fragment is ran on BINGO it will read the temperature of the HDD and make and push the data at an RRD database stored on ELMO.

#!/usr/bin/perl
#
# rrd_hddtemp.pl
 
use IO::Socket;
 
# Remote RRD server and port
my $host = "elmo";
my $port = 13900;
 
my $socket = IO::Socket::INET->new(PeerAddr=> $host,
                                PeerPort=> $port,
                                Proto=> 'tcp',
                                Type=> SOCK_STREAM)
                                or die "Can't talk to $host at $port";
 
&ProcessHDD($socket, "bingo", "hda");
 
close $socket;
 
sub ProcessHDD
{
    my($socket,$server,$hdd) = @_;
 
    my $temp=`/usr/sbin/hddtemp -n /dev/$hdd`;
    $temp =~ s/[\n ]//g;
 
    print "/dev/$hdd : $temp degrees C\n";
 
    print $socket "update $server-$hdd.rrd -t temp N:$temp\n" ;
    $answer = <$socket>;
    print $answer;
}

Crontab entry on BINGO to push data to ELMO every 5 mins.

*/5 * * * * /usr/local/bin/rrd_hddtemp.pl >/dev/null

It wouldn't be much of a solution without being able to chart and graph that data that is being collected. To do this we create a CGI script that will dynamically generate a PNG file of the data.

This scripts will accept two arguments as part of the URL - the server and harddrive we are charting. So a query URL will look like this

http://localhost/hdd_temp.cgi?server=bingo&drive=hda

hddtemp.cgi - Place this script in /var/www/html

#!/usr/bin/perl
 
use RRDs;
use CGI qw/:standard/;
 
my $rrd = '/var/lib/rrd';
my $img = '/var/www/html';
 
print header;
 
$server = param('server');
$drive = param('drive');
 
if ("$server" && "$drive" ) {
    &ProcessHDD($server, $drive);
} else {
    print "Invalid query parameters";
}
 
sub ProcessHDD
{
    my($server, $hdd) = @_;
 
    &CreateGraph($server, $hdd, "day" );
    &CreateGraph($server, $hdd, "week");
    &CreateGraph($server, $hdd, "month");
    &CreateGraph($server, $hdd, "year");
 
    &HTML_Page($server, $hdd);
}
 
sub HTML_Page
{
    my ($server, $name) = @_;
 
    print start_html(-title=>"$server HDD temps",
                     -meta=>{'refresh'=>'200',
                             'cache-control'=>'no-cache',
                             'pragma'=>'no-cache'},
                     ),
    h1("$server HDD temps"),
    h2('Daily Graph (5 minute averages)'),   img{src=>"$server-$name-day.png"},
    h2('Weekly Graph (30 minute averages)'), img{src=>"$server-$name-week.png"},
    h2('Monthly Graph (2 hour averages)'),   img{src=>"$server-$name-month.png"},
    h2('Yearly Graph (12 hour averages)'),   img{src=>"$server-$name-year.png"},
    end_html;
}
 
# creates graph
# inputs: $hdd: hdd name (ie, hda, etc)
#         $interval: interval (ie, day, week, month, year)
 
sub CreateGraph
{
  my ($server, $hdd, $interval) = @_;
  RRDs::graph "$img/$server-$hdd-$interval.png",
              "--lazy",
              "-s -1$interval",
              "-t hdd temperature (/dev/$hdd)",
              "-h", "180", "-w", "600",
              "-a", "PNG",
              "-v degrees C",
              "DEF:temp=$rrd/$server-$hdd.rrd:temp:AVERAGE",
              "LINE2:temp#0000FF: (/dev/$hdd)",
              "GPRINT:temp:MIN:  Min\\: %2.lf",
              "GPRINT:temp:MAX: Max\\: %2.lf",
              "GPRINT:temp:AVERAGE: Avg\\: %4.1lf",
              "GPRINT:temp:LAST: Current\\: %2.lf degrees C\\n";
  if ($ERROR = RRDs::error) { print "$0: unable to generate $hdd graph: $ERROR\n"; }
}