blog:remote_monitoring_with_rrdtools

no way to compare when less than two revisions

Differences

This shows you the differences between two versions of the page.


—blog:remote_monitoring_with_rrdtools [2010/01/11 09:59] (current) – created - external edit 127.0.0.1
Line 1: Line 1:
 +====== Remote Monitoring with RRDTOOL ======
 +
 +Problem: I wanted to monitor HDD temperatures and network statistics of machines on my network and feed all the data back to a central point for storage and charting.  This page outlines a prototype for remote monitoring of a single machines HDD.  I'll leave it as an excerise for the reader to extend to networking.
 +
 +{{:rrd.png|}}
 +
 +Reference
 +  * http://martybugs.net/linux/hddtemp.cgi
 +
 +===== RRDTOOLS Server =====
 +
 +rrdtools in server mode require setting up the following:
 +
 +**/etc/xinet.d/rrdsrv**
 +<code>
 +# default: on
 +# description: RRDTool as a service
 +service rrdsrv
 +{
 +        disable         = no
 +        socket_type     = stream
 +        protocol        = tcp
 +        wait            = no
 +        user            = apache
 +        server          = /usr/bin/rrdtool
 +        server_args     = - /var/lib/rrd
 +}
 +</code>
 +
 +We will run the SERVER as a non-privileged user for protection.  We choose the __apache__ user as it's going to need access to the files later to graph them anyway. Make sure the rrdtool logging directory we specified for the server exists.
 +
 +<code>
 +# mkdir /var/lib/rrd
 +# chmod 755 /var/lib/rrd
 +</code>
 +
 +Append the following to the **/etc/services** file.  This is the well known port that rrdtools will use by default.
 +<code>
 +# Local services
 +rrdsrv         13900/tcp                       # RRD server
 +</code>
 +
 +Reload XINETD to start this service.
 +<code>
 +[root@elmo xinetd.d]# service xinetd reload
 +Reloading configuration:                                   [  OK  ]
 +[root@elmo xinetd.d]#
 +</code>
 +
 +Verify that its up and running
 +<code>
 +[root@elmo xinetd.d]# lsof -i:13900
 +COMMAND   PID USER   FD   TYPE DEVICE SIZE NODE NAME
 +xinetd  29549 root    5u  IPv4 184497       TCP *:rrdsrv (LISTEN)
 +</code>
 +
 +===== Remote Server =====
 +
 +==== Database on elmo ====
 +
 +Before we create a cron job and start monitoring on **bingo** we need an initialized .RRD database for bingo to log to.  We setup the server to log all data into /var/lib/rrd so we need to create an initialize an RRD database in this location.  
 +
 +Run the following script __once__ on **ELMO** as root:
 +<code perl>
 +#!/usr/bin/perl
 +use RRDs;
 +
 +my $rrd = '/var/lib/rrd';
 +
 +&ProcessHDD("bingo", "hda");
 +
 +sub ProcessHDD
 +{
 +    my($server,$hdd) = @_;
 +
 +    # if rrdtool database doesn't exist, create it
 +    if (! -e "$rrd/$server-$hdd.rrd")
 +    {
 +        print "creating rrd database for /dev/$hdd...\n";
 +        RRDs::create "$rrd/$server-$hdd.rrd",
 +                        "-s 300",
 +                        "DS:temp:GAUGE:600:0:100",
 +                        "RRA:AVERAGE:0.5:1:576",
 +                        "RRA:AVERAGE:0.5:6:672",
 +                        "RRA:AVERAGE:0.5:24:732",
 +                        "RRA:AVERAGE:0.5:144:1460";
 +    }
 +}
 +</code>
 +
 +This will create the file **/var/lib/rrd/bingo-hda.rrd** however it will be own by root, now remember that the RRDTOOL server is running as __apache__ and will need write access to this file to record data so adjust the owner.
 +<code>
 +# chown apache /var/lib/rrd/bingo_hda.rrd
 +</code>
 +
 +==== Remote logging from bingo ====
 +
 +This perl fragment is ran on **BINGO** it will read the temperature of the HDD and make and push the data at an RRD database stored on **ELMO**.
 +<code perl>
 +#!/usr/bin/perl
 +#
 +# rrd_hddtemp.pl
 +
 +use IO::Socket;
 +
 +# Remote RRD server and port
 +my $host = "elmo";
 +my $port = 13900;
 +
 +my $socket = IO::Socket::INET->new(PeerAddr=> $host,
 +                                PeerPort=> $port,
 +                                Proto=> 'tcp',
 +                                Type=> SOCK_STREAM)
 +                                or die "Can't talk to $host at $port";
 +
 +&ProcessHDD($socket, "bingo", "hda");
 +
 +close $socket;
 +
 +sub ProcessHDD
 +{
 +    my($socket,$server,$hdd) = @_;
 +
 +    my $temp=`/usr/sbin/hddtemp -n /dev/$hdd`;
 +    $temp =~ s/[\n ]//g;
 +
 +    print "/dev/$hdd : $temp degrees C\n";
 +
 +    print $socket "update $server-$hdd.rrd -t temp N:$temp\n" ;
 +    $answer = <$socket>;
 +    print $answer;
 +}
 +</code>
 +
 +Crontab entry on **BINGO** to push data to **ELMO** every 5 mins.
 +<code>
 +*/5 * * * * /usr/local/bin/rrd_hddtemp.pl >/dev/null
 +</code>
 +
 +===== Graphing =====
 +
 +It wouldn't be much of a solution without being able to chart and graph that data that is being collected.  To do this we create a CGI script that will dynamically generate a PNG file of the data.
 +
 +This scripts will accept two arguments as part of the URL - the server and harddrive we are charting.  So a query URL will look like this <code>http://localhost/hdd_temp.cgi?server=bingo&drive=hda</code>
 +
 +**hddtemp.cgi** - Place this script in /var/www/html
 +<code perl>
 +#!/usr/bin/perl
 +
 +use RRDs;
 +use CGI qw/:standard/;
 +
 +my $rrd = '/var/lib/rrd';
 +my $img = '/var/www/html';
 +
 +print header;
 +
 +$server = param('server');
 +$drive = param('drive');
 +
 +if ("$server" && "$drive" ) {
 +    &ProcessHDD($server, $drive);
 +} else {
 +    print "Invalid query parameters";
 +}
 +
 +sub ProcessHDD
 +{
 +    my($server, $hdd) = @_;
 +
 +    &CreateGraph($server, $hdd, "day" );
 +    &CreateGraph($server, $hdd, "week");
 +    &CreateGraph($server, $hdd, "month");
 +    &CreateGraph($server, $hdd, "year");
 +
 +    &HTML_Page($server, $hdd);
 +}
 +
 +sub HTML_Page
 +{
 +    my ($server, $name) = @_;
 +
 +    print start_html(-title=>"$server HDD temps",
 +                     -meta=>{'refresh'=>'200',
 +                             'cache-control'=>'no-cache',
 +                             'pragma'=>'no-cache'},
 +                     ),
 +    h1("$server HDD temps"),
 +    h2('Daily Graph (5 minute averages)'),   img{src=>"$server-$name-day.png"},
 +    h2('Weekly Graph (30 minute averages)'), img{src=>"$server-$name-week.png"},
 +    h2('Monthly Graph (2 hour averages)'),   img{src=>"$server-$name-month.png"},
 +    h2('Yearly Graph (12 hour averages)'),   img{src=>"$server-$name-year.png"},
 +    end_html;
 +}
 +
 +# creates graph
 +# inputs: $hdd: hdd name (ie, hda, etc)
 +#         $interval: interval (ie, day, week, month, year)
 +
 +sub CreateGraph
 +{
 +  my ($server, $hdd, $interval) = @_;
 +  RRDs::graph "$img/$server-$hdd-$interval.png",
 +              "--lazy",
 +              "-s -1$interval",
 +              "-t hdd temperature (/dev/$hdd)",
 +              "-h", "180", "-w", "600",
 +              "-a", "PNG",
 +              "-v degrees C",
 +              "DEF:temp=$rrd/$server-$hdd.rrd:temp:AVERAGE",
 +              "LINE2:temp#0000FF: (/dev/$hdd)",
 +              "GPRINT:temp:MIN:  Min\\: %2.lf",
 +              "GPRINT:temp:MAX: Max\\: %2.lf",
 +              "GPRINT:temp:AVERAGE: Avg\\: %4.1lf",
 +              "GPRINT:temp:LAST: Current\\: %2.lf degrees C\\n";
 +  if ($ERROR = RRDs::error) { print "$0: unable to generate $hdd graph: $ERROR\n"; }
 +}
 +</code>
 +
 +==== Daily Graph (5 minute averages) ====
 +
 +{{:bingo-hda-day.png|}}
 +
 +==== Daily Graph (30 minute averages) ====
 +
 +{{:bingo-hda-week.png|}}
 +
 +{{tag>programming linux}}
  
  • blog/remote_monitoring_with_rrdtools.txt
  • Last modified: 2010/01/11 09:59
  • by 127.0.0.1