Differences

This shows you the differences between two versions of the page.

Link to this comparison view

blog:remote_monitoring_with_rrdtools [2010/01/11 09:59] (current)
Line 1: Line 1:
 +====== Remote Monitoring with RRDTOOL ======
 +
 +Problem: I wanted to monitor HDD temperatures and network statistics of machines on my network and feed all the data back to a central point for storage and charting. ​ This page outlines a prototype for remote monitoring of a single machines HDD.  I'll leave it as an excerise for the reader to extend to networking.
 +
 +{{:​rrd.png|}}
 +
 +Reference
 +  * http://​martybugs.net/​linux/​hddtemp.cgi
 +
 +===== RRDTOOLS Server =====
 +
 +rrdtools in server mode require setting up the following:
 +
 +**/​etc/​xinet.d/​rrdsrv**
 +<​code>​
 +# default: on
 +# description:​ RRDTool as a service
 +service rrdsrv
 +{
 +        disable ​        = no
 +        socket_type ​    = stream
 +        protocol ​       = tcp
 +        wait            = no
 +        user            = apache
 +        server ​         = /​usr/​bin/​rrdtool
 +        server_args ​    = - /​var/​lib/​rrd
 +}
 +</​code>​
 +
 +We will run the SERVER as a non-privileged user for protection. ​ We choose the __apache__ user as it's going to need access to the files later to graph them anyway. Make sure the rrdtool logging directory we specified for the server exists.
 +
 +<​code>​
 +# mkdir /​var/​lib/​rrd
 +# chmod 755 /​var/​lib/​rrd
 +</​code>​
 +
 +Append the following to the **/​etc/​services** file.  This is the well known port that rrdtools will use by default.
 +<​code>​
 +# Local services
 +rrdsrv ​        ​13900/​tcp ​                      # RRD server
 +</​code>​
 +
 +Reload XINETD to start this service.
 +<​code>​
 +[root@elmo xinetd.d]# service xinetd reload
 +Reloading configuration: ​                                  ​[ ​ OK  ]
 +[root@elmo xinetd.d]#
 +</​code>​
 +
 +Verify that its up and running
 +<​code>​
 +[root@elmo xinetd.d]# lsof -i:13900
 +COMMAND ​  PID USER   ​FD ​  TYPE DEVICE SIZE NODE NAME
 +xinetd ​ 29549 root    5u  IPv4 184497 ​      TCP *:rrdsrv (LISTEN)
 +</​code>​
 +
 +===== Remote Server =====
 +
 +==== Database on elmo ====
 +
 +Before we create a cron job and start monitoring on **bingo** we need an initialized .RRD database for bingo to log to.  We setup the server to log all data into /​var/​lib/​rrd so we need to create an initialize an RRD database in this location.  ​
 +
 +Run the following script __once__ on **ELMO** as root:
 +<code perl>
 +#​!/​usr/​bin/​perl
 +use RRDs;
 +
 +my $rrd = '/​var/​lib/​rrd';​
 +
 +&​ProcessHDD("​bingo",​ "​hda"​);​
 +
 +sub ProcessHDD
 +{
 +    my($server,​$hdd) = @_;
 +
 +    # if rrdtool database doesn'​t exist, create it
 +    if (! -e "​$rrd/​$server-$hdd.rrd"​)
 +    {
 +        print "​creating rrd database for /​dev/​$hdd...\n";​
 +        RRDs::​create "​$rrd/​$server-$hdd.rrd",​
 +                        "-s 300",
 +                        "​DS:​temp:​GAUGE:​600:​0:​100",​
 +                        "​RRA:​AVERAGE:​0.5:​1:​576",​
 +                        "​RRA:​AVERAGE:​0.5:​6:​672",​
 +                        "​RRA:​AVERAGE:​0.5:​24:​732",​
 +                        "​RRA:​AVERAGE:​0.5:​144:​1460";​
 +    }
 +}
 +</​code>​
 +
 +This will create the file **/​var/​lib/​rrd/​bingo-hda.rrd** however it will be own by root, now remember that the RRDTOOL server is running as __apache__ and will need write access to this file to record data so adjust the owner.
 +<​code>​
 +# chown apache /​var/​lib/​rrd/​bingo_hda.rrd
 +</​code>​
 +
 +==== Remote logging from bingo ====
 +
 +This perl fragment is ran on **BINGO** it will read the temperature of the HDD and make and push the data at an RRD database stored on **ELMO**.
 +<code perl>
 +#​!/​usr/​bin/​perl
 +#
 +# rrd_hddtemp.pl
 +
 +use IO::Socket;
 +
 +# Remote RRD server and port
 +my $host = "​elmo";​
 +my $port = 13900;
 +
 +my $socket = IO::​Socket::​INET->​new(PeerAddr=>​ $host,
 +                                PeerPort=>​ $port,
 +                                Proto=> '​tcp',​
 +                                Type=> SOCK_STREAM)
 +                                or die "​Can'​t talk to $host at $port";​
 +
 +&​ProcessHDD($socket,​ "​bingo",​ "​hda"​);​
 +
 +close $socket;
 +
 +sub ProcessHDD
 +{
 +    my($socket,​$server,​$hdd) = @_;
 +
 +    my $temp=`/​usr/​sbin/​hddtemp -n /dev/$hdd`;
 +    $temp =~ s/[\n ]//g;
 +
 +    print "/​dev/​$hdd : $temp degrees C\n";
 +
 +    print $socket "​update $server-$hdd.rrd -t temp N:​$temp\n"​ ;
 +    $answer = <​$socket>;​
 +    print $answer;
 +}
 +</​code>​
 +
 +Crontab entry on **BINGO** to push data to **ELMO** every 5 mins.
 +<​code>​
 +*/5 * * * * /​usr/​local/​bin/​rrd_hddtemp.pl >/​dev/​null
 +</​code>​
 +
 +===== Graphing =====
 +
 +It wouldn'​t be much of a solution without being able to chart and graph that data that is being collected. ​ To do this we create a CGI script that will dynamically generate a PNG file of the data.
 +
 +This scripts will accept two arguments as part of the URL - the server and harddrive we are charting. ​ So a query URL will look like this <​code>​http://​localhost/​hdd_temp.cgi?​server=bingo&​drive=hda</​code>​
 +
 +**hddtemp.cgi** - Place this script in /​var/​www/​html
 +<code perl>
 +#​!/​usr/​bin/​perl
 +
 +use RRDs;
 +use CGI qw/:​standard/;​
 +
 +my $rrd = '/​var/​lib/​rrd';​
 +my $img = '/​var/​www/​html';​
 +
 +print header;
 +
 +$server = param('​server'​);​
 +$drive = param('​drive'​);​
 +
 +if ("​$server"​ && "​$drive"​ ) {
 +    &​ProcessHDD($server,​ $drive);
 +} else {
 +    print "​Invalid query parameters";​
 +}
 +
 +sub ProcessHDD
 +{
 +    my($server, $hdd) = @_;
 +
 +    &​CreateGraph($server,​ $hdd, "​day"​ );
 +    &​CreateGraph($server,​ $hdd, "​week"​);​
 +    &​CreateGraph($server,​ $hdd, "​month"​);​
 +    &​CreateGraph($server,​ $hdd, "​year"​);​
 +
 +    &​HTML_Page($server,​ $hdd);
 +}
 +
 +sub HTML_Page
 +{
 +    my ($server, $name) = @_;
 +
 +    print start_html(-title=>"​$server HDD temps",​
 +                     ​-meta=>​{'​refresh'​=>'​200',​
 +                             '​cache-control'​=>'​no-cache',​
 +                             '​pragma'​=>'​no-cache'​},​
 +                     ),
 +    h1("​$server HDD temps"​),​
 +    h2('​Daily Graph (5 minute averages)'​), ​  ​img{src=>"​$server-$name-day.png"​},​
 +    h2('​Weekly Graph (30 minute averages)'​),​ img{src=>"​$server-$name-week.png"​},​
 +    h2('​Monthly Graph (2 hour averages)'​), ​  ​img{src=>"​$server-$name-month.png"​},​
 +    h2('​Yearly Graph (12 hour averages)'​), ​  ​img{src=>"​$server-$name-year.png"​},​
 +    end_html;
 +}
 +
 +# creates graph
 +# inputs: $hdd: hdd name (ie, hda, etc)
 +#         ​$interval:​ interval (ie, day, week, month, year)
 +
 +sub CreateGraph
 +{
 +  my ($server, $hdd, $interval) = @_;
 +  RRDs::graph "​$img/​$server-$hdd-$interval.png",​
 +              "​--lazy",​
 +              "-s -1$interval",​
 +              "-t hdd temperature (/​dev/​$hdd)",​
 +              "​-h",​ "​180",​ "​-w",​ "​600",​
 +              "​-a",​ "​PNG",​
 +              "-v degrees C",
 +              "​DEF:​temp=$rrd/​$server-$hdd.rrd:​temp:​AVERAGE",​
 +              "​LINE2:​temp#​0000FF:​ (/​dev/​$hdd)",​
 +              "​GPRINT:​temp:​MIN: ​ Min\\: %2.lf",​
 +              "​GPRINT:​temp:​MAX:​ Max\\: %2.lf",​
 +              "​GPRINT:​temp:​AVERAGE:​ Avg\\: %4.1lf",​
 +              "​GPRINT:​temp:​LAST:​ Current\\: %2.lf degrees C\\n";
 +  if ($ERROR = RRDs::​error) { print "$0: unable to generate $hdd graph: $ERROR\n";​ }
 +}
 +</​code>​
 +
 +==== Daily Graph (5 minute averages) ====
 +
 +{{:​bingo-hda-day.png|}}
 +
 +==== Daily Graph (30 minute averages) ====
 +
 +{{:​bingo-hda-week.png|}}
 +
 +{{tag>​programming linux}}