vCenter VM monitoring with Graphite

This was done on a CentOS 6.3 server your mileage may vary on another platform.

The problem I was trying to solve was that I wanted to monitor the vitals of all my VM's without having to install collectd into each VM, by talking to vCenter we can pull everything out that we need. This work was inspired by collectd-vcenter which I could not get to work as its assumes you have an ESX Cluster. So I wrote my own.

See details on setting up graphite which I used for my xAP monitoring

We don't use the collectd RPM from the EPEL repository as its too old and does not have the graphite_write plugin.

You will need to have a version of collectd compiled that has the python plugin enabled. This can be down by downloading the collectd source and building it, making sure to have python-devel package installed.

Mathew has a good write on the graphite / collectd compilation and configuration for graphite. http://blog.matthewdfuller.com/2014/06/sending-collectd-metrics-to-graphite.html

Be sure to have this installed before run “./configure”

yum install python
yum install python-devel

The configuration of /opt/collectd/etc/collectd.conf requires the following entries. Adjust the Username, Vcenter and Password to suit your environment.

<LoadPlugin python>
  Globals true
</LoadPlugin>

<Plugin python>
        # vcenter.py is at /usr/lib/python2.6/site-packages
        LogTraces true
        Interactive false
        Import vcenter
        <Module vcenter>
             Username "root"
             Vcenter "vc.local"
             Password "vmware"
             Verbose false
        </Module>
</Plugin>

You'll need to have pysphere installed. Paraphrasing the installation:

yum install python-setuptools
easy_install -U pysphere

The magic script that pulls all the stats we need. This works in my ESX 5.1 lab where I have a single vCenter instance.

vcenter.py

#!/usr/bin/python
# Collect basic stats about power on VM's on a single vCenter
# Brett England - 1-Oct-2014
 
import collectd
from pysphere import VIServer
NAME = 'vCenter'   # Plugin name
 
# Metric and reporting type as a value
# https://collectd.org/wiki/index.php/Data_source
# /opt/collectd/share/collectd/types.db
# https://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/vim.vm.Summary.QuickStats.html
METRIC={"name":None,
        "runtime.powerState":None,
        "summary.quickStats.overallCpuUsage":"cpufreq",  # Mhz
        "summary.quickStats.uptimeSeconds":"gauge",
        "summary.quickStats.guestMemoryUsage":"bytes",
        "summary.quickStats.hostMemoryUsage":"bytes",
        "config.hardware.memoryMB":"bytes"}
 
def connect():
    server = VIServer()
    try:
        server.connect(VCENTER, USERNAME, PASSWORD)
    except:
        logger('warn', "failed to connect to %s" % (VCENTER))
        return None
    return server
 
def get_stats():
    server = connect()
    if server is None:
        return []
    ret = []
    props = server._retrieve_properties_traversal(property_names=METRIC.keys(),
                                                  obj_type="VirtualMachine")
    for obj in props:
        skip_vm = False
        d={}
        for vm_property in obj.PropSet:
            if vm_property.Name == "runtime.powerState":
                if vm_property.Val != "poweredOn":
                    skip_vm = True
                    break
                continue
 
            d[vm_property.Name] = vm_property.Val
 
        if not skip_vm:
            ret.append(d)
    server.disconnect()
    return ret
 
# callback configuration for module
def configure_callback(conf):
  global VCENTER, USERNAME, PASSWORD, VERBOSE_LOGGING
  VCENTER = ''
  USERNAME = ''
  PASSWORD = ''
  VERBOSE_LOGGING = False
 
  for node in conf.children:
    if node.key == "Vcenter":
      VCENTER = node.values[0]
    elif node.key == "Username":
      USERNAME = node.values[0]
    elif node.key == "Password":
      PASSWORD = node.values[0]
    elif node.key == "Verbose":
      VERBOSE_LOGGING = bool(node.values[0])
    else:
      logger('warn', 'Unknown config key: %s' % node.key)
 
# https://collectd.org/wiki/index.php/Naming_schema
# The serialized form of the identifier is:
# host "/" plugin ["-" plugin instance] "/" type ["-" type instance]
#
# We want:   VCENTER / VMNAME / type...
def dispatch_value(vmname, value, key, type):
 
    logger('verb','%s: Sending value: %s=%s' % (vmname, key, value))
    # This is not intuitive but its what we want.
    val = collectd.Values()
    val.host = VCENTER
    val.plugin = vmname
    val.type = type
    val.type_instance = key
    val.values = [value]
    val.dispatch()
 
def read_callback():
  logger('verb', "beginning read_callback")
  info = get_stats()
 
  if not info:
    logger('warn', "No data received")
    return
 
  for vm in info:
      vmname = vm['name']
      for key,value in vm.items():
          type = METRIC[key]
          if type:
              dispatch_value(vmname, value, key, type)
 
# logging function
def logger(t, msg):
    if t == 'err':
        collectd.error('%s: %s' % (NAME, msg))
    elif t == 'warn':
        collectd.warning('%s: %s' % (NAME, msg))
    elif t == 'verb':
        if VERBOSE_LOGGING:
            collectd.info('%s: %s' % (NAME, msg))
    else:
        collectd.notice('%s: %s' % (NAME, msg))
 
# main
collectd.register_config(configure_callback)
collectd.register_read(read_callback)
2014/10/01 02:44 · 0 Linkbacks

PLEX virtual host configuration

I had googled all around trying to find a solution to the problem of setting up a virtual host configuration for a plex installation and hiding the URL for the port and making '/' just do the right thing.

Instead of typing this into a browser

http://cheese.local:32400/web/index.html

I want to be able to enter this

http://plex.local/

This was pretty damn close: http://matt.coneybeare.me/how-to-map-plex-media-server-to-your-home-domain/

However it did not completely work and required a small adjustment for my Plex server running on CentOS.

Modify my DNS server to setup a CNAME for cheese this already hosts an apache webserver on port 80 and its where I also run PLEX.

cheese          IN      A       192.168.1.20
plex            IN      CNAME cheese.local.

Create a file /etc/httpd/conf.d/plex.conf containing the following.

NameVirtualHost *:80
<VirtualHost *:80>
   ServerName cheese.local
</VirtualHost>

<VirtualHost *:80>
    ServerName plex.local

    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    ProxyRequests off
    ProxyPreserveHost On
    ProxyPass / ws://127.0.0.1:32400/
    ProxyPassReverse / ws://127.0.0.1:32400/
    ProxyPass / wss://127.0.0.1:32400/
    ProxyPassReverse / wss://127.0.0.1:32400/
    ProxyPass / http://127.0.0.1:32400/
    ProxyPassReverse / http://127.0.0.1:32400/

    RewriteEngine on
    RewriteCond %{REQUEST_URI} !^/web
    RewriteCond %{HTTP:X-Plex-Device} ^$
    RewriteCond %{QUERY_STRING} ^$
    RewriteRule ^/$ /web/$1 [R,L]
</VirtualHost>

We need to preserve the host cheese so its webserver is not affected so we place this host first, then we define how the plex hostname works. The crucial piece of the puzzle is the QUERY_STRING conditional. We do NOT want to redirect “/?something” into “/web?something” as this will screw things up.

The inclusion of the WebSocket proxying is also crucial for activity feedback. This requires the mod_proxy_wstunnel.so module which is available with apache 2.4+

As I was running CentOS 6.5 x64 which only has apache 2.2.15 I had to compile this module up myself by following the instructions here: http://notmyitblog.blogspot.com.au/2014/03/websockets-support-for-apache-httpd.html

For your convenience the precompiled module mod_proxy_wstunnel.zip which can be unzipped and dropped into /usr/lib64/httpd/modules you may need to chmod this to 755 after unzipping don't forget to enable this as well.

Edit apache configuration to support websockets

# vi /etc/httpd/conf/httpd.conf
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so

I find it useful to know how somebody managed to figure out what everybody else is struggling with so here we go.

This additional QUERY_STRING conditional was figured out by enabling debug on the rewrite rules by putting this into the virtual host rewrite section.

RewriteLog "/var/log/httpd/rewrite.log"
RewriteLogLevel 3

Now we can see what is happening and why it's going wrong. A request to / with a get query string was being redirected and we don't want that.

192.168.1.12 - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (3) applying pattern '^/$' to uri '/'
192.168.1.12 - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (2) rewrite '/' -> '/web/'
192.168.1.12 - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (2) explicitly forcing redirect with http://plex.local/web/
192.168.1.12 - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (1) escaping http://plex.local/web/ for redirect
192.168.1.12 - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (1) redirect to http://plex.local/web/?X-Plex-Client-Identifier=68igxy5
41z4&X-Plex-Product=Plex+Web&X-Plex-Device=Windows&X-Plex-Platform=Firefox&X-Plex-Platform-Version=26.0&X-Plex-Version=2.1.12&X-Plex-Device-Name=Plex+Web+(Firefox) [REDIRECT/3
02]

Once you see that its a small step to know you need a conditional that won't redirect / when a query string is present.

2014/07/28 03:07 · 1 Linkback

Circuit Diagram

I ran across this site http://www.schematics.com which if it grows might be the place to go for open source schematics.

2014/06/18 03:33 · 0 Linkbacks

Apache Redhat 6 make_sock Apache defect

This is a fucker of a defect that affected RedHat 6.3, 6.4, 6.5 - which I've verified. Probably 6.0, 6.1, 6.2 and 6.3 as well. I found this: https://issues.apache.org/bugzilla/show_bug.cgi?id=45863 but it was of no help and was more of a red-herring than anything else.

# httpd -version
Server version: Apache/2.2.23 (Unix)
Server built:   Feb 13 2013 02:41:10

Here is the output that you are confronted with

[Thu Sep 19 14:11:13 2013] [warn] VirtualHost localhost:82 overlaps with VirtualHost localhost:82, the first has precedence, perhaps you need a NameVirtualHost directive
[Thu Sep 19 14:11:13 2013] [warn] VirtualHost localhost:81 overlaps with VirtualHost localhost:81, the first has precedence, perhaps you need a NameVirtualHost directive
(98)Address already in use: make_sock: could not bind to address 127.0.0.1:81
no listening sockets available, shutting down
Unable to open logs

Doing an lsof, netstat, telnet or anything else you can think of on port 81 tells you nothing is listening. OK so why the error message !

The strace is telling its trying to bind TWICE to the same port! WTF.

32709 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
32709 setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
32709 setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
32709 bind(3, {sa_family=AF_INET, sin_port=htons(81), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
32709 listen(3, 511)                    = 0
32709 setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
32709 setsockopt(4, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
32709 setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
32709 bind(4, {sa_family=AF_INET, sin_port=htons(81), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRINUSE (Address already in use)

The solution. The trouble makers were found in /etc/hosts as follows

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Once we had removed 'localhost' and 'localhost.localdomain' from the ::1 definition line, apache/httpd could start successfully.

UPDATE: In light of https://issues.apache.org/bugzilla/show_bug.cgi?id=52884

In my configuration I had this

Listen localhost:81

If I had change it to this it would have also made the problem go away.

Listen 127.0.0.1:81
2014/05/30 03:23 · 0 Linkbacks

Ripping PBS videos

So you've tried streamtransport, rtmpsuck, rtmpdump and various searches on the web but everything came up short or sucked badly. This was what happened to me before I figured out you only need two very simple tools; one to figure out what the video stream URL is the other to grab it.

Examine URL snooper for the .m3u8 stream.

Split for readability. This should be all on one line.

ffmpeg -i "http://ga.video.cdn.pbs.org/cove2.0/art-21/
449de345-bc3e-4bab-8591-6f0ec2efc3b7/2013-11-21-163504/hd-mezzanine-16x9/artf502_hd_mezz-16x9-hls-800k.m3u8"
 -vcodec copy -acodec copy -bsf:a aac_adtstoasc output.mp4

If you inside Australia you'll have an issue accessing the PBS video in the first place due to right restrictions. You can workaround by using host outside of the country.

ffmpeg -i "http://ga.video.cdn.pbs.org/cove2.0/art-21/
449de345-bc3e-4bab-8591-6f0ec2efc3b7/2013-11-21-163504/hd-mezzanine-16x9/artf502_hd_mezz-16x9-hls-800k.m3u8"
 -vcodec copy -acodec copy -bsf:a aac_adtstoasc output.mp4
ffmpeg version N-60456-g4040b56 Copyright (c) 2000-2014 the FFmpeg developers
  built on Feb  8 2014 22:06:34 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aa
cenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavp
ack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 63.101 / 52. 63.101
  libavcodec     55. 50.100 / 55. 50.100
  libavformat    55. 30.100 / 55. 30.100
  libavdevice    55.  7.100 / 55.  7.100
  libavfilter     4.  1.102 /  4.  1.102
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, hls,applehttp, from 'http://ga.video.cdn.pbs.org/cove2.0/art-21/
449de345-bc3e-4bab-8591-6f0ec2efc3b7/2013-11-21-163504/hd-mezzanine-16x9/artf502_hd_mezz-16x9-hls-800k.m3u8':
  Duration: 00:54:30.29, start: 10.000000, bitrate: 0 kb/s
  Program 0
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 640x360 
    [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0:1: Audio: aac ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 174 kb/s
File 'output.mp4' already exists. Overwrite ? [y/N] y
Output #0, mp4, to 'output.mp4':
  Metadata:
    encoder         : Lavf55.30.100
    Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 640x360 
    [SAR 1:1 DAR 16:9], q=2-31, 29.97 fps, 90k tbn, 90k tbc
    Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo, 174 kb/s

Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 3639 fps=0.0 q=-1.0 size=   12337kB time=00:02:01.38 bitrate= 832.5kbits/
frame= 6318 fps=6223 q=-1.0 size=   23575kB time=00:03:30.77 bitrate= 916.3kbits
frame= 9269 fps=6025 q=-1.0 size=   36016kB time=00:05:09.24 bitrate= 954.1kbits
frame=12066 fps=5897 q=-1.0 size=   47824kB time=00:06:42.56 bitrate= 973.2kbits
frame=14893 fps=5832 q=-1.0 size=   59782kB time=00:08:16.89 bitrate= 985.6kbits
frame=17342 fps=5636 q=-1.0 size=   70210kB time=00:09:38.64 bitrate= 994.0kbits
frame=19452 fps=5427 q=-1.0 size=   79173kB time=00:10:49.01 bitrate= 999.3kbits
frame=21528 fps=5261 q=-1.0 size=   87531kB time=00:11:58.28 bitrate= 998.3kbits
frame=23621 fps=4950 q=-1.0 size=   95953kB time=00:13:08.12 bitrate= 997.4kbits
frame=24867 fps=4710 q=-1.0 size=  101288kB time=00:13:49.69 bitrate=1000.1kbits
frame=26539 fps=4586 q=-1.0 size=  108374kB time=00:14:45.48 bitrate=1002.6kbits
frame=28704 fps=4381 q=-1.0 size=  117495kB time=00:15:57.75 bitrate=1005.0kbits
frame=31096 fps=4400 q=-1.0 size=  124199kB time=00:17:17.53 bitrate= 980.6kbits
frame=33050 fps=4363 q=-1.0 size=  132516kB time=00:18:22.73 bitrate= 984.4kbits
frame=35527 fps=4366 q=-1.0 size=  140345kB time=00:19:45.38 bitrate= 969.9kbits
frame=36940 fps=4273 q=-1.0 size=  146141kB time=00:20:32.53 bitrate= 971.3kbits
frame=39001 fps=4261 q=-1.0 size=  154863kB time=00:21:41.30 bitrate= 974.9kbits
frame=41262 fps=4261 q=-1.0 size=  162614kB time=00:22:56.78 bitrate= 967.6kbits
frame=43259 fps=4245 q=-1.0 size=  168863kB time=00:24:03.37 bitrate= 958.4kbits
frame=45448 fps=4220 q=-1.0 size=  174908kB time=00:25:16.42 bitrate= 944.9kbits
frame=47242 fps=4189 q=-1.0 size=  182410kB time=00:26:16.27 bitrate= 948.0kbits
frame=49407 fps=4193 q=-1.0 size=  191603kB time=00:27:28.51 bitrate= 952.1kbits
frame=51301 fps=4174 q=-1.0 size=  199606kB time=00:28:31.71 bitrate= 955.3kbits
frame=53273 fps=4162 q=-1.0 size=  207389kB time=00:29:37.50 bitrate= 955.8kbits
frame=55913 fps=4202 q=-1.0 size=  215037kB time=00:31:05.59 bitrate= 944.2kbits
frame=58006 fps=4192 q=-1.0 size=  223092kB time=00:32:15.45 bitrate= 944.3kbits
frame=60099 fps=4180 q=-1.0 size=  231965kB time=00:33:25.29 bitrate= 947.6kbits
frame=62192 fps=4174 q=-1.0 size=  238270kB time=00:34:35.14 bitrate= 940.6kbits
frame=64199 fps=4167 q=-1.0 size=  245493kB time=00:35:42.07 bitrate= 938.8kbits
frame=65793 fps=4134 q=-1.0 size=  250554kB time=00:36:35.25 bitrate= 935.0kbits
frame=67720 fps=4124 q=-1.0 size=  258386kB time=00:37:39.55 bitrate= 936.8kbits
frame=69045 fps=4078 q=-1.0 size=  263202kB time=00:38:23.76 bitrate= 935.9kbits
frame=70265 fps=4024 q=-1.0 size=  268407kB time=00:39:04.51 bitrate= 937.8kbits
frame=71326 fps=3969 q=-1.0 size=  272860kB time=00:39:39.87 bitrate= 939.2kbits
frame=73255 fps=3963 q=-1.0 size=  281069kB time=00:40:44.24 bitrate= 942.0kbits
frame=75092 fps=3954 q=-1.0 size=  288786kB time=00:41:45.53 bitrate= 944.2kbits
frame=75707 fps=3882 q=-1.0 size=  291433kB time=00:42:06.05 bitrate= 945.1kbits
frame=77142 fps=3853 q=-1.0 size=  297486kB time=00:42:53.93 bitrate= 946.8kbits
frame=78507 fps=3824 q=-1.0 size=  303345kB time=00:43:39.48 bitrate= 948.7kbits
frame=80471 fps=3825 q=-1.0 size=  311651kB time=00:44:45.01 bitrate= 950.8kbits
frame=81029 fps=3761 q=-1.0 size=  314008kB time=00:45:03.66 bitrate= 951.4kbits
frame=81996 fps=3718 q=-1.0 size=  317858kB time=00:45:35.89 bitrate= 951.8kbits
frame=82823 fps=3666 q=-1.0 size=  321344kB time=00:46:03.52 bitrate= 952.6kbits
frame=83421 fps=3606 q=-1.0 size=  323865kB time=00:46:23.44 bitrate= 953.2kbits
frame=85514 fps=3612 q=-1.0 size=  331434kB time=00:47:33.28 bitrate= 951.6kbits
frame=87331 fps=3611 q=-1.0 size=  338803kB time=00:48:33.91 bitrate= 952.5kbits
frame=89102 fps=3600 q=-1.0 size=  346010kB time=00:49:33.00 bitrate= 953.4kbits
frame=91494 fps=3622 q=-1.0 size=  356063kB time=00:50:52.81 bitrate= 955.5kbits
frame=93461 fps=3627 q=-1.0 size=  364248kB time=00:51:58.44 bitrate= 956.9kbits
frame=95082 fps=3586 q=-1.0 size=  371029kB time=00:52:52.54 bitrate= 958.1kbits
frame=97433 fps=3606 q=-1.0 size=  379877kB time=00:54:10.98 bitrate= 957.2kbits
frame=98012 fps=3602 q=-1.0 Lsize=  384807kB time=00:54:30.41 bitrate= 963.9kbits/s
video:318073kB audio:64851kB subtitle:0 data:0 global headers:0kB muxing overhead 0.491705%
2014/02/11 04:11 · 0 Linkbacks

<< Newer entries | Older entries >>