Python decompress PACK_MAGIC

A file compressed with pack format has magic bytes in octal \036\037 or in hex 0x1e1f

GZIP can decode this along with the pcat program. For an exercise I converted the unpack.c module in gzip into its python equivalent.

The slowest part of the code is the look_bits function and this is where you can see how an interpreted language grinds compared to C.

Using the excellent line profiler:

Timer unit: 1e-06 s = 1uS

Total time: 43.3399 s
Function: look_bits at line 36

Line #      Hits         Time  Per Hit   % Time  Line Contents
    36                                               @profile
    37                                               def look_bits(self,bits,mask):
    38    351442       265280      0.8      0.6          while(self.valid < bits):
    39    140575     10361348     73.7     23.9              self.bitbuf <<= 8
    40    140575     14576495    103.7     33.6              self.bitbuf |= next(self.get_byte)
    41    140575       189102      1.3      0.4              self.valid += 8
    42    210867     17947709     85.1     41.4          return (self.bitbuf >> (self.valid - bits)) & mask

2014/10/24 11:09 · 0 Linkbacks

vCenter VM monitoring with Graphite

This was done on a CentOS 6.3 server your mileage may vary on another platform.

The problem I was trying to solve was that I wanted to monitor the vitals of all my VM's without having to install collectd into each VM, by talking to vCenter we can pull everything out that we need. This work was inspired by collectd-vcenter which I could not get to work as its assumes you have an ESX Cluster. So I wrote my own.

See details on setting up graphite which I used for my xAP monitoring

We don't use the collectd RPM from the EPEL repository as its too old and does not have the graphite_write plugin.

You will need to have a version of collectd compiled that has the python plugin enabled. This can be down by downloading the collectd source and building it, making sure to have python-devel package installed.

Mathew has a good write on the graphite / collectd compilation and configuration for graphite.

Be sure to have this installed before run “./configure”

yum install python
yum install python-devel

The configuration of /opt/collectd/etc/collectd.conf requires the following entries. Adjust the Username, Vcenter and Password to suit your environment.

<LoadPlugin python>
  Globals true

<Plugin python>
        # is at /usr/lib/python2.6/site-packages
        LogTraces true
        Interactive false
        Import vcenter
        <Module vcenter>
             Username "root"
             Vcenter "vc.local"
             Password "vmware"
             Verbose false

You'll need to have pysphere installed. Paraphrasing the installation:

yum install python-setuptools
easy_install -U pysphere

The magic script that pulls all the stats we need. This works in my ESX 5.1 lab where I have a single vCenter instance.

# Collect basic stats about power on VM's on a single vCenter
# Brett England - 1-Oct-2014
import collectd
from pysphere import VIServer
NAME = 'vCenter'   # Plugin name
# Metric and reporting type as a value
# /opt/collectd/share/collectd/types.db
        "summary.quickStats.overallCpuUsage":"cpufreq",  # Mhz
def connect():
    server = VIServer()
        server.connect(VCENTER, USERNAME, PASSWORD)
        logger('warn', "failed to connect to %s" % (VCENTER))
        return None
    return server
def get_stats():
    server = connect()
    if server is None:
        return []
    ret = []
    props = server._retrieve_properties_traversal(property_names=METRIC.keys(),
    for obj in props:
        skip_vm = False
        for vm_property in obj.PropSet:
            if vm_property.Name == "runtime.powerState":
                if vm_property.Val != "poweredOn":
                    skip_vm = True
            d[vm_property.Name] = vm_property.Val
        if not skip_vm:
    return ret
# callback configuration for module
def configure_callback(conf):
  VCENTER = ''
  for node in conf.children:
    if node.key == "Vcenter":
      VCENTER = node.values[0]
    elif node.key == "Username":
      USERNAME = node.values[0]
    elif node.key == "Password":
      PASSWORD = node.values[0]
    elif node.key == "Verbose":
      VERBOSE_LOGGING = bool(node.values[0])
      logger('warn', 'Unknown config key: %s' % node.key)
# The serialized form of the identifier is:
# host "/" plugin ["-" plugin instance] "/" type ["-" type instance]
# We want:   VCENTER / VMNAME / type...
def dispatch_value(vmname, value, key, type):
    logger('verb','%s: Sending value: %s=%s' % (vmname, key, value))
    # This is not intuitive but its what we want.
    val = collectd.Values() = VCENTER
    val.plugin = vmname
    val.type = type
    val.type_instance = key
    val.values = [value]
def read_callback():
  logger('verb', "beginning read_callback")
  info = get_stats()
  if not info:
    logger('warn', "No data received")
  for vm in info:
      vmname = vm['name']
      for key,value in vm.items():
          type = METRIC[key]
          if type:
              dispatch_value(vmname, value, key, type)
# logging function
def logger(t, msg):
    if t == 'err':
        collectd.error('%s: %s' % (NAME, msg))
    elif t == 'warn':
        collectd.warning('%s: %s' % (NAME, msg))
    elif t == 'verb':
  '%s: %s' % (NAME, msg))
        collectd.notice('%s: %s' % (NAME, msg))
# main
2014/10/01 02:44 · 0 Linkbacks

PLEX virtual host configuration

I had googled all around trying to find a solution to the problem of setting up a virtual host configuration for a plex installation and hiding the URL for the port and making '/' just do the right thing.

Instead of typing this into a browser


I want to be able to enter this


This was pretty damn close:

However it did not completely work and required a small adjustment for my Plex server running on CentOS.

Modify my DNS server to setup a CNAME for cheese this already hosts an apache webserver on port 80 and its where I also run PLEX.

cheese          IN      A
plex            IN      CNAME cheese.local.

Create a file /etc/httpd/conf.d/plex.conf containing the following.

NameVirtualHost *:80
<VirtualHost *:80>
   ServerName cheese.local

<VirtualHost *:80>
    ServerName plex.local

    <Proxy *>
        Order deny,allow
        Allow from all

    ProxyRequests off
    ProxyPreserveHost On
    ProxyPass / ws://
    ProxyPassReverse / ws://
    ProxyPass / wss://
    ProxyPassReverse / wss://
    ProxyPass /
    ProxyPassReverse /

    RewriteEngine on
    RewriteCond %{REQUEST_URI} !^/web
    RewriteCond %{HTTP:X-Plex-Device} ^$
    RewriteCond %{QUERY_STRING} ^$
    RewriteRule ^/$ /web/$1 [R,L]

We need to preserve the host cheese so its webserver is not affected so we place this host first, then we define how the plex hostname works. The crucial piece of the puzzle is the QUERY_STRING conditional. We do NOT want to redirect “/?something” into “/web?something” as this will screw things up.

The inclusion of the WebSocket proxying is also crucial for activity feedback. This requires the module which is available with apache 2.4+

As I was running CentOS 6.5 x64 which only has apache 2.2.15 I had to compile this module up myself by following the instructions here:

For your convenience the precompiled module which can be unzipped and dropped into /usr/lib64/httpd/modules you may need to chmod this to 755 after unzipping don't forget to enable this as well.

Edit apache configuration to support websockets

# vi /etc/httpd/conf/httpd.conf
LoadModule proxy_wstunnel_module modules/

I find it useful to know how somebody managed to figure out what everybody else is struggling with so here we go.

This additional QUERY_STRING conditional was figured out by enabling debug on the rewrite rules by putting this into the virtual host rewrite section.

RewriteLog "/var/log/httpd/rewrite.log"
RewriteLogLevel 3

Now we can see what is happening and why it's going wrong. A request to / with a get query string was being redirected and we don't want that. - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (3) applying pattern '^/$' to uri '/' - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (2) rewrite '/' -> '/web/' - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (2) explicitly forcing redirect with http://plex.local/web/ - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (1) escaping http://plex.local/web/ for redirect - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (1) redirect to http://plex.local/web/?X-Plex-Client-Identifier=68igxy5
41z4&X-Plex-Product=Plex+Web&X-Plex-Device=Windows&X-Plex-Platform=Firefox&X-Plex-Platform-Version=26.0&X-Plex-Version=2.1.12&X-Plex-Device-Name=Plex+Web+(Firefox) [REDIRECT/3

Once you see that its a small step to know you need a conditional that won't redirect / when a query string is present.

2014/07/28 03:07 · 1 Linkback

Circuit Diagram

I ran across this site which if it grows might be the place to go for open source schematics.

2014/06/18 03:33 · 0 Linkbacks

Apache Redhat 6 make_sock Apache defect

This is a fucker of a defect that affected RedHat 6.3, 6.4, 6.5 - which I've verified. Probably 6.0, 6.1, 6.2 and 6.3 as well. I found this: but it was of no help and was more of a red-herring than anything else.

# httpd -version
Server version: Apache/2.2.23 (Unix)
Server built:   Feb 13 2013 02:41:10

Here is the output that you are confronted with

[Thu Sep 19 14:11:13 2013] [warn] VirtualHost localhost:82 overlaps with VirtualHost localhost:82, the first has precedence, perhaps you need a NameVirtualHost directive
[Thu Sep 19 14:11:13 2013] [warn] VirtualHost localhost:81 overlaps with VirtualHost localhost:81, the first has precedence, perhaps you need a NameVirtualHost directive
(98)Address already in use: make_sock: could not bind to address
no listening sockets available, shutting down
Unable to open logs

Doing an lsof, netstat, telnet or anything else you can think of on port 81 tells you nothing is listening. OK so why the error message !

The strace is telling its trying to bind TWICE to the same port! WTF.

32709 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
32709 setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
32709 setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
32709 bind(3, {sa_family=AF_INET, sin_port=htons(81), sin_addr=inet_addr("")}, 16) = 0
32709 listen(3, 511)                    = 0
32709 setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
32709 setsockopt(4, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
32709 setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
32709 bind(4, {sa_family=AF_INET, sin_port=htons(81), sin_addr=inet_addr("")}, 16) = -1 EADDRINUSE (Address already in use)

The solution. The trouble makers were found in /etc/hosts as follows   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Once we had removed 'localhost' and 'localhost.localdomain' from the ::1 definition line, apache/httpd could start successfully.

UPDATE: In light of

In my configuration I had this

Listen localhost:81

If I had change it to this it would have also made the problem go away.

2014/05/30 03:23 · 0 Linkbacks

<< Newer entries | Older entries >>