Windows patches and SHA1

Windows download URL contain a SHA1 checksum as part of the URL:

http://www.download.windowsupdate.com/msdownload/update/v3-19990518/cabpool/windowsserver2003-kb824141-x86-enu_90853a52ea80f7da3c5460ef102ade3.exe

You can download the file and the use the SHA1 checksum from the URL itself to then validate the file downloaded correctly. Sounds like a good idea. It is until MS screw up the SHA1 on the URL.

# openssl sha1 windowsserver2003-kb824141-x86-enu_90853a52ea80f7da3c5460ef102ade3.exe
SHA1(windowsserver2003-kb824141-x86-enu_90853a52ea80f7da3c5460ef102ade3.exe)= bfa8072aa29dbe552f952cdb42b1f635072ae081

These are a list of filenames that I've discovered where the SHA in the URL file does not match that computed.

['windowsserver2003-kb824141-x86-enu_90853a52ea80f7da3c5460ef102ade3.exe',
 'msjavwu_8073687b82d41db93f4c2a04af2b34d.exe',
 'windowsserver2003-kb835732-x86-enu_9c2348f833ade0cca439ec6b2a92179.exe',
 'windowsmedia9-kb819639-x86-enu_57af369562f19dc35e69681660521fb.exe',
 'windowsserver2003-kb828741-x86-enu_1e3156bf5ec0354f542c38f309bab49.exe',
 'windowsserver2003-kb819696-x86-enu_41cdc8619ebb756106ea383c055530d.exe',
 'windowsserver2003-kb825119-x86-enu_329e94ea193be4c2d2f8d9bfc4daf23.exe',
 'windowsserver2003-kb840374-x86-enu_eeafbc20c2402b1c951d155d3d2cb9c.exe',
 'windowsserver2003-kb837001-x86-enu_0a248bb59a71c52a288c837779ac98e.exe',
 'windowsserver2003-kb823980-x86-enu_7f97e0d2355f670acb9384ad0933515.exe',
 'windowsserver2003-kb824146-x86-enu_f759bdcfdc906b0b35ad697a29ed1a1.exe',
 'windowsserver2003-kb823559-x86-enu_d8d3b25c5678c692e29cf971a6c38fa.exe',
 'windowsserver2003-kb824105-x86-enu_c7fd830ee6b1c3bb594be4f7a61f43c.exe',
 'windowsserver2003-kb828028-x86-enu_52dce385c001ce81c2514c3fb1cac7e.exe',
 'windowsserver2003-kb828035-x86-enu_d1df77e311740d6c012bcda5a7f821f.exe',
 'directx9-kb819696-x86-enu_977f8cc86c1e151a0168d1296210913.exe',
 'windowsserver2003-kb830352-x86-enu_d67acb6c784dd87961c8070943dadd8.exe',
 'sql2000-kb815495-8.00.0818-enu_4c77bb3f492fb1670b90b477d674e7e.exe',
 'windowsserver2003-kb823182-x86-enu_c7ee6a3716815554656d98ed9bc85d5.exe',
 'windowsxp-kb883939-x64-enu_9e1efe32675530155c34f7af1172a6d496e1e5ee.exe',
 'ndp10_sp_q321884_en_0fc8b14a073e01a03c27c948d254feedaa79feae.exe']
2015/05/19 12:07 · 0 Linkbacks

Python decompress PACK_MAGIC

A file compressed with pack format has magic bytes in octal \036\037 or in hex 0x1e1f

GZIP can decode this along with the pcat program. For an exercise I converted the unpack.c module in gzip into its python equivalent.

The slowest part of the code is the look_bits function and this is where you can see how an interpreted language grinds compared to C.

Using the excellent line profiler: https://pypi.python.org/pypi/line_profiler/

Timer unit: 1e-06 s = 1uS

Total time: 43.3399 s
File: unpack.py
Function: look_bits at line 36

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    36                                               @profile
    37                                               def look_bits(self,bits,mask):
    38    351442       265280      0.8      0.6          while(self.valid < bits):
    39    140575     10361348     73.7     23.9              self.bitbuf <<= 8
    40    140575     14576495    103.7     33.6              self.bitbuf |= next(self.get_byte)
    41    140575       189102      1.3      0.4              self.valid += 8
    42    210867     17947709     85.1     41.4          return (self.bitbuf >> (self.valid - bits)) & mask

unpack.zip

2014/10/24 11:09 · 0 Linkbacks

vCenter VM monitoring with Graphite

This was done on a CentOS 6.3 server your mileage may vary on another platform.

The problem I was trying to solve was that I wanted to monitor the vitals of all my VM's without having to install collectd into each VM, by talking to vCenter we can pull everything out that we need. This work was inspired by collectd-vcenter which I could not get to work as its assumes you have an ESX Cluster. So I wrote my own.

See details on setting up graphite which I used for my xAP monitoring

We don't use the collectd RPM from the EPEL repository as its too old and does not have the graphite_write plugin.

You will need to have a version of collectd compiled that has the python plugin enabled. This can be down by downloading the collectd source and building it, making sure to have python-devel package installed.

Mathew has a good write on the graphite / collectd compilation and configuration for graphite. http://blog.matthewdfuller.com/2014/06/sending-collectd-metrics-to-graphite.html

Be sure to have this installed before run “./configure”

yum install python
yum install python-devel

The configuration of /opt/collectd/etc/collectd.conf requires the following entries. Adjust the Username, Vcenter and Password to suit your environment.

<LoadPlugin python>
  Globals true
</LoadPlugin>

<Plugin python>
        # vcenter.py is at /usr/lib/python2.6/site-packages
        LogTraces true
        Interactive false
        Import vcenter
        <Module vcenter>
             Username "root"
             Vcenter "vc.local"
             Password "vmware"
             Verbose false
        </Module>
</Plugin>

You'll need to have pysphere installed. Paraphrasing the installation:

yum install python-setuptools
easy_install -U pysphere

The magic script that pulls all the stats we need. This works in my ESX 5.1 lab where I have a single vCenter instance.

vcenter.py

#!/usr/bin/python
# Collect basic stats about power on VM's on a single vCenter
# Brett England - 1-Oct-2014
 
import collectd
from pysphere import VIServer
NAME = 'vCenter'   # Plugin name
 
# Metric and reporting type as a value
# https://collectd.org/wiki/index.php/Data_source
# /opt/collectd/share/collectd/types.db
# https://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/vim.vm.Summary.QuickStats.html
METRIC={"name":None,
        "runtime.powerState":None,
        "summary.quickStats.overallCpuUsage":"cpufreq",  # Mhz
        "summary.quickStats.uptimeSeconds":"gauge",
        "summary.quickStats.guestMemoryUsage":"bytes",
        "summary.quickStats.hostMemoryUsage":"bytes",
        "config.hardware.memoryMB":"bytes"}
 
def connect():
    server = VIServer()
    try:
        server.connect(VCENTER, USERNAME, PASSWORD)
    except:
        logger('warn', "failed to connect to %s" % (VCENTER))
        return None
    return server
 
def get_stats():
    server = connect()
    if server is None:
        return []
    ret = []
    props = server._retrieve_properties_traversal(property_names=METRIC.keys(),
                                                  obj_type="VirtualMachine")
    for obj in props:
        skip_vm = False
        d={}
        for vm_property in obj.PropSet:
            if vm_property.Name == "runtime.powerState":
                if vm_property.Val != "poweredOn":
                    skip_vm = True
                    break
                continue
 
            d[vm_property.Name] = vm_property.Val
 
        if not skip_vm:
            ret.append(d)
    server.disconnect()
    return ret
 
# callback configuration for module
def configure_callback(conf):
  global VCENTER, USERNAME, PASSWORD, VERBOSE_LOGGING
  VCENTER = ''
  USERNAME = ''
  PASSWORD = ''
  VERBOSE_LOGGING = False
 
  for node in conf.children:
    if node.key == "Vcenter":
      VCENTER = node.values[0]
    elif node.key == "Username":
      USERNAME = node.values[0]
    elif node.key == "Password":
      PASSWORD = node.values[0]
    elif node.key == "Verbose":
      VERBOSE_LOGGING = bool(node.values[0])
    else:
      logger('warn', 'Unknown config key: %s' % node.key)
 
# https://collectd.org/wiki/index.php/Naming_schema
# The serialized form of the identifier is:
# host "/" plugin ["-" plugin instance] "/" type ["-" type instance]
#
# We want:   VCENTER / VMNAME / type...
def dispatch_value(vmname, value, key, type):
 
    logger('verb','%s: Sending value: %s=%s' % (vmname, key, value))
    # This is not intuitive but its what we want.
    val = collectd.Values()
    val.host = VCENTER
    val.plugin = vmname
    val.type = type
    val.type_instance = key
    val.values = [value]
    val.dispatch()
 
def read_callback():
  logger('verb', "beginning read_callback")
  info = get_stats()
 
  if not info:
    logger('warn', "No data received")
    return
 
  for vm in info:
      vmname = vm['name']
      for key,value in vm.items():
          type = METRIC[key]
          if type:
              dispatch_value(vmname, value, key, type)
 
# logging function
def logger(t, msg):
    if t == 'err':
        collectd.error('%s: %s' % (NAME, msg))
    elif t == 'warn':
        collectd.warning('%s: %s' % (NAME, msg))
    elif t == 'verb':
        if VERBOSE_LOGGING:
            collectd.info('%s: %s' % (NAME, msg))
    else:
        collectd.notice('%s: %s' % (NAME, msg))
 
# main
collectd.register_config(configure_callback)
collectd.register_read(read_callback)
2014/10/01 02:44 · 0 Linkbacks

PLEX virtual host configuration

I had googled all around trying to find a solution to the problem of setting up a virtual host configuration for a plex installation and hiding the URL for the port and making '/' just do the right thing.

Instead of typing this into a browser

http://cheese.local:32400/web/index.html

I want to be able to enter this

http://plex.local/

This was pretty damn close: http://matt.coneybeare.me/how-to-map-plex-media-server-to-your-home-domain/

However it did not completely work and required a small adjustment for my Plex server running on CentOS.

Modify my DNS server to setup a CNAME for cheese this already hosts an apache webserver on port 80 and its where I also run PLEX.

cheese          IN      A       192.168.1.20
plex            IN      CNAME cheese.local.

Create a file /etc/httpd/conf.d/plex.conf containing the following.

NameVirtualHost *:80
<VirtualHost *:80>
   ServerName cheese.local
</VirtualHost>

<VirtualHost *:80>
    ServerName plex.local

    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    ProxyRequests off
    ProxyPreserveHost On
    ProxyPass / ws://127.0.0.1:32400/
    ProxyPassReverse / ws://127.0.0.1:32400/
    ProxyPass / wss://127.0.0.1:32400/
    ProxyPassReverse / wss://127.0.0.1:32400/
    ProxyPass / http://127.0.0.1:32400/
    ProxyPassReverse / http://127.0.0.1:32400/

    RewriteEngine on
    RewriteCond %{REQUEST_URI} !^/web
    RewriteCond %{HTTP:X-Plex-Device} ^$
    RewriteCond %{QUERY_STRING} ^$
    RewriteRule ^/$ /web/$1 [R,L]
</VirtualHost>

We need to preserve the host cheese so its webserver is not affected so we place this host first, then we define how the plex hostname works. The crucial piece of the puzzle is the QUERY_STRING conditional. We do NOT want to redirect “/?something” into “/web?something” as this will screw things up.

The inclusion of the WebSocket proxying is also crucial for activity feedback. This requires the mod_proxy_wstunnel.so module which is available with apache 2.4+

As I was running CentOS 6.5 x64 which only has apache 2.2.15 I had to compile this module up myself by following the instructions here: http://notmyitblog.blogspot.com.au/2014/03/websockets-support-for-apache-httpd.html

For your convenience the precompiled module mod_proxy_wstunnel.zip which can be unzipped and dropped into /usr/lib64/httpd/modules you may need to chmod this to 755 after unzipping don't forget to enable this as well.

Edit apache configuration to support websockets

# vi /etc/httpd/conf/httpd.conf
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so

I find it useful to know how somebody managed to figure out what everybody else is struggling with so here we go.

This additional QUERY_STRING conditional was figured out by enabling debug on the rewrite rules by putting this into the virtual host rewrite section.

RewriteLog "/var/log/httpd/rewrite.log"
RewriteLogLevel 3

Now we can see what is happening and why it's going wrong. A request to / with a get query string was being redirected and we don't want that.

192.168.1.12 - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (3) applying pattern '^/$' to uri '/'
192.168.1.12 - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (2) rewrite '/' -> '/web/'
192.168.1.12 - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (2) explicitly forcing redirect with http://plex.local/web/
192.168.1.12 - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (1) escaping http://plex.local/web/ for redirect
192.168.1.12 - - [28/Jul/2014:12:07:53 +091800] [plex.local/sid#7f22bc664b68][rid#7f22bc70c778/initial] (1) redirect to http://plex.local/web/?X-Plex-Client-Identifier=68igxy5
41z4&X-Plex-Product=Plex+Web&X-Plex-Device=Windows&X-Plex-Platform=Firefox&X-Plex-Platform-Version=26.0&X-Plex-Version=2.1.12&X-Plex-Device-Name=Plex+Web+(Firefox) [REDIRECT/3
02]

Once you see that its a small step to know you need a conditional that won't redirect / when a query string is present.

2014/07/28 03:07 · 1 Linkback

Circuit Diagram

I ran across this site http://www.schematics.com which if it grows might be the place to go for open source schematics.

2014/06/18 03:33 · 0 Linkbacks

Older entries >>