So here you go:

• PDF version, ready to be printed onto two side-by-side pages. Not sure why the fonts didn’t completely translate over.
• OpenOffice.org version, ready to be edited.

But it didn’t catch a couple of… terse… commenters on this post about a presentation I was about to give. One commenter seemed rather irritated, and posted a malformed link to some addition recovery place or another. A second commenter appeared to be much friendlier, and posted a few words mildly relevant to the presentation, but also added a link to another treatment center. And it’s just happening on this one post, as far as I can tell.

And this isn’t the garden-variety spam I’m used to seeing in Spam Karma 2’s reports. Which means it’s not getting caught with their Javascript test, their “Flash Gordon was here” test (comment posted just a few seconds after page load), etc. The comments contain one link, nothing formatted with BBCode, no unformatted links, and there are complete sentences attached to them that are just barely related to the post content.

Of all the posts I’ve got here, and all the opportunity for spam it provides, why this post, and why these spammers? The only thing the post and the spams have in common is the word “Tennessee”. Spammer 1 tried to link to an addiction recovery site with the word Tennessee in its URL. Spammer 2 successfully linked to a treatment center with the word Tennessee in its URL.

So I wonder, did I include the word “Tennessee” in this post enough to attract these folks’ attention? Will they post more spam here? Because if they do, what Spam Karma 2 doesn’t catch, I will. Word of advice, guys. I don’t get an enormous volume of comments here. In fact, I get few enough to where I’m normally able to look at them within minutes of their arrival. If they’re spam, they’ll get deleted. If you’re getting paid by the hour, I guess it doesn’t matter. If you’re getting paid by number of valid links left after some period of time, you’re better off spamming elsewhere.

So, why not build a MarcoPolo for Windows? I’ve not yet found one, but I’d be more than happy to be proven wrong there. In the meantime, I’m working on a Python/Windows proof-of-concept that could be the groundwork for a Windows analogue to MarcoPolo.

# Idea for Win32 version of MarcoPolo (Context-aware computing for Mac OS X)

# Contexts
contexts = {
    'Home': 1,
    'Work': 1,
    'pre-Noon': 1,
    'post-Noon': 1,
    'Unknown': 1,
}

import wmi, pybonjour, time, IP4Range
# wmi: http://tgolden.sc.sabren.com/python/wmi.html
# pybonjour: http://o2s.csail.mit.edu/o2s-wiki/pybonjour
# IP4Range: http://code.activestate.com/recipes/466298/

def main():
    c=wmi.WMI()
    # Evidence sources via WMI (direct from MarcoPolo features, some
    # sources may not be available in Win32)

    # Current audio output device (headphones/speakers) -- may not be
    # discoverable on XP and earlier.

    # Discoverable Bluetooth devices -- can find a Bluetooth adapter via
    # USB, but not a connected end-user device

    # Advertised Bonjour (Zeroconf) services -- 'browse_and_resolve.py
    # _daap._tcp' finds a remote iTunes share, but no idea how to look for
    # random services.

    # Attached FireWire devices -- no FireWire-capable computers on hand

    # Assigned IP addresses
    ipList=[]
    for adapter in c.Win32_NetworkAdapterConfiguration(IPEnabled="True"):
        ipList.append(adapter.IPAddress[0])

    # Ambient light level — may not be any standard sensors available to
    # Windows

    # Attached monitors
    monitorList=[]
    for monitor in c.Win32_DesktopMonitor(Availability=3):
        monitorList.append(monitor.Name)

    # Active network links
    networkAdapterList=[]
    for adapter in c.Win32_NetworkAdapter(NetConnectionStatus=2):
        networkAdapterList.append(adapter.Description)

    # Power source (power adapter/battery) — need to test on laptop

    # Running Applications
    applicationList=[]
    for process in c.Win32_Process():
        applicationList.append(process.Name)

    # Time of day
    timeOfDay=time.strftime("%H:%M")

    # Attached USB devices
    usbDeviceList=[]
    for device in c.Win32_USBControllerDevice():
        usbDeviceList.append(device.Dependent.Description)

    # Visible WiFi networks — need to test on laptop

    # Rules (type, criteria, context, confidence)
    ruleList = [
        ('usb', 'Brother HL-5140 series', 'Home', 0.80),
        ('usb', 'Visioneer OneTouch 7300', 'Home', 0.99),
        ('monitor', 'Plug and Play Monitor', 'Home', 0.60),
        ('ip', '192.168.254.4', 'Home', 0.50),
        ('ip', '149.149.254.0/24', 'Work', 0.99),
        ('ssid', 'mike-and-carolyn', 'Home', 0.99),
        ('time', ['00:00', '11:59'], 'pre-Noon', 0.99),
        ('time', ['12:00', '23:59'], 'post-Noon', 0.99),
        ]

    # Decision
    for rule in ruleList:
        itemFound = -1
        (ruleType, ruleItem, ruleContext, ruleStrength) = rule
        if ruleType=='ip':
            ipRange=IP4Range.IP4Range(ruleItem)
            for ip in ipList:
                currentIP=IP4Range.IP4Range(ip.encode('ascii'))
                if currentIP.issubset(ipRange):
                    itemFound = 1

        elif ruleType=='monitor':
            try:
                itemFound = monitorList.index(ruleItem)
            except ValueError:
                pass
        elif ruleType=='adapter':
            try:
                itemFound = networkAdapterList.index(ruleItem)
            except ValueError:
                pass
        elif ruleType=='application':
            try:
                itemFound = applicationList.index(ruleItem)
            except ValueError:
                pass
        elif ruleType=='usb':
            try:
                itemFound = usbDeviceList.index(ruleItem)
            except ValueError:
                pass
        elif ruleType=='time':
            if cmp(timeOfDay,ruleItem[0])>=0 and cmp(timeOfDay,ruleItem[1])<=0:
                itemFound=1
         else:
            print "Unknown rule type %s" % (ruleType)
            continue

        if itemFound!=-1:
            contexts[ruleContext]=contexts[ruleContext]*(1-ruleStrength)

    myLocation='Unknown'
    myConfidence=1-contexts['Unknown']
    for context in contexts:
        if contexts[context]!=1:
            if (1-contexts[context])>myConfidence:
                myLocation=context
                myConfidence = 1-contexts[context]

    print "Your location: %s (confidence %f)" % (myLocation, myConfidence)
    print

if __name__ == "__main__":
    main()

I know my Python is mediocre at best. But this does work so far in my limited testing (”Unknown rule type ssid, Your location: Home (confidence 0.999600)”). Thanks to David Symonds for his assistance in this thread.

Code Example:

/**
 * This holds the version number in a separate file so we can bump it without cluttering the SVN
 */

/**
 * The WordPress version string
 *
 * @global string $wp_version
 */
$wp_version = '2.5.1';

/**
 * Holds the WordPress DB revision, increments when changes are made to the WordPress DB scheme
 * changes.
 *
 * @global int $wp_db_version
 */
$wp_db_version = 7796;

$wpmu_version = '1.5.1';

Math Example:

class cluster-host inherits public-host {
    # ...
    file { "/etc/hosts.equiv":
        source  => "puppet:///files/apps/rsh-server/hosts.equiv.$hostgroup",
        owner   => root,
        group   => root,
        mode    => 644,
        require => Package[rsh-server],
    }
}

Along the lines of Puppet Best Practice 2.0, I’m trying to compartmentalize all my building blocks into separate modules, even the really simple ones that just install a package or two. So here’s the contents of a trivial hello module for Debian:

# /etc/puppet/modules/hello/manifests/init.pp
class hello {
  package { "hello":
    ensure => latest;
  }
}

Nothing complicated here: install this package via the default package provider. Add an include hello to my node’s manifest, and the hello package gets installed.

Now for the “undo” class:

# /etc/puppet/modules/hello/manifests/disabled.pp
class hello::disabled inherits hello {
  Package["hello"] {
    ensure => purged
  }
}

And this is neat. Properly written, the disabling class can undo all the changes made by the parent class. Modify my node’s manifest from include hello to include hello::disabled and I can switch that class on and off like, well, a switch.

It all started so simply: I was going to set up a little Xen instance to be my next cluster submit host, and needed a spare address for it:

1. I started setting up an instance for ch208i.cae.tntech.edu, since it was no longer on the Xen host like it was several months ago. Crap, the reason it’s no longer on the Xen instance is because I moved it to its own dedicated hardware — it’s still my main ftp/mirror server. Ctrl-C that one.
2. Hmm, what’s available from old Xen instances? mail2.cae.tntech.edu.cfg from when I was testing out a new mail server setup last fall — doesn’t ping, doesn’t show up in xm list, no problem.

   xen-create-image --hostname=mail2.cae.tntech.edu --ip=149.149.254.23 \
       --gateway=149.149.254.4 --netmask=255.255.255.0 --size=10Gb --memory=256Mb \
       --swap=1Gb --debootstrap --force

   A few minutes later, my instance is debootstrapped and ready to go.

3. Oh, crap. Why am I getting an error on xm create that says my LVM is already in use on a domU somewhere?
4. Further crap. Looking in /etc/xen/mail.cae.tntech.edu.cfg for the production mail server, it apparently uses the old mail2.cae.tntech.edu LVMs. Wonderful. ssh mail? It works since sshd was already memory-resident, but /root/.profile doesn’t exist. And neither does much of anything else.
5. Great. I’ve just killed the mail server. Off to the Amanda server to do a quick restore of its data. What? I never put mail.cae.tntech.edu into the backup list? Not normally the end of the world, since the mail stores are held accessed over NFS from the main file server, but what about my dovecot and postfix configurations?
6. Oh, well. Time to see how good my puppet manifests are for the mail server.

Not too bad, as it turns out. Total downtime was only a couple hours, including having to redo the postfix and dovecot configurations (which were then copied off to the puppetmaster). I still have a few more things to fix, but mail delivery is up, and imap is running. TLS support for my sending mail from home isn’t up yet, but it’ll be fixed shortly.

I still need to fix that submit host, though. Next time, I think I’ll use an IP address reserved for my office.

Update: after getting a partial TLS/SASL setup going late Wednesday night, I went to sleep without realizing I’d killed mail delivery again. Finally got it straightened out Thursday morning.

• The default inventory reports always come out sorted by inventory tag. I have no idea why. If you have anything more than a handful of items to inventory, you would sort them by room, just like you’d identify them when walking around. Since I have somewhere around 100 items to inventory across 6-7 buildings, it’s a pain and rather error-prone to manually mark the ones in a particular building.
• Even when I got a report sorted by location, that doesn’t help a great deal with items that I only see during inventory time, or during a random audit. My memory just isn’t that good, and occasionally an item will move from one room to another without my knowledge. Photos of the items would help greatly in finding them, but the default reports don’t include any.

So this year, within a few hours of first asking for it, I got a nice CSV file of our inventory from the Business Office (thanks, Matt). It included lines like

74396,"Fork Lift, Electric",SU 803647,F30,CATERPILLAR,5/1/2001,CH106,13900,Center for Manufacturing Research,6700

that I could work with. I had text files from previous years on the old system, but they were much weirder to parse. This opened right up in Excel without issue, so I had hope for a much cleaner implementation of my old Python inventory scripts.

Man, I love Python. There’s a built-in CSV module that parses lines into lists, and separates values on each line into separate items in those lists. No parsing required on my part. There’s a nice third-party module for generating HTML markup (original site is down, but see this link from archive.org for the module). And it’s a mature enough language to where a years-old Usenet post on sorting is still useful.

On a procedural rather than technical front, I started going around a couple of years ago and snapping pictures of items as I inventoried them. The old camera-phone pictures aren’t great, but they’re good enough to identify an item and find its property tag. So now I had a table of inventory data, and a big folder of JPGs named according to the item’s property tag. So now to convert them into useful web pages:

import csv
import os
import string
from HyperText.XHTML import TABLE, TR, TH, TD, IMG, DIV, HEAD, STYLE, BR
from HyperText.Documents import Document

notesDict = {
    '064286':'(2007) Surplused',
    }

"""A set of comparer classes."""
## http://groups.google.com/group/comp.lang.python/msg/42b5a84450278ab2

class CmpComposite:
    """Takes a list of compare functions and sorts in that order."""
    def __init__(self,*comparers):
        self.comparers=comparers
    def __call__(self,a,b):
        for cmp in self.comparers:
            c=cmp(a,b)
            if c:
                return c
        return 0

class CmpInverse:
    """Inverses the effect of a cmp."""
    def __init__(self,cmp):
        self.cmp=cmp
    def __call__(self,a,b):
        return -self.cmp(a,b)

class CmpColumn:
    """Sorts on an index of a sequence."""
    def __init__(self,column):
        self.column=column
    def __call__(self,a,b):
        return cmp(a[self.column],b[self.column])

class CmpAttr:
    “”"Sorts on an attribute.”"”
    def __init__(self, attr):
        self.attr = attr
    def __call__(self, x, y):
        return cmp(getattr(x, self.attr), getattr(y, self.attr))

reader=csv.reader(open(’inventory_control_listing_13900.csv’,'rb’))
equipmentList = []
for row in reader:
    (tag, description, serial, model, manufacturer, arrival,
     location, code, department, cost) = row
    equipmentList.append((tag, description, serial, model, manufacturer,
                          arrival, location, cost))

for sortBy in ['tag', 'location']:
    if sortBy==’tag’:
        equipmentList.sort(CmpColumn(0))
    elif sortBy==’location’:
        equipmentList.sort(CmpColumn(6))

    outFilename = “by-%s.html” % (sortBy)
    print
    print “Would print to %s” % (outFile)
    print

    n=0
    seenTags = []
    inventoryTableList=DIV()
    for equipmentItem in equipmentList:
        (itemTag, itemDescription, itemSerial, itemModel, itemManufacturer,
         itemArrival, itemLocation, itemCost) = equipmentItem

        try:
            seenTags.index(itemTag)
        except ValueError:
            seenTags.append(itemTag)
            n=n+1
            if (n%2)==0:
                inventoryTable=TABLE(border=’1′,style=’page-break-after: always’)
            else:
                inventoryTable=TABLE(border=’1′)

            headerRow=TR()
            headerRow.append(TH(’Tag’))
            headerRow.append(TH(’Item Description’))
            headerRow.append(TH(’Serial Number’))
            headerRow.append(TH(’Model Number’))
            headerRow.append(TH(’Arrival’))
            headerRow.append(TH(’Manufacturer’))
            headerRow.append(TH(’Location’))
            headerRow.append(TH(’Cost’))
            inventoryTable.append(headerRow)

            itemRow=TR()
            if itemTag[0] in string.digits:
                itemTag=”.join(['0',itemTag])
            itemRow.append(TD(itemTag,align=’right’))
            itemRow.append(TD(itemDescription))
            itemRow.append(TD(itemSerial))
            itemRow.append(TD(itemModel))
            itemRow.append(TD(itemArrival,align=’right’))
            itemRow.append(TD(itemManufacturer))
            itemRow.append(TD(itemLocation,align=’center’))
            itemRow.append(TD(”$%.2f” % (float(itemCost)),align=’right’))
            inventoryTable.append(itemRow)

            imageRow=TR()
            if os.path.exists(”%s.jpg” % itemTag):
                imageRow.append(TD(IMG(src=”%s.jpg” % itemTag),
                                   colspan=4,align=’center’,width=’640′))
            else:
                imageRow.append(TD(IMG(src=”missing.gif”),
                                   colspan=4,align=’center’,width=’640′))

            if os.path.exists(”%s-detail.jpg” % itemTag):
                imageRow.append(TD(IMG(src=”%s-detail.jpg” % itemTag),
                                   colspan=4,align=’center’,width=’640′))
            else:
                imageRow.append(TD(IMG(src=”missing.gif”),
                                   colspan=4,align=’center’,width=’640′))

            inventoryTable.append(imageRow)

            notesRow=TR()
            if notesDict.has_key(itemTag):
                notesRow.append(TD(”Notes: %s” % notesDict[itemTag],colspan=8))
            else:
                notesRow.append(TD(”Notes:”,colspan=8))

            inventoryTable.append(notesRow)

            inventoryTableList.append(inventoryTable)
            inventoryTableList.append(BR())

    head=HEAD()
    head.append(STYLE(type=’text/css’))
    htmlOutput=Document(head,inventoryTableList)
    htmlFile=open(outFilename,’w+’)
    htmlOutput.writeto(htmlFile)
    htmlFile.close()

Once this script has run, I have two web pages of the same inventory data: one sorted by location, and another by property tag. I generally only use the location-sorted one, though. The CSS tries to force page breaks every two items, and judicious use of landscape printing and a bit of scaling in Firefox means I can carry around 40 pages of my report and find everything I need pretty quickly.

1. By default, the majority of a researcher’s files are stored on a single drive (hard drive in the office, hard drive at home, or flash drive). A single hardware failure can mean the loss of days, weeks, months, or years of work.
2. If a researcher makes a mistake on a particular file or set of files that can’t be easily undone, they may have considerable difficulty in restoring those files to their former state. Most people don’t have automated backups of any kind, and many just manually save copies of their work onto different folders periodically.
3. Advisors don’t have automatic access to their students’ research files upon graduation. Rarely do they have easy access to them during the research period. Collaboration is often reduced to emailing drafts back and forth, or to shuffling materials around on removable media on an ad hoc basis. Simple supervision uses the same methods, but on an even more infrequent basis.
4. Researchers don’t have a standard storage location for their research materials that is large enough to contain all relevant files, accessible from around the world, and secured against unauthorized access. GMail doesn’t count.
5. Researchers don’t have a standard place to publish works in progress, completed papers, and anything else that would be of use to the larger research community (at TTU and/or elsewhere).
6. Researchers don’t have a facility for others to comment on completed projects, and to collaborate on works in progress.
7. Researchers with computational needs tend to focus on the types of problems they can solve with PCs in ITS labs, PCs on their desk, or PCs they can purchase on a project budget. These PCs are often underpowered, underutilized, and redundant purchases when you consider multiple projects. They also limit the scope and scale of problems that can be solved.

Why would anyone related to TTU’s Engineering research activities care?

1. We shouldn’t limit our computational research unnecessarily. We should work to the limit of our available facilities, and use those facilities as effeciently as possible.
2. Researchers shouldn’t have to worry about their storage media’s integrity. They should be able to trust that if they save files somewhere safe, that they’ll be there the next time they’re needed. They also shouldn’t have to always worry about keeping multiple copies organized.
3. Especially for projects where there is more than one researcher, and also for projects of interest to a supervisor or advisor, the ability to automatically track code and other changes automatically would be great. Even on single-researcher projects, the ability to track all the details of changes means the ability to revert those changes as needed.
4. Many techniques that are new to a particular research group may be established procedure for another. This could include image processing, Groebner bases, boundary element methods, LaTeX tips, etc. If the various research groups can see the works in progress of other groups, then they at least have an opportunity to comment and suggest alternative strategies. Similarly, if groups are in the habit of constantly publishing their daily successes and failures, then others at TTU or worldwide can avoid reinventing the wheel and/or offer suggestions on how to work around the particular problems.

My basic strategy is as follows (some of these have already been done to varying degrees):

1. Define types of services that each of the above goals, and set up the specific hardware and software for those services.
2. Identify faculty and student champions/testers for each of these types of services. Ideally, these researchers would be spread around all departments in Engineering.
3. Tune service offerings to meet unexpected needs from the testing group.
4. Branch out to other faculty and student researchers not in the initial testing group. The testing group would provide some credence to the tested methodologies from step 3, and could demonstrate how the business of conducting research had improved as a result.

So what does step 1 look like? The following table shows the various services already available in the CAE network, and how they help solve the problems given above:

Next up, identifying my champions/testers in different departments, tuning the offerings to meet their unexpected needs, and starting on documentation and publicity materials for the rest of the college.