class cluster-host inherits public-host {
    # ...
    file { "/etc/hosts.equiv":
        source  => "puppet:///files/apps/rsh-server/hosts.equiv.$hostgroup",
        owner   => root,
        group   => root,
        mode    => 644,
        require => Package[rsh-server],
    }
}

Along the lines of Puppet Best Practice 2.0, I’m trying to compartmentalize all my building blocks into separate modules, even the really simple ones that just install a package or two. So here’s the contents of a trivial hello module for Debian:

# /etc/puppet/modules/hello/manifests/init.pp
class hello {
  package { "hello":
    ensure => latest;
  }
}

Nothing complicated here: install this package via the default package provider. Add an include hello to my node’s manifest, and the hello package gets installed.

Now for the “undo” class:

# /etc/puppet/modules/hello/manifests/disabled.pp
class hello::disabled inherits hello {
  Package["hello"] {
    ensure => purged
  }
}

And this is neat. Properly written, the disabling class can undo all the changes made by the parent class. Modify my node’s manifest from include hello to include hello::disabled and I can switch that class on and off like, well, a switch.

It all started so simply: I was going to set up a little Xen instance to be my next cluster submit host, and needed a spare address for it:

1. I started setting up an instance for ch208i.cae.tntech.edu, since it was no longer on the Xen host like it was several months ago. Crap, the reason it’s no longer on the Xen instance is because I moved it to its own dedicated hardware — it’s still my main ftp/mirror server. Ctrl-C that one.
2. Hmm, what’s available from old Xen instances? mail2.cae.tntech.edu.cfg from when I was testing out a new mail server setup last fall — doesn’t ping, doesn’t show up in xm list, no problem.

   xen-create-image --hostname=mail2.cae.tntech.edu --ip=149.149.254.23 \
       --gateway=149.149.254.4 --netmask=255.255.255.0 --size=10Gb --memory=256Mb \
       --swap=1Gb --debootstrap --force

   A few minutes later, my instance is debootstrapped and ready to go.

3. Oh, crap. Why am I getting an error on xm create that says my LVM is already in use on a domU somewhere?
4. Further crap. Looking in /etc/xen/mail.cae.tntech.edu.cfg for the production mail server, it apparently uses the old mail2.cae.tntech.edu LVMs. Wonderful. ssh mail? It works since sshd was already memory-resident, but /root/.profile doesn’t exist. And neither does much of anything else.
5. Great. I’ve just killed the mail server. Off to the Amanda server to do a quick restore of its data. What? I never put mail.cae.tntech.edu into the backup list? Not normally the end of the world, since the mail stores are held accessed over NFS from the main file server, but what about my dovecot and postfix configurations?
6. Oh, well. Time to see how good my puppet manifests are for the mail server.

Not too bad, as it turns out. Total downtime was only a couple hours, including having to redo the postfix and dovecot configurations (which were then copied off to the puppetmaster). I still have a few more things to fix, but mail delivery is up, and imap is running. TLS support for my sending mail from home isn’t up yet, but it’ll be fixed shortly.

I still need to fix that submit host, though. Next time, I think I’ll use an IP address reserved for my office.

Update: after getting a partial TLS/SASL setup going late Wednesday night, I went to sleep without realizing I’d killed mail delivery again. Finally got it straightened out Thursday morning.

• Slides for presentation
• Handouts for presentation

Update: So yesterday, I get an email regarding my presentation (well, the slides, at least). No reason to clutter up the main page with it though, so if you’re not happy with the slides and want to express your displeasure, read the rest after the jump and see if I’ve addressed your concerns already.

Hi Mike,

I’ve visited your site before and found your Debian preseeding info to be useful.

That said, I just went through your presentation slides and must say I’m very disappointed. It contains numerous examples of what gives sysadmins a bad name. Egotistical, “I’m right, you’re stupid”, “I did this because I’m way too busy doing more important things than you”, etc. comments abound.

On several occasions your first bullet-point was “just do it”, or “you need this”. Hey Mike, people don’t come to conferences and presentations to listen to a smart-ass.

You mentioned that the people in #puppet gave you useful feedback. Next time you give a presentation, also get feedback from non-geeks. They’ll help you filter out the cruft that makes you look like a spoiled 5-year old talking about his new widget set at show-and-tell.

I hope for the sake of the attendees that your verbal presentation was better than your slideshow.

To respond to the comments more or less in order:

• Sorry you didn’t like my slides. At first, I thought you were an irritated audience member who waited a couple of weeks before emailing me. But since you’re apparently only judging this based off the slides, that’s different.
• “I’m right, you’re stupid” is in the eye of the reader. Though you can’t tell without the soundtrack, it tended to work out more like “I used to do things one way, which probably is the most common way everyone else does it. It didn’t scale for the following reasons, and here’s what I’m doing instead.”
• “I did it this way because I’m way too busy doing more important things than you” is a bit of an exaggerated inference. Am I busy? Sure. Am I busy doing things that most sysadmins don’t have to deal with? As far as I can tell, yes; most of the sysadmins in my immediate vicinity (and from past experience over the last 15-20 years) don’t have major duties outside systems administration, just like most of the engineers don’t have major duties outside their specialty or lab. These non-sysadmin tasks aren’t necessarily more important per se, but they’re important in my particular job description. The hours these other tasks take up in my week force me to find more efficient methods to do the systems administration tasks; others will possibly hit the same walls I have at different times — maybe when they have to keep track of 300 servers in 10 different roles, where all servers in a particular role have to be interchangeable. Maybe when they get a 1000 node cluster where a particular application has to be installed identically on every node, and on every node to be purchased in the future. My belief is that as time goes on, we’re all going to be managing more systems, not fewer, and that methods we use for managing a few systems relatively well don’t scale to larger groups of computers.
• “Just do it” or “You need this” shows up in three places: using version control, using NTP for time synchronization, and using SMTP for email. I stand by each of those points, being entirely convinced by the verbiage at infrastructures.org that was my primary source material. I cannot fathom why someone would use something other than SMTP for sending email, why they wouldn’t want version control of some form as the code that automates their systems administration tasks grows more complicated, or why they’d use a different protocol to synchronize their system clocks. To me, each of those is as self-evident as “your SAN should have redundant power supplies” and “racks are a good way to house a bunch of servers in a small space”. You may have counterexamples, but since you didn’t provide any, I’m left in the dark.
• The folks on #puppet did give me some useful feedback. As for other feedback, I did ask a coworker to look at the slides, and he saw no problems with them. However, he’s a full-time Windows systems administrator, so his opinion may be suspect. As for non-geeks, they’re really not the intended audience, were generally absent from the conference, and aren’t too likely to be interested in systems administration techniques.
• Cruft in the verbal presentation? Guilty, but some might call it illustrative anecdotes. Personally, I’ve always tried to work in Sidney Harris‘ “I think you should be more explicit here in step two” joke into at least one over-equationed lecture per semester. The students seem to enjoy it:
  
  Cruft in the slides? Matter of opinion, I guess. I did run out of time, but I honestly hadn’t done enough practice runs to see how long it would actually take.
• As for the spoiled five-year-old showing off his new widget set at show-and-tell, I have trouble understanding the issue. Lots of the talks at these conferences are basically a show-and-tell: other talks included “Software Deployment Using Ghost”, “Virtualizing Business Continuity — Getting Your Systems Back Online,” “DBA Task Automation II: Extending the Basics, Best Practices, Processes and Icing,” etc. When I submit an abstract saying that I’m going to give a talk about what goes into a “managed infrastructure,” its benefits over regular administration methods, and talk about a particular tool we use to accomplish some of these tasks, exactly what do I change in material? What do I change in delivery (that you didn’t see)? You’ve never told coworkers “holy crap, this Linux thing is awesome! It’s like Unix, but free and runs on regular PCs!” or similar? Nothing about using PHP to format stuff out of a database for some dynamic web pages? Nothing about a CMS or blogging platform that lets you do all the things a CMS or blog is supposed to do? Nothing about rrdtool, Nagios, cacti, apt-get, Perl/Python/Ruby or some other tool that you didn’t write, but by gosh it’s going to make all of your lives easier?

class jre5 {
  package { "sun-java5-jre":
    ensure => latest;
  }
}

It doesn’t work because Sun wants you to agree to its license before installing the JRE. There’s a couple of ways around this. First, the old-school method:

ssh host "yes | apt-get -y install sun-java5-jre"

where ‘yes’ is a standard Unix program that just prints out “yes” over and over until the program on the other side of the pipe terminates. But “ssh host foo” is not the way of the managed infrastructure.

The second method, much more friendly to centralized management, is to first install debconf-utils on a candidate system, and then install sun-java5-jre on the same system. Once that’s done, you can query the debconf database to see how it stored your answers to the Sun license agreement:

ch226-12:~# debconf-get-selections | grep sun-
sun-java5-bin   shared/accepted-sun-dlj-v1-1    boolean true
sun-java5-jre   shared/accepted-sun-dlj-v1-1    boolean true
sun-java5-jre   sun-java5-jre/jcepolicy note
sun-java5-jre   sun-java5-jre/stopthread        boolean true
sun-java5-bin   shared/error-sun-dlj-v1-1       error
sun-java5-jre   shared/error-sun-dlj-v1-1       error
sun-java5-bin   shared/present-sun-dlj-v1-1     note
sun-java5-jre   shared/present-sun-dlj-v1-1     note

Save those results (debconf seeds) into a file on the gold server. Then we can modify our jre5 class as follows:

class jre5 {
  package { "sun-java5-jre":
    require      => File["/var/cache/debconf/jre5.seeds"],
    responsefile => "/var/cache/debconf/jre5.seeds",
    ensure       => latest;
  }

  file { "/var/cache/debconf/jre5.seeds":
    source => "puppet:///jre5/jre5.seeds",
    ensure => present;
  }
}

Now our class will download the preseeded answers for the Java license, download and install the JRE, and then use the preseeded answers to skip past the license agreement. I had never messed with debconf seeding previously, since I had either just imaged my systems, or provided config files that would be used when I restarted any daemons or programs that depended on those files. Now debconf-utils is part of my standard system class definition.

Note that this method doesn’t work with the default puppet provided in Debian Etch (version 0.20) — the responsefile parameter for Debian packages was only added in puppet 0.22.

If you get an error of err: Could not retrieve catalog: Could not parse for environment development: Syntax error at 'Debian' at /etc/puppet/master/manifests/os/Debian.pp:1 on a Debian.pp that only has one line of class Debian {}, just go ahead and change it to class debian {}, then go on with your life. Yes, I know you tried to be smart and use the output of facter operatingsystem and to make include $operatingsystem work right, but just lowercase the class name.

Carry on.

1. There was one more file to manage outside my regular puppet manifests, and I’d have to remember to keep them both up to date and in sync.
2. There wasn’t an easy way of ensuring that other versions of a particular package got unstowed before deploying out the desired version.
3. The entire stow tree would be copied out to every system, regardless of whether OpenMPI was a good fit for the web server.

So, here’s my new method:

1. Keep my same metastow module loaded on the rsync server. The metastow module contains one top-level directory per puppet architecture (i686, x86_64, etc.). Each of those architecture folders is a stow tree containing every stowed package for that architecture.
2. Add a stowedpackage definition to my puppet manifests as follows:

   define stowedpackage ( $basepackage, $version,
       $rsyncserver='gold.cae.tntech.edu',
       $rsyncmodule='metastow',
       $stowdestdir='/usr/local/stow' ) {
       file { "stow-initiator_${basepackage}-${version}":
           source => "puppet:///files/stow-initiator_${basepackage}-${version}",
           path   => "/etc/puppet/stow-initiator_${basepackage}-${version}",
       }
       exec { download:
           command     => "/usr/bin/rsync -a --delete ${rsyncserver}::${rsyncmodule}/${hardwaremodel}/${basepackage}-${version} ${stowdestdir}",
           refreshonly => true,
           subscribe   => File["stow-initiator_${basepackage}-${version}"],
           alias       => "download_${basepackage}-${version}"
       }
       exec { unstow-others:
           command     => "cd ${stowdestdir} && stow --delete ${basepackage}-*",
           refreshonly => true,
           subscribe   => Exec["download_${basepackage}-${version}"],
           alias       => "unstow-others_${basepackage}-${version}"
       }
       exec { stow:
           command     => "cd ${stowdestdir} ; stow ${basepackage}-${version}",
           refreshonly => true,
           subscribe   => Exec["unstow-others_${basepackage}-${version}"]
       }
   }
   

3. Use the stowedpackage definition in other parts of my manifests:

   # Create OpenMPI installation and configuration.
   class openmpi {
   
       stowedpackage {
           "openmpi-1.0.1":
               basepackage=>"openmpi",
               version=>"1.0.1";
       }
   
   }
   

4. Add a trigger file to the puppetmaster’s /etc/puppet/files folder:

   /etc/puppet/files# date > stow-initiator_openmpi-1.0.1
   

]]> http://blogs.cae.tntech.edu/mwr/2008/02/01/the-autostow-is-dead-long-live-stowedpackage/feed/ 1 http://blogs.cae.tntech.edu/mwr/2007/10/31/client-configuration-management/ http://blogs.cae.tntech.edu/mwr/2007/10/31/client-configuration-management/#comments Thu, 01 Nov 2007 01:17:21 +0000 Mike Renfro http://blogs.cae.tntech.edu/mwr/2007/10/31/client-configuration-management/ Back at the infrastructures.org mothership, client configuration management is described as everything that makes a host unique and/or part of a particular group or domain. And for Unix-like systems, everything pretty much comes down to configuration files, services being enabled/disabled, and cron jobs.

Hmm.

• Configuration files
• Services
• Cron jobs

Looks like Puppet pretty much handles all of that. As long as I can describe aspects of my systems with puppet classes and modules, I’ve got reusable, consistent configurations on any servers I care to manage.

]]> http://blogs.cae.tntech.edu/mwr/2007/10/31/client-configuration-management/feed/ 0 http://blogs.cae.tntech.edu/mwr/2007/10/31/client-file-access/ http://blogs.cae.tntech.edu/mwr/2007/10/31/client-file-access/#comments Thu, 01 Nov 2007 01:00:16 +0000 Mike Renfro http://blogs.cae.tntech.edu/mwr/2007/10/31/client-file-access/ The infrastructures.org folks list two primary goals of what they call “client file access“: first, consistent access to users’ home directories, and second, consistent access to end-user applications. Some of the things they warn against, such as automounters and the /net directory, we never thought of using to begin with. Their need to consider systems with limited disk space and a need to mount a software share via NFS is less of an issue with us, too: disks large enough to hold our regular software load are relatively cheap, so there shouldn’t be much of a problem there. And since we’re supposed to be deploying applications consistently via Puppet, there shouldn’t be any inconsistency in where an application is installed. As far as consistent access to the home directories, we accomplish this with a two-pronged solution in Puppet — one on the file server, and one on the clients:

On the Debian file server, we include the following class:

class nfs-server {
  package { nfs-kernel-server: ensure => installed }

  file { exports:
    path => $operatingsystem ? {
      default => "/etc/exports"
    },
    owner => root, group => root, mode => 644,
    source => "puppet://gold.cae.tntech.edu/files/apps/nfs-server/exports",
    require => Package[nfs-kernel-server]
  }

  service { nfs-kernel-server:
    ensure => running,
    enable => true,
    hasstatus => true,
    subscribe => [Package[nfs-kernel-server], File[exports]]
  }
  service { portmap:
    ensure => running,
    enable => true,
  }
}

We also include the following in its node definition:

    file {
        "/etc/exports.d":
        source  => "puppet:///files/apps/nfs-kernel-server/exports.d",
        ensure  => directory,
        ignore  => ".svn",
        purge   => true,
        recurse => inf;
    }

    concatenated_file {
        "/etc/exports":
            dir => "/etc/exports.d";
    }

Each file dropped in /etc/exports.d is a fragment of /etc/exports. Right now, we’re using one fragment per exported filesystem. As an example, here’s /etc/exports.d/_opt_solaris_stow:

/opt/solaris/stow \\
        149.149.254.12(rw,sync,no_root_squash) \\
        149.149.254.13(rw,sync,no_root_squash) \\
        149.149.254.14(rw,sync,no_root_squash) \\
        149.149.254.15(rw,sync,no_root_squash) \\
        149.149.254.220(rw,sync,no_root_squash)

David Schmitt’s concatenated_file definition combines all the export fragments into one /etc/exports file:

define concatenated_file ($dir, $mode = 0644, $owner = root, $group = root)
{
    file {
        $name:
            ensure   => present,
            checksum => md5,
            mode     => $mode,
            owner    => $owner,
            group => $group;
    }
    exec { "find ${dir} -maxdepth 1 -type f ! -name '*puppettmp' -print0 | sort -z | xargs -0 cat > ${name}":
        refreshonly => true,
        subscribe   => File[$dir],
        alias       => "concat_${name}",
    }
}
define concatenated_file_part ($dir, $content = '', $ensure = present,
                               $mode = 0644, $owner = root, $group = root) {
    file { "${dir}/${name}":
        ensure  => $ensure,
        content => $content,
        mode    => $mode,
        owner   => $owner,
        group   => $group,
        alias   => "cf_part_${name}",
    }
}

On the client side, we just make sure we can mount the NFS share. For example:

    file {
        "/home/CAE":
            ensure => directory,
            owner  => root,
            group  => root,
            mode   => 755,
    }
    mount { "/home/CAE":
        require => File["/home/CAE"],
        atboot  => true,
        fstype  => "nfs",
        device  => "files.cae.tntech.edu:/home/CAE",
        ensure  => mounted,
        options => "defaults",
        dump    => "0",
        pass    => "0",
    }
    package { "nfs-common": ensure => installed }

]]> http://blogs.cae.tntech.edu/mwr/2007/10/31/client-file-access/feed/ 0 http://blogs.cae.tntech.edu/mwr/2007/10/31/file-replication-servers/ http://blogs.cae.tntech.edu/mwr/2007/10/31/file-replication-servers/#comments Thu, 01 Nov 2007 00:30:51 +0000 Mike Renfro http://blogs.cae.tntech.edu/mwr/2007/10/31/file-replication-servers/ Back when the infrastructures.org folks were writing their pages, the page for file replication servers described a need to keep current copies of configuration files in /etc and all programs and other data from /usr/local on all the managed systems. In puppet structures, every file or other resource is just a part of a higher-order class or module. If we’re already keeping our classes and modules up to date on all the managed systems, then old-school file replication goes along automatically.
]]> http://blogs.cae.tntech.edu/mwr/2007/10/31/file-replication-servers/feed/ 0