It all started so simply: I was going to set up a little Xen instance to be my next cluster submit host, and needed a spare address for it:

1. I started setting up an instance for ch208i.cae.tntech.edu, since it was no longer on the Xen host like it was several months ago. Crap, the reason it’s no longer on the Xen instance is because I moved it to its own dedicated hardware — it’s still my main ftp/mirror server. Ctrl-C that one.
2. Hmm, what’s available from old Xen instances? mail2.cae.tntech.edu.cfg from when I was testing out a new mail server setup last fall — doesn’t ping, doesn’t show up in xm list, no problem.

   xen-create-image --hostname=mail2.cae.tntech.edu --ip=149.149.254.23 \
       --gateway=149.149.254.4 --netmask=255.255.255.0 --size=10Gb --memory=256Mb \
       --swap=1Gb --debootstrap --force

   A few minutes later, my instance is debootstrapped and ready to go.

3. Oh, crap. Why am I getting an error on xm create that says my LVM is already in use on a domU somewhere?
4. Further crap. Looking in /etc/xen/mail.cae.tntech.edu.cfg for the production mail server, it apparently uses the old mail2.cae.tntech.edu LVMs. Wonderful. ssh mail? It works since sshd was already memory-resident, but /root/.profile doesn’t exist. And neither does much of anything else.
5. Great. I’ve just killed the mail server. Off to the Amanda server to do a quick restore of its data. What? I never put mail.cae.tntech.edu into the backup list? Not normally the end of the world, since the mail stores are held accessed over NFS from the main file server, but what about my dovecot and postfix configurations?
6. Oh, well. Time to see how good my puppet manifests are for the mail server.

Not too bad, as it turns out. Total downtime was only a couple hours, including having to redo the postfix and dovecot configurations (which were then copied off to the puppetmaster). I still have a few more things to fix, but mail delivery is up, and imap is running. TLS support for my sending mail from home isn’t up yet, but it’ll be fixed shortly.

I still need to fix that submit host, though. Next time, I think I’ll use an IP address reserved for my office.

Update: after getting a partial TLS/SASL setup going late Wednesday night, I went to sleep without realizing I’d killed mail delivery again. Finally got it straightened out Thursday morning.

• make
• Kerberos
• tar
• syslog

We’ve got a central NTP server on campus, and I’m using that to sync from. Puppet handles ntp and ntpdate configuration on the managed systems. Components of that setup:

• ntp.pp and ntpdate.pp classes imported from puppet/classes
• Virtualization-detecting facter recipe (originally from here, but also included below since it’s short and in case the original gets moved). This does two things: first, Xen domUs get their time from the dom0 by default. They won’t fail running ntp, but if dom0 has the wrong time, you’ll have a hard time getting any of the domUs to ever get the right time. So we’ll make sure ntp isn’t running there, as a reminder. Second, according to the virtualization recipe’s author, VMWare guests can’t run ntp at all. So we’ll disable it there, too.

/etc/puppet/facts/virtual.rb

Facter.add("virtual") do
  confine :kernel => :linux
  result = "physical"
  setcode do
    lspciexists = system "which lspci >&/dev/null"
    if $?.exitstatus == 0
      output = %x{lspci}
      output.each {|p|
        # --- look for the vmware video card to determine
        # if it is virtual => vmware.
        # ---     00:0f.0 VGA compatible controller: VMware ...
        result = "vmware" if p =~ /VMware/
        }
    end
    # VMware server 1.0.3 rpm places vmware-vmx in this place,
    # other versions or platforms may not.
    if FileTest.exists?("/usr/lib/vmware/bin/vmware-vmx")
      result = "vmware_server"
    end
    if FileTest.exists?("/proc/sys/xen/independent_wallclock")
      result = "xenu"
    elsif FileTest.exists?("/proc/xen/capabilities")
      txt = File.read("/proc/xen/capabilities")
      if txt =~ /control_d/i
        result = "xen0"
      end
    end
    result
  end
end

/etc/puppet/manifests/classes/ntp.pp

class ntp {
  $ntppackage = $operatingsystem ? {
      Solaris => "SUNWntpu",
      default => "ntp"
  }
  package { $ntppackage:
      ensure => installed,
      provider => $operatingsystem ? {
          Solaris => "sun",
          default => "apt"
      }
  }

  file { ntpconf:
    path => $operatingsystem ? {
      Solaris => "/etc/inet/ntp.conf",
      default => "/etc/ntp.conf"
    },
    owner => root, group => root, mode => 644,
    source => "puppet://REDACTED/ntp.conf",
    require => Package[$ntppackage],
  }

  service { ntp:
    ensure => $virtual ? {
      vmware => stopped,
      xenu => stopped,
      default => running
    },
    enable => $virtual ? {
      vmware => false,
      xenu => false,
      default => true
    },
    subscribe => [Package[$ntppackage], File[ntpconf]]
  }
}

/etc/puppet/manifests/classes/ntpdate.pp

class ntpdate {
  package { ntpdate: ensure => installed }
}

and one entry from /etc/puppet/manifests/site.pp:

node ch405l {
  include ntp, ntpdate
}

Minor annoyances or deviations from the way things used to be configured: as of Debian 4.0, ntpdate is run when network interfaces are brought up, rather than at a user-defined time via the SysV init system. So if a system was installed with a bad time (most commonly on our dual-boot systems) and you want to avoid reboots, you’ll have to run ntpdate-debian once to get the clock in sync with the NTP server before ntpd will do anything right.