So assuming we have a project for ME4444, group 3 (I already had projects defined for groups 1 and 2 from earlier tests):

# grep /home /etc/fstab
/dev/md1000/home        /home   xfs     defaults,usrquota,prjquota      0      1
# echo "me4444-03:/home/projects/me4444-03" >> /etc/projects
# echo "me4444-03:3" >> /etc/projid
# mkdir /home/projects/me4444-03
# xfs_quota -x -c "project -s me4444-03"
Setting up project me4444-03 (path /home/projects/me4444-03)...
Processed 1 /etc/projects paths for project me4444-03
# xfs_quota -x -c "limit -p bsoft=5g bhard=10g me4444-03"
# xfs_quota -x -c "report -p"
Project quota on /home (/dev/md1000/home)
                               Blocks
Project ID       Used       Soft       Hard    Warn/Grace
---------- --------------------------------------------------
me4444-01           0          0    1048576     00 [--------]
me4444-02           0    5242880   10485760     00 [--------]
me4444-03           0    5242880   10485760     00 [--------]

Now group 3 has a 5 GB “soft” quota, can exceed that for up to 7 days at a time, but can never exceed their 10 GB “hard” quota. All that’s left is setting up directory permissions and Samba configuration so that the authorized users can store things there.

It all started so simply: I was going to set up a little Xen instance to be my next cluster submit host, and needed a spare address for it:

1. I started setting up an instance for ch208i.cae.tntech.edu, since it was no longer on the Xen host like it was several months ago. Crap, the reason it’s no longer on the Xen instance is because I moved it to its own dedicated hardware — it’s still my main ftp/mirror server. Ctrl-C that one.
2. Hmm, what’s available from old Xen instances? mail2.cae.tntech.edu.cfg from when I was testing out a new mail server setup last fall — doesn’t ping, doesn’t show up in xm list, no problem.

   xen-create-image --hostname=mail2.cae.tntech.edu --ip=149.149.254.23 \
       --gateway=149.149.254.4 --netmask=255.255.255.0 --size=10Gb --memory=256Mb \
       --swap=1Gb --debootstrap --force

   A few minutes later, my instance is debootstrapped and ready to go.

3. Oh, crap. Why am I getting an error on xm create that says my LVM is already in use on a domU somewhere?
4. Further crap. Looking in /etc/xen/mail.cae.tntech.edu.cfg for the production mail server, it apparently uses the old mail2.cae.tntech.edu LVMs. Wonderful. ssh mail? It works since sshd was already memory-resident, but /root/.profile doesn’t exist. And neither does much of anything else.
5. Great. I’ve just killed the mail server. Off to the Amanda server to do a quick restore of its data. What? I never put mail.cae.tntech.edu into the backup list? Not normally the end of the world, since the mail stores are held accessed over NFS from the main file server, but what about my dovecot and postfix configurations?
6. Oh, well. Time to see how good my puppet manifests are for the mail server.

Not too bad, as it turns out. Total downtime was only a couple hours, including having to redo the postfix and dovecot configurations (which were then copied off to the puppetmaster). I still have a few more things to fix, but mail delivery is up, and imap is running. TLS support for my sending mail from home isn’t up yet, but it’ll be fixed shortly.

I still need to fix that submit host, though. Next time, I think I’ll use an IP address reserved for my office.

Update: after getting a partial TLS/SASL setup going late Wednesday night, I went to sleep without realizing I’d killed mail delivery again. Finally got it straightened out Thursday morning.

• Slides for presentation
• Handouts for presentation

Update: So yesterday, I get an email regarding my presentation (well, the slides, at least). No reason to clutter up the main page with it though, so if you’re not happy with the slides and want to express your displeasure, read the rest after the jump and see if I’ve addressed your concerns already.

Hi Mike,

I’ve visited your site before and found your Debian preseeding info to be useful.

That said, I just went through your presentation slides and must say I’m very disappointed. It contains numerous examples of what gives sysadmins a bad name. Egotistical, “I’m right, you’re stupid”, “I did this because I’m way too busy doing more important things than you”, etc. comments abound.

On several occasions your first bullet-point was “just do it”, or “you need this”. Hey Mike, people don’t come to conferences and presentations to listen to a smart-ass.

You mentioned that the people in #puppet gave you useful feedback. Next time you give a presentation, also get feedback from non-geeks. They’ll help you filter out the cruft that makes you look like a spoiled 5-year old talking about his new widget set at show-and-tell.

I hope for the sake of the attendees that your verbal presentation was better than your slideshow.

To respond to the comments more or less in order:

• Sorry you didn’t like my slides. At first, I thought you were an irritated audience member who waited a couple of weeks before emailing me. But since you’re apparently only judging this based off the slides, that’s different.
• “I’m right, you’re stupid” is in the eye of the reader. Though you can’t tell without the soundtrack, it tended to work out more like “I used to do things one way, which probably is the most common way everyone else does it. It didn’t scale for the following reasons, and here’s what I’m doing instead.”
• “I did it this way because I’m way too busy doing more important things than you” is a bit of an exaggerated inference. Am I busy? Sure. Am I busy doing things that most sysadmins don’t have to deal with? As far as I can tell, yes; most of the sysadmins in my immediate vicinity (and from past experience over the last 15-20 years) don’t have major duties outside systems administration, just like most of the engineers don’t have major duties outside their specialty or lab. These non-sysadmin tasks aren’t necessarily more important per se, but they’re important in my particular job description. The hours these other tasks take up in my week force me to find more efficient methods to do the systems administration tasks; others will possibly hit the same walls I have at different times — maybe when they have to keep track of 300 servers in 10 different roles, where all servers in a particular role have to be interchangeable. Maybe when they get a 1000 node cluster where a particular application has to be installed identically on every node, and on every node to be purchased in the future. My belief is that as time goes on, we’re all going to be managing more systems, not fewer, and that methods we use for managing a few systems relatively well don’t scale to larger groups of computers.
• “Just do it” or “You need this” shows up in three places: using version control, using NTP for time synchronization, and using SMTP for email. I stand by each of those points, being entirely convinced by the verbiage at infrastructures.org that was my primary source material. I cannot fathom why someone would use something other than SMTP for sending email, why they wouldn’t want version control of some form as the code that automates their systems administration tasks grows more complicated, or why they’d use a different protocol to synchronize their system clocks. To me, each of those is as self-evident as “your SAN should have redundant power supplies” and “racks are a good way to house a bunch of servers in a small space”. You may have counterexamples, but since you didn’t provide any, I’m left in the dark.
• The folks on #puppet did give me some useful feedback. As for other feedback, I did ask a coworker to look at the slides, and he saw no problems with them. However, he’s a full-time Windows systems administrator, so his opinion may be suspect. As for non-geeks, they’re really not the intended audience, were generally absent from the conference, and aren’t too likely to be interested in systems administration techniques.
• Cruft in the verbal presentation? Guilty, but some might call it illustrative anecdotes. Personally, I’ve always tried to work in Sidney Harris‘ “I think you should be more explicit here in step two” joke into at least one over-equationed lecture per semester. The students seem to enjoy it:
  
  Cruft in the slides? Matter of opinion, I guess. I did run out of time, but I honestly hadn’t done enough practice runs to see how long it would actually take.
• As for the spoiled five-year-old showing off his new widget set at show-and-tell, I have trouble understanding the issue. Lots of the talks at these conferences are basically a show-and-tell: other talks included “Software Deployment Using Ghost”, “Virtualizing Business Continuity — Getting Your Systems Back Online,” “DBA Task Automation II: Extending the Basics, Best Practices, Processes and Icing,” etc. When I submit an abstract saying that I’m going to give a talk about what goes into a “managed infrastructure,” its benefits over regular administration methods, and talk about a particular tool we use to accomplish some of these tasks, exactly what do I change in material? What do I change in delivery (that you didn’t see)? You’ve never told coworkers “holy crap, this Linux thing is awesome! It’s like Unix, but free and runs on regular PCs!” or similar? Nothing about using PHP to format stuff out of a database for some dynamic web pages? Nothing about a CMS or blogging platform that lets you do all the things a CMS or blog is supposed to do? Nothing about rrdtool, Nagios, cacti, apt-get, Perl/Python/Ruby or some other tool that you didn’t write, but by gosh it’s going to make all of your lives easier?

class jre5 {
  package { "sun-java5-jre":
    ensure => latest;
  }
}

It doesn’t work because Sun wants you to agree to its license before installing the JRE. There’s a couple of ways around this. First, the old-school method:

ssh host "yes | apt-get -y install sun-java5-jre"

where ‘yes’ is a standard Unix program that just prints out “yes” over and over until the program on the other side of the pipe terminates. But “ssh host foo” is not the way of the managed infrastructure.

The second method, much more friendly to centralized management, is to first install debconf-utils on a candidate system, and then install sun-java5-jre on the same system. Once that’s done, you can query the debconf database to see how it stored your answers to the Sun license agreement:

ch226-12:~# debconf-get-selections | grep sun-
sun-java5-bin   shared/accepted-sun-dlj-v1-1    boolean true
sun-java5-jre   shared/accepted-sun-dlj-v1-1    boolean true
sun-java5-jre   sun-java5-jre/jcepolicy note
sun-java5-jre   sun-java5-jre/stopthread        boolean true
sun-java5-bin   shared/error-sun-dlj-v1-1       error
sun-java5-jre   shared/error-sun-dlj-v1-1       error
sun-java5-bin   shared/present-sun-dlj-v1-1     note
sun-java5-jre   shared/present-sun-dlj-v1-1     note

Save those results (debconf seeds) into a file on the gold server. Then we can modify our jre5 class as follows:

class jre5 {
  package { "sun-java5-jre":
    require      => File["/var/cache/debconf/jre5.seeds"],
    responsefile => "/var/cache/debconf/jre5.seeds",
    ensure       => latest;
  }

  file { "/var/cache/debconf/jre5.seeds":
    source => "puppet:///jre5/jre5.seeds",
    ensure => present;
  }
}

Now our class will download the preseeded answers for the Java license, download and install the JRE, and then use the preseeded answers to skip past the license agreement. I had never messed with debconf seeding previously, since I had either just imaged my systems, or provided config files that would be used when I restarted any daemons or programs that depended on those files. Now debconf-utils is part of my standard system class definition.

Note that this method doesn’t work with the default puppet provided in Debian Etch (version 0.20) — the responsefile parameter for Debian packages was only added in puppet 0.22.

class nfs-server {
  package { nfs-kernel-server: ensure => installed }

  file { exports:
    path => $operatingsystem ? {
      default => "/etc/exports"
    },
    owner => root, group => root, mode => 644,
    source => "puppet://gold.cae.tntech.edu/files/apps/nfs-server/exports",
    require => Package[nfs-kernel-server]
  }

  service { nfs-kernel-server:
    ensure => running,
    enable => true,
    hasstatus => true,
    subscribe => [Package[nfs-kernel-server], File[exports]]
  }
  service { portmap:
    ensure => running,
    enable => true,
  }
}

We also include the following in its node definition:

    file {
        "/etc/exports.d":
        source  => "puppet:///files/apps/nfs-kernel-server/exports.d",
        ensure  => directory,
        ignore  => ".svn",
        purge   => true,
        recurse => inf;
    }

    concatenated_file {
        "/etc/exports":
            dir => "/etc/exports.d";
    }

Each file dropped in /etc/exports.d is a fragment of /etc/exports. Right now, we’re using one fragment per exported filesystem. As an example, here’s /etc/exports.d/_opt_solaris_stow:

/opt/solaris/stow \\
        149.149.254.12(rw,sync,no_root_squash) \\
        149.149.254.13(rw,sync,no_root_squash) \\
        149.149.254.14(rw,sync,no_root_squash) \\
        149.149.254.15(rw,sync,no_root_squash) \\
        149.149.254.220(rw,sync,no_root_squash)

David Schmitt’s concatenated_file definition combines all the export fragments into one /etc/exports file:

define concatenated_file ($dir, $mode = 0644, $owner = root, $group = root)
{
    file {
        $name:
            ensure   => present,
            checksum => md5,
            mode     => $mode,
            owner    => $owner,
            group => $group;
    }
    exec { "find ${dir} -maxdepth 1 -type f ! -name '*puppettmp' -print0 | sort -z | xargs -0 cat > ${name}":
        refreshonly => true,
        subscribe   => File[$dir],
        alias       => "concat_${name}",
    }
}
define concatenated_file_part ($dir, $content = '', $ensure = present,
                               $mode = 0644, $owner = root, $group = root) {
    file { "${dir}/${name}":
        ensure  => $ensure,
        content => $content,
        mode    => $mode,
        owner   => $owner,
        group   => $group,
        alias   => "cf_part_${name}",
    }
}

On the client side, we just make sure we can mount the NFS share. For example:

    file {
        "/home/CAE":
            ensure => directory,
            owner  => root,
            group  => root,
            mode   => 755,
    }
    mount { "/home/CAE":
        require => File["/home/CAE"],
        atboot  => true,
        fstype  => "nfs",
        device  => "files.cae.tntech.edu:/home/CAE",
        ensure  => mounted,
        options => "defaults",
        dump    => "0",
        pass    => "0",
    }
    package { "nfs-common": ensure => installed }

Puppet and other configuration excerpts follow.

classes/nagios-nrpe-server.pp:

class nagios-nrpe-server {
    # nagios-nrpe-server for remote monitoring
    package {
        [ "nagios-nrpe-server", "nagios-plugins" ]:
            ensure => installed;
    }
    file {
        "/etc/nagios":
            ensure => directory,
            owner  => root,
            group  => root,
            mode   => 0755;
        "/etc/nagios/nrpe_local.cfg":
            source => "puppet:///files/apps/nagios-nrpe-server/nrpe_local.cfg";
    }
    service {
        "nagios-nrpe-server":
            ensure    => running,
            pattern   => "/usr/sbin/nrpe",
            subscribe => File["/etc/nagios/nrpe_local.cfg"],
            require   => Package["nagios-nrpe-server"];
    }
}

files/apps/nagios-nrpe-server/nrpe_local.cfg:

allowed_hosts=127.0.0.1,NAGIOSIP
command[check_disk_root]=/usr/lib/nagios/plugins/check_disk -w 20 -c 10 -p /
command[check_disk_boot]=/usr/lib/nagios/plugins/check_disk -w 20 -c 10 -p /boot
command[check_disk_tmp]=/usr/lib/nagios/plugins/check_disk -w 20 -c 10 -p /tmp
command[check_disk_var]=/usr/lib/nagios/plugins/check_disk -w 20 -c 10 -p /var
command[check_disk_amanda]=/usr/lib/nagios/plugins/check_disk -w 20 -c 10 -p /opt/amanda
command[check_disk_home]=/usr/lib/nagios/plugins/check_disk -w 20 -c 10 -p /home

excerpt from /etc/nagios2/conf.d/hosts.cfg:

define service{
        use                     generic-service
        host_name               ch226-21, ch226-22, ch226-23, ch226-24, ch226-25, ch226-26, ch226-27, ch226-28, ch226-29, ch226-30, ch226-31, ch226-32
        service_description     Disk Usage - /
        check_command           check_nrpe_1arg!check_disk_root
        contact_groups  admins
}
define service{
        use                     generic-service
        host_name               ch226-21, ch226-22, ch226-23, ch226-24, ch226-25, ch226-26, ch226-27, ch226-28, ch226-29, ch226-30, ch226-31, ch226-32
        service_description     Disk Usage - /tmp
        check_command           check_nrpe_1arg!check_disk_tmp
        contact_groups  admins
}

I tried lots of quick fixes:

• purge, clean, and reinstall — no good.
• purge, clean, clear out /etc/alternatives and /usr/bin of any detritus from earlier package versions — no luck there, either
• edit /var/lib/dpkg/info/hyperworks8.postinst to remove the amfbuilder line — that just moved the error onto another program included in the package

Finally, I found a /var/lib/dpkg/alternatives/hw file and removed it. It contained nothing but my Hyperworks-related settings, so that was safe. Afterwards, no more errors.

• Lots of Linux systems and a few Solaris systems, some of which dual-boot and aren’t accessible as *nix systems during normal hours of the day
• An Active Directory already in place
• An overworked administration team that has lots of other stuff to do besides create user accounts

As I had indicated at the end of the previous authentication server post, my experience with winbind has gotten decidedly more positive lately. My previous experience with it in 2001-2002 was entirely negative, since either due to design limitations or my own incompetence, I was unable to get multiple systems running winbind to agree on what the users’ UIDs and GIDs were, which made NFS unworkable. As of 2007, things have improved vastly.

So what do I need the authentication server structure to do for me?

1. keep a canonical mapping of usernames and groups to UIDs and GIDs
2. keep a list of which users are allowed to access particular hosts or services
3. keep a canonical list of passwords for those users

NSS and Winbind will handle the first goal. PAM and the pam_listfile module will handle the second goal. And PAM and Kerberos will handle the third. Winbind and Kerberos will both verify everything against the existing Active Directory. The gold server will generate a master list of allowed users via puppet and file fragments, and all the clients will use that list by default.

The puppet manifest for an Active Directory member system looks like:

# For our purposes, an Active Directory member should:

# 1. Run winbind to enumerate users and groups beyond what's already in
#    /etc/passwd and /etc/group (requires working machine account in
#    domain)

# 2. Keep a list of valid users for the local system, since not all
#    domain users should get automatic shell access

# 3. Check passwords via Kerberos

class active-directory-member {
    package {
        [ "winbind", "krb5-config", "libpam-krb5" ]:
            ensure    => installed;
    }
    service { [ "winbind" ]:
        pattern   => "/usr/sbin/winbindd",
        ensure    => running,
        subscribe => [ File["/etc/samba/smb.conf"],
                       File["/var/lib/samba/secrets.tdb"] ];
    }
    file {
        "/etc/samba/smb.conf":
            source => "puppet:///files/apps/samba/smb.conf";

        "/var/lib/samba":
            ensure => directory,
            owner  => root,
            group  => root,
            mode   => 755;

        "/var/lib/samba/secrets.tdb":
            source => "puppet:///private/secrets.tdb",
            owner  => root,
            group  => root,
            mode   => 600;

        "/etc/nsswitch.conf":
            source => "puppet:///files/apps/active-directory-member/nsswitch.conf";

        "/etc/pam.d/common-account":
            source => "puppet:///files/apps/active-directory-member/common-account";

        "/etc/krb5.conf":
            source => "puppet:///files/apps/active-directory-member/krb5.conf";

        "/etc/pam.d/common-auth":
            source => "puppet:///files/apps/active-directory-member/common-auth";

        "/etc/security/users.conf":
            source => "puppet:///files/apps/pam_listfile/users.conf.default";
    }
}

Most of this is pretty straightforward Puppet management (”I need winbind, libpam-krb5, and krb5-config installed.” “Make sure winbind is running, and restart it if the machine account file or the Samba configuration file changes.” “Grab these Kerberos config files from the puppet server.”) A few things that might be non-obvious:

I’ve got a private fileserver area on the puppet server. I use this to store system-specific security information, such as ssh keys and the Samba machine account (secrets.tdb) for Active Directory. Barring a puppet-level breakin on the gold server or a root-level breakin on the CVS server, this is safe. Only systems with properly signed keys get access to their specific private folder, and they don’t get access to any other system’s private folder.

The Samba configuration I use for systems that only need winbind running is pretty minimal:

[global]
   netbios name = %h-linux
   workgroup = CAE
   log file = /var/log/samba/log.%m
   max log size = 1000
   syslog = 0
   panic action = /usr/share/samba/panic-action %d
   security = ADS
   realm = CAE.TNTECH.EDU
   allow trusted domains = No
   idmap backend = idmap_rid:CAE=5000-100000000
   idmap uid = 5000-100000000
   idmap gid = 5000-100000000
   template shell = /bin/bash
   winbind use default domain = Yes
   winbind nested groups = Yes
; The following was the default behaviour in sarge
; but samba upstream reverted the default because it might induce
; performance issues in large organizations
; See #368251 for some of the consequences of *not* having
; this setting and smb.conf(5) for all details
;
;   winbind enum groups = yes
;   winbind enum users = yes
   winbind enum users = No
   winbind enum groups = No

but it’s good enough to tell winbind where to look for username, UID, and GID information. netbios name is set differently from the default to ensure that we don’t have a conflict on dual-boot systems between their Windows machine account and their Linux machine account. On Linux-only systems, I could get away with using the default, but this setting works in both cases, so I’ll likely keep it as-is.

Since all my winbind systems use the same idmap backend and read the same RIDs from Active Directory (added in Samba 3.0.8, November 2004), they all calculate the same UIDs and GIDs. No NFS problems there any more. One step that’s not yet automated is the initial creation of an Active Directory machine account. On larger rollouts, I expect I’ll ssh into each system and run that part by hand. Eventually, I plan to either put this into a puppet manifest or into my preseed configuration. I’ll also need to make sure that I periodically sync the machine accounts from the local systems to the gold server, since Active Directory machine accounts change their own passwords from time to time.

nsswitch.conf has added entries for winbind on the passwd, group, and shadow databases:

# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         compat winbind
group:          compat winbind
shadow:         compat winbind

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

common-account and common-auth are where most Debian PAM-aware applications look for account and authentication information, respectively.

#
# /etc/pam.d/common-account - authorization settings common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of the authorization modules that define
# the central access policy for use on the system.  The default is to
# only deny service to users whose accounts are expired in /etc/shadow.
#
account sufficient      pam_unix.so
account required        pam_listfile.so onerr=fail sense=allow file=/etc/security/users.conf item=user
account required        pam_winbind.so

If we’re looking up an account entry (not a password):

1. If they’re in the local passwd file, they’re allowed.
2. If they’re in the list of allowed users, they’re allowed. Otherwise, they’re not, so don’t go any further.
3. If you’ve gotten this far, and they’re in the Active Directory, they’re allowed.

#
# /etc/pam.d/common-auth - authentication settings common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of the authentication modules that define
# the central authentication scheme for use on the system
# (e.g., /etc/shadow, LDAP, Kerberos, etc.).  The default is to use the
# traditional Unix authentication mechanisms.
#
auth    sufficient      pam_unix.so nullok_secure
auth    required        pam_krb5.so use_first_pass

If we’re looking up a password entry:

1. If you can verify them from the local shadow file, they’re allowed. No need to look any further.
2. If you can verify them from the Active Directory via Kerberos, they’re allowed.

krb5.conf is set the same way as I noted in the previous post. Mostly, it just needs to have kdc entries for each Active Directory server, and admin_server set to the Active Directory master, and default_realm set to the Active Directory domain name.

users.conf is generated on the puppetmaster by concatenating one file per user from a users.conf.d directory. Each file contains one line: the user’s username. Eventually, we may modify things to make each file a group of users (one faculty’s advisees, all the members of an advanced class, etc.), but for now, echo bob > /etc/puppet/files/apps/pam_listfile/users.conf.d/bob works ok for us. The puppetmaster’s manifest now includes:

    concatenated_file {
        "/etc/puppet/files/apps/pam_listfile/users.conf.default":
            dir => "/etc/puppet/files/apps/pam_listfile/users.conf.d"
    }

using David Schmitt’s /etc/puppet/manifests/definitions/concatenated_file.pp — at one point, I had the client systems running concatenated_file themselves, but the latency and md5 load for 270+ one-line files on the puppetmaster was ridiculous.

node ch208r {
    include cae-host

    # ch208r (new file server) needs:

    # Disk tools (XFS support, LVM support, parted)
    package {
        [ "xfsdump", "lvm2", "parted" ]: ensure => installed;
    }
    # Backup tools (see amandaconfig module for details)
    include amanda
    amanda::amandaconfig {
        "holding":
            labelstr   => "HOLDING",
            dumpcycle  => 1,
            tapecycle  => 2,
            runtapes   => 1;
        "daily":
            labelstr   => "DAILY",
            dumpcycle  => 35,
            tapecycle  => 10,
            runtapes   => 1;
        "archival":
            labelstr   => "ARCH[2-9]",
            dumpcycle  => 0,
            tapecycle  => 1000,
            runtapes   => 4;
    }

    cron {
        "amdump-daily":
            command => "/usr/local/sbin/mkdisklist > /etc/amanda/daily/disklist; /usr/sbin/amdump daily; /usr/sbin/amdump holding ; du -shc /opt/amanda/work | tail -1 | mutt -s 'Amanda Disk Usage' 1234567890@vtext.com",
            ensure  => present,
            user    => backup,
            hour    => 0,
            minute  => 15;
    }

    file {
        "/etc/amanda/holding/disklist":
            source => "puppet:///files/apps/amanda-server/disklist.holding";
    }

    # Environmental monitoring
    package {
        [ "ipmitool", "openipmi" ]: ensure => installed;
    }
    file {
        "/etc/modules":
            source => "puppet:///files/apps/ipmitool/modules";
    }

    mount { "/home":
        atboot  => true,
        device  => "/dev/md1000/home",
        ensure  => mounted,
        fstype  => "xfs",
        options => "defaults",
        dump    => "0",
        pass    => "1",
        require => [ Package["xfsdump"], Package["lvm2"] ];
    }

    include active-directory-fileserver
    # - NFS export of /home/CAE
    package {
        [ "nfs-kernel-server" ]: ensure => installed;
    }

    # - scponly for remote file access (no shell access allowed)
    package {
        [ "scponly" ]:
            ensure => installed;
    }

}

Since the active-directory-fileserver class ties strongly into an active-directory-member class, and that class warrants an entire followup artcle for my authentication notes, I’ll hold off on that part for now which can be found here. But the Amanda configuration code, that’s worth talking about now.

If you’ve never used Amanda, it’s arguably the best multi-system, multi-platform Free Software backup solution out there. We’ve kept it in constant service since late 2002, making this the third major fileserver we’ve put it on. It’s a little fiddly to get installed properly:

Create the config directory (eg. /usr/local/etc/amanda/confname) and copy the example/ files into that directory. Edit these files to be correct for your site, consulting the amanda(8) man page if necessary. You can also send mail to mailto://amanda-users@amanda.org if you are having trouble deciding how to set things up. You will also need to create the directory for the log and database files for the configuration to use (eg /usr/local/var/amanda/confname), and the work directory on the holding disk. These directories need to agree with the parameters in amanda.conf. Don’t forget to make all these directories writable by the dump user!

but worth it. Since I have no less than three different Amanda configurations to set up (one for rotating daily backups, one for monthly archival backups, and one to back up the Amanda holding disk as disaster insurance), I figured it was time to make Puppet create my configuration files in addition to copying them to the file server. It wasn’t nearly as difficult as I’d expected, and hopefully this will provide an example that’s easy enough to follow, complicated enough to be useful, and uses enough of Puppet’s module capabilities to keep others from hitting the same walls I did. I don’t claim any of them are perfect, but they’re currently working and relatively clean.

First, I made a modules directory in the puppet root (/etc/puppet, in my case). Then I added the following entry into puppet’s fileserver.conf:

[modules]
  # path isn't actually used here
  allow A.B.C.0/24

where A.B.C.0/24 corresponds to the class C subnet that can access the modules. In the modules folder, I created an amandaconfig folder, which currently has the following contents:

/etc/puppet/modules# ls -lR amandaconfig
amandaconfig:
total 12
drwxr-xr-x 3 root root 4096 2007-08-02 15:36 files
drwxr-xr-x 3 root root 4096 2007-08-02 15:36 manifests
drwxr-xr-x 3 root root 4096 2007-08-02 15:36 templates

amandaconfig/files:
total 12
-rw-r----- 1 root root 1827 2007-08-02 09:41 disklist.systems
-rwxr-xr-x 1 root root  416 2007-07-31 14:31 mkdisklist
-rwxr-xr-x 1 root root  390 2007-07-31 14:35 mkdisklist.archival

amandaconfig/manifests:
total 4
-rw-r--r-- 1 root root 2554 2007-08-01 17:21 init.pp

amandaconfig/templates:
total 8
-rw-r--r-- 1 root root 1418 2007-08-02 09:52 amanda.conf.erb
-rw-r--r-- 1 root root   13 2007-07-25 21:05 changer.conf.erb

The files directory should contain auxiliary files for the module that don’t need to be edited on a per-configuration basis. Mine contains a couple of shell scripts to create amanda disklist files by iterating over the file server’s home directories, and a list of system partitions on other servers and workstations I need to back up daily. The manifests directory contains the Puppet code to create a series of working Amanda configurations, and the templates directory contains files that need to have some search/replace operations done on a per-configuration basis.

I also added an import "amandaconfig" to the top of my site.pp to make sure the new Amanda module was usable.

The contents of manifests/init.pp:

# Create a new Amanda configuration set.
class amanda {
    package {
        [ "amanda-server", "amanda-client", "mtx", "dump" ]:
            ensure => latest;
    }

    file {
        "/opt/amanda":
            ensure => directory,
            owner  => root,
            group  => root,
            mode   => 0755;

        "/opt/amanda/work":
            ensure => directory,
            owner  => backup,
            group  => backup,
            mode   => 0700;

        "/usr/local/sbin/mkdisklist":
            source => "puppet:///amandaconfig/mkdisklist",
            owner  => root,
            group  => backup,
            mode   => 0750;

        "/usr/local/sbin/mkdisklist.archival":
            source => "puppet:///amandaconfig/mkdisklist.archival",
            owner  => root,
            group  => backup,
            mode   => 0750;

        "/etc/amanda/disklist.systems":
            source => "puppet:///amandaconfig/disklist.systems",
            owner  => root,
            group  => backup,
            mode   => 0640;
    }

    define amandaconfig (
        $confdir = "/etc/amanda",
        $logdir = "/var/log/amanda",
        $libdir = "/var/lib/amanda",
        $user = "backup",
        $group = "backup",
        $dumpcycle = 1,
        $tapecycle = 2,
        $runtapes = 1,
        $labelstr = "LABEL"
    ) {

        file {
            "$confdir/$name":
                ensure => directory,
                owner  => $user,
                group  => $group,
                mode   => 0770;

            "$confdir/$name/amanda.conf":
                content => template("amandaconfig/amanda.conf.erb"),
                ensure  => present,
                owner   => $user,
                group   => $group,
                mode    => 0755;

            "$confdir/$name/changer.conf":
                content => template("amandaconfig/changer.conf.erb"),
                ensure  => present,
                owner   => $user,
                group   => $group,
                mode    => 0755;

            "$confdir/$name/tapelist":
                ensure => present,
                owner  => $user,
                group  => $group,
                mode   => 0600;

            "$libdir/$name":
                ensure => directory,
                owner  => $user,
                group  => $group,
                mode   => 0770;

            "$logdir/$name":
                ensure => directory,
                owner  => $user,
                group  => $group,
                mode   => 0770;

        }
    }
}

Everything in the amanda class outside the amandaconfig definition gets evaluated when the node manifest gets to its include amanda line. So the necessary holding disks are created, packages are installed, etc. After that, I can answer the questions for what makes each Amanda configuration unique in the amanda::amandaconfig stanza. For example:

• the “holding” configuration that backs up the holding disk each day alternates between two tapes labeled HOLDING001 and HOLDING002.
• the “daily” configuration backs up the file server’s system disks, several other systems’ system disks, and all the users’ and groups’ home directories each day into the holding disk, which we flush off to tape every few days to a series of tapes labeled DAILY001, DAILY002, …
• the “archival” configuration backs up the users’ and groups’ home directories into the holding disk, which is immediately backed up to tapes labeled ARCH201, ARCH202, … ARCH999.

As for the templates, they’re ERB-formatted configuration files. amanda.conf is the only interesting one, since changer.conf.erb is actually a static file and should be moved into the files folder with mkdisklist and friends:

org "TTU CAE Network"
mailto "root"
dumpuser "<%= user %>"

inparallel 4
netusage 600
reserve 0

dumpcycle <%= dumpcycle %> days
tapecycle <%= tapecycle %> tapes

runtapes <%= runtapes %>
tpchanger "/usr/lib/amanda/chg-zd-mtx"
changerdev "/dev/sg3"
changerfile "changer"
tapedev "/dev/nst0"
tapetype PV-124T_LTO3
labelstr "^<%= labelstr %>[0-9][0-9]*$"

infofile "<%= libdir %>/<%= name %>/curinfo"
logdir "<%= logdir %>/<%= name %>"

indexdir "<%= libdir %>/<%= name %>/index"

holdingdisk opt {
    comment "/opt/amanda/work"
    directory "/opt/amanda/work"
    use -1 gb
}

define tapetype PV-124T_LTO3 {
    comment "Dell PowerVault 124T - LTO3"
    length 402432 mbytes
    filemark 0 kbytes
    speed 71064 kps
}

define dumptype holding-full {
    comment "Full dump of the holding disk always"
    program "GNUTAR"
    compress none
    holdingdisk never
    skip-incr yes
    priority high
    dumpcycle 0
}

define dumptype comp-root {
    comment "Entire partitions backed up with dump"
    compress fast
    index yes
    priority low
}

define dumptype homedir-full {
    comment "User home directories backed up with tar"
    program "GNUTAR"
    compress fast
    index yes
    priority medium
    holdingdisk auto
}

define dumptype homedir-full-archival {
    comment "User home directories backed up with tar"
    program "GNUTAR"
    compress fast
    index yes
    priority medium
    record no
}

You can read through the Amanda documentation for what all those settings actually mean. The relevant part is that Puppet will parse through the template looking for variables to replace, and will replace them with either the defaults used in the amandaconfig definition, or with ones passed as arguments to the amandaconfig call.

Remember that no one cares if you can back up — only if you can restore.

d-i partman-auto/disk string /dev/discs/disc0/disc
d-i partman-auto/method string lvm
d-i partman-auto/purge_lvm_from_device boolean true
d-i partman-lvm/confirm boolean true
d-i partman-auto/init_automatically_partition \\
	select Guided - use entire disk and set up LVM

d-i partman-auto/expert_recipe string                         \\
      boot-root ::                                            \\
              40 300 300 ext3                                 \\
                      $primary{ } $bootable{ }                \\
                      method{ format } format{ }              \\
                      use_filesystem{ } filesystem{ ext3 }    \\
                      mountpoint{ /boot }                     \\
              .                                               \\
              500 10000 1000000000 ext3                       \\
                      method{ format } format{ } $lvmok{ }    \\
                      use_filesystem{ } filesystem{ ext3 }    \\
                      mountpoint{ / }                         \\
              .                                               \\
              600000 600000 600000 ext3                       \\
                      method{ format } format{ } $lvmok{ }    \\
                      use_filesystem{ } filesystem{ ext3 }    \\
                      mountpoint{ /opt/amanda }               \\
              .                                               \\
              500 9000 5000 ext3                              \\
                      method{ format } format{ } $lvmok{ }    \\
                      use_filesystem{ } filesystem{ ext3 }    \\
                      mountpoint{ /var }                      \\
              .                                               \\
              500 9000 5000 ext3                              \\
                      method{ format } format{ } $lvmok{ }    \\
                      use_filesystem{ } filesystem{ ext3 }    \\
                      mountpoint{ /tmp }                      \\
              .                                               \\
              64 512 200% linux-swap $lvmok{ }                \\
                      method{ swap } format{ }                \\
              .

d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition \\
	select Finish partitioning and write changes to disk
d-i partman/confirm boolean true

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/ch208r-root
                      230G  707M  218G   1% /
/dev/sda1             274M   17M  243M   7% /boot
/dev/mapper/ch208r-opt+amanda
                      564G  422M  535G   1% /opt/amanda
/dev/mapper/ch208r-tmp
                      4.7G  138M  4.4G   4% /tmp
/dev/mapper/ch208r-var
                      4.7G  264M  4.2G   6% /var