class cluster-host inherits public-host {
    # ...
    file { "/etc/hosts.equiv":
        source  => "puppet:///files/apps/rsh-server/hosts.equiv.$hostgroup",
        owner   => root,
        group   => root,
        mode    => 644,
        require => Package[rsh-server],
    }
}

Along the lines of Puppet Best Practice 2.0, I’m trying to compartmentalize all my building blocks into separate modules, even the really simple ones that just install a package or two. So here’s the contents of a trivial hello module for Debian:

# /etc/puppet/modules/hello/manifests/init.pp
class hello {
  package { "hello":
    ensure => latest;
  }
}

Nothing complicated here: install this package via the default package provider. Add an include hello to my node’s manifest, and the hello package gets installed.

Now for the “undo” class:

# /etc/puppet/modules/hello/manifests/disabled.pp
class hello::disabled inherits hello {
  Package["hello"] {
    ensure => purged
  }
}

And this is neat. Properly written, the disabling class can undo all the changes made by the parent class. Modify my node’s manifest from include hello to include hello::disabled and I can switch that class on and off like, well, a switch.

It all started so simply: I was going to set up a little Xen instance to be my next cluster submit host, and needed a spare address for it:

1. I started setting up an instance for ch208i.cae.tntech.edu, since it was no longer on the Xen host like it was several months ago. Crap, the reason it’s no longer on the Xen instance is because I moved it to its own dedicated hardware — it’s still my main ftp/mirror server. Ctrl-C that one.
2. Hmm, what’s available from old Xen instances? mail2.cae.tntech.edu.cfg from when I was testing out a new mail server setup last fall — doesn’t ping, doesn’t show up in xm list, no problem.

   xen-create-image --hostname=mail2.cae.tntech.edu --ip=149.149.254.23 \
       --gateway=149.149.254.4 --netmask=255.255.255.0 --size=10Gb --memory=256Mb \
       --swap=1Gb --debootstrap --force

   A few minutes later, my instance is debootstrapped and ready to go.

3. Oh, crap. Why am I getting an error on xm create that says my LVM is already in use on a domU somewhere?
4. Further crap. Looking in /etc/xen/mail.cae.tntech.edu.cfg for the production mail server, it apparently uses the old mail2.cae.tntech.edu LVMs. Wonderful. ssh mail? It works since sshd was already memory-resident, but /root/.profile doesn’t exist. And neither does much of anything else.
5. Great. I’ve just killed the mail server. Off to the Amanda server to do a quick restore of its data. What? I never put mail.cae.tntech.edu into the backup list? Not normally the end of the world, since the mail stores are held accessed over NFS from the main file server, but what about my dovecot and postfix configurations?
6. Oh, well. Time to see how good my puppet manifests are for the mail server.

Not too bad, as it turns out. Total downtime was only a couple hours, including having to redo the postfix and dovecot configurations (which were then copied off to the puppetmaster). I still have a few more things to fix, but mail delivery is up, and imap is running. TLS support for my sending mail from home isn’t up yet, but it’ll be fixed shortly.

I still need to fix that submit host, though. Next time, I think I’ll use an IP address reserved for my office.

Update: after getting a partial TLS/SASL setup going late Wednesday night, I went to sleep without realizing I’d killed mail delivery again. Finally got it straightened out Thursday morning.

• Software was more important than hardware.
• Some software ran only under Windows, some had no Windows version at all.
• Of the non-Windows software people cared about, there was always a version for Solaris. There was often a version for most other Unixes, but regardless of the company, they always had a Solaris version.
• Sun’s matching grant program for education was awesome.

And to be fair, for some operations, our Sun Blade 1000 workstations blow the doors off of our Dell Precision Workstations with 3x the clock cycles. We’ve had very little hardware trouble from the Suns, and the aforementioned matching grant program and judicious use of third-party upgrade vendors let us buy two decked out Ultra 80 workstations on a budget that was originally allocated for one decked out workstation and one considerably lower-specced one.

But there’s little to no excuse for the following:

• patchadd rewrites every byte of /var/sadm/install/contents every time you do a file operation. During jumpstarts, I manage to put that file in a tmpfs for faster access, but before that, I couldn’t do a single Solaris-only Jumpstart install in less than half a day.
• Solaris 10 includes Samba. Solaris 10’s Samba includes winbind, which is what I use on my Debian systems to convert Active Directory accounts to Unix ones. But the Solaris 10 winbind doesn’t include the idmap_rid backend for consistently converting an Active Directory RID into a Unix UID, which confuses NFS mightily. I thought blastwave’s or sunfreeware’s Samba packages might be better, but they weren’t. I found these instructions for configuring winbind and idmap_rid for Solaris, but they’re squirreled off in a manual for Sun Cluster Data Services. What reason might they have for not compiling in idmap_rid by default? Am I the only person who uses Active Directory to generate UIDs for a central NFS and Samba server?
• Today, during an attempt to install and test Matlab 7.6, I found that X11 forwarding is broken on recently-patched Solaris systems like mine. A similar bug came up in 2005 and sat unfixed for a few months. The usual fix of telling sshd to only listen on IPv4 interfaces in sshd_config isn’t enough, though. You actually have to add the -4 argument to the sshd service file.

I hate throwing away tens of thousands of dollars of perfectly functional hardware. I could install Debian’s sparc port on them, but why? I’d lose access to Ansys, Matlab, and all the other packages that are the reason I have these systems in the first place. And letting them languish like they did for years before I got into the managed infrastructure business seems a waste. Solaris 10, puppet, and the newer firmware that allows PXE booting is such a vast improvement over earlier versions for what I need to do, but there’s still some distance to go before it’s up to Debian standards.

If we went through the same evaluation process in 2002, I’d probably not have any Solaris sytems at all. Matlab, Maple, Ansys, Abaqus, etc. were all coming out with (or had already come out with) Linux versions. We’d have spent a lot less on hardware, and some jobs just love the extra clock cycles available on an Intel CPU.

The packages

I already use Blastwave and pkg-get to install third-party free software applications, so I figured it would be easiest to use the same tools on my packaging. So for a first example, I installed Maple 11.00 manually into /opt/maple/11 on a Solaris 10 system. Then I made a temporary working folder and build folder, made an opt folder there, and moved the maple folder from the regular opt to my build folder’s opt. I also made a usr/local/bin in my build folder, and made relative symlinks from the main Maple executables to their assumed homes in usr/local/bin. The abridged results from the temporary working folder looked like this:

# pwd
/root/work/maple11-11.00
# ls -l
total 6
drwxr-xr-x   4 root     root         512 May 22 09:27 build
-rw-r--r--   1 root     root          41 May 21 17:59 copyright
-rw-r--r--   1 root     root           0 May 22 09:36 depend
-rw-r--r--   1 root     root         143 May 22 09:35 pkginfo
# cat copyright
Copyright MapleSoft, All Rights Reserved
# cat pkginfo
PKG=MAPLmaple11
NAME=maple11
VERSION=11.00
ARCH=sun4u
DESC=Interactive computer algebra system
BASEDIR=/
CATEGORY=application
VENDOR=MapleSoft
EMAIL=renfro@tntech.edu
# ls -al build/opt/maple/11
total 504
drwxrwxr-x  17 root     other        512 May 22 08:43 .
drwxrwxr-x   3 root     other        512 May 22 08:42 ..
drwxr-xr-x   2 root     other        512 May 22 08:42 afm
drwxr-xr-x   2 root     other        512 May 22 08:42 bin
drwxr-xr-x   3 root     other       2048 May 22 08:42 bin.SUN_SPARC_SOLARIS
drwxr-xr-x   9 root     other        512 May 22 08:42 data
drwxr-xr-x   2 root     other        512 May 22 08:42 etc
drwxr-xr-x   2 root     other       3072 May 22 08:42 examples
drwxr-xr-x   3 root     other        512 May 22 08:42 extern
-rw-r--r--   1 root     other     153861 May 21 14:12 Install.html
drwxr-xr-x   2 root     other       1536 May 22 08:42 java
drwxrwxr-x   7 root     other        512 May 22 08:42 jre.SUN_SPARC_SOLARIS
drwxr-xr-x   4 root     other       1536 May 22 08:43 lib
drwxr-xr-x   2 root     other        512 May 22 08:43 license
drwxr-xr-x   3 root     other        512 May 22 08:43 man
-rw-rw-r--   1 root     other      60064 May 21 14:15 Maple_11_InstallLog.log
-rw-r--r--   1 root     other      10285 May 21 14:12 readme.txt
drwxr-xr-x   6 root     other        512 May 22 08:43 samples
drwxr-xr-x   2 root     other        512 May 22 08:43 test
drwxr-xr-x   2 root     other        512 May 22 08:42 X11_defaults
# ls -al build/usr/local/bin
total 10
drwxr-xr-x   2 root     root         512 May 22 08:56 .
drwxr-xr-x   3 root     root         512 May 22 08:47 ..
lrwxrwxrwx   1 root     root          31 May 22 08:55 maple11 -> ../../../opt/maple/11/bin/maple
lrwxrwxrwx   1 root     root          30 May 22 08:56 mint11 -> ../../../opt/maple/11/bin/mint
lrwxrwxrwx   1 root     root          32 May 22 08:55 xmaple11 -> ../../../opt/maple/11/bin/xmaple

Now, given that folder structure, I could adapt Blastwave’s package creation instructions to create some workable Solaris packages:

# (echo "i pkginfo"; echo "i copyright" ; echo "i depend" ; cd build ; find . | pkgproto ) > prototype
# pkgmk -b / -a `uname -p`
# filename=maple11-11.00-SunOS`uname -r`-`uname -p`.pkg
# pkgtrans -s /var/spool/pkg /root/$filename MAPLmaple11
# cd /root
# gzip $filename

Once mkpkg is all done with its work, I have a valid maple11-11.00-SunOS5.10-sparc.pkg.gz Solaris package in my /root folder. After testing it with regular pkgadd, I’m ready to put it into a private pkg-get repository.

The pkg-get repository

Compared to a Debian repository, a pkg-get repository is pretty simple. From the top-level folder in the repository on the ftp server:

# find sparc -print
sparc
sparc/5.10
sparc/5.10/maple11-11.00-SunOS5.10-sparc.pkg.gz
sparc/5.10/descriptions
sparc/5.10/catalog

A pkg-get repository’s top-level folders are named by processor type, i.e., the results of uname -p. Each processor type folder contains folders for each OS release level (from uname -r). Each release level folder contains packages for that CPU and OS, plus a descriptions and a catalog file.

The catalog file is created with Phil Brown’s makecontents script. It could potentially handle creating the descriptions, file, too, but I guess he never needed them. But the pkg-get script I got from blastwave.org definitely wants a descriptions file, so I’ll need to create that, too.

The way I’m creating the descriptions file is with the following script (on a Debian ftp server, so there may be some GNU-isms or bash-isms in the following code):

#!/bin/sh
PKG_GET_DIR=/wherever/has/the/sparc/and/i386/folders
cd ${PKG_GET_DIR}
for name in sparc i386; do
    if [ -d $name ]; then
        cd $name
        for version in 5*; do
            if [ -d $version ]; then
                cd $version
                for package in *.gz; do
                    name=`grep $package catalog | awk '{print $1}'`
                    echo -ne "$name - "
                    zcat $package | head | strings | grep DESC= | cut -d= -f2-
                done > descriptions
                cd ..
            fi
        done
        cd ..
    fi
done

which leaves me with a catalog file containing (so far, since I’ve only made one package):

maple11 11.00 MAPLmaple11 maple11-11.00-SunOS5.10-sparc.pkg.gz

and a descriptions file containing:

maple11 - Interactive computer algebra system

And now I can install them on a second host that’s never seen Maple installed before with:

pkg-get -s ftp://host/path/to/repository/ -U ; pkg-get -s ftp://host/path/to/repository/ install maple11

and afterwards get:

# which maple11
/usr/local/bin/maple11
# maple11
    |\\^/|     Maple 11 (SUN SPARC SOLARIS)
._|\\|   |/|_. Copyright (c) Maplesoft, a division of Waterloo Maple Inc. 2007
 \\  MAPLE  /  All rights reserved. Maple is a trademark of
 <____ ____>  Waterloo Maple Inc.
      |       Type ? for help.
> quit
bytes used=412112, alloc=393144, time=0.07

• Slides for presentation
• Handouts for presentation

Update: So yesterday, I get an email regarding my presentation (well, the slides, at least). No reason to clutter up the main page with it though, so if you’re not happy with the slides and want to express your displeasure, read the rest after the jump and see if I’ve addressed your concerns already.

Hi Mike,

I’ve visited your site before and found your Debian preseeding info to be useful.

That said, I just went through your presentation slides and must say I’m very disappointed. It contains numerous examples of what gives sysadmins a bad name. Egotistical, “I’m right, you’re stupid”, “I did this because I’m way too busy doing more important things than you”, etc. comments abound.

On several occasions your first bullet-point was “just do it”, or “you need this”. Hey Mike, people don’t come to conferences and presentations to listen to a smart-ass.

You mentioned that the people in #puppet gave you useful feedback. Next time you give a presentation, also get feedback from non-geeks. They’ll help you filter out the cruft that makes you look like a spoiled 5-year old talking about his new widget set at show-and-tell.

I hope for the sake of the attendees that your verbal presentation was better than your slideshow.

To respond to the comments more or less in order:

• Sorry you didn’t like my slides. At first, I thought you were an irritated audience member who waited a couple of weeks before emailing me. But since you’re apparently only judging this based off the slides, that’s different.
• “I’m right, you’re stupid” is in the eye of the reader. Though you can’t tell without the soundtrack, it tended to work out more like “I used to do things one way, which probably is the most common way everyone else does it. It didn’t scale for the following reasons, and here’s what I’m doing instead.”
• “I did it this way because I’m way too busy doing more important things than you” is a bit of an exaggerated inference. Am I busy? Sure. Am I busy doing things that most sysadmins don’t have to deal with? As far as I can tell, yes; most of the sysadmins in my immediate vicinity (and from past experience over the last 15-20 years) don’t have major duties outside systems administration, just like most of the engineers don’t have major duties outside their specialty or lab. These non-sysadmin tasks aren’t necessarily more important per se, but they’re important in my particular job description. The hours these other tasks take up in my week force me to find more efficient methods to do the systems administration tasks; others will possibly hit the same walls I have at different times — maybe when they have to keep track of 300 servers in 10 different roles, where all servers in a particular role have to be interchangeable. Maybe when they get a 1000 node cluster where a particular application has to be installed identically on every node, and on every node to be purchased in the future. My belief is that as time goes on, we’re all going to be managing more systems, not fewer, and that methods we use for managing a few systems relatively well don’t scale to larger groups of computers.
• “Just do it” or “You need this” shows up in three places: using version control, using NTP for time synchronization, and using SMTP for email. I stand by each of those points, being entirely convinced by the verbiage at infrastructures.org that was my primary source material. I cannot fathom why someone would use something other than SMTP for sending email, why they wouldn’t want version control of some form as the code that automates their systems administration tasks grows more complicated, or why they’d use a different protocol to synchronize their system clocks. To me, each of those is as self-evident as “your SAN should have redundant power supplies” and “racks are a good way to house a bunch of servers in a small space”. You may have counterexamples, but since you didn’t provide any, I’m left in the dark.
• The folks on #puppet did give me some useful feedback. As for other feedback, I did ask a coworker to look at the slides, and he saw no problems with them. However, he’s a full-time Windows systems administrator, so his opinion may be suspect. As for non-geeks, they’re really not the intended audience, were generally absent from the conference, and aren’t too likely to be interested in systems administration techniques.
• Cruft in the verbal presentation? Guilty, but some might call it illustrative anecdotes. Personally, I’ve always tried to work in Sidney Harris‘ “I think you should be more explicit here in step two” joke into at least one over-equationed lecture per semester. The students seem to enjoy it:
  
  Cruft in the slides? Matter of opinion, I guess. I did run out of time, but I honestly hadn’t done enough practice runs to see how long it would actually take.
• As for the spoiled five-year-old showing off his new widget set at show-and-tell, I have trouble understanding the issue. Lots of the talks at these conferences are basically a show-and-tell: other talks included “Software Deployment Using Ghost”, “Virtualizing Business Continuity — Getting Your Systems Back Online,” “DBA Task Automation II: Extending the Basics, Best Practices, Processes and Icing,” etc. When I submit an abstract saying that I’m going to give a talk about what goes into a “managed infrastructure,” its benefits over regular administration methods, and talk about a particular tool we use to accomplish some of these tasks, exactly what do I change in material? What do I change in delivery (that you didn’t see)? You’ve never told coworkers “holy crap, this Linux thing is awesome! It’s like Unix, but free and runs on regular PCs!” or similar? Nothing about using PHP to format stuff out of a database for some dynamic web pages? Nothing about a CMS or blogging platform that lets you do all the things a CMS or blog is supposed to do? Nothing about rrdtool, Nagios, cacti, apt-get, Perl/Python/Ruby or some other tool that you didn’t write, but by gosh it’s going to make all of your lives easier?

class jre5 {
  package { "sun-java5-jre":
    ensure => latest;
  }
}

It doesn’t work because Sun wants you to agree to its license before installing the JRE. There’s a couple of ways around this. First, the old-school method:

ssh host "yes | apt-get -y install sun-java5-jre"

where ‘yes’ is a standard Unix program that just prints out “yes” over and over until the program on the other side of the pipe terminates. But “ssh host foo” is not the way of the managed infrastructure.

The second method, much more friendly to centralized management, is to first install debconf-utils on a candidate system, and then install sun-java5-jre on the same system. Once that’s done, you can query the debconf database to see how it stored your answers to the Sun license agreement:

ch226-12:~# debconf-get-selections | grep sun-
sun-java5-bin   shared/accepted-sun-dlj-v1-1    boolean true
sun-java5-jre   shared/accepted-sun-dlj-v1-1    boolean true
sun-java5-jre   sun-java5-jre/jcepolicy note
sun-java5-jre   sun-java5-jre/stopthread        boolean true
sun-java5-bin   shared/error-sun-dlj-v1-1       error
sun-java5-jre   shared/error-sun-dlj-v1-1       error
sun-java5-bin   shared/present-sun-dlj-v1-1     note
sun-java5-jre   shared/present-sun-dlj-v1-1     note

Save those results (debconf seeds) into a file on the gold server. Then we can modify our jre5 class as follows:

class jre5 {
  package { "sun-java5-jre":
    require      => File["/var/cache/debconf/jre5.seeds"],
    responsefile => "/var/cache/debconf/jre5.seeds",
    ensure       => latest;
  }

  file { "/var/cache/debconf/jre5.seeds":
    source => "puppet:///jre5/jre5.seeds",
    ensure => present;
  }
}

Now our class will download the preseeded answers for the Java license, download and install the JRE, and then use the preseeded answers to skip past the license agreement. I had never messed with debconf seeding previously, since I had either just imaged my systems, or provided config files that would be used when I restarted any daemons or programs that depended on those files. Now debconf-utils is part of my standard system class definition.

Note that this method doesn’t work with the default puppet provided in Debian Etch (version 0.20) — the responsefile parameter for Debian packages was only added in puppet 0.22.

1. There was one more file to manage outside my regular puppet manifests, and I’d have to remember to keep them both up to date and in sync.
2. There wasn’t an easy way of ensuring that other versions of a particular package got unstowed before deploying out the desired version.
3. The entire stow tree would be copied out to every system, regardless of whether OpenMPI was a good fit for the web server.

So, here’s my new method:

1. Keep my same metastow module loaded on the rsync server. The metastow module contains one top-level directory per puppet architecture (i686, x86_64, etc.). Each of those architecture folders is a stow tree containing every stowed package for that architecture.
2. Add a stowedpackage definition to my puppet manifests as follows:

   define stowedpackage ( $basepackage, $version,
       $rsyncserver='gold.cae.tntech.edu',
       $rsyncmodule='metastow',
       $stowdestdir='/usr/local/stow' ) {
       file { "stow-initiator_${basepackage}-${version}":
           source => "puppet:///files/stow-initiator_${basepackage}-${version}",
           path   => "/etc/puppet/stow-initiator_${basepackage}-${version}",
       }
       exec { download:
           command     => "/usr/bin/rsync -a --delete ${rsyncserver}::${rsyncmodule}/${hardwaremodel}/${basepackage}-${version} ${stowdestdir}",
           refreshonly => true,
           subscribe   => File["stow-initiator_${basepackage}-${version}"],
           alias       => "download_${basepackage}-${version}"
       }
       exec { unstow-others:
           command     => "cd ${stowdestdir} && stow --delete ${basepackage}-*",
           refreshonly => true,
           subscribe   => Exec["download_${basepackage}-${version}"],
           alias       => "unstow-others_${basepackage}-${version}"
       }
       exec { stow:
           command     => "cd ${stowdestdir} ; stow ${basepackage}-${version}",
           refreshonly => true,
           subscribe   => Exec["unstow-others_${basepackage}-${version}"]
       }
   }
   

3. Use the stowedpackage definition in other parts of my manifests:

   # Create OpenMPI installation and configuration.
   class openmpi {
   
       stowedpackage {
           "openmpi-1.0.1":
               basepackage=>"openmpi",
               version=>"1.0.1";
       }
   
   }
   

4. Add a trigger file to the puppetmaster’s /etc/puppet/files folder:

   /etc/puppet/files# date > stow-initiator_openmpi-1.0.1
   

]]> http://blogs.cae.tntech.edu/mwr/2008/02/01/the-autostow-is-dead-long-live-stowedpackage/feed/ 1 http://blogs.cae.tntech.edu/mwr/2007/10/31/client-configuration-management/ http://blogs.cae.tntech.edu/mwr/2007/10/31/client-configuration-management/#comments Thu, 01 Nov 2007 01:17:21 +0000 Mike Renfro http://blogs.cae.tntech.edu/mwr/2007/10/31/client-configuration-management/ Back at the infrastructures.org mothership, client configuration management is described as everything that makes a host unique and/or part of a particular group or domain. And for Unix-like systems, everything pretty much comes down to configuration files, services being enabled/disabled, and cron jobs.

Hmm.

• Configuration files
• Services
• Cron jobs

Looks like Puppet pretty much handles all of that. As long as I can describe aspects of my systems with puppet classes and modules, I’ve got reusable, consistent configurations on any servers I care to manage.

]]> http://blogs.cae.tntech.edu/mwr/2007/10/31/client-configuration-management/feed/ 0 http://blogs.cae.tntech.edu/mwr/2007/10/31/client-file-access/ http://blogs.cae.tntech.edu/mwr/2007/10/31/client-file-access/#comments Thu, 01 Nov 2007 01:00:16 +0000 Mike Renfro http://blogs.cae.tntech.edu/mwr/2007/10/31/client-file-access/ The infrastructures.org folks list two primary goals of what they call “client file access“: first, consistent access to users’ home directories, and second, consistent access to end-user applications. Some of the things they warn against, such as automounters and the /net directory, we never thought of using to begin with. Their need to consider systems with limited disk space and a need to mount a software share via NFS is less of an issue with us, too: disks large enough to hold our regular software load are relatively cheap, so there shouldn’t be much of a problem there. And since we’re supposed to be deploying applications consistently via Puppet, there shouldn’t be any inconsistency in where an application is installed. As far as consistent access to the home directories, we accomplish this with a two-pronged solution in Puppet — one on the file server, and one on the clients:

On the Debian file server, we include the following class:

class nfs-server {
  package { nfs-kernel-server: ensure => installed }

  file { exports:
    path => $operatingsystem ? {
      default => "/etc/exports"
    },
    owner => root, group => root, mode => 644,
    source => "puppet://gold.cae.tntech.edu/files/apps/nfs-server/exports",
    require => Package[nfs-kernel-server]
  }

  service { nfs-kernel-server:
    ensure => running,
    enable => true,
    hasstatus => true,
    subscribe => [Package[nfs-kernel-server], File[exports]]
  }
  service { portmap:
    ensure => running,
    enable => true,
  }
}

We also include the following in its node definition:

    file {
        "/etc/exports.d":
        source  => "puppet:///files/apps/nfs-kernel-server/exports.d",
        ensure  => directory,
        ignore  => ".svn",
        purge   => true,
        recurse => inf;
    }

    concatenated_file {
        "/etc/exports":
            dir => "/etc/exports.d";
    }

Each file dropped in /etc/exports.d is a fragment of /etc/exports. Right now, we’re using one fragment per exported filesystem. As an example, here’s /etc/exports.d/_opt_solaris_stow:

/opt/solaris/stow \\
        149.149.254.12(rw,sync,no_root_squash) \\
        149.149.254.13(rw,sync,no_root_squash) \\
        149.149.254.14(rw,sync,no_root_squash) \\
        149.149.254.15(rw,sync,no_root_squash) \\
        149.149.254.220(rw,sync,no_root_squash)

David Schmitt’s concatenated_file definition combines all the export fragments into one /etc/exports file:

define concatenated_file ($dir, $mode = 0644, $owner = root, $group = root)
{
    file {
        $name:
            ensure   => present,
            checksum => md5,
            mode     => $mode,
            owner    => $owner,
            group => $group;
    }
    exec { "find ${dir} -maxdepth 1 -type f ! -name '*puppettmp' -print0 | sort -z | xargs -0 cat > ${name}":
        refreshonly => true,
        subscribe   => File[$dir],
        alias       => "concat_${name}",
    }
}
define concatenated_file_part ($dir, $content = '', $ensure = present,
                               $mode = 0644, $owner = root, $group = root) {
    file { "${dir}/${name}":
        ensure  => $ensure,
        content => $content,
        mode    => $mode,
        owner   => $owner,
        group   => $group,
        alias   => "cf_part_${name}",
    }
}

On the client side, we just make sure we can mount the NFS share. For example:

    file {
        "/home/CAE":
            ensure => directory,
            owner  => root,
            group  => root,
            mode   => 755,
    }
    mount { "/home/CAE":
        require => File["/home/CAE"],
        atboot  => true,
        fstype  => "nfs",
        device  => "files.cae.tntech.edu:/home/CAE",
        ensure  => mounted,
        options => "defaults",
        dump    => "0",
        pass    => "0",
    }
    package { "nfs-common": ensure => installed }

]]> http://blogs.cae.tntech.edu/mwr/2007/10/31/client-file-access/feed/ 0