Automated migration of systems to a new puppet master server

I recently built a new puppet master server and wanted to migrate all of my systems from the old master server to the new master server. Googling a bit shows various migration strategies, all of which are fairly horrible hacks, and didn’t quite do what I wanted. So I came up with my own horrible hack, in the form of a puppet module to do the work.

So here’s the recipe. I’ll refer to the old puppet server to which all systems are subscribed as oldserver, and the server to which I want to migrate the systems to as newserver. In my case oldserver had a DNS CNAME of puppet, and so puppet.conf on the client machines didn’t contain any “server=foo” lines. However, I think the recipe below is entirely general.

On oldserver I created a new modules containing the following (this was stored in /etc/puppet/modules/puppet/manifests/init.pp):

class puppet::migrate ( $puppetmaster ) {

  augeas {'puppet.conf.migrate':
    context => '/files/etc/puppet/puppet.conf/main',
    changes => ["set server ${puppet::migrate::puppetmaster}",
		]
  }

  # These next two objects handle migration to a new puppet master
  # server - if the value of $puppetmaster is updated, the
  # puppet-clear-certs.sh script is executed.
  file {'/var/lib/puppet/lib/puppet-clear-certs.sh':
    owner  => 'root',
    group  => 'root',
    mode   => 700,
    source => 'puppet:///modules/puppet/puppet-clear-certs.sh',
  }

  exec {'/var/lib/puppet/lib/puppet-clear-certs.sh':
    path    => ['/usr/bin', '/bin', '/usr/sbin', '/sbin'],
    require => [File ['/var/lib/puppet/lib/puppet-clear-certs.sh'],
                Augeas ['puppet.conf.migrate'],
                ],
    unless  => ["openssl x509 -text -in /var/lib/puppet/ssl/certs/ca.pem | grep ${puppet::migrate::puppetmaster} >/dev/null 2>&1",
                "openssl x509 -text -in /var/lib/puppet/ssl/certs/${fqdn}.pem | grep ${puppet::migrate::puppetmaster} >/dev/null 2>&1",
                ]
  }
}

The magic ingredient here is the file puppet-clear-certs.sh which was inspired by this blog post by Ryan Uber. This script forks a shell on the client which waits until puppet has finished its catalogue run on the client and then deletes all the SSL certificates on the client, preparing the way for registration with the new server. This file looks like this:

#!/bin/bash

# This script is a hack to remove SSL certificates from a puppet
# client to prepare it for migration to a new puppet master server
# after puppet has altered the puppet.conf file to point to the new
# puppet master server.
#
# Normally, if you subscribe the puppet service to the puppet.conf
# file, the puppet service will be restarted too soon, interrupting
# the current puppet run. Various attempts at using
# configure_delayed_restart among other things have not proven to be
# 100% effective.  This script will watch the puppetdlock file, which
# can determine whether or not there is a run in progress. If there is
# a run in progress, we sleep for a second and then test again until
# the process is unlocked. Once unlocked, we can safely delete
# certificates and call a puppet restart. The checker process itself
# gets forked into the background. If it were not forked into the
# background, the puppet run would sit and wait for the process to
# return, or for the exec timeout, whichever came first. This would
# cause serious trouble if timeouts were disabled or very long periods
# of time.
#
# This script was inspired by this blog post by Ryan Uber:
# http://www.ryanuber.com/puppet-self-management.html
#


# Begin waiting for the current puppet run to finish, then restart.
/bin/sh -c "
    until [ ! -f /var/lib/puppet/state/puppetdlock ]
    do
        sleep 1
    done
    /sbin/service puppet stop
    rm -f /var/lib/puppet/ssl/certs/*
    rm -f /var/lib/puppet/ssl/certificate_requests/*
    rm -r /var/lib/puppet/ssl/crl.pem
    /sbin/service puppet start
" &
 
# Always return true, since this script just forks another process.
exit 0
 
# EOF

So, when this script is executed on the client after the puppet manifest delivered from oldserver modifies puppet.conf to point to newserver it waits until the puppet agent run finishes, then stops the puppet daemon, deletes all the SSL certs, and then restarts the puppet agent which will now connect to newserver and issue a certificate signing request.

So, having created that module, I rsync the puppet modules and manifests from oldserver onto newserver, and then edit the node entries on the oldserver to look like

node somenode.somewhere.com {
  class {'puppet::migrate':
    puppetmaster => 'newserver.somewhere.com',
  }
}

Well, in actual fact I added that class to the basenode from which all other nodes inherit, but you get the idea. Once this is active, nodes migrate and generate signing requests at the next puppet run, and you should see the signing requests on the newserver. Job done!

I should also point out the “unless” guards on the exec in the above init.pp file. Since I rsynced across the migration module, I wanted to protect myself from inadvertently re-triggering a migration from newserver to newserver on every puppet run, so those are there as a safeguard.