Day 13 in the #vDM30in30
Pretty quick one, but I thought I’d write it up because I hadn’t heard about it before.
So, normally when trying to diagnose a puppet issue, one normally runs
puppet agent -t or
puppet agent --test. However, the
--test flag comes with a set of predefined flags:
Enable the most common options used for testing. These are ‘onetime’, ‘verbose’, ‘no-daemonize’, ‘no-usecacheonfailure’, ‘detailed-exitcodes’, ‘no-splay’, ‘show_diff’, and ‘no-use_cached_catalog’.
At the time, we were trying to diagnose an issue with cached catalogs and how they worked on failure. However, we couldn’t reproduce it with
puppet agent -t because it explicitly disabled using the cache on failure and using the cached catalog at all.
At first we just set the
runinterval to a minute and waited for the run to happen, but it turns out there’s a way of actually triggering a proper daemonized agent run immediately, running a kill with
SIGUSR1? It’s a reserved unix signal.
If you’re less familiar with core unix processes, you’ve probably ran a unix signal without knowing it:
Typing certain key combinations at the controlling terminal of a running process causes the system to send it certain signals: Ctrl-C (in older Unixes, DEL) sends an INT signal ("interrupt", SIGINT); by default, this causes the process to terminate. Ctrl-Z sends a TSTP signal ("terminal stop", SIGTSTP); by default, this causes the process to suspend execution. Ctrl-\ sends a QUIT signal (SIGQUIT); by default, this causes the process to terminate and dump core.
Like a lot of people, I knew to run Ctrl-C in a terminal if I wanted to jump out of a process, but I had never really thought about how it works until someone told be about unix signals.
So what about
SIGUSR1? Well, basically SIGUSR1 and SIGUSR2 are reserved for the developer to define the behaviour:
The SIGUSR1 and SIGUSR2 signals are set aside for you to use any way you want. They’re useful for simple interprocess communication, if you write a signal handler for them in the program that receives the signal.
OK, so basically it’s up to the developers to define what the behaviour is.
For Puppet, these are defined as the following:
SIGUSR1 Immediately retrieve and apply configurations from the puppet master. SIGUSR2 Close file descriptors for log files and reopen them. Used with logrotate.
So, we can trigger a “fresh” puppet agent run just using
pkill -SIGUSR1 puppet-agent
We can see that in action here:
Send the puppet process the
[root@homebox centos]# pkill -SIGUSR1 puppet
/var/log/messagesand see a deamonised puppet run is immediately triggered by the signal:
Nov 20 19:10:40 homebox puppet-agent: Caught USR1; storing reload Nov 20 19:10:41 homebox puppet-agent: Processing reload Nov 20 19:11:16 homebox puppet-agent: Applied catalog in 16.12 seconds