Debugging puppet queueing

Today we ran into a problem where the data put in ActiveMQ by the puppetmaster seemed corrupted in some way. When running the puppet queue daemon on the foreground (with –debug –verbose –no-daemonize), we noticed messages like these:

info: Loaded queued catalog in 22.16 seconds
debug: Searched for resources in 0.31 seconds
err: Could not save queued catalog for web1.ourserver.com: syntax error on line 68, col 34: `  serverversion: 2.7.6  sshdsakey: [long string]'
notice: Processing queued catalog for web1.ourserver.com in 0.41 seconds

It seemed like for some reason there’s a newline missing there, but what exactly is it trying to do? It would be helpful if we could check the message in total, to see which resource is doing this. Python to the rescue!

On the machine that’s running the ActiveMQ, install python-stompy (we’re on Debian Squeeze). Open a python interactive shell and do this:

>>> from stompy.simple import Client
>>> stomp = Client()
>>> stomp.connect()
>>> stomp.subscribe("/queue/catalog")
>>> message = stomp.get_nowait()
>>> f = open("message","w")
>>> f.write(message.body)
>>> f.close()
>>> stomp.unsubscribe("/queue/catalog")
>>> stomp.disconnect()

You know have a file called message that contains the message. You might want to make the file a little bit easier to read by executing the following: sed -i 's/{/\n{/g' message, which adds a newline in front of each opening accolade. Now to search for the problem and the resource that causes it.

I hope this helps someone!