Python: Inheritance is not templating

This is something I forgot today, which bugged me a lot. Honestly, I don't program a lot, but it's a bit bothersome that I didn't have this piece of knowledge on the top of my mind. The gist is simple, inheritance doesn't work like templating. There's a difference between local variables and class variables.

What I tried the first time:

class Base: x = []

class A(Base): pass

class B(Base): pass

a_inst = A()

b_inst = B()

b_inst.x.append("foo")

a_inst.x.append("bar")

print a_inst.x

print b_inst.x

This resulted in the output:

['foo','bar']

['foo','bar']

That's not what I expected. After asking around a little on #python, I was pointed to my mistake. The code should've been:


class Base:
    def __init__(self):
        self.x = []


class A(Base):
    def __init__(self):
        Base.__init__(self)


class B(Base):
    def __init__(self):
        Base.__init__(self)


a_inst = A()

b_inst = B()



b_inst.x.append("foo")

a_inst.x.append("bar")



print a_inst.x

print b_inst.x

And this give me the expected output:

['bar']

['foo']

Hope this helps someone!

Debugging puppet queueing

Today we ran into a problem where the data put in ActiveMQ by the puppetmaster seemed corrupted in some way. When running the puppet queue daemon on the foreground (with –debug –verbose –no-daemonize), we noticed messages like these:

info: Loaded queued catalog in 22.16 seconds

debug: Searched for resources in 0.31 seconds

err: Could not save queued catalog for web1.ourserver.com: syntax error on line 68, col 34: ` serverversion: 2.7.6 sshdsakey: [long string]'

notice: Processing queued catalog for web1.ourserver.com in 0.41 seconds

It seemed like for some reason there’s a newline missing there, but what exactly is it trying to do? It would be helpful if we could check the message in total, to see which resource is doing this. Python to the rescue!

On the machine that’s running the ActiveMQ, install python-stompy (we’re on Debian Squeeze). Open a python interactive shell and do this:

>>> from stompy.simple import Client

>>> stomp = Client()

>>> stomp.connect()

>>> stomp.subscribe("/queue/catalog")

>>> message = stomp.get_nowait()

>>> f = open("message","w")

>>> f.write(message.body)

>>> f.close()

>>> stomp.unsubscribe("/queue/catalog")

>>> stomp.disconnect()

You know have a file called message that contains the message. You might want to make the file a little bit easier to read by executing the following: sed -i 's/{/\n{/g' message, which adds a newline in front of each opening accolade. Now to search for the problem and the resource that causes it.

I hope this helps someone!

Cryptostick on Ubuntu

We use the following puppet recipe to get our CryptoStick (v1.2) working on Ubuntu. This enables the gpg-agent to be used as ssh-agent as well. There might be unnecessary code in here, but I'm too lazy to remove that :) Do leave me a message if you find mistakes or just if it helped you! You need to run puppet as root (just sudo is enough) to deploy the modification the script makes. Reboot after it's been applied.

## VARIABLES - CHANGE THESE

$user = "tim"

You probably no longer need to change these, but check them just in case

$vendorId = "04e6"

$productId = "5115"

DON'T CHANGE BELOW THIS LINE

define line ( $ensure = "present", $file, $content = false ) {

if ! $content {
    $content = $name
}

case $ensure {
    default: {
        fail("Unknown ensure value: ${ensure}.")
    }
    present: {
        exec { "line $name":
            command => "/bin/echo '${content}' >> '${file}'",
            unless  => "/bin/grep -Fx '${content}' '${file}'";
        }
    }
    absent: {
        exec { "line $name":
            command => "/usr/bin/perl -ni -e 'print unless /^\\Q${content}\\E\$/' '${file}'",
            onlyif  => "/bin/grep -Fx '${content}' '${file}'";
        }
    }
}

}

line { "Start of default.desktop": content => "[Desktop Entry]", file => "/usr/share/xsessions/default.desktop"; "default.desktop, Encoding": content => "Encoding=UTF-8", require => Line["Start of default.desktop"], file => "/usr/share/xsessions/default.desktop"; "default.desktop, Name": content => "Name=default", require => Line["Start of default.desktop"], file => "/usr/share/xsessions/default.desktop"; "default.desktop, Comment": content => "Comment=Default session", require => Line["Start of default.desktop"], file => "/usr/share/xsessions/default.desktop"; "default.desktop, Exec": content => "Exec=default", require => Line["Start of default.desktop"], file => "/usr/share/xsessions/default.desktop"; "default.desktop, Type": content => "Type=Application", require => Line["Start of default.desktop"], file => "/usr/share/xsessions/default.desktop"; }

Not sure if this part is still needed, but does no harm.

line { "Xsession: OPTIONFILE": content => 'OPTIONFILE=${OPTIONFILE:-/etc/X11/Xsession.options}', file => "/etc/X11/Xsession.d/01x11-common_setup-vars"; "Xsession: SYSRESOURCES": content => 'SYSRESOURCES=${SYSRESOURCES:-/etc/X11/Xresources}', file => "/etc/X11/Xsession.d/01x11-common_setup-vars"; "Xsession: USRRESOURCES": content => 'USRRESOURCES=${USRRESOURCES:-$HOME/.Xresources}', file => "/etc/X11/Xsession.d/01x11-common_setup-vars"; "Xsession: SYSSESSIONDIR": content => 'SYSSESSIONDIR=${SYSSESSIONDIR:-/etc/X11/Xsession.d}', file => "/etc/X11/Xsession.d/01x11-common_setup-vars"; "Xsession: USERXSESSION": content => 'USERXSESSION=${USERXSESSION:-$HOME/.xsession}', file => "/etc/X11/Xsession.d/01x11-common_setup-vars"; "Xsession: USERXSESSIONRC": content => 'USERXSESSIONRC=${USERXSESSIONRC:-$HOME/.xsessionrc}', file => "/etc/X11/Xsession.d/01x11-common_setup-vars"; "Xsession: ALTUSERXSESSION": content => 'ALTUSERXSESSION=${ALTUSERXSESSION:-$HOME/.Xsession}', file => "/etc/X11/Xsession.d/01x11-common_setup-vars"; "Xsession: ERRFILE": content => 'ERRFILE=${ERRFILE:-$HOME/.xsession-errors}', file => "/etc/X11/Xsession.d/01x11-common_setup-vars"; }

We need these packages.

package { ["gnupg-agent","pinentry-gtk2","openssh-client","gpgsm","pcscd"]: ensure => installed, }

line { "Disable use-ssh-agent in Xsession.options": ensure => absent, file => "/etc/X11/Xsession.options", content => "use-ssh-agent"; "Personal xsession start": ensure => present, file => "/home/${user}/.xsession", content => "#!/bin/bash"; "Start gpg-agent with the correct variables from .Xsession": ensure => present, file => "/home/${user}/.xsession", content => "eval $(/usr/bin/gpg-agent --daemon --sh --write-env-file=/home/${user}/.gnupg/gpg-agent-info-$(hostname) --enable-ssh-support)", require => Line["Personal xsession start"]; "Make sure X11 reads user options": ensure => present, file => "/etc/X11/Xsession.options", content => "allow-user-xsession"; "Kill scdaemon when we insert a new card in the SCM reader": ensure => present, file => "/etc/udev/rules.d/smartcard.rules", content => "ACTION==\"add\", SUBSYSTEM==\"usb\", SYSFS{idVendor}==\"${vendorId}\", SYSFS{idProduct}==\"${productId}\", PROGRAM==\"/usr/bin/killall -9 scdaemon\""; "Kill scdaemon when we insert a cryptocard": ensure => present, file => "/etc/udev/rules.d/smartcard.rules", content => 'ACTION=="add", SUBSYSTEM=="usb", ATTR{product}=="Crypto Stick v1.2", PROGRAM=="/usr/bin/killall -9 scdaemon"'; "Make gnupg use the agent": ensure => present, file => "/home/${user}/.gnupg/gpg.conf", content => "use-agent"; }

file { "/home/${user}/.xsession": ensure => file, mode => 755, owner => "${user}", }

Remove this file, please.

file { "/etc/xdg/autostart/gnome-keyring-ssh.desktop": ensure => absent; "/etc/xdg/autostart/gnome-keyring-gpg.desktop": ensure => absent; }

Change the gnome settings

exec { "/bin/su - ${user} -c '/usr/bin/gconftool-2 --set -t bool /apps/gnome-keyring/daemon-components/ssh false'": unless => "/bin/su - ${user} -c '[ $(/usr/bin/gconftool-2 --get /apps/gnome-keyring/daemon-components/ssh) == \"false\" ]'", }

Check SSL certificates

This post is mostly a collection of commands to check SSL certificates and make sure they are what you think they are. Especially when things do not go as expected, these commands are handy to have around. First, some definitions. We call the signed certificate cert.crt, the private key server.key, the certificate sign request cert.csr and any intermediate/chain certificates chain.pem. Substitude in the commands below with your files. It’s assumed you have all these certificates in the PEM format, for easy use with Apache’s mod_ssl.

Checking if the CSR is actually a public key from your serverkey

You need to check the modulo of the private key and the certificate sign request. The output of these two commands should be the same if the csr is made with this server key.

$ openssl rsa -noout -modulus -in server.key | openssl md5
$ openssl req -noout -modulus -in cert.csr | openssl md5

Checking if a signed certificate is actually created from the CSR that you created

You need to check the modulo of both files. The output of the two commands should be the same.

$ openssl x509 -noout -modulus -in cert.crt | openssl md5
$ openssl req -noout -modulus -in cert.csr | openssl md5

Checking if a signed certificate is actually the public key from your serverkey

This should be obvious if you read the two items above. The output of both commands should be the same.

$ openssl x509 -noout -modulus -in cert.crt | openssl md5
$ openssl rsa -noout -modulus -in server.key | openssl md5

Checking if the chain file actually applies to the signed certificate

openssl verify -CAfile chain.pem -verbose cert.crt

Output the details from a certificate sign request

openssl req -text -in cert.csr

Output the details from a signed certificate

openssl x509 -text -in cert.crt

Another job opening

Kumina is again looking for a new full-time junior systems administrator per July or August 2011. Are you the person we’re looking for?

We’re looking for someone who…

  • … doesn’t quit when the going gets tough
  • … has an interest in system maintenance
  • … is comfortable with responsibility
  • … wants to go the extra mile if that results in higher quality of the end-product
  • … is versatile and wants to learn new things all the time
  • … can work with a team, but not necessarily in a team

Linux knowledge is not necessarily required, if you’re willing to learn fast. You can find info about what we do on our website. Some keywords:

We’re looking for a full-time employee starting July or August 2011. Interested? Send your resumé and an introductory letter to jobs@kumina.nl!

Cassandra: java.io.IOError: java.io.IOException: Map failed

Today we ran into a problem with Cassandra in which Cassandra failed to start with the following stack trace:

ERROR 2011-04-18 12:53:01,759 Fatal exception in thread Thread[FlushWriter:1,5,main]

java.io.IOError: java.io.IOException: Map failed

at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:172)

at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:149)

at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:190)

at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:169)

at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:163)

at org.apache.cassandra.db.Memtable.access$000(Memtable.java:51)

at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:176)

at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

Caused by: java.io.IOException: Map failed

at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)

at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:164)

... 10 more

Caused by: java.lang.OutOfMemoryError: Map failed

at sun.nio.ch.FileChannelImpl.map0(Native Method)

at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)

... 11 more


Additional strange errors in the logs were:

Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.

It took us a while to figure out, but after using strace, some help in #cassandra on freenode and a knowledgeable guy (thanks Ed), we determined the problem was with Cassandra trying to mmap too many files. This limit is enforced by the kernel in the /proc/sys/vm/max_map_count. Eventually an strace pointed it out to us:

mmap(NULL, 135168, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = -1 ENOMEM (Cannot allocate memory)

According to man 2 mmap:

ENOMEM No memory is available, or the process’s maximum number of mappings would have been exceeded.

So we ended up increase max_map_count with:

sysctl -w vm.max_map_count = 131072

It’s a bit hackish (I personally don’t think Cassandra should mmap that many files), but it solved our problem. Hope it helps someone!

Published our repositories

So we’ve finally gotten around to publishing our puppet generic and kbp repositories, as described in this previous post. They still need a lot of work (and a lot of documentation!), but it’s a start. And having them public gives us the extra push to add the documentation. You can find them here:

Not all in there is originally ours, but most of it is.

Feel free to send pull requests if you fix stuff!

Our fix for Squeeze-backports “Expired Release file”

This morning there have been problems with the Release file for Squeeze backports. The main problem being that it is automatically updated when a new package is added, but there haven’t been any new packages lately, so the Release file expired. They’re working on fixing the problem. In the meantime, we’re happy we got both our own repository and use approx almost everywhere.

Fixing it is as easy as editing the Release file for squeeze-backports in your approx cache (in our setup, it’s in /var/cache/approx/backports/dists/squeeze-backports/Release) and increase the Valid-Until date. Then resign the Release file with the key you use for your repository (mostly as a convenience, since all machines will have that key in their apt config) and renaming the resulting detached signature to Release.gpg. Now approx will serve a valid Release file, signed by a key it knows.

If you don’t have your own repository, you can sign the file with your own gpg key and import your public key into apt with the following command:
sudo apt-key adv --keyserver your.default.keyserver --recv-key 12345678 (with 12345678 being your key ID of course)

Once the squeeze-backports repository is fixed, approx will notice a newer Modified-Since header and overwrite the files again.

Update 11:32 They fixed the squeeze-backports repositories.