Twitter Weekly Updates for 2011-04-24

• Had a wonderful evening at Zinc in Rotterdam! Time for bed, though. #
• Missed the train by a hair :-( I swear they left early! Bah. #
• Off to home for my first (short) vacation in a long time. Even though it's only a long weekend, I'm fairly nervous about it! #
• Stiekem loop je nog best veel tijdens zo'n dagje Efteling. Dit is alleen de middag: http://t.co/IlKvWce #

Powered by Twitter Tools

Tags: tweets

Cassandra: java.io.IOError: java.io.IOException: Map failed

Today we ran into a problem with Cassandra in which Cassandra failed to start with the following stack trace:

ERROR 2011-04-18 12:53:01,759 Fatal exception in thread Thread[FlushWriter:1,5,main]
java.io.IOError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:172)
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:149)
at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:190)
at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:169)
at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:163)
at org.apache.cassandra.db.Memtable.access$000(Memtable.java:51)
at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:176)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:164)
... 10 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
... 11 more

Additional strange errors in the logs were:

Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.

It took us a while to figure out, but after using strace, some help in #cassandra on freenode and a knowledgeable guy (thanks Ed), we determined the problem was with Cassandra trying to mmap too many files. This limit is enforced by the kernel in the /proc/sys/vm/max_map_count. Eventually an strace pointed it out to us:

mmap(NULL, 135168, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = -1 ENOMEM (Cannot allocate memory)

According to man 2 mmap:

ENOMEM No memory is available, or the process’s maximum number of mappings would have been exceeded.

So we ended up increase max_map_count with:

sysctl -w vm.max_map_count = 131072

It’s a bit hackish (I personally don’t think Cassandra should mmap that many files), but it solved our problem. Hope it helps someone!

Twitter Weekly Updates for 2011-04-17

• @bartcortooms Zin om nog een keertje gezellig hier mee te lunchen en bij te praten, nu je in NL zit? Je bent altijd welkom! #
• Anyone know how to make an excerpt mandatory in #wordpress before publishing a post? #lazyweb #
• @sanchothefat There doesn't happen to be some kind of easily installable plugin that you know of, by any chance? in reply to sanchothefat #
• @davecoveney Awesome! Much appreciated! in reply to davecoveney #
• @sanchothefat Awesome! Thanks man. Now for my next question… where do I put the file? :) in reply to sanchothefat #
• @TubbyNL Orderbevestiging is niet ondertekend, of wel? Gewoon kijken naar de koopovereenkomt. Als dat de orderbevestiging is, klaar. IANAL in reply to TubbyNL #
• @unixdaemon Congratulations!! … I think… in reply to unixdaemon #
• De dag is bewolkt en de zon is afwezig. Daar begon hij gisteravond al mee. Hopelijk snel verbetering, zou me goed doen. #

Powered by Twitter Tools

Tags: tweets

Published our repositories

So we’ve finally gotten around to publishing our puppet generic and kbp repositories, as described in this previous post. They still need a lot of work (and a lot of documentation!), but it’s a start. And having them public gives us the extra push to add the documentation. You can find them here:

• Kumina’s generic modules
• Kumina’s best practises in modules

Not all in there is originally ours, but most of it is.

Feel free to send pull requests if you fix stuff!

Twitter Weekly Updates for 2011-04-10

• Apparantly we need to shop for fish with a GM-counter from now on. Thanks, nuclear power. #
• @patrickdebois Give metrics or we won't believe you! ;-) in reply to patrickdebois #
• @EdSchouten Gast, het is al 25 over 2, ben je nog niet klaar met dat tentamen? #
• Train arrived 8 minutes early. That's new. #
• Read an article once about project planning and how you should not allocate slack time at end but at beginning. Anyone know the article? #

Powered by Twitter Tools

Tags: tweets

Our fix for Squeeze-backports “Expired Release file”

This morning there have been problems with the Release file for Squeeze backports. The main problem being that it is automatically updated when a new package is added, but there haven’t been any new packages lately, so the Release file expired. They’re working on fixing the problem. In the meantime, we’re happy we got both our own repository and use approx almost everywhere.

Fixing it is as easy as editing the Release file for squeeze-backports in your approx cache (in our setup, it’s in /var/cache/approx/backports/dists/squeeze-backports/Release) and increase the Valid-Until date. Then resign the Release file with the key you use for your repository (mostly as a convenience, since all machines will have that key in their apt config) and renaming the resulting detached signature to Release.gpg. Now approx will serve a valid Release file, signed by a key it knows.

If you don’t have your own repository, you can sign the file with your own gpg key and import your public key into apt with the following command:
sudo apt-key adv --keyserver your.default.keyserver --recv-key 12345678 (with 12345678 being your key ID of course)

Once the squeeze-backports repository is fixed, approx will notice a newer Modified-Since header and overwrite the files again.

Update 11:32 They fixed the squeeze-backports repositories.

On our puppet module design

Most puppeteers seem to work for either one company or only deploy one type of service to different companies. You can notice this in the way they build their modules. Most people tend to create one module that contains everything that is needed for a service or application, since they only need to have it (mostly) the same. We at Kumina work for a lot of different companies and try to be as flexible as possible. This makes our module design a fair bit different than most others, in my experience. This post tries to explain some of the choices we’ve made with regards to module design.

We work at three different levels, which we gave separate names:

• The first one is the most generic and we call it… generic.
• The second one builds on top of a generic module and implements our best practises. We call it “Kumina Best Practises” or kbp for short.
• The final one is the customer specific layer, which actually implements our kbp modules with the correct variables for a specific customer.

When designing our modules, we keep several guidelines in mind:

• Prefer to include instead of inheriting. Inheritance is no longer needed since we can pass parameters to classes.
• In generic, the modules name should be the name of the application. So if you’re creating a module for Apache, call it “gen_apache”.
• In generic, you should only add resources that are needed for that exact application, nothing else.
• In generic, you can only rely on functions that are part of puppet or the gen_common module. Nothing else.
• In kbp, you can only rely on functions in gen_common and any included class from generic or kbp.
• In kbp, make sure the module is totally self-contained. This means that if it uses defined types or functions from other modules, it should include those modules explicitely.
• The kbp module should setup monitoring and trending explicitely using the kbp_icinga and kbp_munin classes (or whichever modules we use per default for those).
• The customer specific modules are only allowed to include kbp modules or customer specific modules from the same customer (actually, each customer has her own environment, so they cannot access specific modules from other customers).
• The customer specific modules should be divided in actual service, where the class includes everything that’s needed for a service. So if you have multiple PHP websites, each site has a separate class and each class includes everything it needs, even if that means that both site classes will include the mysql server, for example.
• Anything slightly generic should be built in kbp, not in the customer specific modules.

That looks like a fairly long list, but most of it seems rather logical once you’re working with it. These guidelines make sure that our entire team can quickly and easily work on each customer’s setup, where needed. Keeping the environments separate from each other also allows us to easily see the impact certain changes will make. In practise, most of the resources will be defined in the kbp layer. But the generic layer is still important, because we try to create an API-like approach the applications of the same service type. I’ve described how to do that a while ago on the Puppet mailinglist (I should probably write a blog post about that too, but not today). The main advantage of this being that you should be able to easily replace for example apache with nginx.

Using the above guidelines, setting up our webserver is simply a block like this:

node 'web.kumina.nl' inherits 'kumina_default' {
  include site::www_kumina_nl
  include site::www_twenty_five_nl
  include site::blog_kumina_nl
  include mail::incoming
}

And everything is setup as we need it. The class itself looks as follows (for example):

class site::www_kumina_nl {
  include kbp_httpd
  include site::common

  kbp_httpd::simple_site { "www.kumina.nl":
    ensure => 'present',
    documentroot => '/srv/www/www.kumina.nl/',
  }
}

class site::www_twenty_five_nl {
  include kbp_httpd
  include site::common
  include kbp_httpd::php
  include kbp_mysql

  kbp_httpd::simple_site { "www.twenty-five.nl":
    ensure => 'present',
    documentroot => '/srv/www/www.twenty-five.nl/',
  }

  kbp_mysql::db_with_user { "tf_interface":
    password_hash => 'very_secret',
  }
}

class site::blog_kumina_nl {
  include kbp_httpd
  include site::common
  include kbp_httpd::php
  include kbp_mysql

  kbp_httpd::simple_site { "blog.kumina.nl":
    ensure => 'present',
    documentroot => '/srv/www/blog.kumina.nl/',
  }

  kbp_mysql::db_with_user { "kumiblog":
    password_hash => 'very_secret',
  }
}

If we decide to ever move the blog to a separate server, we can simply do:

node 'web.kumina.nl' inherits 'kumina_default' {
  include site::www_kumina_nl
  include site::www_twenty_five_nl
  include mail::incoming
}

node 'blog.kumina.nl' inherits 'kumina_default' {
  include site::blog_kumina_nl
}

Aside from manually moving the data in the database, everything should work as expected. This allows us to easily move sites (or other applications) from one machine to another.

This way of building your modules either appeals because of the flexibility or seems a horribly inefficient use of your time. We find it’s a nice way to keep some order without losing too much flexibility.

Twitter Weekly Updates for 2011-04-03

• I need more hours in the day. Anyone got any to spare? #
• @KrisBuytaert Make him run then! :) in reply to KrisBuytaert #
• @domfarr That would probably fix global warming, but not add hours to the day, I'm afraid. Moving the moon further out might help, though… in reply to domfarr #
• @patrickdebois We're going to introduce another nifty new thing for that: telefax! We hear it's going to be all the rage. in reply to patrickdebois #
• Always love the April Fools RFC: http://tools.ietf.org/html/rfc5984 #

Powered by Twitter Tools

Tags: tweets

ANNOUNCEMENT: Microsoft Windows support coming soon!

Kumina is happy to announce it’s in the process of switching all its managed servers from Debian Linux to Microsoft Windows. Due to high demand, it was decided last January that this would be a good time to switch from Linux to Windows.

The last few months we’ve been working hard to migrate our best practices over to this new and exciting platform. All of our employees have followed a week long course to learn about all the licensing issues surrounding the Microsoft platform, and by now we’re comfortable with advising you about all the benefits Microsoft licensing can offer you!

Most of our best practices have been migrated by now, so we feel confident that we can provide you with the same or better service level as you’re used from us by now.

We will inform our customers separately of the intended migration schedule as it affects them. We’re happy and proud to finally be part of the Microsoft world!

If you have any questions, feel free to contact us!

PS
Yes! We’re so excited!