Sorting Apache log files

Last week I had to sort several months worth of Apache log files to feed them to Webalizer. These came from several servers, so it was fairly difficult to get the data together and in the correct format. I ended up using a small python sorting script, especially written for this. Maybe it’ll help someone else, so I’m sharing it here. Just pipe the logs through the script and direct the output to a file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

import sys
 
data = sys.stdin.readlines()
 
def compare_apache_dates (date1, date2):
	str1 = date1.split()[3]
	str2 = date2.split()[3]
	months = ["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"]
 
	if str1[8:12] > str2[8:12]:
		return 1
	elif str1[8:12] < str2[8:12]:
		return -1
	elif months.index(str1[4:7]) > months.index(str2[4:7]):
		return 1
	elif months.index(str1[4:7]) < months.index(str2[4:7]):
		return -1
	elif str1[1:3] > str2[1:3]:
		return 1
	elif str1[1:3] < str2[1:3]:
		return -1
	elif str1[13:15] > str2[13:15]:
		return 1
	elif str1[13:15] < str2[13:15]:
		return -1
	elif str1[16:18] > str2[16:18]:
		return 1
	elif str1[16:18] < str2[16:18]:
		return -1
	elif str1[19:21] > str2[19:21]:
		return 1
	elif str1[19:21] < str2[19:21]:
		return -1
	else:
		return 0
 
data.sort(compare_apache_dates)
for line in data:
	if line[1:4] == "var":
		# Small hack to fix output lines from egrep
		sys.stdout.write( line.split(":",1)[1] )
	else:
		sys.stdout.write( line )

Tags: apache, log, python, sort, webalizer

Puppet Tips&Tricks: testing your regsubst replacings

This is part of an ongoing series. Check this for the complete series!

Regular Expressions are important for us. We use them a lot, mostly because it’s such a powerful tool. So our puppet recipes contain several regsubst calls too. One problem is usually that regex can be fairly complex and you’d like a nice way to check it out. After some talk on IRC (#puppet on freenode), monarchus gave me some tips for this. Simply use the interactive Ruby shell, irb, for this.

Now, I wanted to check whether a certain string ended in “:ssl” or not. I tested my regex replacement as follows:

$ irb
>> s1="www.kumina.nl:ssl"
=> "www.kumina.nl:ssl"
>> s2="www.kumina.nl"
=> "www.kumina.nl"
>> s1.sub(/.*:(ssl)$/, "\\1")
=> "ssl"
>> s2.sub(/.*:(ssl)$/, "\\1")
=> "www.kumina.nl"
>>

From this I gathered that the resulting regsubst call would be something like this:

if regsubst($name, '.*:(ssl)$', '\1') == "ssl" { ... do stuff ... }

Awesome! Now, if you want to try out a global replace, instead of sub, use gsub.

If you want to use regex in your selector, you can simply use egrep on the commandline, like so:

echo "foo" | egrep "foo|bar"

You can also try it in irb, with the following:

>> a = "foo"
=> "foo"
>> a.match("foo|bar")
=> #
>> b = "beastieboys"
=> "beastieboys"
>> b.match("foo|bar")
=> nil
>>

Hope this helps someone!

Checking visitor IPs with apachetop

We have one customer’s site that’s running on a, well, let’s call it a not-so-very-good platform. If it gets too many hits, the database tends to get slow very quickly, which results in a severe congestion. I was looking for a way to easily check if a certain IP has been hitting the same site a lot, since that’s usually the cause (someone who wants to download the entire site or something). Thanks to a tip on IRC, I finally found the tool for that and it turns out it was installed already!

The solution: apachetop. I tend to start it as follows: apachetop -H 5000 -d 2 and when it has started, I press ‘d’ once, which will make it show me the source ip addresses. Nice!

PS. Another solution: tail -10000 access.log | cut -d ' ' -f 1 | sort | uniq -c | sort -nr | head -n 10

Tags: apache, apachetop, hits, ip, traffic, work

Puppet Tips&Tricks: Converting booleans to numbers

This post is a part of my series about tips and tricks for puppet, the configuration management tool we prefer to use here at Kumina.

Puppet has nice support for Nagios via its Nagios-specific resources. However, this requires you to use “0″ and “1″ instead of “true” and “false” for booleans in Nagios. Because we like uniformity, I’ve created a little function that simply converts a named boolean to a numerical. Check it out:

module Puppet::Parser::Functions
  newfunction(:bool2num, :type => :rvalue) do |args|
    case args[0]
      when "true"  then "1"
      when "false" then "0"
      when "1"     then "1"
      when "0"     then "0"
      when true    then "1"
      when false   then "0"
      else raise Puppet::ParseError, "Either specify true, false, 1 or 0."
      end
  end
end

Hope this helps someone!

Puppet Tips&Tricks: Getting the version from a package

Sometimes you need to know the version of a package before you can do anything useful. For example, we need to make sure the augeas-lenses package is 0.6.0 or newer, otherwise it doesn’t include the aptpreferences lens. We need that lens to modify /etc/apt/preferences in a nice way.

It would be very nice if there was some sort of function that could check on the client which version a certain package was, but during the design of puppet it was decided that puppet is not allowed to execute functions on a client, all functions are executed on the puppetmaster. So that’s not a valid option.

Another solution would be to create puppet facts for every package installed containing their version. Although that would help, it would clutter the facts a bit and probably make each puppetd run quite a bit longer. Also, all those facts would have to be stored in the database, if you’re using storeconfigs, which would add a lot of useless data there (although I can imagine setups which would actually appreciate this). So I chose for a simpler way, creating a fact specifically for augeas-lenses.

Getting the version from the commandline is easy on Debian, simply execute dpkg-query -W -f='${Version}' augeas-lenses. Creating a fact from that, is in its turn easy too:

if  FileTest.exists?("/usr/bin/dpkg-query")
	Facter.add("augeas_version") do
		setcode do
			%x{/usr/bin/dpkg-query -W -f='${Version}' augeas-lenses}
		end
	end
end

I created a file in our common module, lib/facter/augeas_version.rb, since that module is included on every host and the fact isn’t specific for a certain module. Preferably, you should add the fact to the module which uses it, of course, to keep things separated.

Now I can simply do:

# We need to compare the current augeas version, to see if we have the
# aptpreferences lens available. It's only been included since 0.6.0.
if versioncmp($augeas_version, '0.6.0') < 0 {
  # Work with templates and concatenated files
  notify { "Using concatenated files, augeas-lenses version $augeas_version":; }
} else {
  # Work with augeas
  notify { "Using augeas, augeas-lenses version $augeas_version":; }
}

Awesome puppet! Hope this helps someone.