LVM + Xen + drbd + heartbeat2 + partition resize

Quite a lot of layers. But as we shall see, layers are A Good Thing™. Although there is some downtime involved in this. So, about the setup:

  • Two identical physical machines, setup mostly identical, we call them 'node1-dom0' and 'node2-dom0'
  • Xen for the virtual servers, which we will call 'node1-domu' and 'node2-domu'
  • LVM used in the dom0 for the partitions of the domU
  • DomU only sees block devices, no LVM
  • Drbd partition with external metadata on a separate partition
  • Heartbeat2 in both domUs, one active (node1-domu), one dormant (node2-domu)
  • ... and now we need to resize the drbd partition

If you take a look at the IT department blog, you'll see a HowTo that describes how to do it when you do not have the metadata for the drbd partition on a separate partition. Our way is much easier. Make backups before you try this.

First, login to node1-domu. We're going to disable the resource group like so:

$ sudo crm_resource -p is_managed -r name_of_resource_group -t primitive -v off

This will not shut down the service, just make it unmanaged, so we can do with it what we want. For the resize we need to unmount the partition and be able to manipulate it. Continue with the following on node1-domu:

$ sudo /etc/init.d/server-using-the-drbd-partition stop

$ sudo umount /mounted/drbd/partition

$ sudo drbdadm down drbd-resource-number

Replace above as required. The drbd resource number is the one you can find in your resource definition (in /etc/drbd.conf). Now let's manipulate the partition. Log into node1-dom0 and do the following:

$ sudo xm block-list node1-domu

You'll get a list of the block devices that are used by the domU. It's a bit of a bother that you cannot see the device names that are given to them inside the domU, but you can presume that this list is in exactly the same order as the partitions are listed in your node1-domu.cfg, which should be located in /etc/xen/{domains/,}node1-domu.cfg. Make sure you get the last number of the line for the partition corresponding with the partition that we just unmounted inside the domU. Lets say it has number 5024. Do the following:

$ sudo xm block-detach node1-domu 5024

Now we can simply resize it with the default LVM tools. Something like this:

$ sudo lvresize -L +10G domU/node1-drbd

Of course you'll need to replace the actually partition name. Now fsck the disk, resize the filesystem and attach it back to the domU:

$ sudo fsck -f /dev/mapper/domU-node1--drbd

$ sudo resize2fs /dev/mapper/domU-node1--drbd

$ sudo xm block-attach node1-domu phy:/dev/domU/node1-drbd sda4 w

For the last line, you can find those settings in the aforementioned node1-domu.cfg. Make sure you use the exact same settings, or things will break.

We're almost done. As you noticed, we've kept node2 out of the picture until now. This is important, so we can always revert to a know working condition in case of failure. But we cannot keep the service running on node2 while we change node1, since data will probably change during the short period that we resize the partition on node1 and those changes will be lost once we startup node1 again. I wouldn't trust a sync from node2 to node1 when node1's partition is of a different size. I'm not sure if that would work as you'd expect, since it would overwrite the filesystem info and probably do a full resync. Will get back to you on that a bit later.

But to keep our fallback operational, we need to make sure it will not try to sync with primary. Log into node2-domu and disable the drbd connection:

$ sudo drbdadm down drbd-resource-number

If you get an error here, check to make sure your service isn't running, since you don't want anything using the partition. It shouldn't be, though. Now that the network connection is severed, we can start the service again on node1. Log into node1-domu and do the following:

$ sudo drbdadm up drbd-resource-number

$ sudo crm_resource -p is_managed -r name_of_resource_group -t primitive -v on

Use crm_mon to check if everything starts okay. If not, disable the resource group (by changing the 'on' in the last line to 'off') and fix it manually. I had to do that, but I forgot to enable the drbd interface, which probably was the cause of it not working.

Once it's running, check your service. Everything should be working as expected. If not, revert using your fallback on node2. I leave that part as an exercise for the reader (mainly because I didn't need to do it myself).

Everything working fine? Great! Let's fix the failover.

On node2-domu, the partition should already be unmounted and we already disabled the drbd layer for this partition. So we can directly log into node2-dom0 and issue the following commands (make sure you replace the values of the arguments with your own!):

$ sudo xm block-detach node2-domu 5024

$ sudo lvresize -L +10G domU/node2-drbd

$ sudo fsck -f /dev/mapper/domU-node2--drbd

$ sudo resize2fs /dev/mapper/domU-node2--drbd

$ sudo xm block-attach node2-domu phy:/dev/domU/node2-drbd sda4 w

Now log into node2-domu and do the following:

$ drbdadm up drbd-resource-number

... And that's it! Test your failover and make sure everything works as expected. But this is how I did it and this worked for me.

Although reminiscing about this, I figure it might be possible to do this with very little downtime, by using online resizing and using drbd's feature that it doesn't care how much extra diskspace is available. Will try that soon, just to see if it would work reliably.


Comments powered by Disqus