Booting Xen DomU with pygrub on Debian Etch dom0

I have a hosted system which runs Xen 3.0 with Debian Etch as the dom0 (host OS), which was first installed a couple of years ago. At the time it was easiest to set things up to boot each domU (virtual machine) using the same kernel as was on the host machine as that was easy to get working (just specify path to kernel and initrd in the Xen configuration file for the DomU). Now that I want to upgrade the domU virtual machines incrementally, with the aim to eventually upgrading the whole machine, I'd like to use pygrub to boot individual kernels out of each virtual machine.

pygrub is in /usr/lib/xen-3.0.3-1/bin/pygrub, in xen-utils on Debian Etch, but has some issues which prevent it from working -- primarily the python import path is set incorrectly and the ext2 file system modules aren't installed.

The import path issue manifests itself like this:

ewen@dl380g3:~$ /usr/lib/xen-3.0.3-1/bin/pygrub -h
Traceback (most recent call last):
  File "/usr/lib/xen-3.0.3-1/bin/pygrub", line 25, in ?
    import grub.GrubConf
ImportError: No module named grub.GrubConf
ewen@dl380g3:~$

and can be fixed by correcting the system import path in pygrub. The missing file system modules are due to the package being built without sufficient file system libraries installed, so nothing gets detected.

Debian bug #390678 contains a patch to fix both these issues. I fixed them both manually by:

editing /usr/lib/xen-3.0.3-1/bin/pygrub and changing line 23 (which updates sys.path) to read
```
sys.path.append('%s/../lib/python' % sys.path[0])
```
building the necessary file system modules on a Debian Etch system (from xen-utils-3.0.3-1 as follows) then copying them into /usr/lib/xen-3.0.3-1/lib/python/grub/fsys on the Xen Dom0.

apt-get source xen-utils-3.0.3-1
cd xen*/tools/pygrub
sudo aptitude install python2.4-dev e2fslibs-dev
make
cd build/lib.linux-i686-2.4/grub/fsys
tar -cvf /tmp/pygrub-3.0.3-ext2-modules.tar ext2

(NOTE: This doesn't build ReiserFS support, because the right sysetm libraries aren't available in Debian Etch -- Debian Etch seems to have Reiser4 libraries, not Reiser3 ones, at least in main. Since I didn't use ReiserFS I didn't worry about this.)

At this point pygrub will work, and will boot a kernel if there is someone around to press enter. Unfortunately while it will read the timeout value from the /boot/grub/menu.lst file, and carefully count down the time remaining, after the timer expires it then goes into a non-timed wait for a keypress... instead of just booting the kernel.

The fix for this issue is to replace the code at line 377 in /usr/lib/xen-3.0.3-1/bin/pygrub which increments the timer count with:

            if c == -1:
                # Timed out waiting for a keypress
                if mytime != -1:
                    mytime += 1
                    if mytime >= int(timeout):
                        self.isdone = True
                        break
            else:
                # received a keypress: stop the timer
                mytime = -1
                self.screen.timeout(-1)

(taken from pygrub in xen-utils version 3.2, the one packaged in Debian Lenny.)

This way when the timeout hits we declare that to be the final answer, and if any key is pressed we then cancel the timeout completely.

The /boot/grub/menu.lst in the DomU needs to look something like this:

# Basic grub file for pygrub

default 0
timeout 3

title   Debian GNU/Linux, with Linux 2.6.18-6-xen-686
root    (hd0,0)
kernel  /boot/vmlinuz-2.6.18-6-xen-686 root=/dev/sda1 ro
initrd  /boot/initrd.img-2.6.18-6-xen-686

title   Debian GNU/Linux, with Linux 2.6.18-6-xen-686 (recovery mode)
root    (hd0,0)
kernel  /boot/vmlinuz-2.6.18-6-xen-686 root=/dev/sda1 ro single
initrd  /boot/initrd.img-2.6.18-6-xen-686

(NOTE: The update-grub in Debian Sid will use a much newer version of grub which uses a different configuration format that is not compatible, but writes to /boot/grub/grub.conf -- so we can maintain this pygrub compatible version in /boot/grub/menu.lst by hand.)

After that it's just a matter of ensuring that initramfs-tools, and the right Xen kernel image and kernel modules files are installed, and changing the Xen configuration file to read:

# Boot via PyGrub instead of using Dom0 hosted kernel/initrd
bootloader = '/usr/lib/xen-3.0.3-1/bin/pygrub'

rather than specifying a kernel and initrd to load from the dom0 filesystem.

(FTR, patched version of pygrub for Xen 3.0.3 on Debian Etch.)

Edited to add: Unfortunately the Debian Lenny 2.6.26 kernel was built with backwards compatibility for Xen 3.1 and later (CONFIG_XEN_COMPAT_030100_AND_LATER=y), but Debian Etch 2.6.18 is Xen 3.0.3, which needs backwards compatibility for Xen 3.0.2 and later (CONFIG_XEN_COMPAT_030002_AND_LATER), and that is explicitly not set in the Debian Lenny kernels :-( (The symptom of this incompatibility appears to be the kernel crashing immediately on starting booting, without any useful error messages, so it took about 4 hours to discover this likely cause :-( ) It may be possible to make the Lenny kernel work on a Etch dom0 by rebuilding with all the Xen backwards compatibility options enabled; I'll try this at some point.