Joggler: Debian kernel doesn't boot

This is part of my series on running an unmodified Debian on the Joggler. See here for other posts on the same topic.

Now that the Joggler is booting rEFIt with a functioning keyboard, experimenting with things to boot is easy. The ideal thing to boot would be the standard Debian kernel. If that works, then pretty much any stock Linux distribution kernel should also work. In order to boot it you need an EFI GRUB. I grabbed a premade binary from the Joggler wiki for now (we'll come back to this later).

Unfortunately, while the GRUB binary worked fine, the kernel doesn't boot, with no indication of why. This is tricky to debug; since the Joggler doesn't have standard VGA, you don't get any console messages on the screen until it's booted far enough to activate efifb, the framebuffer driver which talks to the EFI video output system. It also doesn't have a serial port (at least, not that anyone has found) so you can't get the log messages out that way either, and after experimenting with netconsole (sending console messages over UDP) I concluded that it was failing before initialising networking as well. The only other early boot debugging system that Linux explicitly supports is a weird host-to-host USB debug interface that I don't have and isn't cheap.

So, I grabbed a working Joggler kernel config from one of the Joggler Ubuntu images available on the forum, and built Debian's kernel sources using that configuration. That worked fine, so I bisected the interesting differences between the kernel configurations until I tracked down the single configuration option which stopped it from working: CONFIG_PHYSICAL_ALIGN. This one is going to take some explaining. :)

UPDATE: this is now fixed, see my post here.

So, a bit of back-story here. On x86, the main portion (the protected mode part) of the Linux kernel was loaded at 1MB from the start of memory. All bootloaders behaved the same, and the kernel was compiled specifically to run at that address. Eventually this became inconvenient for various reasons (such as wanting to be able to boot a second kernel loaded elsewhere in memory if the first one crashed, in order to dump the state of memory for debugging purposes), and CONFIG_RELOCATABLE was implemented.

RELOCATABLE allows the kernel to be loaded at different addresses; when started, if it's not been loaded at the address it's compiled to run at, the kernel is patched to run from the address it's loaded at using relocation data attached to the binary. This is enabled by default, since it doesn't actually cost very much; the patching takes place at boot time, so there's no runtime performance cost, the relocation data can be discarded after it's used, so there's no runtime memory cost, and in many cases the kernel will be loaded at the address it's compiled for anyway, so it won't actually cost any time at boot either.

However, the load address has to be aligned to at least a certain size. The configuration value which sets the kernel's preferred alignment is CONFIG_PHYSICAL_ALIGN, and on recent kernels this defaults to 16MB. Older kernels had a default of 1MB. On the Joggler, a 1MB aligned kernel works, but a 16MB aligned kernel does not.

Why does this break things? Well, the alignment is stored in the kernel header so the bootloader can see it; the idea is that if the kernel wants to be aligned to 16MB, the bootloader will load it to 16MB (or 32MB, or 48MB, or any other aligned address) instead of the default 1MB. The bootloader knows what memory exists and what is in use for other things, so it's in the best position to figure this out. However, GRUB doesn't do this yet (as of version 1.99). GRUB just ignores the alignment and loads the kernel at 1MB.

But wait, you say, wouldn't that break every computer, not just the Joggler? You might think that, but the kernel allows for the possibility that it's being loaded by an older bootloader that doesn't know about alignment. If it finds itself at a non-aligned address, it rounds the address up to the next aligned one and moves itself there before applying the relocations. So, on a normal computer, GRUB loads the kernel to 1MB, then the kernel moves itself to 16MB and runs from there, aligned as it wants. The same thing happens on the Joggler, but when it moves itself to 16MB the system crashes.

Why? Well, EFI has something called "runtime services", which are bits of EFI code that can be called by an operating system at runtime to do certain things. The code for these services is in memory somewhere, and if you overwrite it and then call the services, it will crash. Some of the Joggler's EFI services are at 16MB, and thus when the kernel moves itself there, it breaks everything.

There is an EFI memory map passed to the bootloader which tells it where all the allocated memory areas are so the bootloader can avoid overwriting them. The kernel can't easily consult this when it's deciding where to locate itself, because at that point it doesn't have much initialised yet. So, you need the bootloader to not only choose an appropriately aligned address, but also to avoid the runtime services while doing so.

So, for my next post: making GRUB smarter.