Joggler: sorry for the delay!

I've been really busy lately working on other stuff, which you might get some posts about soon. Debian has added 64-bit EFI support to d-i and I intend to look at the Joggler again soon to see if the installer can be prodded into working a bit better on it now.

Joggler: booting Linux directly

This is part of my series on running an unmodified Debian on the Joggler. See here for other posts on the same topic.

Thanks to Ben Hutchings (see this bug), the standard Debian kernels in unstable now have the EFI boot stub enabled. This means that you can boot them directly on the Joggler without needing to use a bootloader at all. This isn't all that great for regular booting, because there's no sensible way to pass a command line to the kernel without typing it manually on the shell, but it works very well for debian-installer. All you need to do is download the latest kernel and initrd.gz from the d-i nightly site, rename the kernel from "linux" to "linux.efi", put them on a FAT-formatted stick, and create a file on the stick called boot.nsh containing:

fs1:

linux initrd=\initrd.gz

and then the Joggler should boot directly to the installer. The issue with the keyboard not working when booting from boot.nsh doesn't apply, because Linux doesn't depend on EFI services for input. It also gets loaded at a suitable address without any issues.

For the actual installed system, GRUB is still preferable, since it allows much more flexible boot configuration; my plan there is to wait until GRUB 2.00 is released and makes it into Debian so that it can be installed with the regular grub-efi package.

Joggler: No more GRUB patches!

This is part of my series on running an unmodified Debian on the Joggler. See here for other posts on the same topic.

Just a quick update: you can now build and run an unmodified GRUB on the Joggler! A finished version of the patch I referred to in my previous post has been committed and is present in version 2.00~beta2, which you can get here. You still need to build it yourself, but it will work without any source modifications. Once the final release of 2.00 comes out, there should be a Debian package which will work :)

Thanks to Matthew Garrett and Vladimir Serbinenko for finalising and committing this patch.

Joggler: eMMC hilarity

This is part of my series on running an unmodified Debian on the Joggler. See here for other posts on the same topic.

Since my last post, the helpful folks at Debian have added the missing sdhci_pci module to the installer packages, so now, daily builds of debian-installer do indeed detect the Joggler's internal storage. However, it doesn't quite work right. (shock!)

If you boot up d-i and install to the internal storage, everything will appear to be working fine, and it will install... but if you switch to tty4 where syslog is being displayed, you'll see about a million copies of this message:
mmc2: Too large timeout requested for CMD25!
Every single write to the internal storage causes this message to be printed (CMD25 is the write command). It doesn't prevent the install from working, but when you boot the actual system the console is just filled with this message and you can't see what you're doing. You can change the kernel loglevel so it doesn't spam those messages to the console (adding loglevel=4 to the kernel command line is sufficient here, to disable messages of level warning and below), but they are still getting spewed into klogd, which will, via syslogd... write them to your logfiles.

Yes, you guessed it: this causes more MMC writes, which causes more log spam, which causes more MMC writes... so the system starts spending an unfortunate amount of its resources spiralling its logs out of control. Oh dear. It's not as bad as it could be because the log is buffered, but it's still rather unfortunate. It's possible to hack the logging configuration to "solve" this in various ways, but rather than mess about with that, I wanted to look into why the kernel was printing this message in the first place.

Joggler: Trying debian-installer

This is part of my series on running an unmodified Debian on the Joggler. See here for other posts on the same topic.

Now that I've compiled a GRUB binary that works on the Joggler, I can try booting debian-installer. Being able to run this would be nice, since it means that instead of preparing the system on another machine using debootstrap, you can just install it the usual way. I'm going to be trying to install wheezy, the current testing release; it's too late to fix any issues that come up during a stable install. The development version of debian-installer also has several features that are helpful.

I usually use netboot images to install Debian, though I don't actually boot them from the network. You can boot the netboot kernel and initrd from any existing Linux bootloader, and as long as your network interface is supported by the drivers included, the rest of the install can proceed directly from a Debian mirror, with no need to download an ISO. To try this, you need four things:

1. A USB stick with a small FAT16 partition.
2. An EFI GRUB binary built with the patch from my previous post.
3. The linux and initrd.gz from the d-i daily builds archive, and a grub.cfg which will load them (you don't need any command line arguments).
4. The firmware for the Joggler's wireless interface: download the firmware archive here and extract the firmware-ralink udeb.

Joggler: Improving GRUB

This is part of my series on running an unmodified Debian on the Joggler. See here for other posts on the same topic.

As explained in my last post, the problem with running a standard Debian kernel (or most other prebuild modern Linux kernels) is that GRUB loads them at the "wrong" address - that is, the address that older versions of the Linux kernel boot protocol expected. Because of the Joggler's weird (and possibly broken) EFI memory map, the kernel needs to be loaded using the new boot protocol 2.10, so that GRUB can choose a correctly aligned address that doesn't overlap any of the EFI mapped areas. However, GRUB 1.99 doesn't support this.

I spoke to Matthew Garrett about this problem, who knows a great deal about EFI booting (and about hideous workarounds for broken hardware). He had a patch in progress already to add this support to GRUB, which I experimented with and gave a few tweaks until it worked on the Joggler. I gave him the modified patch back and hopefully he'll be able to get it tested on more platforms and integrated into GRUB proper, but until then, it does work on the Joggler.

So, if you want to build a GRUB that works on the Joggler right now, just grab the GRUB 1.99 tarball and apply my version of Matthew's patch to it. You don't need any other patches; the Joggler GRUB patch on other sites is used to add EFI GOP video support, which is already included in 1.99.

With this patch, a normal modern kernel which requests 16MB alignment and is relocatable will be loaded at 32MB, to avoid the area at 16MB which is in use by EFI. This is aligned, so the kernel won't attempt to move itself at start, but it won't be the address the kernel is compiled to run at so there will be a tiny (seriously, very tiny) amount of time spent at boot applying the relocations to make it run from 32MB. The standard Debian kernel used to run the installer works fine, and if you also include one of the installer initrd images then the installer boots. There are several problems with trying to actually use the installer, however, so while it boots and you can get to a shell to explore the device, I haven't run out of problems to address yet.

So, as far as GRUB is concerned, if this patch (or a derivative of it) can get into the GRUB tree, then future versions of GRUB (including, say, Debian prebuilt binary packages) will work just fine on the Joggler without any special preparation.

UPDATE: This has now happened, see my post here :)

Joggler: Debian kernel doesn't boot

This is part of my series on running an unmodified Debian on the Joggler. See here for other posts on the same topic.

Now that the Joggler is booting rEFIt with a functioning keyboard, experimenting with things to boot is easy. The ideal thing to boot would be the standard Debian kernel. If that works, then pretty much any stock Linux distribution kernel should also work. In order to boot it you need an EFI GRUB. I grabbed a premade binary from the Joggler wiki for now (we'll come back to this later).

Unfortunately, while the GRUB binary worked fine, the kernel doesn't boot, with no indication of why. This is tricky to debug; since the Joggler doesn't have standard VGA, you don't get any console messages on the screen until it's booted far enough to activate efifb, the framebuffer driver which talks to the EFI video output system. It also doesn't have a serial port (at least, not that anyone has found) so you can't get the log messages out that way either, and after experimenting with netconsole (sending console messages over UDP) I concluded that it was failing before initialising networking as well. The only other early boot debugging system that Linux explicitly supports is a weird host-to-host USB debug interface that I don't have and isn't cheap.

So, I grabbed a working Joggler kernel config from one of the Joggler Ubuntu images available on the forum, and built Debian's kernel sources using that configuration. That worked fine, so I bisected the interesting differences between the kernel configurations until I tracked down the single configuration option which stopped it from working: CONFIG_PHYSICAL_ALIGN. This one is going to take some explaining. :)

UPDATE: this is now fixed, see my post here.

Joggler: Installing rEFIt

This is part of my series on running an unmodified Debian on the Joggler. See here for other posts on the same topic.

In the last part I explained how the Joggler boots its retail firmware, and the ways in which the boot process can be interfered with. The first step to getting somewhere is to get a proper bootloader on there; one that can give you access to an EFI shell and tools, and can boot other things without fiddly editing of scripts or pressing ESC repeatedly.

rEFIt is a Mac EFI bootloader which gives you a nice graphical menu of possible EFI programs to boot, and comes with a shell and tools. It's not the ideal bootloader for Linux since it can't directly load a Linux kernel, but it can load an EFI GRUB binary for that. I don't want to rely on rEFIt permanently because I don't know how to build it or where to obtain a binary of guaranteed provenance; this defeats the point of the "booting stock Debian" exercise; but it's a useful tool while messing around with the boot configuration.

Joggler: How it boots

This is part of my series on running an unmodified Debian on the Joggler. See here for other posts on the same topic.

Okay, so the first thing that's important in running something other than the stock software is understanding exactly how the device boots normally, and at what points in that process you can easily/safely interfere.

The Joggler is mostly just a PC in a weird box: it has an Atom processor that runs normal x86 code. However, unlike most x86 computers, it doesn't have a traditional PC-compatible BIOS: it uses EFI. Some very modern PCs boot with EFI, but they generally have BIOS compatibility code that allows them to run traditional bootloaders and operating systems, and EFI is not widely used to actually boot on them just yet. The only platforms I know of where this is already used by large numbers of users are the x86-based Macs.

This means that the early stages of boot proceed very differently to the usual process on a PC, and to make things more annoying, the Joggler doesn't bother to display any hints as to what it's actually doing; the screen just displays a static logo on poweron (either the OpenPeak logo or the O2 logo, depending exactly which hardware variant you have) and leaves it there until the OS has booted far enough to take control of the display and display its own boot animation. As far as I know there's nothing you can press/do to see any actual boot progress; the logo is all you get.

Fortunately, other people already figured out pretty much everything it does during boot, though not all the details seem to be well-explained in the same place. To try and help out with that, here's my explanation of how it boots its factory-shipped OS. I've included notes that compare this to PC BIOS booting in red and notes on how you can encourage it to deviate from the normal boot sequence in blue.

Project intro: Joggler

This is the most recent project I've started working on: I bought a Joggler a couple of weeks ago. It's a weird device that looks like a rather thick digital photo frame, but contains a reasonably complete Atom computer, with the only peripherals being a touchscreen, a single USB port, wifi, ethernet and audio out. It was sold as a "lifestyle organiser", with the intention that it would serve as a hub for a family who used their O2 mobile phones to organise their lives. Nobody was interested, and they ended up selling the remainder of the stock off cheap before abandoning it. Their software is kinda crappy (written in Flash running on Linux) and many of the backend services are broken at this point, so it's not particularly useful running the factory software.

I originally bought it with various random purposes vaguely in mind, but by the time it actually arrived I'd thought of better ways to accomplish several of those things, so right now I'm just fiddling with it for fun.

They are pretty easy to repurpose; there's no secure boot system or anything else protecting the device, so you can just boot your own OS on it. What does "protect" the device is the obscure way in which it boots; more about that in a future post, but it's been worked around by others already, so it's not a real barrier.

These devices are quite popular to use to run other Linux-y stuff, so there's a pretty good community out there (mostly around this wiki, as far as I can tell) who have ported various distributions and applications to run on the device. However, most of the available software distributions for it are based on various collections of patches, nonstandard configurations, and occasionally odd binaries with unclear origins; a standard Linux distribution doesn't quite run on it out of the box. I get irritated by that kind of thing, so my first goal is to get Debian (my preferred distribution for non-desktop uses) running correctly on it with no local modifications. This looks like it will involve submitting various interesting changes to Debian and other upstream projects, so I'm volunteering myself to do that. :)

I have a Trello board here for my tasks to get this to work. I've already made some progress in getting it working with less hackery, as you can see there, and I'll blog about the parts I've done before considering them finished.

insert blog here

I'm going to try blogging about some of the software and gadget-hacking projects I've been working on, instead of just talking about them on IRC. So, this is that.

Over the next couple of days/weeks/months (depending on motivation) I'll talk a bit about the projects I'm currently working on, and then going forward cover stuff that's interesting as I do it.

Hope to get some interesting feedback!