Enabling HYP mode on the Raspberry Pi 2

The newest iteration of the wonderful machine designed by Raspberry Pi Foundation, the Raspberry Pi 2, sports a Broadcom BCM2836 SoC, with four Cortex-A7 cores. The Cortex-A7, being the little brother Cortex-A15, features the ARM Virtualization Extensions, so both Xen and KVM based virtualization should work on it.

At this point, you probably are wondering why would someone want to use virtualization on a RPi2. In addition to the usual “because you can!” answer, there’s a pretty good reason for it. Imagine you want to use the RPi2 as a media center and, at the same time, you want to run some personal services (like ownCloud or Pydio) on it. Instead of polluting the media center image, you can run an isolate, secure, virtual machine for such purpose. And, using my VEXPRESS_KVM port, you can even provide those services running NetBSD! ;-)

The first step towards being able to use virtualization on the Raspberry Pi 2, is finding a way to boot the kernel in HYP mode. Let’s see how can we do that.

HYP mode and the Raspberry Pi 2 bootloader

To be able to use the Virtualization Extensions on the RPi2 (and, by extension, on any ARM based machine), the bootloader needs to jump into HYP mode before passing the control to Linux (this is generally done at the same time the bootloader jumps from secure to non-secure world). If that’s not the case, as happens with the default RPi2’s bootloader, Linux will print this message, and no virtualization support will be available:

[    0.154021] CPU: All CPU(s) started in SVC mode.

Luckily, this post in the forums points that it’s possible to obtain control of the board when it’s still in secure mode, by setting the kernel_old=1 option in config.txt. So we just need to build a bootloader which properly sets HYP mode before jumping to 0x8000, and then bundle it our Linux’s zImage.

The easy way. Using a prebuilt image

I’ve uploaded a prebuilt image here (md5sum: 356788d260e1a93376c5b8acbb63da13), which contains the bootloader and ~32k of padding (to be sure the zImage starts at 0x8000). You just need to concatenate it with your current kernel (save a copy first!):

mv /boot/kernel7.img /boot/kernel7.img.bak
cat bootblk.bin /boot/kernel7.img.bak > /boot/kernel7.img

Then you’ll need to add the kernel_old=1 option to your config.txt:

echo "kernel_old=1" >> /boot/config.txt

That’s it! On the next boot, Linux should say something like this:

[    0.154131] CPU: All CPU(s) started in HYP mode.
[    0.154158] CPU: Virtualization extensions available.

Building it yourself

If you want to build the bootloader by yourself, you’ll need to grab an ARM cross toolchain. I’m using the one from the GCC ARM Embedded project, which works just fine. Then add it to your $PATH, grab the code for my GitHub’s repo, and build it:

export PATH=$PATH:/home/slp/sources/gcc-arm-none-eabi-4_9-2014q4/bin
git clone https://github.com/slp/rpi2-hyp-boot.git
cd rpi2-hyp-boot
make

That should produce a bootblk.bin, with contains the boot code and 32k of padding.

Next steps

With this custom bootloader, we’ve been able to boot Linux in HYP mode, but there’s still some work to be done to be able to run our first guest. The RPi2’s bcm2836 SoC doesn’t provide a GIC (Generic Interrupt Controller), which would provide a very helpful vGIC (Virtual GIC) interface, so we need to find a way to provide some mechanism for interrupting the guest (for rescheduling or servicing an IRQ) without relying on it.

As the documentation for the bcm2836/bcm2709 is nowhere to be found, this is somewhat harder, but I hope we’ll find a way.