As I wrote on my previous post, Enabling HYP mode on the Raspberry Pi 2, the newest machine from the Raspberry Pi Foundation features a Cortex-A7 with Virtualization Extensions, but it isn’t possible to make use of such feature out of the box.
In that article I showed that it was possible to start the kernel in HYP mode. Now, I’ll cover the rest of steps needed for enabling KVM virtualization and running your first guest OS.
Isolating a core
With KVM on ARM, interrupts are dealt in a very particular way. If a hardware interrupt is received when a core in running in guest mode, the execution is interrupted, a full world switch takes place returning to the kernel in host mode, then interrupts are reenabled and the core traps again, this time following the usual route.
Apparently, the BCM2836 (the core of the RPi2) doesn’t like the behavior, and if you try to run a guest on a core that can receive physical interrupts, you’ll find your RPi2 hangs within a few seconds. Without a JTAG of a full development board, debugging this problem is very, very hard.
Luckily, there’s a pretty simple workaround. Using the kernel option
isolcpus when can isolate a core, so Linux doesn’t assign any tasks (including IRQ lines) to it.
In our case, we’re going to isolate the core number 3 (starting from zero). This is pretty straightforward, just edit
/boot/cmdline.txt and add
isolcpus=3 at the of the line.
A kernel with VGIC emulation for the RPi2
On my previous article about RPi2 emulation, I wrote about the BCM2836 and its lack of a GIC, which makes emulation a bit harder. To work around this issue, I’ve implemented VGIC emulation inside the Linux kernel.
To be able to use both the customized bootloader and this modified version of the kernel, we need to generate a bundle with both components, and the dtb concatenated at end.
The easy way. Using a prebuilt image
I’ve uploaded a prebuilt image here (md5sum: 356788d260e1a93376c5b8acbb63da13), which contains the bootloader, some padding (to put the
0x8000), the kernel and the dtb. Simply replace your kernel with it (save a copy first!):
mv /boot/kernel7.img /boot/kernel7.img.bak cp kernel7.img /boot
Then you’ll need to add the
kernel_old=1 option to your
echo "kernel_old=1" >> /boot/config.txt
That’s it! On the next boot, Linux should say something like this:
[ 0.154131] CPU: All CPU(s) started in HYP mode. [ 0.154158] CPU: Virtualization extensions available.
Building it yourself (Part I: the bootloader)
If you want to build the image by yourself, you’ll need to grab an ARM cross toolchain. I’m using the one from the GCC ARM Embedded project, which works just fine. Then add it to your
$PATH, grab the code for my repo, and build it:
export PATH=$PATH:/home/slp/sources/gcc-arm-none-eabi-4_9-2014q4/bin git clone https://github.com/slp/rpi2-hyp-boot.git cd rpi2-hyp-boot make
That should produce a
bootblk.bin, with contains the boot code and
32k of padding.
Building it yourself (Part II: kernel and dtb)
You can find a complete guide for building the kernel on a variety of host systems here Raspberry Pi Kernel Compilation. Read that and the grab the sources from my repo and make sure the following options are enabled:
Patch physical to virtual translations at runtime
General setup -> Control Group support
System Type -> Support for Large Physical Address Extension
Boot options -> Use appended device tree blob to zImage (EXPERIMENTAL)
Boot options -> Supplement the appended DTB with traditional ATAG information
Device Drivers -> Block devices -> Loopback device support
Virtualization -> Kernel-based Virtual Machine (KVM) support (NEW)
Virtualization -> KVM support for Virtual GIC
Virtualization -> KVM support for Emulated GIC
Now build both kernel and dtb:
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- zImage dtbs
Building it yourself (Part III: bundling all pieces)
This is the easy part, just concatenate them:
cat rpi2-hyp-boot/bootblk.bin linux/arch/arm/boot/zImage linux/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb > kernel7.img
Now you can use it in the same way as the prebuilt image provided above.
Patched QEMU installation
To be able to launch our first Guest, we need a recent and patched (to ensure the guest is running in the isolated core) QEMU. The one that comes with Raspbian is quite old, so you have to build a newer one:
Download the build dependencies:
pi@raspberrypi:~$ sudo apt-get build-dep qemu pi@raspberrypi:~$ sudo apt-get install libpixman-1-dev
Then, create a directory for the sources, download QEMU 2.2 and uncompress it:
pi@raspberrypi:~$ mkdir srcs pi@raspberrypi:~$ cd srcs pi@raspberrypi:~/srcs$ wget http://wiki.qemu-project.org/download/qemu-2.2.0.tar.bz2 pi@raspberrypi:~/srcs$ tar xf qemu-2.2.0.tar.bz2
We need to apply this patch to make sure the QEMU runs our guest in the core we isolated with option
pi@raspberrypi:~/srcs$ cd qemu-2.2.0/ pi@raspberrypi:~/srcs/qemu-2.2.0$ patch -p1 < ~/qemu-cpu-affinity.patch
Now run the
configure script with options for enabling KVM (it is enabled automatically if supported, but this way we ensure we get warned if it’s not going to be build with it) and for building the ARM emulation target only:
pi@raspberrypi:~/srcs/qemu-2.2.0$ ./configure --enable-kvm --target-list=arm-softmmu
Finally, build and install it (by default, the new binaries will reside in
pi@raspberrypi:~/srcs/qemu-2.2.0$ make pi@raspberrypi:~/srcs/qemu-2.2.0$ sudo make install
A kernel for the Guest
When running QEMU with KVM, the hardware emulated is a Versatile Express A15, one the reference platforms provided by ARM Holdings. So, for our Guest we need a kernel which supports this board, and the corresponding dtb.
If you want to build the kernel yourself, you’ll need a vanilla kernel (the one from RPi2’s repo wouldn’t build). In case of doubt, just grab the lastest stable from kernel.org. Use the default configuration for Vestatile Express A15 (
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- vexpress_defconfig) and make sure you enable these options:
General setup -> Configure standard kernel features (expert users)
General setup -> open by fhandle syscalls
Enable the block layer -> Support for large (2TB+) block devices and files
Then build both kernel and dtb:
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- zImage dtbs
This will generate
linux/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dtb. Copy both files to your Raspberry.
Running our first Guest
In addition to the kernel, we also need a root filesystem with the distribution we want to use for our userland. As we’re going to virtualize an ARMv7 CPU, the best option is using a earmv7hf distribution. In this guide, we’re going to use a minimal OpenSuSE image (JeOS), but you can choose the one of your preference.
Create a directory with the files needed by the Guest:
pi@raspberrypi:~$ mkdir -p ~/kvm-arm/opensuse pi@raspberrypi:~$ cd ~/kvm-arm/opensuse pi@raspberrypi:~/kvm-arm/opensuse$
Now we create a raw image with OpenSuSE’s userland:
pi@raspberrypi:~/kvm-arm/opensuse$ wget http://download.opensuse.org/ports/armv7hl/factory/images/openSUSE-Factory-ARM-JeOS.armv7-rootfs.armv7l-Current.tbz pi@raspberrypi:~/kvm-arm/opensuse$ qemu-img create -f raw opensuse-factory.img 1G Formatting 'opensuse-factory.img', fmt=raw size=1073741824 pi@raspberrypi:~/kvm-arm/opensuse$ sudo losetup /dev/loop0 opensuse-factory.img pi@raspberrypi:~/kvm-arm/opensuse$ sudo mkfs.ext4 /dev/loop0 mke2fs 1.42.5 (29-Jul-2012) Discarding device blocks: done Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 65536 inodes, 262144 blocks 13107 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=268435456 8 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done pi@raspberrypi:~/kvm-arm/opensuse$ sudo mount /dev/loop0 /mnt pi@raspberrypi:~/kvm-arm/opensuse$ sudo tar xf openSUSE-Factory-ARM-JeOS.armv7-rootfs.armv7l-Current.tbz -C /mnt pi@raspberrypi:~/kvm-arm/opensuse$ sudo umount /mnt pi@raspberrypi:~/kvm-arm/opensuse$ sudo losetup -d /dev/loop0
And finally, launch your first Guest:
pi@raspberrypi:~/kvm-arm/opensuse$ sudo qemu-system-arm -enable-kvm -smp 1 -m 256 -M vexpress-a15 -cpu host -kernel /home/pi/vexpress-zImage -dtb /home/pi/vexpress-v2p-ca15-tc1.dtb -append "root=/dev/vda console=ttyAMA0 rootwait" -drive if=none,file=/home/pi/opensuse-factory.img,id=factory -device virtio-blk-device,drive=factory -net nic,macaddr=02:fd:01:de:ad:34 -net tap -monitor null -serial stdio -nographic audio: Could not init `oss' audio driver Booting Linux on physical CPU 0x0 Initializing cgroup subsys cpuset Linux version 3.19.1 (slp@linux-ni2o) (gcc version 4.5.3 (GCC) ) #1 SMP Wed Mar 18 10:52:22 CET 2015 CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache Machine model: V2P-CA15 Memory policy: Data cache writealloc PERCPU: Embedded 9 pages/cpu @8fddd000 s7232 r8192 d21440 u36864 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 65024 Kernel command line: root=/dev/vda console=ttyAMA0 rootwait PID hash table entries: 1024 (order: 0, 4096 bytes) Dentry cache hash table entries: 32768 (order: 5, 131072 bytes) Inode-cache hash table entries: 16384 (order: 4, 65536 bytes) Memory: 252936K/262144K available (4839K kernel code, 184K rwdata, 1324K rodata, 256K init, 147K bss, 9208K reserved, 0K cma-reserved) ... [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Welcome to openSUSE Factory "Tumbleweed" - Kernel 3.19.1 (ttyAMA0). linux login:
Now we can start playing with this Guest. The default credentials are
linux login: root Password: Last login: Tue Mar 17 18:01:41 on ttyAMA0 Have a lot of fun... linux:~ # cat /proc/cpuinfo processor : 0 model name : ARMv7 Processor rev 5 (v7l) BogoMIPS : 38.40 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xc07 CPU revision : 5 Hardware : ARM-Versatile Express Revision : 0000 Serial : 0000000000000000
And that’s it, now you can extract the full potential of your Raspberry Pi 2!
Speaking for myself, I’m going to work on getting MMIO support on NetBSD (without this, the performance is really awful), and then I’ll be able to build an hybrid Linux/NetBSD SD card image, which was my initial motivation for all this work 😉