Enabling KVM virtualization on ARM (Allwinner A20)
Some time ago, ARM Holdings presented the new virtualization extensions for its processor architecture, which are now present on some models of the Cortex family, like the Cortex-A7 and Cortex-A15. Though it’s a quite recent technology, both KVM and Xen hypervisors already support such extensions, allowing to run virtualized Guests in the same way you can already do on x86.
It’s true that current SoCs (System-on-Chip) and development boards doesn’t provide a number of cores and RAM memory that invite to run a significant number of Guests on them, but these are the first steps towards stabilization of ARM virtualization, paving the way for the future server-oriented ARM processors. On the other hand, this is also an interesting option for running alternative operating systems (like the *BSD family) on ARM hardware, without dealing with the extremely heterogeneous nature of it.
In this guide, where going to see how you can enable KVM virtualization on the Olinuxino-A20-MICRO development board.
About the Olinuxino-A20-MICRO board
This is one of the latest ARM-based boards produced by Olimex, an European company specialized in electronics and devoted to Open Hardware. This board is powered by an Allwinner A20 SoC with a dual-core Cortex-A7 processor and 1 GB of RAM memory.
This is pretty similar to the popular Cubieboard2, and most of the steps described here also can be applied to this one with minor variations.
Requirements
* An OlinuXino-A20-MICRO board (it should work with Olinuxino-A20-LIME too, but I haven't tested it).
* A 4GB (or bigger) SD card.
* (OPTIONAL but encouraged) An USB to UART adapter. The one sold by Olimex ([USB-Serial-Cable-F](https://www.olimex.com/Products/Components/Cables/USB-Serial-Cable/USB-Serial-Cable-F/)) works nicely.
* A PC with a recent GNU/Linux distribution.
Step 1: Installing an ARM cross-compiler
To be able to compile u-boot and Linux, we need a cross-compiler for ARM with EABI and Hard Float options. Most recent distributions provide this king of cross-compiler and all its dependencies on their repositories. In our case, using Ubuntu 14.04, the main package is called gcc-arm-linux-gnueabihf:
slp@ubuntu1404:~$ sudo apt-get install gcc-arm-linux-gnueabihf
Step 2: Compiling the boot loader (u-boot)
When we power our board, the initial boot loader, residing in SoC’s internal memory, searches for a secondary boot loader on the SD card.
The official Debian images provided by Olimex already provide this boot loader, but we need to replace it for a version which detects when the CPU is on HYP mode, and avoids dropping to SVC32 (a requirement for KVM initialization).
We create a directory to store all the software we’re going to build, and then we get the u-boot code from the appropriate git repository:
slp@ubuntu1404:~$ mkdir kvm-arm
slp@ubuntu1404:~$ cd kvm-arm/
slp@ubuntu1404:~/kvm-arm$ git clone -b sunxi-next --depth=1 git://github.com/jwrdegoede/u-boot-sunxi.git
Clonar en «u-boot-sunxi»...
remote: Counting objects: 9155, done.
remote: Compressing objects: 100% (7854/7854), done.
remote: Total 9155 (delta 1708), reused 3512 (delta 1113)
Receiving objects: 100% (9155/9155), 13.73 MiB | 1.58 MiB/s, done.
Resolving deltas: 100% (1708/1708), done.
Checking connectivity... hecho.
Then we build u-boot, specifying the configuration for our board, and the cross-compiler prefix:
slp@ubuntu1404:~/kvm-arm/u-boot-sunxi$ make CROSS_COMPILE=arm-linux-gnueabihf- A20-OLinuXino_MICRO_config
Configuring for A20-OLinuXino_MICRO - Board: sun7i, Options: A20_OLINUXINO_M,CONS_INDEX=1,STATUSLED=226,SPL,SUNXI_EMAC
slp@ubuntu1404:~/kvm-arm/u-boot-sunxi$ make CROSS_COMPILE=arm-linux-gnueabihf-
If we’re building on a multicore machine, we should consider adding the option -j4 for compiling in parallel.
If the build ends successfully, we’ll have a file named u-boot-sunxi-with-spl.bin, ready to be written to the SD card.
Step 3: Preparing an startup script
To avoid the need of manually entering all the commands on each boot, u-boot searches for a file named boot.scr on various block devices, and loads it if found. This file contains a list of commands to be run one by one, in binary format. We’re going to create our own startup script with the appropriate commands.
On the u-boot-sunxi, open your favorite text editor and create the file boot.cmd, with the following contents:
setenv kernel_addr_r 0x46000000 # 8M
setenv fdt_addr 0x49000000 # 2M
setenv fdt_high 0xffffffff # Load fdt in place instead of relocating
fatload mmc 0 ${kernel_addr_r} /uImage
setenv bootargs "console=ttyS0,115200 ro root=/dev/mmcblk0p2 rootwait"
fatload mmc 0 ${fdt_addr} /sun7i-a20-olinuxino-micro.dtb
bootm ${kernel_addr_r} - ${fdt_addr}
Save the file, and create its binary counterpart with mkimage:
slp@ubuntu1404:~/kvm-arm/u-boot-sunxi$ ./tools/mkimage -C none -A arm -T script -d boot.cmd boot.scr
Image Name:
Created: Wed Jul 23 13:48:35 2014
Image Type: ARM Linux Script (uncompressed)
Data Size: 365 Bytes = 0.36 kB = 0.00 MB
Load Address: 00000000
Entry Point: 00000000
Contents:
Image 0: 357 Bytes = 0.35 kB = 0.00 MB
Now we have our startup script on the file boot.scr.
Paso 4: Building the kernel
As its sadly common in the ARM world, the SoC maker (Allwinner, in this case), releases an initial version of Linux with support for a certain model, updates it for a few months, and then stops maintaining it. It doesn’t invest any resources into trying to get the support pushed to mainline Linux neither. The result is that kernel support for such SoC gets “stuck” on a certain Linux version, lacking support for newer features, like KVM.
The goods news is that some hackers at the linux-sunxi community have been working on getting support for Allwinner SoCs into mainline. At the moment of writing this, Linux 3.16 already supports a big part of the hardware, and also features KVM support, among other cool stuff.
So let’s get the latest kernel (3.16-rc6 in our case) and uncompress it on the previously created sources directory:
slp@ubuntu1404:~/kvm-arm$ wget https://www.kernel.org/pub/linux/kernel/v3.x/testing/linux-3.16-rc6.tar.xz
slp@ubuntu1404:~/kvm-arm$ tar xf linux-3.16-rc6.tar.xz
Next, we create a default configuration file, and launch the kernel configuration menu (if this last step fails, you’ll probably need to install the libncurses5-dev package, or the equivalent on your distributions):
slp@ubuntu1404:~/kvm-arm$ cd linux-3.16-rc6/
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make CROSS_COMPILE=arm-linux-gnueabihf- ARCH=arm sunxi_defconfig
#
# configuration written to .config
#
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make CROSS_COMPILE=arm-linux-gnueabihf- ARCH=arm menuconfig
On the configuration menu, enable to following options:
* `General setup -> Control Group support`
* `System Type -> Support for Large Physical Address Extension`
* `Device Drivers -> Block devices -> Loopback device support`
* `Virtualization`
* `Virtualization -> Kernel-based Virtual Machine (KVM) support (NEW)`
Exit from menuconfig, saving the new configuration.
Finally, launch a build for kernel and DTBs (Flattened Device Tree):
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ export PATH=$PATH:/home/slp/kvm-arm/u-boot-sunxi/tools
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- LOADADDR=0x40008000 uImage dtbs
This will take a few minutes (less that the usual x86 kernel build, due to the smaller number of options built in). When finished, you’ll have the kernel (arch/arm/boot/uImage) and the DTB (arch/arm/boot/dts/sun7i-a20-olinuxino-micro.dtb).
Step 5: Preparing the SD card
Now that we have all the pieces ready, we need to prepare an SD card for booting our new kernel. If you already have an SD with the official Debian image from Olimex, you can reuse it. If this is not your case, download it from the Olinuxino-A20-MICRO page and dump it to the SD card.
Once you have an SD card with such image, write the new u-boot to it (we assume the device block for the SD card is mmcblk0, you should check which one is in your case):
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ cd ~/kvm-arm
slp@ubuntu1404:~/kvm-arm$ sudo dd if=u-boot-sunxi/u-boot-sunxi-with-spl.bin of=/dev/mmcblk0 bs=1k seek=8
Next, mount the first partition and copy the startup script, the kernel and the DTB:
slp@ubuntu1404:~/kvm-arm$ sudo mount /dev/mmcblk0p1 /mnt
slp@ubuntu1404:~/kvm-arm$ sudo cp u-boot-sunxi/boot.scr /mnt
slp@ubuntu1404:~/kvm-arm$ sudo cp linux-3.16-rc6/arch/arm/boot/uImage /mnt
slp@ubuntu1404:~/kvm-arm$ sudo cp linux-3.16-rc6/arch/arm/boot/dts/sun7i-a20-olinuxino-micro.dtb /mnt
slp@ubuntu1404:~/kvm-arm$ sudo umount /mnt
Now, the SD card is ready for the first boot.
Step 6: First boot and QEMU installation
If you have the USB-to-UART cable, when booting you’ll be able to see the messages coming from u-boot and Linux. If you don’t have one, wait a few minutes and try connecting with SSH (we assume you’ve already configured your network before).
To be able to launch our first Guest, we need a recent QEMU version. The one that comes with Debian “wheezy” is quite old, so you have to build a newer one:
Download the build dependencies:
slp@a20:~$ sudo apt-get build-dep qemu
slp@a20:~$ sudo apt-get install libpixman-1-dev
Then, create a directory for the sources, download QEMU 2.0 and uncompress it:
slp@a20:~$ mkdir srcs
slp@a20:~$ cd srcs
slp@a20:~/srcs$ wget http://wiki.qemu-project.org/download/qemu-2.0.0.tar.bz2
--2014-07-24 06:28:00-- http://wiki.qemu-project.org/download/qemu-2.0.0.tar.bz2
Resolving wiki.qemu-project.org (wiki.qemu-project.org)... 140.211.15.109
Connecting to wiki.qemu-project.org (wiki.qemu-project.org)|140.211.15.109|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12839647 (12M) [application/x-bzip2]
Saving to: `qemu-2.0.0.tar.bz2'
100%[======================================>] 12,839,647 2.08M/s in 8.2s
2014-07-24 06:28:09 (1.49 MB/s) - `qemu-2.0.0.tar.bz2' saved [12839647/12839647]
slp@a20:~/srcs$ tar xf qemu-2.0.0.tar.bz2
Now run the configure script with options for enabling KVM (it is enabled automatically if supported, but this way we ensure we get warned if it’s not going to be build with it) and for building the ARM emulation target only:
slp@a20:~/srcs$ cd qemu-2.0.0/
slp@a20:~/srcs/qemu-2.0.0$ ./configure --enable-kvm --target-list=arm-softmmu
Finally, build and install it (by default, the new binaries will reside in /usr/local/bin):
slp@a20:~/srcs/qemu-2.0.0$ make
slp@a20:~/srcs/qemu-2.0.0$ sudo make install
Step 7: Building a kernel for the Guest
When running QEMU with KVM, the hardware emulated is a Versatile Express A15, one the reference platforms provided by ARM Holdings. So, for our Guest we need a kernel which supports this board.
Return to the Linux directory we used on Step 3, load the default configuration for Versatile Express A15, and launch menuconfig:
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- vexpress_defconfig
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig
Enable these options:
* `General setup -> Configure standard kernel features (expert users)`
* `General setup -> open by fhandle syscalls`
* `Enable the block layer -> Support for large (2TB+) block devices and files`
And generate a kernel in zImage format:
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage
Step 8: Running our first Guest
In addition to the kernel, we also need a root filesystem with the distribution we want to use for our userland. As we’re going to virtualize an ARMv7 CPU, the best option is using a earmv7hf distribution. In this guide, we’re going to use a minimal OpenSuSE image (JeOS), but you can choose the one of your preference.
Create a directory with the files needed by the Guest:
slp@a20:~/srcs/qemu-2.0.0$ mkdir -p ~/kvm-arm/opensuse
slp@a20:~/srcs/qemu-2.0.0$ cd ~/kvm-arm/opensuse
slp@a20:~/kvm-arm/opensuse$
Copy the kernel built on Step 7, and the DTB for the Versatile Express A15:
slp@a20:~/kvm-arm/opensuse$ scp slp@10.x.x.x:~/kvm-arm/linux-3.16-rc6/arch/arm/boot/zImage .
slp@10.x.x.x's password:
zImage 100% 2558KB 2.5MB/s 00:00
slp@a20:~/kvm-arm/opensuse$ scp slp@10.x.x.x:~/kvm-arm/linux-3.16-rc6/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dtb .
slp@10.x.x.x's password:
vexpress-v2p-ca15-tc1.dtb 100% 12KB 12.0KB/s 00:00
Now we create a raw image with OpenSuSE’s userland:
slp@a20:~/kvm-arm/opensuse$ wget http://download.opensuse.org/ports/armv7hl/factory/images/openSUSE-Factory-ARM-JeOS.armv7-rootfs.armv7l-1.12.1-Build175.2.tbz
slp@a20:~/kvm-arm/opensuse$ qemu-img create -f raw opensuse-factory.img 1G
Formatting 'opensuse-factory.img', fmt=raw size=1073741824
slp@a20:~/kvm-arm/opensuse$ sudo losetup /dev/loop0 opensuse-factory.img
[sudo] password for slp:
slp@a20:~/kvm-arm/opensuse$ sudo mkfs.ext4 /dev/loop0
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
slp@a20:~/kvm-arm/opensuse$ sudo mount /dev/loop0 /mnt
slp@a20:~/kvm-arm/opensuse$ sudo tar xf openSUSE-Factory-ARM-JeOS.armv7-rootfs.armv7l-1.12.1-Build175.2.tbz -C /mnt
slp@a20:~/kvm-arm/opensuse$ sudo umount /mnt
slp@a20:~/kvm-arm/opensuse$ sudo losetup -d /dev/loop0
And finally, launch your first Guest:
slp@a20:~/kvm-arm/opensuse$ sudo qemu-system-arm -enable-kvm -m 512m -M vexpress-a15 -cpu host -kernel zImage -dtb vexpress-v2p-ca15-tc1.dtb -append "root=/dev/vda console=ttyAMA0 rootwait" -drive if=none,file=opensuse-factory.img,id=factory -device virtio-blk-device,drive=factory -net nic -net user -monitor null -serial stdio -nographic
Booting Linux on physical CPU 0x0
Initializing cgroup subsys cpuset
Linux version 3.16.0-rc6 (slp@ubuntu1404) (gcc version 4.8.2 (Ubuntu/Linaro 4.8.2-16ubuntu4) ) #12 SMP Thu Jul 24 11:48:10 CEST 2014
CPU: ARMv7 Processor [410fc074] revision 4 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine model: V2P-CA15
Memory policy: Data cache writealloc
PERCPU: Embedded 7 pages/cpu @9fbdf000 s7552 r8192 d12928 u32768
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: root=/dev/vda console=ttyAMA0 rootwait
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 513156K/524288K available (4604K kernel code, 188K rwdata, 1284K rodata, 235K init, 150K bss, 11132K reserved)
...
[ OK ] Started OpenSSH Daemon.
[ OK ] Reached target Multi-User System.
[ OK ] Reached target Graphical Interface.
Welcome to openSUSE Factory "Bottle" - Kernel 3.16.0-rc6 (ttyAMA0).
linux login:
Now we can start playing with this Guest. The default credentials are root/linux:
linux login: root
Password: linux
Last login: Thu Jul 24 11:57:53 on ttyAMA0
Have a lot of fun...
linux:~ # uname -a
Linux linux 3.16.0-rc6 #12 SMP Thu Jul 24 11:48:10 CEST 2014 armv7l armv7l armv7l GNU/Linux
linux:~ # cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 4 (v7l)
Features : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xc07
CPU revision : 4
Hardware : ARM-Versatile Express
Revision : 0000
Serial : 0000000000000000
That’s all, now you’re ready for ARM virtualization! ;-)