Using microVMs for Gaming on Fedora Asahi

It’s been almost a year since I transitioned from the Virtualization to the Automotive team at Red Hat with the goal of ensuring RHIVOS ships with a powerful Virtualization stack. While there’s a large overlap between a Virtualization stack for Servers and the one for Automotive platforms, the latter is much more demanding on one particular aspect: GPU acceleration.

For me, personally, that meant having to delve into the Linux graphics stack, both kernel (DRM, GEM, KMS…) and userspace (Mesa, virglrenderer…), something in which, so far, I only had a superficial knowledge. And, when facing a new field, my preferred approach is looking for an interesting project that would both motivate me to play with those software components.

While looking for such project, I came across Rob Clark’s XDC 2022 talk about Virtgpu DRM native context, a novel technique that brings close-to-native GPU performance to VMs, which he implemented for the freedreno Mesa driver. This is something definitely worth exploring, but which project could I spin off from it?

A page size problem in Asahi Linux

Then I remembered how, months ago, Eric Curtin asked to me if it would be possible to use libkrun to address one particular problem Asahi Linux was facing. Apple Silicon devices are 16K page size systems, meaning they require running a Linux kernel configured for that page size (Linux is not very flexible in this regard).

Running a 16K page size kernel implies some userspace applications that assume that the page size is 4K will break. While the Asahi Team did an amazing work fixing many of them, there’s little anyone can do (at least, at a scale) to fix applications that ship in binary form. And the prime example of this are x86_64 (a 4K page size platform) games.

While it’s technically possible to run Apple Silicon devices with a 4K page size kernel, that would require a number of (potentially controversial upstream) changes in the kernel and will probably have a significant impact on the performance. A better approach would be running the host with a 16K kernel, and use a 4K kernel in a VM for those problematic workloads, as long the performance of the VM was good enough for the use case.

When Eric first asked me about this, my first reaction was “well, we would need to import virtio-gpu, but even then the performance wouldn’t be good enough for gaming”. But, what if we could bring Rob Clark’s DRM native context to Asahi?

Virtgpu DRM native context for Asahi

The effort to accomplish this project is quite significant and involves changes in multiple components. On one hand, we need to implement the equivalent of Rob Clark’s work for freedreno but for the asahi/agx drivers in both Mesa in virglrenderer.

On the other, we need to import virtio-gpu+rutabaga_gfx into libkrun. Later I found Steam didn’t play nice with TSI, but luckily Matej Hrica was already working on virtio-net+passt support. While not strictly needed, supporting virtio-snd would nice to have audio support (which is quite important for gaming ;-).

Given the amount of work needed and the significant grade of uncertainty surrounding the project, driven my lack of knowledge of the graphics stack, I decided to do first a quick’n’dirty Proof-of-Concept, which turned out to be quite successful and an amazing learning opportunity.

Current status of DRM native context for Asahi

On the libkrun front, virtio-net support has been merged, and we have a PR for virtio-gpu. I also have a branch with virtio-snd support that will be soon ready to be reviewed.

The changes for Mesa and virglrenderer still need some polishing, but are getting very close to be ready for a draft MR.

Meanwhile, since there are many moving pieces, I put together some binary packages in a COPR repository and an OCI image of Fedora 38 with FEX-Emu (an great piece of software that enables users to run x86_64 binaries on aarch64 with pretty good performance) that can be consumed by krunvm. This allows me to have a stable base to develop and test each component individually. But, since those repositories are public, you can also give it a try if you want.

Trying out DRM native context for Asahi

UPDATE 1: I’ve just found that, for the moment, this only works while running Plasma on the host. With GNOME you’ll get this error: MESA: error: failed to map handle: Invalid argument.

UPDATE 2: As expected, the root cause was a 16K misalignment. For Plasma, the keymap blob’s size is 16K page aligned, while in GNOME isn’t. Forcing the alignment in sommelier fixes the issue. If you’re following this instructions for the first time, you don’t need to worry, I’ve updated the OCI image to include the fix. Otherwise, you can update the package in the VM.

UPDATE 3: Teoh Han Hui noticed the issue with passt and SELinux is already fixed, meaning it’s no longer necessary disabling the latter.

Please note this code is very immature and the most likely outcome is that nothing will work for you, so adjust your expectations accordingly.

This assumes you have a fresh Fedora Asahi Linux installation on your Apple Silicon system. I’ve only tested this one on my MacBook Air M1, but AFAIK this code isn’t tied to this particular model in any way.

• Enable the COPR repo, install krunvm (which will bring libkrun and libkrunfw) and upgrade virglrenderer:

1
2
3

$ sudo dnf copr enable slp/asahi-krun fedora-38-aarch64
$ sudo dnf install krunvm
$ sudo dnf install --allowerasing virglrenderer-1.0.0~asahikrun

• Temporarily disable SELinux to work around a problem with passt and newer kernels. This change is non-persistent, so you either need to do it after each reboot, or edit /etc/selinux/config to make the change persistent (not recommended):

• Create a new microVM with 6 vCPUs and 6 GB of RAM (my MacBook only has 8 GB in total, you can choose to use a larger number if your system has more memory available):

1
2

$ buildah unshare
# krunvm create --cpus 6 --mem 6144 --name asahi-krun quay.io/slopezpa/asahi-krun

• Start the microVM and test OpenGL acceleration with glmark2 (you should get around 270 FPS on the terrain benchmark):

1
2
3

# krunvm start asahi-krun
asahi-krun$ glmark2 -b terrain:duration=20 --annotate
asahi-krun$ exit

• Start the microVM again, this time using /bin/bash as entrypoint to avoid switching into an unprivileged used in the guest, and install Steam:

1
2
3
4
5

# krunvm start asahi-krun /bin/bash
asahi-krun# dhclient
asahi-krun# curl -OL https://mirrors.cloud.tencent.com/rpmfusion/nonfree/fedora/steam/38/x86_64/s/steam-1.0.0.78-1.fc38.i686.rpm
asahi-krun# rpm2cpio steam-1.0.0.78-1.fc38.i686.rpm | cpio -idvm
asahi-krun# exit

• Start the microVM, this time using the default entrypoint, and run Steam:

1
2

# krunvm start asahi-krun
asahi-krun$ FEXBash /usr/bin/steam

If you’re lucky enough, you should eventually get greeted by the Steam login screen. If you have 2FA enabled and have trouble login in, with the authentication session expiring before you approve it, try clicking on “Enter a code instead”.

After a reboot, you should be able to start Steam again by doing this:

1
2
3

$ buildah unshare
# krunvm start asahi-krun
asahi-krun$ FEXBash /usr/bin/steam

If this happens to work for you, and you’re able to run some game with decent performance, please consider sharing your joy with me on Mastodon or X. Thanks!