Synchronous virtio-fs yields 30% better performance

Virtio devices normally work asynchronously: the guest submits a request, the host processes it, and an interrupt tells the guest that the result is ready. This gives the guest an opportunity to do something else while the device is busy. But what if the application is just waiting for that one request to finish?

That’s a common situation with filesystem operations, and the reason I’ve implemented experimental synchronous queues for virtio-fs in libkrun. The idea is to let the host thread running the vCPU process the request itself before returning to the guest.

In my tests, this brought a libkrunfw build down from 7m43s to 5m34s, improved synchronous random I/O throughput, and reduced average file deletion latency by 36.1%. Let’s look at the path we’re shortening.

The asynchronous path

Virtio-fs lets a guest access a filesystem shared by the host. Requests and replies travel through virtqueues, queues in guest memory that both sides can access.

With normal queues, a vCPU enqueues a request and kicks the queue. The notification crosses into the host, where the VMM arranges for a filesystem worker to handle it. The vCPU can return to the guest while that work happens. Once the worker has processed the request and published its reply, a completion interrupt tells the guest to collect the result.

This makes sense when there’s useful work to overlap with the I/O. But an application doing a blocking filesystem operation can’t continue until the reply arrives. If the guest has nothing else runnable, the vCPU may go idle while it waits.

Meanwhile, these stages can run on different physical host cores. Handing work between threads, handling the completion interrupt, and waking idle cores all have a cost. With modern cores aggressively entering idle states, the path around a short filesystem operation can be expensive in its own right. This is the motivation described in the proposal.

Keeping the request on the vCPU thread

The new path uses the experimental VIRTIO_FS_F_SYNC feature. The device offers a dedicated synchronous queue for each configured vCPU, in addition to its normal queues. A guest driver that supports the feature negotiates it and uses those queues.

When the guest kicks a synchronous queue, the VM exit takes the vCPU thread directly into the virtio-fs implementation. That same host thread processes the request, performs the host filesystem operation, and publishes the reply. Only then does it resume guest execution. The guest immediately checks the queue and finds the result waiting for it.

Here, the vCPU thread is a host thread executing VMM code. It temporarily stops running guest instructions while it handles the request. The host scheduler can still move that thread between physical cores, and a host filesystem call can still block it.

This also explains the tradeoff: that vCPU can’t run other guest tasks while it’s servicing the request. Short operations with a waiting caller are promising candidates. Long operations, or workloads with plenty of other runnable guest work, may benefit from asynchronous processing.

The implementation keeps the high priority and normal request queues on the worker path. Synchronous queues run their handlers after releasing the bus device lock, so a host filesystem operation doesn’t hold that shared lock throughout the request. Handlers on different synchronous queues can therefore run concurrently on their respective vCPU threads. On KVM, their notifications also bypass ioeventfd registration, ensuring the kick reaches the VMM on the notifying vCPU thread.

Building libkrunfw

For a workload closer to everyday development, I built libkrunfw 5.6.2 inside a VM with 8 vCPUs on an Apple M3 Pro, using make -j8. These are the timings from the benchmark run.

That’s 27.9% less elapsed time, or about 38.7% higher build throughput if expressed as the inverse of the runtime. The elapsed time saving is just over two minutes.

Synchronous random I/O

To exercise blocking I/O more directly, I used fio 3.39 with a mixed random read/write workload, 4 KiB blocks, the pvsync engine, and an I/O depth of one per job. Each configuration ran for 60 seconds, with the best of three runs shown below. The command for one job was:

1
2
3
4

fio --name=virtiofs-test --directory=/root \
    --rw=randrw --bs=4k --size=1G --numjobs=1 \
    --time_based --runtime=60 --group_reporting \
    --ioengine=pvsync --iodepth=1 --direct=1

For the four job test, I changed --numjobs=1 to --numjobs=4.

With one job, read bandwidth improves by 32.4% and write bandwidth by 32.1%. With four jobs, both improve by 24.2%. These comparisons include both the change in request processing and the use of per-vCPU queues; they don’t separate the effects of each.

These are comparisons for this workload and setup. Selecting the best of three runs gives a useful initial result, but doesn’t show run to run variability.

File deletion latency

Finally, I used a synthetic benchmark that first creates 50,000 files, then measures the latency of deleting them with unlinkat in a separate phase. This exercises metadata operations rather than bulk data transfer.

Where this could go

This is still an experimental feature, with the implementation proposed in libkrun PR 933. It needs support in both the VMM and the guest virtio-fs driver to negotiate and use the synchronous queues.

The results make it worth exploring where else the same approach might help. Read requests on virtio-blk are one possibility: whenever the caller must wait, the cost of handing the work away and signalling its completion deserves a closer look.

The benchmark data includes the original measurements and the PR revision used for these comparisons.