# Synchronous virtio-fs yields 30% better performance


Virtio devices normally work asynchronously: the guest submits a request, the
host processes it, and an interrupt tells the guest that the result is ready.
This gives the guest an opportunity to do something else while the device is
busy. But **what if the application is just waiting for that one request to
finish?**

That's a common situation with filesystem operations, and the reason I've
implemented **experimental synchronous queues for virtio-fs in
[libkrun](https://github.com/libkrun/libkrun/pull/933)**. The idea is to let the
host thread running the vCPU process the request itself before returning to
the guest.

In my tests, this brought a libkrunfw build down from **7m43s to
5m34s**, improved synchronous random I/O throughput, and reduced average
file deletion latency by **36.1%**. Let's look at the path we're shortening.

<!--more-->

## The asynchronous path

Virtio-fs lets a guest access a filesystem shared by the host. Requests and
replies travel through *virtqueues*, queues in guest memory that both sides
can access.

With normal queues, a vCPU enqueues a request and kicks the queue. The
notification crosses into the host, where the VMM arranges for a filesystem
worker to handle it. The vCPU can return to the guest while that work happens.
Once the worker has processed the request and published its reply, a
completion interrupt tells the guest to collect the result.

{{< svg-figure src="async-path.svg" height="847" alt="Asynchronous path: the guest vCPU submits a request and kicks the queue; a VM exit notifies a host filesystem worker and the vCPU returns to the guest; the worker handles the request and publishes a reply; a completion interrupt lets a guest vCPU collect it. The requesting task waits while other guest work may run." caption="Normal queues hand work to a filesystem worker and signal completion with an interrupt. The steps show ordering, not time to scale." >}}

This makes sense when there's useful work to overlap with the I/O. But an
application doing a blocking filesystem operation can't continue until the
reply arrives. If the guest has nothing else runnable, the vCPU may go idle
while it waits.

Meanwhile, these stages can run on different physical host cores. Handing
work between threads, handling the completion interrupt, and waking idle
cores all have a cost. With modern cores aggressively entering idle states,
**the path around a short filesystem operation can be expensive in its own
right**. This is the motivation described in the
[proposal](https://github.com/libkrun/libkrun/pull/933).

## Keeping the request on the vCPU thread

The new path uses the experimental **`VIRTIO_FS_F_SYNC`** feature. The device
offers a dedicated synchronous queue for each configured vCPU, in addition
to its normal queues. A guest driver that supports the feature negotiates it
and uses those queues.

When the guest kicks a synchronous queue, the VM exit takes the vCPU thread
directly into the virtio-fs implementation. That same host thread processes
the request, performs the host filesystem operation, and publishes the reply.
Only then does it resume guest execution. The guest immediately checks the
queue and finds the result waiting for it.

{{< svg-figure src="sync-path.svg" height="721" alt="Synchronous path: the guest vCPU submits a request on its dedicated queue and kicks it; a VM exit enters the virtio-fs handler on the same host vCPU thread; that thread handles the request and publishes the reply; the guest resumes and immediately collects the result, without a completion interrupt." caption="With synchronous queues, the notifying vCPU thread does the work before returning to the guest. The VM exit is still there; the worker handoff and completion interrupt are removed from this path." >}}

Here, **the vCPU thread is a host thread executing VMM code**. It temporarily
stops running guest instructions while it handles the request. The host
scheduler can still move that thread between physical cores, and a host
filesystem call can still block it.

This also explains the tradeoff: **that vCPU can't run other guest tasks
while it's servicing the request**. Short operations with a waiting caller
are promising candidates. Long operations, or workloads with plenty of
other runnable guest work, may benefit from asynchronous processing.

The [implementation](https://github.com/libkrun/libkrun/pull/933/files) keeps
the high priority and normal request queues on the worker path. Synchronous
queues run their handlers after releasing the bus device lock, so a host
filesystem operation doesn't hold that shared lock throughout the request.
Handlers on different synchronous queues can therefore run concurrently on
their respective vCPU threads.
On KVM, their notifications also bypass `ioeventfd` registration, ensuring
the kick reaches the VMM on the notifying vCPU thread.

## Building libkrunfw

For a workload closer to everyday development, I built **libkrunfw 5.6.2**
inside a VM with **8 vCPUs on an Apple M3 Pro**, using `make -j8`. These are
the timings from the [benchmark run](https://github.com/libkrun/libkrun/pull/933).

{{< svg-figure src="build-time.svg" height="380" alt="Elapsed build time, lower is better: asynchronous queues 463.826 seconds; synchronous queues 334.348 seconds. Synchronous queues save 129.478 seconds, a 27.9 percent reduction." caption="The build finishes 2m09.478s sooner, a 1.39× speedup. Bars start at zero." >}}

| Time | Async queues | Sync queues |
| :--- | ---: | ---: |
| Elapsed | 7m43.826s | 5m34.348s |
| User CPU | 12m47.002s | 13m16.870s |
| System CPU | 25m02.745s | 11m03.636s |

That's **27.9% less elapsed time**, or about **38.7% higher build throughput**
if expressed as the inverse of the runtime. The elapsed time saving is just
over two minutes.

## Synchronous random I/O

To exercise blocking I/O more directly, I used **fio 3.39** with a mixed
random read/write workload, 4 KiB blocks, the `pvsync` engine, and an I/O depth
of one per job. Each configuration ran for 60 seconds, with the **best of
three runs** shown below. The command for one job was:

```sh
fio --name=virtiofs-test --directory=/root \
    --rw=randrw --bs=4k --size=1G --numjobs=1 \
    --time_based --runtime=60 --group_reporting \
    --ioengine=pvsync --iodepth=1 --direct=1
```

For the four job test, I changed `--numjobs=1` to `--numjobs=4`.

{{< svg-figure src="fio-bandwidth.svg" height="590" alt="fio bandwidth, higher is better. With one job, read bandwidth rises from 41.7 to 55.2 MiB/s and write bandwidth from 41.7 to 55.1 MiB/s. With four jobs, read and write bandwidth each rise from 120 to 149 MiB/s." caption="Read and write bandwidth are shown separately. The four job results are aggregated across all four jobs. Bars start at zero." >}}

| Jobs | Queues | Read MiB/s | Write MiB/s | Read kIOPS | Write kIOPS |
| ---: | :--- | ---: | ---: | ---: | ---: |
| 1 | Async | 41.7 | 41.7 | 10.7 | 10.7 |
| 1 | Sync | 55.2 | 55.1 | 14.1 | 14.1 |
| 4 | Async | 120 | 120 | 30.6 | 30.6 |
| 4 | Sync | 149 | 149 | 38.2 | 38.2 |

With one job, read bandwidth improves by **32.4%** and write bandwidth by
**32.1%**. With four jobs, both improve by **24.2%**. These comparisons
include both the change in request processing and the use of per-vCPU
queues; they don't separate the effects of each.

These are comparisons for this workload and setup. Selecting the best of
three runs gives a useful initial result, but doesn't show run to run
variability.

## File deletion latency

Finally, I used a synthetic benchmark that first creates **50,000 files**,
then measures the latency of deleting them with `unlinkat` in a separate
phase. This exercises metadata operations rather than bulk data transfer.

{{< svg-figure src="unlink-latency.svg" height="420" alt="File deletion latency, lower is better: average latency falls from 161.458 to 103.191 microseconds and median latency from 152.625 to 95.708 microseconds." caption="Average unlink latency falls by 36.1%, and the median by 37.3%. Bars start at zero." >}}

| Latency | Async queues | Sync queues |
| :--- | ---: | ---: |
| Average | 161.458 µs | 103.191 µs |
| Median | 152.625 µs | 95.708 µs |
| Standard deviation | 32.369 µs | 32.229 µs |
| Minimum | 113.875 µs | 72.166 µs |

## Where this could go

This is still an **experimental feature**, with the implementation proposed
in [libkrun PR 933](https://github.com/libkrun/libkrun/pull/933). It needs
support in both the VMM and the guest virtio-fs driver to negotiate and use
the synchronous queues.

The results make it worth exploring where else the same approach might
help. Read requests on virtio-blk are one possibility: whenever the caller
must wait, the cost of handing the work away and signalling its completion
deserves a closer look.

The [benchmark data](benchmarks.json) includes the original measurements and
the PR revision used for these comparisons.