Is vsock just slow on Linux? (EDIT2: Maybe so...)

[lrowe]

UPDATE2: Seems that epoll accept loop patterns that are fast for TCP and UDS are slow for VSOCK

The root cause here seems to be the pattern of calling accept multiple times per epoll loop until the backlog is drained. This is the normal pattern for TCP and UDS but this seems to be slow for VSOCK when every other non blocking accept returns EAGAIN.

UPDATE: In fact it is tokio-vsock that is slow. See: rust-vsock/tokio-vsock#71

I've been trying to benchmark firecracker for an ephemeral use case and assumed that vsock communication would be faster than tcp. However I see 2 ms latency when measuring an http connection with a single request over vsock vs ~150 us latency with tcp in firecracker. Interestingly I also see 1-2 ms latency for loopback vsock connections running the same program on the host.

Is something up with my system or is this just the expected state of vsock on Linux?

Results and repro are here: https://github.com/lrowe/firecracker-snapshots

(Requires Docker and sqfstar to build roots fs and deno for the hello world example server.)

Host kernel: 6.14.0 (Ubuntu 24.04)
Guest kernel: 6.1.164 (from spec.ccfc.min bucket)
Firecracker: 1.15.0
CPU: AMD Ryzen 7 7840HS

[Manciukic]

Hey, I think we have noticed a similar thing in #5603. If you have found anything interesting, let us know!
It's not clear from your updates if your issue is caused by tokio-vsock or not. When you mention the epoll accept loop patterns, what do you actually mean?

[lrowe]

So it turns out my issue was caused by a bug in the Linux Kernel that only affected non-blocking vsock accept calls when the listening socket's backlog was empty (epoll servers tend to use non-blocking sockets and eagerly accept all connections until the backlog is empty and accept returns EAGAIN.) This affected both loopback vsock usage on the host and host unix socket connections to firecracker vsock usage.

Patch is: https://lore.kernel.org/virtualization/20260402044637.73531-1-laurencerowe@gmail.com/T/#u

Repro tools are at : https://github.com/lrowe/linux-vsock-accept-timeout-investigation

I still see some additional latency going from host unix socket -> guest vsock (200 us vs 15 us) but have not yet tried host vsock to guest vsock with qemu to compare.