CUDA vs Vulkan in llama.cpp on an RTX 5080: five wrong turns and 21% more speed
After I unlocked the power limit on the graphics card, a local Qwen3.8-27B was giving me 31 tokens per second. Then unsloth re-released the same quant using a new method, I switched to the fresh one — 35. And I took that for the ceiling.
Then I did the arithmetic against memory bandwidth: the card was delivering 46% of what it could. Half of it was idling.
The obvious suspect is Vulkan instead of CUDA. I had already tried that, during those same two days of dead ends before the power unlock. There was no difference, so I closed the question. Turns out I shouldn't have.
Wrong turn one: the compiler didn't know my card
That measurement was broken. Every NVIDIA card has a version number for its instruction set — the compute capability, written as compute_90 or sm_90. You have to ask the compiler explicitly to build for a given version, and it simply doesn't know about versions newer than itself.
The system nvcc 12.4 goes up to compute_90 — that is Hopper, the server-grade H100s. And an RTX 5080 Laptop is compute_120, consumer Blackwell. The build was running through intermediate PTX translated on the fly, without native kernels. Claude Code and I had measured the fallback path and taken it for CUDA.
Wrong turn two: the installer that silently waited for Enter
There are no official llama.cpp builds with CUDA for Linux at all, only for Windows. So build it myself. I downloaded NVIDIA's 5 GB installer — and it silently hung, having opened a tiny X11 window asking me to press Enter. You need the --nox11 flag.
And later it turned out the installer isn't needed either: the same components ship as separate archives, and 0.86 GB is enough instead of five gigabytes.
Wrong turn three: CUDA won't build against a new glibc
CUDA 12.8 does not build with glibc 2.43. The new glibc added the functions cospi, sinpi and rsqrt, while CUDA's own headers declare them too, with a different exception specification. I patched six declarations by hand (well, by Claude Code's hand) — the same fix, as I found out afterwards, is described in llama.cpp's own documentation.
Wrong turn four, the dangerous one: the benchmark that lied
Built it, measured it: CUDA gives +20% on generation. I was about to switch my working scripts over. Then I measured on a real prompt of 64 thousand tokens — CUDA came out 45% slower.
I caught that same reversal three times in one day: speculative decoding through MTP gave +25% on a short request and −12% on a long one, n-gram lookup +400% and −34%, the DFlash2 draft model +29% and −43%. On this workload any short benchmark inverts your conclusions at real length.
Wrong turn five: a conclusion I didn't like
The conclusion was this: CUDA has no fast path for a quantized KV cache, so we stay on Vulkan. I didn't like that conclusion and asked to dig once more.
Turns out the path is there, and the kernel is built by default. It is slow because of a specific bug — one that already has a report filed, on the same architecture and the same Linux kernel as mine, and an open fix.
What came out of it
Rebuilt: CUDA 13.3, the branch with the fix, native sm_120.
| Before | After | |
|---|---|---|
| Prefill at 64k tokens | 555 tok/s | 768 tok/s |
| One agent step | 149 s | 123 s |
| 12-task benchmark | 12 min | 9 min |
Minus 21% of the time on a real step. The context did not suffer for it — the same 196 thousand tokens.
The ceiling turned out to be a false one
The conclusion I ended up with: the ceiling turns out to be a false one every time. There is always a bit more room above it, you just have to knock.
A few days ago it was 20.8 tokens per second, and that was the limit: the card was running at 80 watts out of the 175 it is rated for. I unlocked the power — 31. Unsloth re-released the same quant using a new method, calibrated for agentic coding — the same model, just rebuilt — I switched to it, and got 35.
And CUDA added its share somewhere I wasn't expecting. Generation on a short request rose from 35 to 44 tokens per second, but on the real 64-thousand-token prompt the gain is modest: from 27.8 to 29.5. What did speed up is the prefill — processing the prompt itself before the answer — from 555 to 768 tokens per second, almost 40%.
And for coding that is worth more than a gain on generation. An agent reads a lot and writes little: 64 thousand tokens of prompt against roughly a thousand tokens of answer, so three quarters of the time goes to reading. That is where the 21% off the whole step comes from.
From 20.8 to 44 tokens per second, plus 38% on prompt processing — in a few days. On live work an agent step now fits into two minutes instead of two and a half. And that is something you can work with, rather than sit and wait for.