It fits in 16 GB. That was the easy question

Assumes you've run llama.cpp and know what the KV cache is.

"Will it fit?" is the question everyone asks about a model and a graphics card, because it has a clean yes-or-no answer. A friend asked it about a 16 GB card: can Qwen3.8-27B, or its Swift-1.5 tune, at IQ4_XS (13.3 and 14.4 GiB of weights) run fully on the GPU if the KV cache goes to system RAM? He sent the flags he'd run it with, -t 8 -ctk q8_0 -ctv q8_0, and the question was what adding -nkvo does.

The answer is yes. I don't have that card, so everything below was measured on the bench box: an RTX 3090 at 280 W, a Ryzen 5 5600G with 6 cores, 31 GiB of RAM, llama.cpp build f95b0d9. I watched peak VRAM, and the fit claims below compare it against 16,384 MiB. The speeds are this machine's and nobody else's.

The yes turned out to be the least useful part of the answer.

What -nkvo moves#

The flag's help text says only "whether to enable KV cache offloading". The field it sets is documented in llama.h:

bool offload_kqv; // offload the KQV ops (including the KV cache) to GPU

So -nkvo moves more than the cache: the attention operations that read it stay on the host too. And this model has more on the host than a cache. Qwen3.8-27B is a hybrid: its GGUF lists 64 layers plus one MTP layer, with full attention every fourth layer. Only those 16 layers keep a KV cache. The other 48 carry a fixed-size recurrent state instead, and in this build the same flag decides where that state lives too. With -nkvo, the weights sit in VRAM, while the 16 attention layers' cache and attention ops and the 48 recurrent states all sit in system RAM, every generated token.

It fits, with room to spare#

Peak memory.used from nvidia-smi, sampled once a second. The 3090 drives no display on this box, so it idles at 1 MiB and each peak is the llama.cpp process alone. A card that also runs a desktop has less to give.

Everything stays under 16,384. The KV type makes no difference to VRAM, 6 MiB between f16 and q8_0, because the cache isn't in VRAM to begin with. Depth still costs some: about 1,000 MiB more at 100k and 200k than at the shallow runs, for both models.

One side result he didn't ask for. The base model with the cache on the GPU, at f16 and up to 32k of depth, peaked at 15,792 MiB: it fits without -nkvo at all, with 592 MiB spare. Swift-1.5's extra gigabyte of weights pushes the same test to 16,954 MiB, over the line. With the cache on the GPU at 100k and 200k depth, both models needed 21,088 to 22,266 MiB, which no 16 GB card has.

Prompt reading barely notices#

Prompt processing is batched, and it held up. The friend's flags at depth 0 gave 1,011 tokens/s for the base model with -nkvo against 1,270 with the cache on the GPU. At 100k tokens of depth it was 422 against 583, at 200k 261 against 379. 31 percent slower at worst. If prompt speed were the whole story, the answer would stop at yes.

Generation pays at once, then for every token of context#

With his flags and an empty context, the base model generated 42.02 tokens/s with the cache in VRAM and 15.14 with -nkvo. That loss is immediate. The context was empty, so the cost isn't the cache's size. What changed is where the attention and the recurrent state live; I didn't measure which of the two costs what.

Then depth adds its own cost, and it compounds:

Swift-1.5 landed within 0.01 of it: 2.77 at 100k, 1.54 at 200k. The deep rows are one run each (-r 1); they take long enough that one was what I had.

At 1.55 tokens/s a 1,000-token answer takes about eleven minutes. At 2.77, the 100k case, about six. At 15 tokens/s the same answer takes a little over a minute.

His q8_0 cache turned out to matter, though not for the reason you'd set it. With -nkvo it saves no VRAM. It does cut the cache to about half the bytes the CPU has to read per token, and at 32k depth q8_0 generated 6.14 tokens/s against 4.20 for f16, on the base model in a separate llama-bench run at default threads. Swift-1.5 showed the same gap: 6.11 against 4.14.

MTP costs the VRAM he doesn't have, and pays at depth#

The Qwen3.8 tunes ship a multi-token prediction head that drafts tokens for speculative decoding. On the bench it's on by default, as the post about running without it explains. On a 16 GB card it's the first thing you'd turn off, since the draft head needs memory. llama-bench has no speculative decoding, so I measured it through llama-server: 131,072 context, -nkvo, three draft tokens, one request per depth.

Generation tokens/s, one stream each. Draft acceptance averaged 0.46 for Swift-1.5 and 0.48 for the base model, and the gain grows with depth: nothing at 817 tokens, about 1.4 times at 77k.

The bill comes in two places:

• VRAM: 954 MiB for Swift-1.5, 1,058 for the base model. Swift-1.5's server peaked at 16,204 MiB, 180 MiB under 16,384. That is not a margin I'd call fitting on a card with any other tenant.
• Prompt reading: 29 to 36 percent slower at every depth. Time to first token at 76,971 tokens went from 112.7 to 160.1 seconds for Swift-1.5 and from 115.4 to 163.3 for the base model.

So the trade is three-quarters of a minute more waiting on a long prompt, for answers that arrive about 40 percent faster once it's read. The base model has the VRAM for it with a gigabyte to spare. Swift-1.5 barely does.

What these numbers don't say#

• Nothing ran on a 16 GB card. The fit is VRAM accounting on a 24 GB card with no display attached. Speeds depend on this CPU, this RAM and this PCIe link; his box will differ.
• No agent ran with -nkvo. These are llama-bench rows and a single-request server probe, not pass rates.
• The 100k and 200k rows are one run each. The 8k and 32k rows are two; the depth-0 rows with his flags ran at llama-bench's default repeat count.

What I sent back: yes, it fits, and keep it under about 32k of context if you want to talk to it. Past that, the card holds the model and the CPU does the waiting.

Filed under: local-llm, llm, benchmarks, llama.cpp

Read next#

• My benchmark served the model at half speed — what MTP is worth with the cache on the GPU, and how the bench lost it.
• 2.2x faster prefill, 1.04x faster agent — why prompt speed alone doesn't predict how long an agent takes.

Sources#

• llama.cpp include/llama.h — the offload_kqv field: "offload the KQV ops (including the KV cache) to GPU". The build measured here, f95b0d9, carries the same line, and its common/arg.cpp maps -nkvo / --no-kv-offload onto it.
• Benchmark artifacts (local, not published): llama-bench output for both models at depth 0/8k/32k (f16 and q8_0 KV, with and without -nkvo) and at 100k/200k with the friend's flags, the llama-server MTP probe log, per-run VRAM samples, 2026-10-19.