Image input on a 27B coder costs 1.1 GiB, once
Contents
Assumes you've served a GGUF with llama.cpp. Part 1 introduced the bench.
I had the memory risk of image input in the wrong place. On a 24 GB RTX 3090 that already holds a 27B coding model with a 128k-token context, I expected the dangerous request to be the full one: an image plus a context filled to the brim. Adding image input to Qwen3.8-27B makes sense: the base model is a vision-language model, and a coding agent that can read a screenshot of a failing UI or a terminal is worth having. The question was whether it still fits on the card.
This series runs local models on that one card as coding agents. I serve them as named keys: a key is one serving profile, a model file plus its context size and server settings, and six of mine run Qwen3.8-27B or a tune of it at 128k tokens. The full case I worried about was a context filled to 121k tokens with a screenshot in it, on a key that already used 22,802 of the card's 24,576 MiB with image input loaded. That case added 118 MiB. The cost of image input turned out to be a fixed charge at load, 1,136 to 1,138 MiB on every model I tried, and once it's paid the context can fill up without the memory moving.
What image input needs in llama.cpp#
In llama.cpp, a vision-language model comes as two files. The GGUF holds the language model. A second, smaller GGUF, the multimodal projector or mmproj, holds the vision encoder and the projection that hands its output to the language model. llama-server loads it with --mmproj <file>. Without it the server is text-only: its /props endpoint reports vision false, and on the laptop described further down, a request with an image in it came back as an HTTP 500.
The projector has to match the model. Here is what the model cards ship for the 27B and its tunes:
- Qwen3.8-27B, Unsloth's GGUF repo:
mmproj-F16(927.6 MB) and a BF16 one. - Swift 1.5, ThinkingCap and Qwen3.8-27B-pi, three fine-tunes: each ships its own projector, and the files differ from the base model's and from each other. ThinkingCap's card calls it a vision-language model and evaluates it on RealWorldQA, an image benchmark, with its projector loaded.
- ISTA-DASLab's GSQ-RCO quant of the base model, a non-uniform quantization that gives each tensor its own bit width: a BF16 projector. I call these GSQ for short below.
- Swift's GSQ quant: none. Its card says a Swift 27B vision projector hasn't been verified for this release, so none is included.
I paired the Swift GSQ quant with Swift's own projector. It worked. For the base model's GSQ quant I used Unsloth's F16 projector, the same file as the base key, not ISTA's BF16 one.
The test: load twice, read a cut list#
Each serving profile loaded twice on llama.cpp ddf48f85, the build from part 25: once without the projector and once with it, with the bench's usual settings: q8_0 KV cache, and MTP drafting of 3 tokens on every profile except the two GSQ keys. MTP is the model's own extra layer drafting tokens ahead for the full model to check; it speeds up generation but makes single requests noisy. Both loads ran the same text speed probe. The load with the projector then read three images of the same six-line woodworking cut list: a clean render, a photo-style one and one skewed by 7 degrees, 514 to 684 prompt tokens each, at temperature 0. The score is the share of the cut list's numbers and words of three or more letters that appear in the reply.
1,136 MiB on every model, give or take 2#
| Profile | Text only | With projector (peak) | OCR |
|---|---|---|---|
| Qwen3.8-27B IQ4_XS 128k | 19,772 | 20,908 (21,006) | 1.00 x3 |
| Swift 1.5 IQ4_XS 128k | 20,816 | 21,952 (22,056) | 1.00 x3 |
| ThinkingCap IQ4_XS 128k | 20,816 | 21,954 (22,062) | 1.00 x3 |
| Qwen3.8-27B-pi IQ4_XS 128k | 21,664 | 22,802 (22,896) | 1.00 x3 |
| Qwen3.8-27B GSQ IQ3_S 128k | 16,390 | 17,526 (17,788) | 1.00 x3 |
| Swift GSQ IQ3_S 128k | 16,390 | 17,526 (17,788) | 1.00 x3 |
| ThinkingCap 256k, q4_0 KV | 22,352 | 23,490 (23,598) | 1.00 x3 |
VRAM in MiB for the whole card, from nvidia-smi; the card has 24,576. The peak is the highest reading after the three image requests.
The projector adds 1,136 to 1,138 MiB at load on every profile. Encoding the images added another 94 to 262 MiB on top of that. Every model scored a recall of 1.00 on every image, the borrowed Swift projector included. Each image request took 3 to 10 seconds, and the replies were plain transcriptions of about 150 tokens.
The tightest row is the last one: ThinkingCap at a 256k context with a q4_0 KV cache. With the projector loaded and an image encoded it peaked at 23,598 MiB, which leaves 978 MiB of the card.
Text speed showed no consistent drop with the projector loaded. On the MTP keys, single requests moved both ways, from 9% slower (ThinkingCap at 128k) to 15% faster (the base key at 32k). One request per setting can't separate that from MTP noise. The two GSQ keys run without MTP and are steadier. The base model's GSQ key read 41.3, 41.2 and 35.5 tokens per second at 1k, 1k and 32k, both without the projector and with it. Swift GSQ read 41.3, 41.2 and 35.4 without it and 41.2, 41.1 and 35.4 with it.
A screenshot, then a full context#
A review of these results pointed out the obvious gap: the cut-list images are small, so "256k fits" was only shown for small pictures. The follow-up used a 1,600x1,000 screenshot of a terminal, 29 lines of kubectl output, which came to 1,622 prompt tokens. This ran on the build from part 26, adopted that afternoon.
- ThinkingCap 256k, q4_0 KV: peak 23,620 MiB, 22 more than the small images on the earlier build. Recall 1.00, 12.5 seconds.
- Qwen3.8-27B-pi 128k: peak 22,920 MiB, recall 1.00, 12.0 seconds.
The encoder's working memory barely grew from about 600 image tokens to about 1,600. The 256k profile still had 956 MiB to spare.
Before giving the serving proxy vision keys, one more question: is that peak still the peak when the context is full? One request on the pi key at 128k: 119,449 tokens of my project docs with a "vault code" planted halfway through, plus the same screenshot, 121,103 prompt tokens in all. The question asked for two things, which node a named pod runs on, readable only from the image, and the vault code, readable only from the text.
Both answers came back right. VRAM went from 22,802 MiB at load to a peak of 22,920, 118 MiB more, the same peak as the screenshot alone on an empty context. The prompt read at 798 tokens per second, the answer generated at 54, and the request took 155 seconds.
The reason is how llama.cpp allocates: the KV cache for the whole configured context is reserved when the server starts. A filled context uses memory that was already taken. What fits at load fits at 121k tokens; the image adds its encoder's working memory on top, and that stayed at 262 MiB or less for every image here.
Not tested on the 3090: a 256k context with a q8_0 KV cache (23.1 GB text-only, which I don't expect to fit with a projector), images above about 1,600 tokens, charts, and image input through OpenCode.
On a laptop iGPU the projector costs speed#
The same projectors also ran on an older AMD laptop with a Radeon 680M iGPU, on a Vulkan build of llama.cpp (b11803), with a larger question set: 14 models, an OCR recall of 1.00 on all of them, and 54 of 56 exact-answer questions about images right. Memory went up 1.1 to 1.5 GiB, counted as GPU memory plus GTT, the system RAM the iGPU borrows.
There, text generation slowed down with the projector loaded, by 4% to 18%: Qwen3.8-27B by 17.5%, Swift by 15.4%, Swift GSQ by 4.2%. The likely reason, not measured there, is that the 27B models already fill the iGPU's memory pool and the extra 1.1 GiB pushes work out of it. The smaller GSQ quant lost least. An image took 15 to 20 seconds, the 1,600x1,000 screenshot about 50. One model crashed on load with DeviceLost until the server ran with --no-host.
On the 3090, with its own 24 GB, none of that showed up beyond single-request noise. The projector costs memory there, and no speed I could measure.
Two config details#
OpenCode needs the model to declare image input. On the laptop, with OpenCode 1.19.3, an image reached the model only when the model's entry in the config listed "image" under modalities.input. Setting attachment: true alone did nothing.
Separate keys, not a switch. The serving proxy now has four -vision twins of the 128k keys, Qwen3.8-27B, pi, Swift and ThinkingCap, each loaded with its projector. The text keys stay as they were. That was my choice: two keys per model, rather than one key that always carries the projector.
Filed under: local-llm, llm, llama.cpp, benchmarks
Read next#
- One open PR made MoE prompts 12-30% faster on my 3090: the build the screenshot and full-context runs ran on.
- It fits in 16 GB. That was the easy question: the same 27B on a smaller card, where fitting the KV cache costs most of the speed.
- MTP acceptance is a side effect, not a target: the MTP drafting these 128k keys run with.
Sources#
- unsloth/Qwen3.8-27B-GGUF: base model GGUFs and
mmproj-F16/mmproj-BF16. - ukisai/Swift-1.5-Qwen3.8-27B-GGUF and ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF: Swift 1.5 with its projector, and the GSQ quant whose card says no projector was verified.
- bottlecapai/ThinkingCap-Qwen3.8-27B-GGUF: ThinkingCap with its projector and the RealWorldQA evaluation.
- bytkim/Qwen3.8-27B-pi-GGUF: Qwen3.8-27B-pi with its BF16 projector.
- ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF: GSQ quants of the base model and a BF16 projector.
- My own bench data (local, not published): load and OCR logs per profile, the screenshot and full-context runs on the 3090, and the laptop's results, 2 November 2026.