The screenshot opencode never sent

Assumes you've run a local model behind a coding agent. Part 1 has the laptop and its memory pool; the router manual has the serving setup.

The obvious suspect for anything new on this laptop is the GPU. It's a Radeon 680M with no memory of its own: on the boot these tests ran on, it borrowed a 4 GiB carve-out plus 13.6 GiB of system RAM it can map (GTT), out of the laptop's 32 GB, and this series has already spent a fair share of its words on what does and doesn't fit in it. So the place to expect trouble from letting local coding models look at screenshots is right there: a picture means extra weights, extra tokens and extra memory, on a GPU that has none to spare.

The GPU did fine. The models read 15-pixel text off a build log. The trouble came from the coding agent, opencode, which in the versions I ran (1.19.3, then 1.19.5) took my screenshot, decided on the model's behalf that it couldn't see, sent the model a note saying so, and reported success. I have a soft spot for software that knows something is wrong and tells the one party who can't do anything about it.

The setup, for anyone landing here first: llama.cpp's llama-server (build b11803, Vulkan backend) runs in router mode, one process holding many named model presets and one model in memory at a time, and opencode 1.19.3, a terminal coding agent, talks to it as an OpenAI-compatible provider. The laptop is a Ryzen 7 PRO 6850U with 32 GB of LPDDR5-6400.

A vision model is two files: the weights and a projector#

A vision model in GGUF form is the language model file plus a separate projector file, mmproj-*.gguf, 0.6 to 1.2 GB, that turns image pixels into tokens the model reads. llama-server loads it with --mmproj <file>, and a router preset takes the same key:

mmproj = /path/to/file.gguf

Of the 19 presets I serve, I checked every repository on Hugging Face through its tree API: 11 ship a projector, 8 don't. The eight are two LFM2.5 models, Mellum, Flint, GLM-4.7-Flash, Sage, and two fine-tunes that have no projector of their own: AgentWorld and Swift GSQ.

Those two fine-tunes borrowed their base model's: Qwen-AgentWorld-35B got the Qwen3.6-35B-A3B projector, and Swift GSQ the Swift-27B one. It isn't as reckless as it sounds, since fine-tunes already share projectors in the wild: the Qwen3.6-35B-A3B base projector and FableVibes' are byte-identical, same sha256. The precondition is that the projector's output dimension equals the language model's embedding_length: 2048 for the 35B-A3B family, 5120 for the 27B. Both numbers sit in the GGUF headers, so read them before downloading anything. Matching dimensions make it load. Only a test says whether it's any good.

I downloaded nine projector files, about 8.3 GB in all, and checked each against the sha256 in Hugging Face's LFS metadata.

Test images with answers nobody can guess#

A vision test with a fixed answer key mostly tests whether the model has seen the answer before, or can guess it from the question. So every run generated five fresh images, each with a random answer:

1. code: a red box with a random code QZ-#### in white 64 px text, and a blue circle.
2. ocr: a six-line woodworking cut list ("Oak shelf 3 pcs 812 x 240 x 18 mm") with random numbers, scored by recall of its words (three letters or more) and numbers.
3. chart: five green bars with random four-letter labels: which bar is tallest?
4. term: a dark terminal screenshot of a failed Gradle bootRun with a BindException on a random port: which port is the error about?
5. big: a 1600x1000 build-log screenshot, 44 lines of 15 px monospace text, one red FAILED line with a random six-digit request id: what is the request id?

These are one set of renders; each run drew new random values. Two controls ran per model, because a test a model can pass without the input isn't testing the input. First, the code question with no image attached: the model must not produce the code. Second, the same weights served without the projector, where llama-server must refuse the image, and it does, with an HTTP 500:

image input is not supported - hint: if this is unexpected, you may need to provide the mmproj

Temperature was 0, and thinking was off (enable_thinking: false) to keep it fast. Each model also generated 512 tokens of plain text with and without the projector loaded, to measure what carrying it costs.

54 of 56 answers right, both misses one character short#

Across 14 presets, the four single-answer images gave 56 answers, and 54 were right. OCR recall was 1.00 on every model. No model produced the code without the image, and every projector-less server refused it.

The two misses each dropped one character: FableVibes 14B read the chart label BMHM as BHM, and Ornith 35B, in its Q4 quant, read request id 241163 as 24163. That's the whole error list, which is a shorter list than this series has trained me to expect.

What the projector costs in memory, seconds and speed#

Seconds per image are the average over the five images and the maximum, which was the 1600x1000 screenshot every time. Generation is tokens per second on the 512-token text request, without the projector and with it.

The preset names are mine: Gemma 4 26B A4B is a mixture-of-experts model with 4B active parameters; Qwen3.6 35B A3B, Nail, Tiel, Ornith and AgentWorld are 35B mixtures of experts with 3B active; Groot is a 19B one with 3B active; Swift 9B, the Swift 27Bs and Qwen3.8 27B are dense.

Memory sorts the table. Where the pool had room, loading the projector added 1.1 to 1.5 GiB of GPU memory, and the mixture-of-experts models' generation barely moved: 2% or less. Three 35B presets, AgentWorld, Tiel and Nail in its larger quant, already used 16.6 GiB without it; with the projector more of the model spilled to the CPU, and they lost about 10%. The dense 27Bs lost 4 to 18%. The biggest file, Ornith's 21.7 GB Q4_K_M, lost 32% and got no image preset; its Q2 quant, which fits, lost 0.7%.

The big screenshot is where the time goes. Qwen-family projectors turned it into up to 1592 tokens. Gemma 4 stopped at 519 in my table, because llama-server caps it at load:

cap image_max_tokens (original=1120) to 512 (n_ubatch = 512) because model needs non-causal attention on image ... increase n_ubatch (-ub) to increase vision token budget

It read the 15 px text anyway, at 9.7 s for the big screenshot, the fastest in the table. A cap that costs nothing on the test is the best kind of cap, and also the kind I trust least.

One model crashed the GPU while loading. A dense 27B in IQ4_XS, a 15.5 GB file, took the projector on top, and the server log showed the Vulkan driver's "radv/amdgpu: Not enough memory for command submission", followed by ErrorDeviceLost. I've met this error before on other big models, and the cure was the same flag, llama.cpp's --no-host, as a preset line:

no-host = true

With it, the same model went 5 of 5.

Two smaller notes. Presets with speculative decoding through MTP draft heads worked with a projector loaded. And the 35B runs left 0 GB of system RAM available, since the GPU pool is system RAM; the laptop kept going, which I'll file under things it did for me without being asked.

opencode sent an error message instead of the image#

With llama-server answering images over the API, the next step was the agent. I pointed opencode at a fake OpenAI-compatible server that records every request body, which needs no GPU and shows exactly what goes over the wire, and attached a screenshot with -f:

opencode run "<question>" -f shot.png

The image never left opencode. In its place, the request carried this text:

ERROR: Cannot read "shot.png" (this model does not support image input). Inform the user.

And the run exited 0. The model, which in the real setup could read a 15 px request id off a build log, received a note about a picture it was never shown, with instructions to pass the bad news on. Whatever it answered, it answered from that. So, opencode: you had the file, a server that takes images and a model that reads them, and you sent a memo.

The default comes from the model entry. In my tests, opencode treated a custom provider's model as text-only unless its entry said otherwise, and the key that says otherwise is modalities:

{
  "modalities": {"input": ["text", "image"], "output": ["text"]}
}

With that in the model entry, the image goes out as an OpenAI image_url part with a base64 data URI. "attachment": true on its own changes nothing. I looked for modalities on opencode's models, providers and config pages on 2 November; none of them mention it, and the providers page's llama.cpp example covers token limits and no input types.

Which versions this covers. Everything above ran on opencode 1.19.3. I re-ran it the same evening on a fresh install of 1.19.5, against the same recording server with the same -f shot.png, two requests per setting: identical. Without modalities the request carried no image and the "does not support image input" text; with it, the image went out. The Strix Halo machine below saw the same split on 1.18.35.

One report said the opposite until later the same evening. A third setup, opencode 1.19.5 behind an OpenAI-compatible proxy with an RTX 3090 behind that, saw the image reach the model with and without modalities in the model entry. The cause was a draft opencode.jsonc in a parent directory of the test, which declared modalities; opencode merged it in, and opencode debug config showed modalities back on the very entry the test had stripped. Re-run in a directory with no config above it, same version and proxy: no image sent, and the model said it doesn't support image input. All three setups now agree on 1.18.35, 1.19.3 and 1.19.5. My favourite kind of config bug: a test of "no modalities" quietly overruled by a file one directory up.

So: declare modalities for a vision model, then check what actually goes out. opencode debug config shows the merged config the client will use, parent-directory files included, so you can see whether an entry really lacks the key. A request capture shows the body. Failing those, a test image with an answer nobody could guess: a model that reads a random QZ-#### code back, with no tool call that could have read the file, has seen the picture, and a model that says it saw the picture has only said so. The tool-call condition matters; the three-machine section below has two models that read the code without seeing anything.

The same key governs the agent opening images itself. With modalities set, the model called opencode's Read tool on term.png and got the picture back. Gemma 4 answered the port question right; Nail in IQ3 answered it right too, after about 13 minutes, most of them spent reading opencode's roughly 12,000-token first prompt and thinking.

One more trap on the way in, the argument order. opencode run -f shot.png "question" fails with File not found: question, because -f takes every following argument as a file. The message goes first. The Strix Halo machine's test script, below, had it the wrong way round too, and was fixed before it ran.

The fix went into my own config generator, the script that writes opencode's model list from the router presets. Without the modalities key it writes perfectly valid entries for models that can see and never get shown anything, and I can't blame opencode for the half of the arrangement I wrote. Now a preset with an mmproj = line gets modalities in opencode's config, and each vision model is offered twice, gemma4-26b and gemma4-26b-vl, so the projector's memory is spent only when a session picks the -vl one.

Real screenshots with thinking on#

Generated images with clean text and one answer each are the easy case, so the next check used real screens. I captured them on the same laptop with no desktop session running: a headless sway compositor (wlroots' headless backend, pixman renderer) with the foot terminal and grim for htop, btop and a live tail -f of the llama-server log, at 1920x1080; headless Firefox with a throwaway profile (firefox --headless --screenshot) for the llama.cpp GitHub releases page, a DuckDuckGo results page and a Hugging Face model page, at 1600x1000.

The 18 questions had answers read off those same screens. Among them: the top CPU process in htop (a busy-loop process started under a random name), battery percentage, uptime, a model name and a tokens-per-second value in the log, the option a log warning suggests, CPU model, disk-used percentage, the newest release tag and its short commit hash, an open-issue count, a discussion number in a search result, a result's domain, monthly downloads, a file size, an architecture and a like count.

This time thinking stayed on, the way the agent runs, with default sampling and at most 8,192 output tokens.

Qwen3.6's miss is the one I'd worry about in an agent: six minutes of thinking about htop and an empty answer at the end, with the cap doing the stopping. One option stays untried. At load, llama-server warns that "Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks" and suggests --image-min-tokens 1024; my presets don't set it.

Two other machines: the same trap, the same memory bill#

Two other machines ran their own versions of this test, and parts of my test design were borrowed from their plans: the random red-box code and the no-projector control from one, the OCR recall scoring from the other. Their results came in the same evening.

The 3090 had room, so nothing moved. It ran the dense Qwen3.8-27B family, base Qwen3.8-27B, Swift-1.5, ThinkingCap and Qwen3.8-27B-pi at 128k context, an ISTA GSQ quant, Swift GSQ, and ThinkingCap again at 256k with a q4_0 KV cache; the other six used a q8_0 KV cache, and all ran MTP drafting of 3 tokens. Each transcribed three cut-list images, one clean, one photo-style and one skewed 7 degrees, at temperature 0, and every config got recall 1.00 on all three. Swift GSQ ran on Swift's borrowed projector there too, so that borrow works on two machines. The projector added 1136 MiB at load and another 94 to 262 MiB while encoding an image: base Qwen3.8-27B at 128k went from 19772 to 20908 MiB, peaking at 21006. Image requests took 3 to 10 s, and text generation stayed within the spread speculative decoding gives anyway. On the 680M the dense 27B models from the same family already fill the pool, so the projector pushes layers to the CPU and generation drops 4 to 18%. Same model family, same extra gigabyte; the difference is whether there was a gigabyte free. The 3090 side has its own write-up, Image input on a 27B coder costs 1.1 GiB, once, with one more run: a context filled to 121k tokens plus a screenshot peaked only 118 MiB above the load figure, because llama.cpp reserves the whole KV cache when the server starts.

The Strix Halo machine tested the paths. Its fixture was a Chromium screenshot with a random QZ-nnnn code in a red box, and a pass meant the exact code in the reply. Each model went down five paths: the raw API; the same prompt with no image; opencode run -f with a stock model entry; opencode with modalities declaring image input; and Claude Code, a second coding agent, in print mode with only its Read tool allowed, through a small proxy that rewrites tool schemas, because llama-server rejects Claude Code's raw schemas with HTTP 400. Qwen3.8-Flash-Next (IQ4_XS), Qwen3.6-35B-A3B and Ornith-35B saw the code on the raw API, opencode with modalities and Claude Code's Read, and not on the no-image prompt or stock opencode. The same weights without the projector saw it on none of the five.

Two models "passed" without seeing anything. Swift-1.5 and ISTA-Coder (IQ1_M) matched that pattern except on stock opencode, where they also returned the right code. Told by opencode that they couldn't view the image, they ran the tesseract OCR tool through bash on their own, read the code off the file, and said so in the reply. It isn't vision. Score only the final answer and it's a pass. The other engine, a HIP fork with a CPU image encoder, saw the image on the raw, modalities and Claude Code paths, and its own tesseract detour on stock opencode read the code as OZ-7131: wrong, and still not vision. A model resourceful enough to work around a missing image is exactly the model that makes a missing image hard to notice, so read the tool calls before the answer.

The Strix Halo machine had a trap of its own. Jobs started from a queue or a systemd unit with a minimal PATH got exit 127 for both opencode and Claude Code, so only the raw API path ran in its first round. Across all three machines, no no-image prompt ever produced the code.

What this doesn't show#

• Only screenshots and drawn images. No camera photos, diagrams or blurry images.
• One run per model per image set. The answers are random, but each question was asked once.
• No real agent task yet. Nothing here has a coding agent checking its own UI from a screenshot; that's the use I'm after, and it's untested.
• opencode 1.18.35, 1.19.3 and 1.19.5 only. Other versions may behave differently; the check above works on any of them.
• The other two machines tested less. The 3090 ran only the Qwen3.8-27B family, only through the plain API, on three cut-list images. The Strix Halo machine tested one generated screenshot per model: no real UI screenshots, no opencode Read tool, no memory or speed cost. Real screenshots and opencode's Read tool were tested on the 680M only.

The part that couldn't see#

On a laptop whose GPU borrows every byte it uses, the vision models cost me a gigabyte and a bit, a few seconds per image and, where they fit, a percent or two of speed, and read 15 px text off a build log. In my setup the only part of the stack that couldn't see the screenshot was the one holding it, and on another machine two models covered for it with OCR. I went in suspecting the GPU, as usual, and the GPU did its part. The client wrote "Inform the user" to a model, informed nobody, and exited 0. New rule, effective now: before I grade what a model saw, I read what it was sent.

Filed under: local-llm, llama.cpp, opencode, ai-agents, vision

Read next#

• Part 1: On a laptop iGPU, memory bandwidth picked the model: the memory pool behind the 10% losses above, and why generation speed follows active parameters.
• Part 2: Twenty-one agent runs on a laptop iGPU, none merge-ready: five of these models as coding agents, graded blind.
• Part 3: Faster prompts, slower agents: Nail on a second inference engine, for comparison with llama.cpp.

Sources#

• My own test runs, logs, fake-server captures and screenshots on the laptop (local, not published), 2 November 2026.
• llama.cpp docs/multimodal.md: --mmproj loads the multimodal projector next to the model.
• opencode providers documentation: the custom-provider and llama.cpp example, checked on 2 November 2026 for input types.
• opencode config documentation and models documentation: checked on 2 November 2026 for modalities.