To a coding agent, 503 means try again

· 10 min read
Contents

opencode run printed its banner, > build · mock-err503, and then nothing. No error, no exit code, no output, until the run was killed. The mock server behind it answered every request with a 503 in my LAN proxy's error format: the status the proxy sent at the time for any model it couldn't serve, whether the GPU box was down, the engine switch had failed or a benchmark owned the GPU for the next few hours. opencode was retrying, and it had no plans to stop.

TL;DR — If your self-hosted LLM endpoint can't serve a request and a retry won't fix that, don't answer 503. opencode retries every 5xx with no attempt limit, and Claude Code makes 11 attempts over three minutes. Answer a 4xx that isn't 408, 409 or 429, and add X-Should-Retry: false; my proxy uses 424 for "that model isn't available" and 423 for "the GPU is busy". If you make a client wait, put 200 and SSE comments on the wire well before 300 seconds, or clients on Node's default fetch drop the request (and some send it again). And the client will pick model names and samplers of its own, so plan for those too.

The proxy sits in front of an RTX 3090 box and a Strix Halo box and switches llama.cpp engines on demand (how it got there). Most of what follows comes from running the real clients against mock servers on 127.0.0.1, which can answer whatever I want, and from the clients' source at the versions I run where it's public (opencode 1.18.4, pi 0.82.0, Node 22's fetch, the SDKs). For Claude Code 2.1.310 that means its docs, the mock runs and the strings in its binary.

A 503 is a retry instruction#

opencode's retry code (session/retry.ts at v1.18.4) says so in a comment:

// 5xx errors are transient server failures and should always be retried,
// even when the provider SDK doesn't explicitly mark them as retryable.
if (!error.data.isRetryable && !(status !== undefined && status >= 500)) return undefined

The delay doubles from two seconds:

export const RETRY_INITIAL_DELAY = 2000
export const RETRY_BACKOFF_FACTOR = 2
export const RETRY_MAX_DELAY_NO_HEADERS = 30_000 // 30 seconds
export const RETRY_MAX_DELAY = 2_147_483_647 // max 32-bit signed integer for setTimeout

The 30-second ceiling applies only when the error came without response headers, like a cut stream. An HTTP 503 has headers, so its ceiling is setTimeout's maximum, about 24.9 days, and nothing in the retry policy limits attempts. The mock saw the main turn again after 2, 4, 8, 16 and 32 seconds before the run was stopped. After its twenty-first failure opencode would wait 24 days before trying again. In a terminal you'd notice. In a headless benchmark run it holds the harness until its wall clock runs out.

Claude Code gives up, eventually. Its docs say it "retries transient failures up to 10 times with exponential backoff". Against a 503 from its own mock, this one worded the way the proxy refused a GPU that a benchmark queue owned, the mock logged 11 POSTs over 181 seconds, the gaps growing from 0.6 s to about 37 s, and then Claude Code printed the mock's message with its own advice appended:

API Error: 503 model 3090-qwen38-27b-128k: switching host 3090 to engine mainline failed: engine mainline stopped: GPU owned by bench queue. This is a server-side issue, usually temporary — try again in a moment. If it persists, check your inference gateway (127.0.0.1:18766).

"Usually temporary" is exactly wrong for a GPU that a benchmark queue owns. The same 503 with one more header, x-should-retry: false, got one POST and the same message in 0.7 seconds. The OpenAI Python SDK reads that header too:

should_retry_header = response.headers.get("x-should-retry")

# If the server explicitly says whether or not to retry, obey.
if should_retry_header == "true":
    log.debug("Retrying as header `x-should-retry` is set to `true`")
    return True
if should_retry_header == "false":
    log.debug("Not retrying as header `x-should-retry` is set to `false`")
    return False

424 and 423, borrowed from WebDAV#

opencode retries whatever the AI SDK marks retryable, and the AI SDK's rule is broader than "5xx":

isRetryable = statusCode != null &&
  (statusCode === 408 || // request timeout
    statusCode === 409 || // conflict
    statusCode === 429 || // too many requests
    statusCode >= 500), // server error

So 408, 409 and 429 are out as well. What's left are plain 4xx codes with a meaning close enough. RFC 4918 defines 424 Failed Dependency, "the requested action depended on another action and that action failed", and 423 Locked. The proxy now answers 424 when the model can't be served (the host is down, the switch to its engine failed), 423 when something else owns the GPU, 404 for a name it doesn't know and 403 for admin paths it refuses, every one with X-Should-Retry: false and in Anthropic's error shape on /v1/messages. From its journal the afternoon the change went in:

2026/10/24 15:55:17 GET /props model="3090-laguna-xs" host=- status=424 bytes=220 dur=1ms
2026/10/24 15:55:17 GET /v1/models model="" host=- status=200 bytes=49393 dur=1ms
2026/10/24 15:55:17 POST /v1/messages model="3090-nope" host=- status=404 bytes=110 dur=0s
2026/10/24 15:55:17 DELETE /models model="" host=- status=403 bytes=116 dur=0s
2026/10/24 15:55:17 OPTIONS /v1/chat/completions model="" host=- status=204 bytes=0 dur=0s
2026/10/24 15:55:17 host a9: switching to engine mainline for a9-laguna-xs-q8
2026/10/24 15:55:17 POST /v1/chat/completions model="a9-laguna-xs-q8" host=- status=423 bytes=242 dur=175ms

None of the retry rules above covers 423 or 424: opencode needs a 5xx or one of the AI SDK's three, and Claude Code and the OpenAI SDK read X-Should-Retry as well. I haven't pointed a client at the proxy's 423 or 424 itself; that part rests on the code above. (Claude Code does send a request a second time after a 404, without streaming, but that's its non-streaming fallback, not a retry loop.) One 5xx is left. When a GPU box refuses the connection or drops it before answering, the proxy returns 502, and opencode retries a 502 whatever the header says.

Five minutes of silence is a timeout#

Node's fetch is undici, and undici's headersTimeout defaults to 300e3 milliseconds. A test server that held its headers for 320 seconds:

ERROR after 301 s: fetch failed | cause: UND_ERR_HEADERS_TIMEOUT Headers Timeout Error

The same wait, but with 200, Content-Type: text/event-stream sent at once and an SSE comment line every 20 seconds:

status 200 headers after 0 s
body done after 320 s; 18 lines; last: data: {"choices":[{"delta":{"content":"hi"}}]} | data: [DONE]

That covers a lot of clients: Cline and anything else on Node's default fetch; Kilo, which sets its own 300-second limit to the first byte; and pi, whose httpIdleTimeoutMs defaults to 300,000 and which then re-sends three times. Claude Code doesn't give up at 300 seconds: a mock that held its headers for 330 seconds got its answer through. After about six minutes it aborts and sends the request again: against a mock that held its headers for 700 seconds, the second request went out 361 seconds after the first. opencode has no timeout for a custom provider at all: a 330-second wait passed.

The proxy is silent while it switches engines (a few seconds for a llama.cpp build, about a minute for Strata, plus however long the running requests take), and llama.cpp sends nothing while its router loads a model or while a request queues for the one slot. So a streaming request that has waited 10 seconds now gets the proxy's own 200 and text/event-stream, a : comment every 15 seconds saying what it waits for, and then the real stream on the same response. A line that starts with : is a comment, and the SSE spec tells every parser to ignore it.

It costs something. Once the 200 has gone out, an error can't be a status code any more, so it arrives as an SSE error event in the request's own shape: data: {"error":…} for chat completions, event: error for Anthropic and the Responses API. And Claude Code answers an SSE error by re-sending the request without streaming, which then gets no keepalive at all.

Model names you didn't send#

Claude Code asks for Anthropic model ids by itself. With the base URL changed and no model set, the first request asked for claude-opus-5-5, which the proxy doesn't have:

There's an issue with the selected model (claude-opus-5-5). It may not exist or you may not have access to it. Run --model to pick a different model.

The default is only one of them. Auto mode's permission classifier first asks for a hard-coded claude-sonnet-5, and some side requests use a small-model id. Every one of those names needs to point at the same model, or a llama.cpp router that holds one model at a time swaps models for every side request. An unknown name also makes Claude Code assume a 200k context window. The full set of variables is in the Claude Code manual.

Samplers you didn't send#

My llama.cpp presets carry the sampler settings for the models that need them, and the clients are meant to send none. opencode sends some anyway, chosen by the model's name:

export function topP(model: Provider.Model) {
  const id = model.id.toLowerCase()
  if (id.includes("qwen")) return 1
  if (["minimax-m2", "gemini", "kimi-k2.5", "kimi-k2p5", "kimi-k2-5"].some((s) => id.includes(s))) {
    return 0.95
  }
  return undefined
}

model.id is the key in your opencode config. A key containing qwen gets top_p: 1 on every request, and a request field beats the server's preset: my preset for one Qwen model says top-p 0.95. Against the mock, a key named mock-qwen3-think-tool sent top_p: 1 on every request; a key named mock-q36 with "id": "mock-qwen36" sent the Qwen name and no top_p. In my real config, 24 of 88 model keys matched: 23 with qwen, one minimax-m2. The proxy's config generator now leaves those words out of the key and puts the real name in "id".

Errors a client can act on#

Two llama.cpp errors get rewritten, on /v1/messages only. The first: Claude Code sends a role: "system" message after the first user turn when it doesn't recognise the model, and stock Qwen chat templates refuse that with System message must be at the beginning., which llama.cpp reports as a 500 (the same wall from the benchmark side). Against a mock returning that 500, Claude Code made 11 attempts over 184 seconds and printed:

API Error: 500 Jinja Exception: System message must be at the beginning. This is a server-side issue, usually temporary — try again in a moment. If it persists, check your inference gateway (127.0.0.1:18766).

A 400 whose message matches Claude Code's own pattern, role 'system' must precede an assistant message, does something useful instead: Claude Code folds the system text into the user turn and sends it once more. Against the mock that took two requests and 0.6 seconds. The proxy turns the template's 500 into that 400, and llama.cpp's context-overflow wording into Claude Code's prompt is too long: N tokens > M maximum, which its compaction looks for (that one I have only from the strings in its binary, not from a run).

None of these clients is wrong. Each was built against an API where a 503 really is temporary, a model name is a contract, and five minutes without a header means the server died. A self-hosted endpoint can say otherwise. It has to say it in the client's language.

Filed under: local-llm, llama.cpp, opencode, claude-code, http

Sources#