ai-agents
- The model card said think harder. The audits got worse
Qwen3.8's card defaults to xhigh and warns lower effort can mean more failures. xhigh passed 1 of 24 audits and 24 of 36 coding runs, against 4 and 27.
- The hard tier sorted by a rollback, not by speed
Five fast MoEs passed 1 of 45 hard-tier runs and skipped the same rollback in 33 of 45 ledger runs. A fast 2-bit model passed 7 of 9.
- The hidden test asked for more than the prompt
39 agent runs on a Java ledger task were marked down by one hidden check. The prompt never asked for it, and the project's README steered them into failing it.
- Empty diff, finished code: 124,000 characters of thinking
Two K2-Horizon agent runs scored 29 with an empty diff. The code was in the reasoning: the Go cell's, pasted into the repo, passes 27 of 27 hidden tests.
- The harness decided how long the model thought
One 27B model, one llama.cpp server: 250 s per agent run in OpenCode, 116 s in Pi. Under OpenCode the model reasoned 3.6 times as much.
- How to tell when your coding agent lied, not just failed
Grade what a coding agent says, not only its code: buggy code is a bug, promised code that doesn't exist is a lie. The script, the rule, the checker's traps.
- 29 of 34 passing agent runs had a blocker the hidden tests never saw
Blind review packets, a merge rubric and probes that must run: 34 agent-written money transfers passed the hidden tests, and review found a blocker in 29.
- Seven FAILs, one setting: rerunning agents at 262144 tokens
Seven agent cells on the A9 failed at a 131072-token context. Rerun at 262144, six passed. Four of the seven never needed the extra room.
- Tabby turned file contents into numbers
TabbyAPI's qwen3_coder parser runs every tool argument through json.loads and strip(). A string "123" arrives as 123, and file content loses its edges.
- The models deleted the benchmark's log
Coding agents found the harness's live log in their workspace, read it as another agent's run, and deleted it. Three rows recorded zero tool calls.