benchmarks
- A Broken AOT Cache Still Serves 200
By default the JVM logs one error and keeps running when its AOT cache is broken, so a Leyden tier without AOTMode=on can measure plain java -jar.
- Empty diff, finished code: 124,000 characters of thinking
Two K2-Horizon agent runs scored 29 with an empty diff. The code was in the reasoning: the Go cell's, pasted into the repo, passes 27 of 27 hidden tests.
- The harness decided how long the model thought
One 27B model, one llama.cpp server: 250 s per agent run in OpenCode, 116 s in Pi. Under OpenCode the model reasoned 3.6 times as much.
- Same Builder, Different Binary
One GraalVM builder image, one banner on every build log, 18 native binaries. Twelve came out PIE and six didn't, and nobody had touched a linker option.
- One rebuild moved gpt-oss off ROCm. Two models stayed
Part 6 sent gpt-oss-120b to ROCm because Vulkan prefill collapsed at depth. One llama.cpp rebuild later it is 3.1x faster at 64k; two models stay on ROCm.
- A -> B -> C -> A: benchmarking a kernel cmdline change you have to reboot for
A repeat boot with the same kernel command line ran one row 6 % faster than the first. How to A/B a boot parameter: four boots, sysfs proof, and drift.
- Every way my GPU job queue went quiet
A file-based job queue ran one GPU box unattended for six days. It never crashed. Here are six ways it went quiet, or nearly did, and the fixes.
- Seven FAILs, one setting: rerunning agents at 262144 tokens
Seven agent cells on the A9 failed at a 131072-token context. Rerun at 262144, six passed. Four of the seven never needed the extra room.
- The models deleted the benchmark's log
Coding agents found the harness's live log in their workspace, read it as another agent's run, and deleted it. Three rows recorded zero tool calls.
- LFM2.5 is fast. Gemma 4 finishes the job
LFM2.5-24B-A2B vs Gemma 4 26B-A4B as local coding agents, 3 runs per task: LFM wins on speed; on Spring Boot only Gemma wrote tests.