The hidden test asked for more than the prompt

· 7 min read
Contents

Assumes you've seen a hidden-test agent benchmark: the model gets a task prompt and a repo, a scorer runs tests the model never saw.

When most models fail the same check, the check starts to look like the hard part of the task. It sits in the results as a wall: four tunes of one 27B model hit it, six other models hit it, run after run. The natural reading is that the models are missing something.

My Java ledger task had 49 full-prompt runs scored before I changed this check, from 12 models in three agent harnesses. 39 of them were marked down by it. It turned out to test something the prompt never asked for. And the repository the models were working in steered them to exactly the code that failed it.

The blank-key check#

The task, t13, asks for a money transfer in a small Maven ledger module: LedgerService.transfer(fromId, toId, amount, idempotencyKey), all or nothing, idempotent per key, safe under concurrency. The scorer drops a hidden JUnit suite into the finished workspace. A failing core_* test makes the run FAIL; a failing detail_* test makes it PARTIAL.

One detail test, since the repository's first commit on 6 October, looked like this:

@Test
void detail_validation_keyIsRequired() {
    AccountId a = open("A", EUR, "100.00");
    AccountId b = open("B", EUR, "0");
    transferFails(a, b, eur("1.00"), null);
    transferFails(a, b, eur("1.00"), "   ");
    assertEquals(eur("100.00"), bal(a));
}

A null key must be rejected. So must a key of three spaces. Every run that failed it got a note in its result row naming it, validation: keyIsRequired, and at best a PARTIAL, even when the money moved correctly.

What the Java prompt says about the key#

Here is the line in the Java prompt:

- Idempotent per key. The key is supplied by the client and is required. Calling transfer again

"Required." Nothing about blank. Now the same rule in t14, the Go port of the task:

- Idempotent per key. The key is supplied by the client and is required: an empty or blank key is
  rejected with a new sentinel ErrIdempotencyKeyRequired. Calling Transfer again with a key that

The Go prompt was written on 8 October, two days after the Java test existed, and it spells the blank case out. The Java prompt never got the sentence. For ten days the Go version said out loud what the Java version only graded.

Seven models wrote the same line#

The Java models weren't skipping validation. The fixture's README lists the conventions "enforced in review", and number 3 ends with this:

   status codes. Programming errors (a `null` argument) are `Objects.requireNonNull` NPEs.

So the models did what the house style says. In the first repetition of the latest re-baseline, all seven models wrote this line in LedgerService, byte for byte the same:

Objects.requireNonNull(idempotencyKey, "idempotencyKey");

That rejects null. It lets " " straight through. Counting the 14 runs scored after the fix as well, 49 of all 63 full-prompt runs let a blank key through, and 47 of those guard the key with Objects.requireNonNull and nothing else. The other two never produced a public four-argument transfer at all, so there was no guard to look at.

Read the README, follow convention 3, fail the hidden check. The test was marking the models down for following the documentation.

Passing meant guessing right#

The 14 runs that did reject a blank key had to infer the rule. Thirteen of them kept the null check and added a blank check after it, like this run of swift15-27b-128k:

Objects.requireNonNull(idempotencyKey, "idempotencyKey");
if (idempotencyKey.isBlank()) {
    throw new IllegalArgumentException("idempotencyKey must not be blank");
}

Twelve of the 14 threw a plain IllegalArgumentException. The other two reused InvalidAmountException, a ledger exception for a bad amount. None added a dedicated exception for a missing key, which is what the same convention 3 asks for a new failure mode. The hidden test accepted all of them, because transferFails catches any RuntimeException.

Rejecting a blank key is probably what a careful reviewer would ask for. I'd ask for it. But "probably" is an inference, and a benchmark verdict shouldn't hang on whether the model guessed the same inference as the test author.

Grade only what the prompt states#

There were two ways to make the contract agree. One was to add the Go sentence to the Java prompt. That is the cleaner spec, and it would have made every recorded full-prompt row incomparable with every run after the change: a different prompt is a different task. On 18 October I chose the other way: keep the prompt, and grade only what it states.

The fix (commit eccdcdc, 18 October) splits the test in two. detail_validation_keyIsRequired now checks null only, which the prompt does state. The blank case moved to its own test:

// The prompt says only that the key "is required" (t14's Go prompt names empty/blank keys; this one does not),
// so rejecting a blank key is an inference: info row, never the verdict (2026-10-18).
@Test
void info_validation_blankKeyRejected() {
    AccountId a = open("A", EUR, "100.00");
    AccountId b = open("B", EUR, "0");
    transferFails(a, b, eur("1.00"), "   ");
    assertEquals(eur("100.00"), bal(a));
}

An info_* test still runs and still lands in every row, as its own info_pass metric. It never touches the verdict. The stricter check is still visible: anyone who wants to rank models on blank-key handling can, from the same rows.

The scorer self-test passed after the change: the reference still PASSes, the four broken implementations (non-atomic, not idempotent, no locks, deadlock-prone lock order) still FAIL, the four that are correct but short of the spec (no tests, useless tests, no docs, conventions ignored) still PARTIAL.

Rescoring the saved workspaces#

Every run keeps its workspace, so nothing had to be rerun. The rescore re-ran the new scorer against each saved workspace and rewrote the rows. It refuses to change a verdict without a recorded reason, so each moved row now carries a RE-SCORED note naming the commit.

Result: 16 rows went from PARTIAL to PASS, 10 from the older Java runs and 6 from the current re-baseline. No other verdict moved. No FAIL became anything else, and no PASS went down.

In the re-baseline, three of the four 27B tunes changed their Java score: swift15 from 1 of 3 to 3 of 3, thinkingcap from 0 of 3 to 3 of 3, qwen38pi from 2 of 3 to 3 of 3. Those are the numbers I'd have used to pick a Java model, and they were wrong by up to all three runs.

Not everything in the 39 was a scorer artifact. The 16 are PASS now. Nine stayed PARTIAL on other detail checks, and 14 were FAIL all along. Nine of those have otherwise working code and fail a core test on the all-or-nothing rule: the store throws halfway and the source stays debited, or the same key can't be retried afterwards. Three more fail every hidden test. Those failures are the models'. Of the nine PARTIALs, four had missed only docs checks besides the blank key, a README section or a Mermaid diagram. Since 22 October the scorer records docs checks without counting them, so those four are PASS too, leaving five.

Read the prompt beside the test#

The check that marked down 39 runs was a property of the test, the prompt and a README that disagreed with each other. The models sat downstream of the disagreement, and the ones that followed the written conventions most closely were the ones that failed.

When most runs fail the same check, put the prompt next to the test before opening a single model's code.

Filed under: local-llm, llm, ai-agents, benchmarks, java

Sources#

All from my local benchmark repository (not public), under agent-playground/bench/:

  • tasks/t13-ledger.txt and tasks/t14-ledger-go.txt: the Java and Go prompts, quoted above.
  • fixtures/t13-ledger/README.md: the module's review conventions, including convention 3.
  • scorers/hidden/t13/ZzHiddenTransferAcceptanceTest.java, its history (first commit 2026-10-06) and commit eccdcdc (2026-10-18): the old and new key tests; scorers/t13.py for the verdict rule and the info_pass metric.
  • forensics/hardbase-20261018/README.md: the re-baseline results, the artifact note on Objects.requireNonNull, the per-model Java scores before and after, and the record of the keep-the-prompt decision.
  • Commits e287362 and 4d2c348 and the rescore logs: the 10 and 6 PARTIAL-to-PASS rows.
  • results*.jsonl and the saved t13-ledger workspaces: the 63 full-prompt rows, their info_pass values, and the key guard in each LedgerService.