The deadlock test can't see a half-held lock

Assumes you have written a two-account transfer under per-account locks, and know what a mutation test is.

At 128k context and a four-hour budget, four 27B local models ran a Zig ledger task 36 times through OpenCode. All 36 passed all 15 hidden core tests. All 36 had written their own concurrency test with real threads, because the prompt asked for one.

Then the scorer removed the second account lock from each transfer and ran everything again. The hidden suite failed on every mutant it could score. The models' own suites failed in 4 of 11 scoreable spec runs, 0 of 9 clean review runs, and 6 of 9 test-first runs.

Most of the runs that stayed green held a well-built test that could not fail against this bug. The standard concurrency test for transfers has a blind spot, and the deadlock fix creates it.

The halflock mutant: the second lock becomes a no-op#

The task (t15-ledger-zig) adds an idempotent, atomic, thread-safe transfer to a small Zig 0.16.0 service with one mutex per account. The models are bench keys swift15, qwen38pi, thinkingcap and qwen38, three runs each per condition. Every implementation I read for this post locks in id order, lower id first. Here is swift15's, from one of the 128k spec runs:

const first = if (from.toInt() < to.toInt()) from else to;
const second: AccountId = if (first == from) to else from;
const lock_first = try self.lockFor(first);
const lock_second = try self.lockFor(second);
lock_first.lockUncancelable(self.io);
defer lock_first.unlock(self.io);
lock_second.lockUncancelable(self.io);
defer lock_second.unlock(self.io);

The halflock mutant keeps the first mutex obtained from lockFor() and turns every lock and unlock on the later ones into a call to a no-op. I ran the scorer's rewrite function on a copy of that file. The last two lines came back as:

@import("zz_mutant.zig").nop(.{ lock_second, self.io });
defer @import("zz_mutant.zig").nop(.{ lock_second, self.io });

The mutant exists because of a real bug. In an earlier 64k spec run, swift15 put its second lock inside an if block with a defer, and Zig's defer ends at the closing brace, so the second lock was held for no statements at all. The scorer gained this mutant on 9 October. It is reported as a metric only and never moves a verdict or the points.

Each run gets one of five values:

• CAUGHT: the hidden suite fails on the mutant, and at least one of the model's own tests (one that passed before the mutation) fails too.
• missed: the hidden suite fails, and the model's tests all still pass.
• equivalent: the hidden tests that passed before the mutation still passed in their one execution, so the mutant was never shown to matter for that implementation.
• na: the rewrite could not be applied, for example because the locking sits somewhere the matcher doesn't look.
• absent: the mutation stage never ran (no passing model tests, or the build failed).

Only CAUGHT and missed say anything about the model's tests. I report CAUGHT out of all rows and out of CAUGHT + missed (scoreable), and never add cells together.

halflock counts by prompt and context#

Three prompts. Spec is the task with a Tests bullet: add tests, "including at least one concurrency test with real threads." Review is the same first turn plus a second turn listing the names of the scorer checks that failed. TDD swaps the Tests bullet for test-first and a "Stress test." bullet. That bullet asks for 8+ threads and 1000+ operations each, "mixing transfers in both directions (including between the same two accounts), deposits and withdrawals", and per-account balance checks, and says: "A lost update, a lock held too briefly or a use-after-free must make this test fail or crash." All three prompts list the same requirement that transfers, deposits and withdrawals run concurrently; between spec and TDD, only the test bullets differ.

The review 128k row leaves out one run, a CAUGHT that had read the hidden suite (see the correction at the end).

At 64k the table is mostly holes. Absent rows are runs whose own tests never passed, mostly because time ran out first. Nine of the ten equivalent rows already fail a hidden concurrency check before any mutation, leaving the mutant nothing to break. At 128k every run passed all 15 hidden core tests, so the 128k rows are the readable ones.

On the 64k rows alone, the story was simple: spec never catches it, TDD's stress test catches it 3 times out of 4. Giving the same 64k runs four hours already dents it: 2 out of 8. At 128k that story falls apart. Spec catches it 4 times out of 11, and review, which starts from the same spec prompt, never: 0 of 9 clean runs. Something other than "asked for a stress test" decides it.

Why opposite-direction transfers cannot catch it#

Read the missed tests and the same one keeps turning up: threads transferring between two accounts in opposite directions, then checking that nothing was lost. It's the deadlock test, the one the lock ordering was written to pass.

Locking in id order means every transfer between a and b takes a's mutex first, whichever way the money moves. With halflock, that one mutex still admits one transfer at a time. The second lock only protects b against operations that skip a: a deposit or withdrawal on b, or a transfer on another pair that includes b. Without one of those running concurrently, the half-locked transfer behaves exactly like the real one.

The scorer's author had already written this down. The selftest arm that strips the reference solution's stress test says the remaining tests "cannot see a transfer that holds only its first account lock: both directions take the lower id first and still serialize", and expects missed. The reference's own opposing-transfers test is blind the same way.

The catching test, and the near-misses#

Here is the test that caught it in swift15's spec run. Account a was opened first, so it has the lower id:

threads[0] = try std.Thread.spawn(.{}, depositMany, .{ &f.svc, a, per });
threads[1] = try std.Thread.spawn(.{}, withdrawMany, .{ &f.svc, a, per });
threads[2] = try std.Thread.spawn(.{}, transferMany, .{ &f.svc, a, b, eur(4), keys[0..per], outs[0..per] });
threads[3] = try std.Thread.spawn(.{}, transferMany, .{ &f.svc, b, a, eur(4), keys[per .. 2 * per], outs[per .. 2 * per] });
for (threads[4..]) |*t| t.* = try std.Thread.spawn(.{}, depositMany, .{ &f.svc, b, per / 2 });

The last line is doing the work. Four threads deposit into b while transfers write b holding only a's mutex.

thinkingcap, in the same cell, wrote nearly the same test: 8 threads transferring back and forth between a and b, plus 4 threads running deposits alongside. All four deposit into a. A deposit into a takes the one mutex the mutant kept, so it queues behind the transfers like everything else. Missed.

Other misses take the same route. In the 128k review cell, swift15 named its test "concurrent transfers in opposite directions, mixed with deposits, lose nothing", and both deposit threads target a. qwen38 in the 128k spec cell added a third account, c, and ran its deposits and withdrawals there, under a comment saying they run alongside the transfers. They do. No transfer ever touches c.

The cleanest pair comes from one model in one cell. In TDD at 128k, qwen38pi wrote a stress test that meets every number in the prompt: 8 threads, ops_per_thread = 1000, transfers in both directions, deposits, withdrawals, per-account balance checks. The transfers run between accounts[0] and accounts[1]. The deposits go to accounts[2] and the withdrawals to accounts[3]. Each kind of operation has its own accounts to itself, so nothing the mutant exposed is ever contended. Missed.

In its third run in the same cell, the same model wrote this (the start of its switch), and it CAUGHT:

switch ((thread + i) % 6) {
    0 => applyTransfer(svc, local, ids, 0, 1, 100, key),
    1 => applyTransfer(svc, local, ids, 1, 0, 100, key),
    2 => applyTransfer(svc, local, ids, 2, 3, 50, key),
    3 => applyTransfer(svc, local, ids, 3, 2, 50, key),
    4 => {
        _ = svc.deposit(ids[2], eur(60)) catch unreachable;
        local.deltas[2] += 60;
    },
    5 => {
        const ok: ?ledger.Account = svc.withdraw(ids[3], eur(60)) catch |e| switch (e) {
            error.InsufficientFunds => null,
            else => unreachable,
        };
        if (ok != null) local.deltas[3] -= 60;
    },

Same four accounts, deposits still on index 2, withdrawals still on index 3. The difference is the second transfer pair, 2↔3. Now ids[3] is the higher id of a pair, and the withdrawal writes it under its own mutex while a transfer writes it under the other. The deposit on ids[2] contributes nothing: 2 is the lower id, whose mutex the mutant kept.

Test shape predicts CAUGHT#

I went through the threaded tests of the 39 clean scoreable runs and sorted them by one question: does some test touch the higher-id account of a transfer pair while transfers run, by a path that doesn't take the lower id's mutex? Call that the right shape. It's a deposit or withdrawal on that account, or transfers across overlapping pairs.

No wrong-shape test caught the mutant in any cell, which is what the mechanism predicts. The column that varies is how often each prompt produced the right shape.

The TDD test bullet names deposits and withdrawals, and 8 of 9 scoreable TDD runs at 128k put one where it mattered. The spec test bullet asks for "at least one concurrency test with real threads", and the models answered with the deadlock test: two accounts, both directions, a conservation check. A good test, of a different bug.

Review can't change the picture by construction. Its second turn is built from the names of failing scorer checks, and halflock isn't a check. Ten of the twelve 128k review prompts contain no word about concurrency, locks, threads or stress. The one run told that "behaviour around concurrency is not right" fixed its code, finished 15/15 on hidden core, and kept its two-account test.

The TDD bullet asks for both directions "including between the same two accounts", which is the shape that can't see this bug. It never says the deposits and withdrawals must hit accounts that transfers are using. qwen38pi followed it to the letter.

Two right-shaped tests that still passed#

Two TDD runs at 128k had the right shape and still missed. qwen38 rang transfers around six accounts with deposits and withdrawals on all of them, 8 threads × 1200 operations. thinkingcap withdrew from b every fourth operation while transfers ran both ways, also 8 × 1200.

I haven't diagnosed either. The scorer runs each mutant once, and a race that doesn't fire in one execution reads as missed. Re-running them needs builds on the box that serves the benchmark, and it was busy with the next queue while this was written.

What three runs per model can't say#

• Three runs per model per cell. Twelve rows per cell is small. Spec's 4 of 11 against review's 0 of 9 at 128k fits noise as well as anything; both arms start from the same prompt.
• 128k also meant four hours. The 128k arms changed the context window and the wall-clock budget together. TDD at 64k with four hours separates them: more runs got far enough to score (8 vs 4), and the catch rate fell to 2 of 8, so the hour limit was hiding misses, not catches.
• Equivalent, na and absent rows say nothing about tests. At 64k they are most of each cell (the table's right-hand columns). At 128k there are na rows (1 in spec, 2 in review, 3 in TDD) where the rewrite didn't recognise the locking.
• The shape classification is mine. I read each test against its own service.zig. It is a reading of 39 test files, not a scorer metric.

On the test side, swift15's spec run shows the cheapest change: point a deposit thread at the account the transfer locks second. It is necessary, not sufficient, as the two misses above show. A concurrency test only checks the locks its threads contend on.

Correction, 16 October: review's one catch had read the hidden suite#

The first version of this post gave review at 128k one CAUGHT, 1 of 10 scoreable. That run, thinkingcap's first 128k review run, is tainted. The harness scored turn 1 into a directory next to the workspace and left the scorer's work copy there. In turn 2 the model found it and grepped the hidden suite's helper file, zz_hidden_helpers.zig. That file also holds the lockstep barrier the hidden concurrency checks use to start every worker at once. Then it deleted its run logs, the only thing the answer-key scan reads. With the run out, review at 128k has no clean CAUGHT: 0 of 9.

The record is kinder than that sounds, and the run still doesn't count. Its greps went after memory-leak and receipt code, and none of their output shows the barrier. Its turn-1 score sheet, written at the end of turn 1, before the model could read the copy, already reports halflock CAUGHT by its own test "transfers run concurrently with deposits and withdrawals", and the final code has a test of that name that deposits into a and withdraws from b, the higher id. A run that read the suite is out regardless. The bench closed the leak on 16 October.

Filed under: zig, concurrency, testing, llm, local-llm

Read next#

• Zig's defer ends at the closing brace: the real bug this mutant copies, a lock released before the statements it was meant to guard.
• I blamed OpenCode; a 2-bit model passed Zig in it: the same Zig task at 64k, and the client that turned out not to be the variable.

Sources#

• Zig 0.16.0 language reference: defer: "Executes an expression unconditionally at scope exit."
• Mutation testing (Wikipedia): goals include identifying "weak tests (those that never kill mutants)"; definition of equivalent mutants.
• Benchmark artifacts (local, not published): t15-ledger-zig results rows for tags gz11/gz21/gz22, steerrevz11/z21/z22, tdd11-13, speclong11-13, steerrevlong11-13 and tddlong11-13; per-run score sheets, model test files and service.zig; the scorer's apply_halflock and the t15-nostress selftest arm; prompts t15-ledger-zig.txt and t15-ledger-zig.tdd.txt.