Four Bugs, Three Exit Codes From a Benchmark Orchestrator

· 6 min read
Contents

The orchestrator reported ALL_DONE after one branch. The map held fourteen branches and 36 scenarios.

Twenty-four framework branches. Docker Compose up, k6 ramp to 200 VUs, Prometheus scrape, tear down, next branch. The loop script is a few dozen lines of bash — read a branch map, check out worktrees, run benchmark-all.sh, collect results. None of this is complicated, which is exactly why all four bugs took so long to find.

Exit 0: The Loop That Ran Once#

# simplified
while read -r branch scenarios; do
  log "Starting $branch ..."
  cd "$WORKTREE_DIR/$branch"
  bash scripts/benchmark-all.sh $scenarios
  log "Done: $branch"
done < branches.map

Fourteen branches in branches.map. One branch runs. ALL_DONE. Exit 0.

The bug: docker reads stdin. So does benchmark-all.sh, which calls docker compose up, which calls docker, which — on some code paths — inherits the shell's stdin. That stdin is the file descriptor pointing at branches.map. Docker reads a few bytes from it (or its subprocesses do, or the PTY layer does — the exact mechanism varies by Docker version and terminal state). The file offset advances. By the time the first benchmark-all.sh finishes, read sees EOF on the map file.

One branch ran. The loop exited cleanly. The log said ALL_DONE. No error, no warning, no indication that thirteen branches were silently skipped.

The fix is two lines:

while read -r -u 9 branch scenarios; do
  log "Starting $branch ..."
  cd "$WORKTREE_DIR/$branch"
  bash scripts/benchmark-all.sh $scenarios </dev/null
  log "Done: $branch"
done 9< branches.map

File descriptor 9 feeds the loop. stdin (</dev/null) feeds the child. Docker can read /dev/null all day — it will never advance the branch map.

The -u 9 and </dev/null are belt-and-suspenders. Either one fixes it. Use both, because the next person to add a command inside that loop won't know why the file descriptor trick exists.

Exit 123: Dead in Under a Second#

The orchestrator iterates all twenty-four branches. Six of them are "plain" branches (imperative-jdbc, reactive-r2dbc) — no mock server, no synthetic scenarios. Their benchmark-all.sh has this near the top:

set -euo pipefail

_SYNTH=$(ls "$SCRIPT_DIR"/bench/*-synth-*.sh 2>/dev/null \
  | xargs -n1 basename 2>/dev/null \
  | sed 's/\.sh$//' | sort | tr '\n' ' ')
SCENARIOS="${*:-${_SYNTH:-native-unlimited native-128mb ...}}"

On a plain branch, there are no *-synth-*.sh files. The glob stays literal. ls produces nothing. xargs receives no input — and without -r (GNU --no-run-if-empty), it runs basename with zero arguments. basename exits 1. xargs translates that to exit 123 (its code for "a child exited 1-125"). pipefail makes the pipeline return the status of the last command that failed, the rightmost non-zero one: ls exited 2, xargs 123, and everything after xargs succeeded, so the pipeline returns 123. set -e kills the script.

Exit 123. In under a second. No scenario runs. My orchestrator logged 123 as branch one's return code; one with its own set -e would have stopped the entire run right there.

The irony: 2>/dev/null hides the stderr from ls, but xargs is the one producing the fatal exit code. The script looks like it ran and found nothing. It died before evaluating SCENARIOS.

The bash fix went into the orchestrator: sed -i 's/set -euo pipefail/set -uo pipefail/' scripts/benchmark-all.sh in each worktree before running it, since the patched script already wraps each scenario in bash "$script" || log. That made the crash go away, and with it the only thing keeping plain branches from running. Without -e, the empty-glob line falls through to the default scenarios, and on a plain branch the defaults run without the mock, against the real upstream URLs. The relaunched run then load-tested api.nasa.gov from two plain branches before anyone noticed; the previous part has that story. The sed stayed. The real fix was upstream of it: never hand a plain branch to the orchestrator, and pass synth scenarios explicitly.

Exit 143: The Cleanup That Killed Itself#

After a run, you want to kill the orchestrator and all its children. The obvious line:

pkill -f 'rerun-all.sh'

This works. Too well. pkill -f matches the pattern against the full command line of every process. pkill never matches itself, but it does not skip its ancestors, and the shell running the cleanup carries rerun-all.sh in its own command line: the pattern is right there in the text of the command. So pkill sends that shell SIGTERM along with the orchestrator. The cleanup dies on that line, and every command after it, the container and worktree removal included, never runs.

A bash parent reports the dead shell as exit 143: 128 + 15, SIGTERM, the default signal (not the -9 in the fix below). The cleanup stopped at the pkill line and reported nothing but a number.

Fix: make the pattern text not match itself.

pkill -9 -f 'rerun-all[.]sh'

The regex [.] matches a literal dot — so rerun-all[.]sh still matches processes whose cmdline contains rerun-all.sh. But any process whose cmdline contains the literal text rerun-all[.]sh (the pkill command, its wrappers) doesn't match, because [ is not a dot.

One character class. The difference between a cleanup that runs to the end and one that stops at its first line. procps-ng also has a flag for exactly this: pkill -A (--ignore-ancestors) skips every ancestor of the pkill process.

Exit 0: Half the Data Is Gone#

This one doesn't crash. It doesn't even error. It just silently drops data.

The benchmark report is evidence-first: it walks benchmark-results/<branch>/ and picks the newest timestamped directory as the "latest" run. Each run directory contains one subdirectory per scenario (native-256mb/, jvm-synth-256mb/, etc.) with prom.csv, k6.txt, and OOM markers.

You rerun two scenarios that had issues — maybe a container OOM'd, maybe k6 timed out. You run them into a fresh timestamped directory because you want a clean run:

RESULTS_DIR=benchmark-results/micronaut-synth/20260702-150000 \
  bash scripts/benchmark-all.sh jvm-synth-256mb jvm-synth-512mb

The new directory becomes the latest. It has two scenario subdirectories. The previous run had the full set. The report picks the new directory, finds two scenarios, reports two scenarios. The rest silently vanish from the report.

No error. The report generated successfully. It generated less.

The fix is copy-refresh:

PREV=$(ls -td benchmark-results/$BRANCH/*/ | head -1)
NEW="benchmark-results/$BRANCH/$(date +%Y%m%d-%H%M%S)"
cp -a "$PREV" "$NEW"
rm -rf "$NEW/jvm-synth-256mb" "$NEW/jvm-synth-512mb"
RESULTS_DIR="$NEW" bash scripts/benchmark-all.sh jvm-synth-256mb jvm-synth-512mb

Copy the previous complete run. Delete the scenarios you're rerunning. Run into the copy. The new directory is complete: the preserved scenarios plus two fresh ones. The old directory stays as a backup.

The Pattern#

Four bugs. Two report exit 0. One reports 123. One reports 143. All of them look like successful completion if you're not watching closely. The full rerun was estimated at 34 to 40 hours, and its failure mode is less data, silently.

There is no substitute for counting the output. How many branches ran? How many scenarios per branch? How many data points in the report? If the orchestrator says ALL_DONE and the answer to any of those is "fewer than last time," something ate your benchmarks.

Filed under: bash, benchmarks, docker, devops

Sources#