200 Virtual Users vs api.nasa.gov (DEMO_KEY)

· 4 min read
Contents

The fix for exit 123 was one sed. It took the crash away, and the crash had been the only thing standing between a load generator and api.nasa.gov.

The Branch Map Nobody Filtered#

The benchmark suite has twenty-four branches. Eighteen are "synth" branches: their *-synth-* scenario scripts add a docker-compose-synth.yml overlay to COMPOSE_FILE, which starts a Go mock server and points every client at it. From the Micronaut overlay:

services:

  mock:
    build:
      context: ./mock-server
    # ... per-upstream latency and failure-rate knobs, healthcheck

  app:
    environment:
      MICRONAUT_HTTP_SERVICES_NASA_APOD_URL: http://mock:8080
      MICRONAUT_HTTP_SERVICES_NASA_NEO_URL: http://mock:8080
      MICRONAUT_HTTP_SERVICES_ISS_URL: http://mock:8080
      MICRONAUT_HTTP_SERVICES_OPEN_METEO_URL: http://mock:8080
    depends_on:
      mock:
        condition: service_healthy

The other six are "plain": imperative-jdbc and reactive-r2dbc, one pair per framework. No overlay, no mock, and an application.yml whose defaults are the real thing. From micronaut/reactive-r2dbc:

micronaut:
  # ...
  http:
    services:
      nasa-apod:
        url: https://api.nasa.gov
      nasa-neo:
        url: https://api.nasa.gov
      iss:
        url: http://api.open-notify.org
      open-meteo:
        url: https://api.open-meteo.com

The benchmark is synth-only. The plain branches were never in the report and never meant to be load-run. The rerun orchestrator, rerun-all.sh, iterated all twenty-four anyway.

A Crash That Was Doing a Guard's Job#

Two days earlier, a different mistake had already burned this suite: synth branches run with base scenario names, which skip the overlay. Sixteen of eighteen upstream-calling runs in the results had gone out mock-less and had to be redone. The follow-up commit taught benchmark-all.sh to pick the synth scenarios by itself when called with no arguments:

_SYNTH=$(ls "$SCRIPT_DIR"/bench/*-synth-*.sh 2>/dev/null \
  | xargs -n1 basename 2>/dev/null \
  | sed 's/\.sh$//' | sort | tr '\n' ' ')
SCENARIOS="${*:-${_SYNTH:-native-unlimited native-128mb native-256mb jvm-unlimited jvm-256mb jvm-vthreads-unlimited jvm-vthreads-256mb}}"

On a plain branch there are no *-synth-*.sh files, and the script starts with set -euo pipefail. The _SYNTH line fails, set -e ends the script with exit 123 in under a second, and no scenario runs. The next part takes that line apart.

That is exactly what happened to branch one of the rerun, micronaut/imperative-jdbc. It looked like a bash bug, so it got a bash fix: the orchestrator now seds set -euo pipefail down to set -uo pipefail in each worktree before running it. Without -e, the failed _SYNTH line leaves the variable empty, ${_SYNTH:-...} falls through to the base scenarios, base scenarios use base compose, and base compose has postgres, app and k6. No mock.

The relaunch worked. Branch one ran all seven base scenarios to completion, clean exit, against the live APIs.

Branch Five: The Second Plain Branch#

Branches two to four were synth and behaved. Branch five was micronaut/reactive-r2dbc, the second plain branch, and it was well into its k6 run when I asked the question that should have come before the relaunch: is this run using the mock at all?

One look at that branch's compose file and application.yml answered it. The k6 script on those branches ramps to 200 virtual users, then holds 500 iterations a second with up to 300. Behind those requests, the app's clients fanned out to api.nasa.gov, api.open-notify.org and api.open-meteo.com.

NASA's DEMO_KEY allows 30 requests per IP address per hour and 50 per day. Open Notify is a small free ISS position API. Open-Meteo is free and has limits of its own. Whatever those runs measured, it was not the app on its own.

The Fix: Fail Closed#

I killed the run, deleted the two plain-branch result directories it had written, and relaunched with an orchestrator that can only see synth branches and refuses to guess. Condensed:

branches=$(git for-each-ref --format='%(refname:short)' refs/heads \
  | grep -E '^(micronaut|quarkus|springboot)/' | grep synth)
for b in $branches; do
  wt="$WT_ROOT/$(echo "$b" | tr '/' '_')"
  git worktree add --quiet "$wt" "$b" || continue
  ( cd "$wt"
    # explicit synth scenarios only: never fall back to base (real-upstream) scenarios
    scen=$(ls scripts/bench/*-synth-*.sh 2>/dev/null | xargs -n1 basename | sed 's/\.sh$//' | sort | tr '\n' ' ')
    if [ -z "$scen" ]; then echo "NO_SYNTH_SCENARIOS -> abort branch"; exit 9; fi
    bash scripts/benchmark-all.sh $scen
  )
  git worktree remove --force "$wt"
done

Two filters, both explicit. for-each-ref lists local branch names with nothing else on the line, so grep synth keeps eighteen. And the scenario list comes from the branch's own synth scripts, passed as arguments, so benchmark-all.sh never reaches its default. A branch without synth scripts aborts.

The principle generalizes: a load harness should fail closed to its mocks. If the mock isn't there, the run stops. A fallback that can reach the internet is one missing overlay away from load-testing a stranger's service, and the crash I "fixed" was the only thing that had been failing closed.

Filed under: benchmarks, docker, java, devops

Sources#