Quarkus 12,563 vs Spring 820 (Both Numbers Are Wrong)

· 5 min read · java, graalvm, native-image, benchmarks, prometheus, observability
The Memory Benchmark · Part 18
Contents

Part 18 of the Micronaut native image series. Follows Part 17, which found that reactive and imperative hit the same throughput ceiling. Assumes familiarity with Prometheus counters.

Two numbers that can’t both be right

The framework comparison benchmark finished. Three frameworks, same app, same k6 profile, same 256 MB native image container. The analysis script printed:

Spring Boot:  HTTP served:  820 requests   avg=2.341ms
Quarkus:      HTTP served:  12,563 requests  avg=1.892ms

Quarkus crushes Spring by 15x. Write the blog post. Ship the chart.

Except k6 — the load generator, the thing that actually sent the requests — reported this:

Spring Boot:  http_reqs: 67,334   (103.2 req/s)
Quarkus:      http_reqs: 12,392   (19.1 req/s)

Spring served 5x more. The analysis script didn’t just get the magnitude wrong. It inverted the winner.

The counter

The analysis script computed throughput from an app-side Prometheus counter: http_server_requests_seconds_count. A Prometheus scraper polled the app every 2 seconds and wrote each value to a CSV. The script’s logic was three lines of awk:

NR>1 {
    last_http=$21
    if(first_http=="") { first_http=$21 }
}
END {
    reqs = last_http - first_http
}

Last value minus first value. For a monotonically increasing counter, this is correct. For a counter that resets to zero mid-run, this computes garbage.

What happened to Spring

Spring Boot’s native image OOM-killed under the 300-VU peak. GraalVM’s Serial GC defaulted the max heap to ~80% of the 256 MB container — roughly 205 MB. Heap plus Netty channel buffers plus native image overhead breached 256 MB at peak concurrency.

The docker-compose file had restart: on-failure. So the container came back up. And got killed again. Five times in the last three minutes of the run.

The counter trajectory looked like this:

scrape #    http_server_requests_seconds_count
   1                  3
  ...               ...
  147            56,012
  148                 0     ← restart
  149               187
  ...               ...
  162             4,891
  163                 0     ← restart
  ...  (3 more resets)
  201               823     ← last scrape

823 - 3 = 820. That’s the “throughput.”

The real total — the sum of all the segments between resets — was somewhere north of 60,000. The script saw 820 because it only looked at two numbers.

Why Quarkus looked right

Quarkus didn’t crash. Its counter was monotonic for the entire run. 12,563 - 171 = 12,392 — close enough to k6’s 12,392. Correct by luck. If Quarkus had restarted once, it would have lied too.

The comparison was wrong in both directions: Spring underreported by 82x, Quarkus happened to tell the truth. A report that’s correct by coincidence is worse than a report that’s always wrong — at least consistent wrongness gets investigated.

The tell

Cross-checking the app counter against k6’s http_reqs showed the disagreement: 820 vs 67,334. An 82x gap.

The instinct was “wrong metric” — maybe the counter tracks something different from what k6 measures. But the average latencies matched: both sources agreed on ~2.3 ms per request. Same metric family, same shape, wildly different count. That pattern has one explanation: the counter reset.

“Looks fine”

The Prometheus scraper also polled container RSS via docker stats every 2 seconds. The maximum recorded value: 233 MB. Well under the 256 MB limit.

Container RSS: avg=198 MB  max=233 MB

By RSS alone, the run “looked fine.” The OOM spike happened between two-second samples. The container breached 256 MB, got killed, restarted, and was back under 200 MB before the next sample landed. Docker stats caught nothing.

The restart count — which the script didn’t report — was 5. That was the actual evidence.

The fix

Six lines of awk, replacing three:

NR>1 {
    cur_http=$21
    if(prev_http!="") {
      dh = cur_http - prev_http
      if(dh >= 0) http_total += dh
      else resets++
    }
    prev_http = cur_http
}
END {
    if(http_total > 0)
      printf "HTTP served:  %.0f requests\n", http_total
    if(resets > 0)
      printf "App restarts: %d (counter reset — OOM/crash)\n", resets
}

Instead of last - first, sum every positive per-scrape delta. A negative delta means the counter reset — the process restarted. Count those resets and print them.

The restart count is the OOM canary. Before this fix, a crash-looping container was invisible to the analysis script. It just silently produced a small number. After: App restarts: 5 (counter reset — OOM/crash) — hard to ignore.

The charts were already correct

The cruelest part: the HTML report’s throughput charts were fine. They pulled requests-per-second from k6’s summary output — awk '/Requests\/sec \(avg\):/{print $NF}' — not from the app counter. Only the text analysis function (analyze_prom()) used the broken last - first math.

Two views of the same benchmark, one correct and one wrong. If you looked at the chart, Spring was winning. If you read the analysis text, Spring was dead. The chart and the text sat in the same report.

This is what made the bug survivable for weeks: someone always looked at one or the other, never both at once, and each view was internally consistent.

Four rules for counter-based benchmarks

1. A monotonic counter delta lies across restarts. last - first is only valid if the process ran continuously. If you can’t guarantee that — and restart: on-failure means you can’t — sum positive deltas instead.

2. The load generator is ground truth for throughput. k6, wrk, hey — whatever sent the requests knows how many it sent. The server’s self-reported counter is a useful cross-check, not a source of truth. Especially not a source of truth when the server is crash-looping.

3. Coarse sampling hides spikes. Two-second polling missed a 256 MB OOM spike entirely. The max RSS read 233 MB. The restart count was 5. Prefer event-based evidence (OOM-kill events, restart counts) over sampled evidence when diagnosing crashes.

4. When two numbers disagree by 82x, suspect a reset. The natural instinct is “wrong metric” or “misconfigured scraper.” But if the shape matches (latency averages agree, metric names are right) and only the count diverges by orders of magnitude, a counter reset is the parsimonious explanation. Check for restarts before questioning the metric identity.

Sources