Health 200, API 200, Prometheus 000

· 10 min read
Contents

Health 200, API 200, and one line missing#

This is the result file of a smoke test against the Spring Boot reactive-synth-more-io native image, freshly rebuilt on the GraalVM 25i4 builder:

up rc=0
healthy=1 after 2x2s
health: HTTP 200  api: HTTP 200
restarts=0 oom=false status=running
exceptions: 6
[...]
down rc=0

Healthy on the second poll. No restarts, no OOM. The script also writes a prom HTTP <code> line for /prometheus, and that line is not in the file. It is not a failure line. It is no line at all.

A probe of my own against the same container, with a 30-second cap, printed this (the FAILED: prefix comes from an output wrapper on that shell, not from curl):

FAILED: curl HTTP 000 bytes=0 t=30.001809
rc=28
[...]

Zero bytes in 30.0018 seconds, exit code 28, curl's timeout. In the same minute the app answered /api/observatory-conditions with a 200 in 0.54 s. Every endpoint a health check looks at was fine. The only endpoint the metrics scraper reads, every two seconds for the whole run, never answered.

A smoke test with no deadline held the docker lock for two hours#

The missing prom HTTP line is missing because the curl that should have written it never returned. The smoke script had no timeout on any curl:

curl -s -w '\nHTTP %{http_code}\n' http://localhost:8181/health > $L/health.txt 2>&1
curl -s -w '\nHTTP %{http_code}\n' "http://localhost:8181$api" > $L/api.txt 2>&1
curl -s -o $L/prom.txt -w 'prom HTTP %{http_code}\n' http://localhost:8181/prometheus >> $L/result.txt

Upgrading 18 branches means a lot of image builds and smoke runs competing for one Docker daemon, so every one of them runs under flock on a shared lock file. A curl that waits forever inside flock holds that lock forever. pstree and /proc/locks showed the queue:

[...]
bash,1187652 -c...
  `-flock,1187653...
      `-smoke.sh,1207335...
          `-curl,1208292 -s -o...
1: FLOCK  ADVISORY  WRITE 1187653 00:33:312489 0 EOF
1: -> FLOCK  ADVISORY  WRITE 1188239 00:33:312489 0 EOF
1:  -> FLOCK  ADVISORY  WRITE 1189488 00:33:312489 0 EOF
1:   -> FLOCK  ADVISORY  WRITE 1193044 00:33:312489 0 EOF
1:    -> FLOCK  ADVISORY  WRITE 1207337 00:33:312489 0 EOF
[...]

One holder, four waiters on the same inode: three docker compose build jobs and another batch of smoke tests. The app's log puts the first scrape at 12:13:53 UTC. I killed curl 1208292 at 14:22:02. That is two hours and eight minutes of a healthy container, a stuck curl, and a build queue that did not move, and nobody looked at it in between. The kill let the script finish, write the result file above, and run down -v.

So that is two bugs: a scrape that never answers, and a harness that let it block everything else. The second one has a one-flag fix. The first one is a sequel.

Part 15's silent metric, now with a hang#

Error Is Not Exception, part 15 of this series, covered this exact method: Micrometer's ProcessorMetrics reads OperatingSystemMXBean.getProcessCpuTime() through reflection, and Micrometer's own native metadata registers the type without that method. On GraalVM 25.0.2, getMethod("getProcessCpuTime") on a partially registered type threw NoSuchMethodException, Micrometer caught it, and process_cpu_time_ns_total was quietly never registered. A missing metric, nothing else.

The 2026-09-01 benchmark run was built on 25.0.2, and its raw scrapes still show that. Micronaut reactive-r2dbc-synth, native at 256 MB:

$ ls raw-prom | wc -l
213
$ grep -l "^process_cpu_time" raw-prom/* | wc -l
0
$ grep -h "^process_cpu" raw-prom/20260901-223852.prom
process_cpu_usage 6.531678641410843E-4

213 scrapes, all answered, none with CPU time. The same gap on 25i4 produced this:

org.graalvm.nativeimage.MissingReflectionRegistrationError: Cannot reflectively invoke method 'public abstract long com.sun.management.OperatingSystemMXBean.getProcessCpuTime()'. To allow this operati
[...]
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.reflect.MissingReflectionRegistrationUtils.reportInvokedExecutable(MissingReflectionRegistrationUtils.java:111)
	at java.base@25.0.4.1.1/java.lang.reflect.Method.acquireMethodAccessor(Method.java:144)
	at java.base@25.0.4.1.1/java.lang.reflect.Method.invoke(Method.java:562)
	at io.micrometer.core.instrument.binder.system.ProcessorMetrics.invoke(ProcessorMetrics.java:148)
	at io.micrometer.core.instrument.binder.system.ProcessorMetrics.lambda$bindTo$2(ProcessorMetrics.java:138)
[...]

Read the frames bottom-up. Line 138 of ProcessorMetrics in Micrometer 1.17.1 is the FunctionCounter lambda, and that counter is only registered if detectMethod returned a non-null Method when the binder was constructed, long before any scrape. Line 148 is the method.invoke(...) inside invoke(), which runs on every scrape. So on 25i4 the bind-time lookup succeeded, the counter exists, and the failure moved to the scrape-time call. The message says so in its own words: cannot reflectively invoke.

Part 15's table had three rows. The middle row, type registered and method not, used to read "NoSuchMethodException, caught, metric absent". On this builder it reads "Method returned, invoke() throws an Error". Micrometer 1.17.1 still catches IllegalAccessException | IllegalArgumentException | InvocationTargetException around that call, so the Error goes straight through. Part 15 had already hit this path once, after a metadata regeneration left the method registered for query but not for invoke, and the app crashed. That was one regenerated metadata file, caught at once. On 25i4 the same path reaches every native binary built with the gap, through a builder default.

The GraalVM 25.1 release notes contain the one change that matches: "Made --future-defaults=complete-reflection-types default. All reflective operations on types registered for reflection will now return complete results." 25i4 is GraalVM 25.4.4.1.1, so it carries that default. Complete results for a registered type means getMethod finds a method that has no invocation metadata, and the reachability-metadata docs say Method#invoke without that metadata throws MissingReflectionRegistrationError. That chain is consistent with the stack trace and the error text. It is still an inference from the release note and the docs; I have not read the GraalVM code that implements the default.

Why Reactor turns a LinkageError into a request that never ends#

An Error in a scrape could have been a 500. On Spring WebFlux it became silence, and the log says who decided that:

[boundedElastic-1] WARN reactor.core.Exceptions - throwIfFatal detected a jvm fatal exception, which is thrown and logged below:

followed, in the same millisecond, by Scheduler worker in group main failed with an uncaught exception. Spring's WebFlux actuator runs the blocking scrape() on boundedElastic through subscribeOn (the ElasticSchedulerInvoker frame in the trace). MissingReflectionRegistrationError extends LinkageError, and reactor-core 3.8.7, the version both apps resolve, classifies that as fatal to the JVM:

public static boolean isJvmFatal(@Nullable Throwable t) {
	return t instanceof VirtualMachineError ||
		t instanceof ThreadDeath ||
		t instanceof LinkageError;
}

and throwIfFatal throws it rather than signalling it:

if (isJvmFatal(t)) {
	LOGGER.warn("throwIfFatal detected a jvm fatal exception, which is thrown and logged below:", t);
	throw (Error) t;
}

The Javadoc spells out the consequence: such errors "would always be thrown by operators instead of propagation through onError, potentially interrupting progress of Flux/Mono sequences." No onError means no error handler, and no error handler means nobody writes a status line. The worker task dies, the Mono behind the HTTP exchange never completes, and the connection stays open until the client gives up. With no --max-time, the client was never going to.

Micronaut fails fast, Spring MVC doesn't fail at all#

The same upgrade hit Micronaut's reactive-r2dbc-synth: health 200, API 200, scrape 000. The smoke's log scan:

[...]
--- /health
{"status":"UP"}
--- /api/iss/current
http=200 bytes=287 t=0.005790
--- /prometheus
[...]
prometheus http=000 lines= process_cpu=
PROM_FAIL
MissingReflectionRegistrationError count: 1
--- log scan
[...]
101:java.io.IOException: Failed to stream Prometheus scrape
130:Caused by: org.graalvm.nativeimage.MissingReflectionRegistrationError: Cannot reflectively invoke method 'public abstract long com.sun.management.OperatingSystemMXBean.getProcessCpuTime()'. To allow this operation, add the following to the 'reflection' section of 'reachability-metadata.json' and rebuild the native image:
[...]

Micronaut wraps the Error in an IOException, and the connection ends before a status line is sent. curl never created its output file (the two elided lines after --- /prometheus are the shell complaining about that). This one was not a hang: the app logged "Startup completed" at 14:40:41.8, and the smoke's output file, written last after down -v, is stamped 14:40:44.5. From startup to teardown took under three seconds, scrape included, so no 30-second timeout was involved.

The Micronaut gap is the same gap; only the reason the registration was missing differs. Its five sibling branches got a /prometheus test and a tracing-agent metadata regeneration back in May, which registers getProcessCpuTime. This branch never did, and its app metadata had no OperatingSystemMXBean entry at all, leaving only Micrometer's partial one.

The three Spring MVC branches were never broken. The MVC native returned 200 before any fix, with process_cpu_time_ns_total 1.7E8 in the body. The registration comes from neither the app nor Micrometer. The GraalVM reachability-metadata repository's tomcat-embed-core entry registers getProcessCpuTime, conditioned on org.apache.tomcat.util.modeler.Registry being reached. I checked that entry in the 1.1.8 metadata repository zip; the 1.1.14 zip is not cached here, so the proof for the version in use is the MVC body itself. A raw scrape from the September MVC native run has process_cpu_time_ns_total too. A WebFlux app has no Tomcat, so it never gets the entry. I have not tested what MVC does without it.

Only one of the three Spring reactive natives was smoked before the fix. The other two share the gap (same Micrometer, same WebFlux actuator, no registration) and got the same file before their first native smoke.

It was not broken all along#

My first write-up of the Micronaut failure said the branch had been hanging since it was created, and that every past native row for it had been scraped through a dead endpoint. That would have invalidated a lot of published numbers. The 2026-09-01 results say otherwise: all five native scenario directories for that branch have 213 prom.csv rows, every one with an uptime value. The 213 raw scrapes above answered. On 25.0.2 the gap cost one metric. On 25i4 it cost the endpoint.

The fix: ten lines of JSON in a directory Spring can't overwrite#

The Spring reactive branches now carry this, at src/main/resources/META-INF/native-image/dev.tsvinc/micrometer-process-cpu-time/reachability-metadata.json:

{
  "reflection": [
    {
      "type": "com.sun.management.OperatingSystemMXBean",
      "methods": [
        { "name": "getProcessCpuTime", "parameterTypes": [] }
      ]
    }
  ]
}

It sits in its own directory because Spring's process-aot writes its own reachability-metadata.json into the app's existing META-INF/native-image path and replaces whatever was there. The same overwrite had been eating a native-image flag on these branches since September; it gets its own post, The Flag Was Verified. The File Was Never Read. Micronaut's fix (c8251989) is the May treatment it missed: a /prometheus test plus a fresh tracing-agent run. After both, every native returns /prometheus 200:

reactive-synth-more-io native: PROM_ASSERT=PASS health: HTTP 200  api: HTTP 200 exceptions: 0  MissingReflectionRegistrationError: 0 
reactive-r2dbc-synth native: PROM_ASSERT=PASS health: HTTP 200  api: HTTP 200 exceptions: 0  MissingReflectionRegistrationError: 0 
reactive-synth native: PROM_ASSERT=PASS health: HTTP 200  api: HTTP 200 exceptions: 0  MissingReflectionRegistrationError: 0 
[...]

Every curl gets a deadline, and the scrape gets an assertion#

The harness fix matters more than the JSON, because the next missing registration will not announce itself either. Every curl in the smoke scripts now carries --max-time, and /prometheus is no longer something the smoke records; it is something the smoke checks:

pc=$(curl --max-time 30 -s -o $L/prom.txt -w '%{http_code}' http://localhost:8181/prometheus)
echo "prom HTTP $pc" >> $L/result.txt
[ "$pc" = 200 ] && echo "PROM_ASSERT=PASS" >> $L/result.txt || echo "PROM_ASSERT=FAIL" >> $L/result.txt

The old smoke waited on the endpoint a load balancer cares about and only recorded the one the benchmark depends on. The first one passed on the second poll. The second one was given two hours.

Filed under: java, graalvm, native-image, micrometer, spring-boot, micronaut, reactive, debugging, testing

Sources#