Every Route Registered. Every Request 404'd.

· 6 min read · java, micronaut, reactor, reactive, benchmarks, debugging
The Memory Benchmark · Part 25
Contents

Thirty-three routes registered. Startup log confirms it. The /routes management endpoint lists every one. And under load, every single request returns 404.

This is the reactive Micronaut branch of the space observatory benchmark — 28 upstream API calls, 10 HTTP clients, all pointed at a Go mock server that deliberately injects failures (HTTP 500, 504-after-delay, abrupt connection close). The imperative sibling branch works. The lighter synth variant works. This one: 100% 404.

The graveyard of wrong hypotheses

I spent fifteen build/test cycles chasing route registration. The reasoning was obvious: 404 means the route doesn’t exist. So:

  • Code regression? Diffed against the working sibling. Identical controller.
  • Docker layer cache? Rebuilt with --no-cache. Same 404.
  • AOT compilation? Disabled entirely (-Dmicronaut.aot.enabled=false). Same 404.
  • Bean definition missing from jar? Extracted it: SurveyController$Definition.class present.
  • Filters or @Requires blocking the route? None matched.

Each experiment took 3–5 minutes (native image rebuild, container restart, curl). Fifteen cycles. An hour of proving the route exists.

One trace log

The breakthrough was not cleverness. It was finally pointing the right diagnostic at the problem.

LOGGER_LEVELS_IO_MICRONAUT_HTTP_SERVER=TRACE

One request. Two log lines:

TRACE RequestLifecycle  - Matched route GET /api/earth-survey to controller class SurveyController
DEBUG RoutingInBoundHandler - Response 404 - GET /api/earth-survey

The route matched. The controller ran. The 404 came after dispatch.

In Micronaut’s reactive stack, an empty Mono — a Mono that completes without emitting a value — renders as HTTP 404. The framework interprets “I have nothing to return” as “this resource does not exist.” The controller returned Mono<EarthSurveyResponse>. Under mock-injected failures, it returned Mono.empty(). The 404 was Micronaut doing exactly what it was told.

How Mono.zip eats a signal

The fan-out aggregates 15 upstream calls using Mono.zip:

Mono<Tuple8<...>> block1 = Mono.zip(
    seismicClient.earthquakes()
        .doOnNext(r -> successes.incrementAndGet())
        .onErrorReturn(EMPTY_EARTHQUAKE),
    seismicClient.volcanoes()
        .doOnNext(r -> successes.incrementAndGet())
        .onErrorReturn(EMPTY_VOLCANO),
    // ... 13 more
);

return Mono.zip(block1, block2)
    .flatMap(tuple -> {
        EarthquakeResponse eq = tuple.getT1().getT1();
        // aggregate, compute, persist, return response
    });

The resilience strategy: each client call has .onErrorReturn(EMPTY_X) — if the upstream errors, substitute an empty sentinel and keep going. Graceful degradation. The survey returns partial data rather than failing.

One problem. onErrorReturn handles errors. It does not handle empty completion.

Reactor’s Mono.zip has a documented behavior that bites exactly here: if any source Mono completes empty, the entire zip completes empty. No error signal. No partial result. The remaining sources get cancelled. The zip produces Mono.empty(), the flatMap never runs, and the controller returns nothing.

With 15 zipped calls and the mock injecting failures on roughly 18% of requests, the probability of all 15 succeeding is roughly 0.82^15 = 5%. The probability of at least one empty is 95%. But since the zip batches calls into two Tuple groups that are themselves zipped, any single empty source kills the whole thing. In practice: 100% of requests 404’d.

What Micronaut @Client does with upstream failures

The root cause behind the empty: Micronaut’s declarative @Client does not surface upstream HTTP failures the same way Spring’s RestClient or Quarkus’s REST client do.

Spring throws on 5xx — HttpServerErrorException from RestClient, WebClientResponseException from WebClient. Quarkus throws WebApplicationException. Both frameworks surface the failure as an error signal. onErrorReturn catches them.

Micronaut’s reactive @Client (5.0.0) returns Mono.empty(). This is by design — the internal ReactorReactiveClientResultTransformer maps Reactor types to behave like RxJava’s Maybe, where an absent value is empty completion rather than an error. There’s no way to opt into error-on-failure with Mono/Flux return types without replacing that transformer.

So the chain becomes:

Upstream 500 → @Client returns Mono.empty()
             → onErrorReturn(EMPTY_X) does nothing (no error to catch)
             → Mono.zip sees empty source → cancels all others → Mono.empty()
             → flatMap never runs → controller returns Mono.empty()
             → Micronaut renders 404

A registered route. A matching request. A controller that ran. A 404.

The fix: 73 lines

seismicClient.earthquakes()
    .doOnNext(r -> successes.incrementAndGet())
    .onErrorReturn(EMPTY_EARTHQUAKE)
    .defaultIfEmpty(EMPTY_EARTHQUAKE),  // ← this

Every client call gets .defaultIfEmpty(EMPTY_X) after .onErrorReturn(EMPTY_X). Seventy-three call sites across five survey services, plus one in HealthService:

// Before
return source.timeout(TIMEOUT).map(r -> "UP").onErrorReturn("DOWN");

// After
return source.timeout(TIMEOUT).map(r -> "UP").onErrorReturn("DOWN").defaultIfEmpty("DOWN");

The fix is mechanical. The diagnosis was not.

The imperative sibling: same cause, different shape

The blocking version of the same fan-out submits each client call to a newVirtualThreadPerTaskExecutor:

private <T> Future<T> submit(Callable<T> call, T fallback, AtomicInteger counter) {
    return executor.submit(() -> {
        try {
            T result = call.call();
            counter.incrementAndGet();
            return result;
        } catch (Exception e) {
            return fallback;
        }
    });
}

Same resilience strategy: catch exception, return fallback. Same gap: Micronaut’s blocking @Client returns null on upstream failure — not an exception. The null flows past the catch, gets counted as a success, and reaches computeSeismicRisk(eq, ...) where eq is null.

NullPointerException: Cannot invoke EarthquakeResponse.magnitude() because "eq" is null

At an ~18% per-call failure rate across 15–28 calls, roughly 95% of survey requests hit at least one null and 500’d. The fix:

T result = call.call();
if (result == null) {
    return fallback;
}
counter.incrementAndGet();
return result;

Three lines. Five files. Same root cause as the reactive bug — Micronaut @Client expresses upstream failure as absence (null/empty) rather than exception, wearing a different hat.

The debugging lesson

For a registered route returning 404, trace request dispatch first. The question is not “did the route register?” — the question is “did the handler run, and what did it return?”

LOGGER_LEVELS_IO_MICRONAUT_HTTP_SERVER=TRACE

Matched route → Response 404 means the handler ran and returned empty. That points straight at the reactive return path. I burned an hour on build/registration hypotheses before enabling dispatch tracing — the trace would have answered the question in one request.

The broader pattern

In Reactor, error and empty are distinct signals. Resilience code that only covers errors (onErrorReturn, onErrorResume, doOnError) silently breaks on empty completion. Mono.zip makes this lethal: one empty source poisons an entire aggregation.

If you’re porting a fan-out from a framework that throws on upstream failure (Spring, Quarkus) to one that returns empty (Micronaut with Reactor), every onErrorReturn needs a matching defaultIfEmpty. The type system won’t warn you. The tests that don’t inject failures won’t catch it. The benchmark that doesn’t use the mock won’t see it.

And when it breaks, it looks like your routes don’t exist.

Sources