Two ways to mistranslate a retry

Part 21 of the Micronaut native image series. Follows Part 20, where two stacked bottlenecks throttled Quarkus. Assumes familiarity with declarative HTTP clients and retry semantics.

Porting 10 HTTP clients from Micronaut to Quarkus and Spring Boot produced two retry bugs. One framework over-retried by ~50%. The other silently never retried HTTP 5xx at all. Both ports passed every functional test. Both shipped code comments claiming parity with the original.

The bugs were mirror images, caused by completely different mechanisms, and invisible without a benchmark that injects failures and counts upstream calls.

The reference

Micronaut’s declarative HTTP clients carry their retry policy right on the interface:

@Client("science")
@Retryable(attempts = "2", delay = "500ms")
public interface SolarClient {
    @Get("/api/v1/solar-flares")
    SolarFlareResponse solarFlares();
}

attempts = "2" means 2 total invocations – 1 initial call + 1 retry. When you omit attempts and only set delay and multiplier:

@Client("nasa-apod")
@Retryable(delay = "500ms", multiplier = "2.0")
public interface NasaApodClient { /* ... */ }

the default is attempts = "3" – 3 total calls with exponential backoff. Micronaut retries on any exception, including HttpClientResponseException from HTTP 5xx responses. A 500 from the server is an exception. Clean.

Ten clients across five groups. Each one with a .onErrorReturn(EMPTY_*) fallback downstream, so failures degrade gracefully instead of propagating.

Bug 1: Quarkus over-retried

MicroProfile Fault Tolerance’s @Retry looks like a direct translation:

// WRONG: naive 1:1 port
@RegisterRestClient(configKey = "science")
@Retry(maxRetries = 2, delay = 500, delayUnit = ChronoUnit.MILLIS)
public interface SolarClient { /* ... */ }

The parameter even sounds compatible. It isn’t.

maxRetries is the number of retries after the initial attempt. @Retry(maxRetries = 2) = 3 total calls (1 + 2). Micronaut’s @Retryable(attempts = "2") = 2 total calls. The naive port added one extra retry to every single client.

The NASA clients were worse. Micronaut’s default attempts = "3" became @Retry(maxRetries = 3) = 4 total calls instead of 3.

Every failure path burned ~50% more upstream calls than the original. On a 28-API fan-out with 1-3% injected failures, that’s not theoretical – it reshapes the tail latency distribution and the mock server’s hit counters.

The fix is arithmetic:

// FIXED: maxRetries = attempts - 1
@RegisterRestClient(configKey = "science")
@Retry(maxRetries = 1, delay = 500, delayUnit = ChronoUnit.MILLIS)
public interface SolarClient { /* ... */ }

The NASA clients needed one more thing. MicroProfile’s @Retry only does fixed delay – there’s no multiplier parameter. Exponential backoff requires SmallRye’s @ExponentialBackoff:

@RegisterRestClient(configKey = "nasa-apod")
@Retry(maxRetries = 2, delay = 500, delayUnit = ChronoUnit.MILLIS)
@ExponentialBackoff  // factor=2 by default, matches Micronaut's multiplier="2.0"
public interface NasaApodClient { /* ... */ }

One minor divergence left unfixed: SmallRye’s @Retry adds +/-200ms jitter by default. Micronaut’s doesn’t. The jitter makes the port better at avoiding thundering herds, so it stays.

Bug 2: Spring under-retried

Spring Boot doesn’t have declarative retry annotations on HTTP interfaces. The retry lives in a WebClient ExchangeFilterFunction:

// WRONG: only retries transport errors
private static ExchangeFilterFunction retryFilter(Retry retry) {
    return (request, next) -> next.exchange(request)
            .retryWhen(retry);
}

Three lines. Looks right. A code comment above it said “matching Micronaut’s effective behaviour.”

It didn’t.

WebClient’s exchange() emits HTTP 4xx/5xx as a normal onNext(ClientResponse), not as onError. The response is a value, not an exception. retryWhen only subscribes to error signals. It never sees a 500.

The synth mock injects three failure modes: immediate HTTP 500, 2-second-then-504, and connection reset. Micronaut retries all three – every 5xx is an HttpClientResponseException. The Spring port retried only the connection reset (a transport error, an actual onError). Two out of three failure modes were fast-failed to the fallback without a single retry attempt.

The effect: ~66% of injected failures that Micronaut would have recovered from were silently abandoned by Spring. Different effective upstream load. Different success/fail counts persisted to the database. The benchmark results told the story – the numbers disagreed between ports in exactly the wrong way.

The fix adds a flatMap that converts error responses to error signals before retryWhen can see them:

// FIXED: convert HTTP errors to onError before retry
private static ExchangeFilterFunction retryFilter(Retry retry) {
    return (request, next) -> next.exchange(request)
            .flatMap(resp -> resp.statusCode().isError()
                    ? resp.createException().flatMap(Mono::error)
                    : Mono.just(resp))
            .retryWhen(retry);
}

createException() drains the response body, so there’s no connection leak. After the fix, retryWhen sees an onError(WebClientResponseException) for every 5xx – same as Micronaut seeing HttpClientResponseException.

Spring’s retry counts were already correct. Retry.max(1) = 1 retry. Retry.backoff(2, Duration.ofMillis(500)) = 2 retries with exponential backoff. The Reactor Retry API counts retries, not total attempts – same as MicroProfile, different from Micronaut. But the count was irrelevant when the retry never fired.

Why both were invisible

Every client has a fallback:

// Micronaut
solarClient.solarFlares()
    .onErrorReturn(SolarFlareResponse.EMPTY)

// Quarkus (Mutiny)
solarClient.solarFlares()
    .onFailure().recoverWithItem(SolarFlareResponse.EMPTY)

// Spring (Reactor)
solarClient.solarFlares()
    .onErrorReturn(SolarFlareResponse.EMPTY)

Over-retry? The extra attempt either succeeds (no visible difference) or fails again (fallback returns empty). Under-retry? The fallback fires immediately instead of after a retry. Either way: the response is well-formed, the HTTP 200 goes to the caller, the test assertions pass.

The only signal is quantitative. Quarkus hit the mock server ~50% more on failure paths. Spring hit it ~66% less. Neither showed up in any test that checks “does the endpoint return valid JSON.” You’d need a test that asserts: “given a server that fails once then succeeds, the client calls the server exactly twice.” Neither port had one.

The count

Two lessons that look obvious in a table but were not obvious in a 10-client port:

Count base differs. Micronaut’s attempts is total invocations. MicroProfile’s maxRetries and Reactor’s Retry count retries after first. The mapping is maxRetries = attempts - 1. Off-by-one per client, compounding across 28 APIs.

Signal model differs. In Micronaut and Quarkus, an HTTP 5xx is an exception. In Spring WebClient, it’s a successful response carrying an error status code. A retry operator that watches for exceptions will never see it. The fix is three lines of flatMap, but you have to know you need them.

Verify retry behavior empirically. Count server hits. Don’t trust the annotation.

Sources

• Micronaut @Retryable API – attempts = total invocations
• MicroProfile @Retry API – maxRetries = retries after initial
• Quarkus SmallRye Fault Tolerance guide
• Baeldung: Retry in Spring WebFlux – retryWhen only reacts to onError
• Spring WebClient retry with ExchangeFilterFunction