Four Bugs the Tests Couldn't See

2026-07-05 · 7 min read · java, spring-boot, quarkus, graalvm, native-image, performance, testing

Contents

Part 24 of the memory benchmark series. Follows Part 23, where six HTTP client swaps each OOM’d differently. Assumes familiarity with R2DBC, Hibernate Reactive, and connection pooling.

The senior code review on the first reactive port – Spring Boot 4.1 WebFlux and Quarkus 3.33 Hibernate Reactive, same app, same tests – came back clean. Field mapping: correct. Fan-out concurrency: correct. Scoring logic: correct. Native image hints: correct. Retry semantics: fixed (that was Part 21). All tests green.

Then the native-synth-256mb benchmark ran. Four bugs. Each invisible to mvnw test, each invisible to the reviewer, each requiring concurrent load against real Postgres to exist.

Bug 1: the type the driver never told you about

Spring R2DBC. The correlation table has a created_at column: timestamp with time zone. The entity field is Instant. Micronaut’s R2DBC layer converts between them transparently. Spring Data R2DBC does not.

The r2dbc-postgresql driver returns OffsetDateTime for timestamptz columns. The entity expects Instant. There is no built-in converter. Every read of a persisted row throws:

ConverterNotFoundException: No converter found capable of converting
from type [java.time.OffsetDateTime] to type [java.time.Instant]

The /api/correlate/recent endpoint – the only one that reads back persisted timestamp columns – returned 500 on every call.

Why tests missed it: the controller tests @MockitoBean the service layer. The mock returns hand-built CorrelationResponse objects. The real findRecent() query – the one that maps a Postgres row to an entity – never runs. The more-io reference port never hit it either: its single survey_result table was write-only. No SELECT, no conversion, no error.

The fix is 28 lines of boilerplate:

@Configuration(proxyBeanMethods = false)
public class R2dbcConversionConfig {

    @Bean
    public R2dbcCustomConversions r2dbcCustomConversions(ConnectionFactory cf) {
        R2dbcDialect dialect = DialectResolver.getDialect(cf);
        return R2dbcCustomConversions.of(dialect,
                OffsetDateTimeToInstant.INSTANCE,
                InstantToOffsetDateTime.INSTANCE);
    }

    @ReadingConverter
    enum OffsetDateTimeToInstant implements Converter<OffsetDateTime, Instant> {
        INSTANCE;
        @Override public Instant convert(OffsetDateTime source) {
            return source.toInstant();
        }
    }

    @WritingConverter
    enum InstantToOffsetDateTime implements Converter<Instant, OffsetDateTime> {
        INSTANCE;
        @Override public OffsetDateTime convert(Instant source) {
            return source.atOffset(ZoneOffset.UTC);
        }
    }
}

Micronaut does this for you. Quarkus (Hibernate Reactive) does this for you. Spring makes you write it, and doesn’t tell you until a persisted row comes back.

Quarkus Hibernate Reactive. The correlateRecent endpoint loads two referenced events by ID to build the response. The natural Mutiny translation of Mono.zip(findById(a), findById(b)) is:

Uni.combine().all().unis(
    EventLog.findById(eventAId),
    EventLog.findById(eventBId)
).asTuple()

Two concurrent queries on the same reactive session. Hibernate Reactive’s Session is not thread-safe – and “concurrent Unis on the same session” counts. The session’s internal JdbcValuesSourceProcessingState stack gets corrupted:

java.lang.IllegalStateException: Illegal pop() with non-matching
JdbcValuesSourceProcessingState

The connection leaks. Under 300 VUs, the leaked connections drain the reactive pool. Rare enough to miss in a quick smoke test, frequent enough to starve the pool under sustained load.

Spring R2DBC doesn’t have this bug. It leases a connection per query, not a shared session. Two concurrent Mono.zip(findById(a), findById(b)) calls each get their own connection. No shared state to corrupt.

The fix: load sequentially.

EventLog.<EventLog>findById(corr.getEventAId())
    .onItem().transformToUni(eventA ->
        EventLog.<EventLog>findById(corr.getEventBId())
            .onItem().transform(eventB ->
                new CorrelationResponse(/* ... */)));

Why tests missed it: single-threaded. One request at a time, one session operation at a time – the Uni.combine only fires concurrently under actual concurrent load. And the controller tests never open a real session at all: they @InjectMock the service, so the findById pair never touches Hibernate.

Bug 3: 61 rps with every gauge at zero

Quarkus reactive-synth. Same app shape as Spring (4 API clients, 4 database tables). Spring hit 270.6 rps. Quarkus hit 61.

CPU: 2.7%. HTTP client queue size: 0. HTTP client active connections: 0. Every metric said nothing was saturated.

Every metric was lying.

The point-in-time gauges – http_client_queue_size, http_client_active_connections – drain between the 2-second Prometheus scrapes. A connection is acquired, used for an API call, and released before the next scrape arrives. The gauge reads 0. In this run, http_client_active_connections sat at 0 regardless of load – a pure red herring.

The truth was in the cumulative counter: http_client_queue_delay_seconds_sum divided by _count. For the open-meteo client:

Mean connection-lease wait: 1407 ms

1407 milliseconds of every request was waiting for a connection. Not doing work. Not waiting for the API. Waiting for a permit to ask.

The cause: /api/survey – the heaviest endpoint in the k6 weighted mix – fires the weather grid. Each request fans out 20-40 concurrent calls to the open-meteo API. At 300 VUs, that’s thousands of meteo calls queued behind a default pool of 50.

A closed-loop single-endpoint probe showed 484 rps and found nothing wrong. The starvation only appears under the open-loop mixed workload where the survey endpoint’s multiplied fan-out competes with every other endpoint for the same 50 connections.

Fix: quarkus.rest-client.open-meteo.connection-pool-size=400. Tried 1000 – OOM crash-loop, 8 restarts at 256 MB. 400 was the sweet spot: 255 rps, RSS 161 MB, 0 restarts.

The diagnostic lesson is portable. Whenever you suspect connection-pool starvation and the gauges say “everything is fine”: ignore the gauges. Find the cumulative delay counter. Divide sum by count. If the connection-lease wait dominates the total request time, the pool is the bottleneck – even if the point-in-time snapshots never catch it in the act.

Bug 4: the upsert that deadlocked

Both frameworks. The correlate endpoint builds correlation pairs and upserts them:

INSERT INTO correlation (event_a_id, event_b_id, correlation_type, ...)
VALUES (?, ?, ?, ...)
ON CONFLICT (event_a_id, event_b_id, correlation_type) DO NOTHING

ON CONFLICT DO NOTHING is idempotent. It should be safe under concurrent load.

It is – for the row. Not for the index. Postgres locks unique-index entries during INSERT ... ON CONFLICT. Two transactions inserting overlapping key sets in different orders acquire those index locks in different orders. Classic deadlock:

ERROR: deadlock detected (40P01)
DETAIL: Process 142 waits for ShareLock on transaction 8471;
blocked by process 147.
Process 147 waits for ShareLock on transaction 8470;
blocked by process 142.

Single-threaded tests insert one batch at a time. No lock ordering conflict. Under 300 VUs, multiple requests propose the same (event_a_id, event_b_id, correlation_type) tuples concurrently.

The fix: sort the batch by the conflict key before inserting, and insert sequentially within each transaction. Every transaction acquires locks in the same order, so they wait instead of deadlocking.

List<Correlation> ordered = correlations.stream()
    .sorted(Comparator.comparing(Correlation::getEventAId)
            .thenComparing(Correlation::getEventBId)
            .thenComparing(Correlation::getCorrelationType))
    .toList();

Then Flux.fromIterable(ordered).flatMap(c -> upsert(c), 1) – concurrency 1, so effectively concatMap (or Mutiny’s .concatenate()). Sequential. Deterministic lock order. No deadlocks.

The pattern

Four bugs. Four different failure modes. One common trait: none of them can exist in a unit test.

Bug	Requires
ConverterNotFoundException	A real Postgres row read through the R2DBC driver
Illegal pop()	Concurrent reactive session operations under load
Pool starvation	Open-loop mixed workload at 300 VUs
Deadlock 40P01	Concurrent transactions on the same unique index

The code review verified that the port was a faithful translation of the Micronaut source. It was. The reviewer checked field mappings, null handling, fan-out bounds, scoring math, retry semantics. All correct. The four bugs live in the gap between “correct translation” and “correct behavior under concurrent load against real Postgres.”

After finding these, each port got a SurveyPersistenceTest – an integration test that calls the real service with dead upstream URLs (fan-out fails fast to fallbacks) and asserts that a row lands in the database. It catches Bug 1 (the converter) and would catch future mapping regressions. It can’t catch Bugs 2-4. Those need load.

The benchmark isn’t a nice-to-have validation step at the end. It’s the first test that runs the actual code path.

Sources

Spring Data R2DBC – R2dbcCustomConversions – registering custom converters for type pairs the driver doesn’t handle
r2dbc-postgresql – Data Type Mapping – timestamptz maps to OffsetDateTime, not Instant
Hibernate Reactive – Sessions and Transactions – session is not thread-safe, one operation at a time
PostgreSQL Error Codes – 40P01 – deadlock_detected
Quarkus REST Client – Connection Pool – per-interface connection-pool-size configuration