Eight Connections, Zero Available

Part 18 of the Micronaut native image series. Follows Part 17, which found that reactive and imperative produced identical throughput on the 28-API workload. Assumes familiarity with R2DBC and Project Reactor.

60.8 and 59.9

Two frameworks. Different HTTP servers, different dependency trees, different teams maintaining them. Under load, both reactive stacks land at almost the same throughput:

Imperative hits 220–252. Quarkus reactive hits 259. Micronaut and Spring Boot reactive: 60.

The coincidence is too tight to be a coincidence. Two independent frameworks don’t land at 60.8 and 59.9 by accident. Something downstream is shared.

That something is r2dbc-pool.

The Fingerprint Nobody Reads

Look at the p95 column again. 15000.6 ms. The load generator — k6 — uses a 15-second request timeout. At the 95th percentile, responses take exactly the timeout and get aborted.

At the 256 MB tier, where there’s more headroom, Micronaut reactive still shows the same signature:

More memory buys more throughput — 98.5 vs 60.8 — but the p95 is still 15000.1 ms. The ceiling isn’t memory. The ceiling is the timeout.

False Lead: Memory Pressure

First instinct: 128 MB is too tight. The fan-out workload calls upstream APIs, aggregates data, writes to Postgres. Native images are lean, but 128 MB under 500 req/s might just be too little.

Ran it again with no memory limit at all. Same stall. Same ~60 rps wall. Same pool wedge.

Memory pressure modulates when the stall starts — tight limits push responses past 15 seconds sooner — but the failure mode is purely a pool bug. Remove the limit and you just delay the inevitable by a few seconds.

Autopsy on a Wedged Instance

The service was still running. Health endpoint: 200 OK. CPU at 2%. RSS stable. Nothing panicked. Nothing restarted. From every operational signal, this was a healthy pod.

$ curl -w '%{time_total}s\n' http://localhost:8181/api/survey
{"surveys":[...]}
0.190s

$ curl -w '%{time_total}s\n' http://localhost:8181/api/scan
# ... hangs forever

/api/survey doesn’t touch the database. It returns in 190 milliseconds. /api/scan queries Postgres. It hangs indefinitely.

On the Postgres side:

SELECT state, wait_event_type, query
FROM pg_stat_activity
WHERE datname = 'space';

Eight connections. All state idle. No active transaction. No wait event. The wire is healthy — Postgres has nothing to do and nobody asking.

But r2dbc-pool thinks all eight connections are borrowed. It’s waiting for one to come back. No maxAcquireTime is configured, so it waits forever. New requests queue behind the acquire call. The queue grows. Nothing moves.

Eight connections. Zero available. Both sides of the pool disagree about who’s holding what.

1,220 Curls That Proved Nothing

To reproduce the leak, the cancel needs to arrive while r2dbc-pool is mid-query. The timing window is narrow.

First attempt: 320 rapid-fire curl requests, each aborted immediately:

for i in $(seq 1 320); do
  timeout 0.05 curl -s http://localhost:8181/api/scan > /dev/null
  sleep 0.01
done

No leak. The service processes each request in ~150 ms, and timeout 0.05 kills curl before the response arrives — but by then the HTTP layer has already read the response off the wire. The cancel signal reaches the reactive pipeline after the DB call has completed and the connection has been returned. Too late to leak.

Second attempt: 900 aborted requests in contention waves — bursts of 20 concurrent curls with 50ms timeouts, repeated 45 times:

for wave in $(seq 1 45); do
  for i in $(seq 1 20); do
    timeout 0.05 curl -s http://localhost:8181/api/scan > /dev/null &
  done
  wait
  sleep 0.1
done

No leak. Same problem — the mock upstream responds in ~150 ms total, so even under contention the DB window is too brief. The cancel lands after the connection is already back in the pool.

The bug requires sustained pressure that pushes response times past the cancel threshold. Not a curl with a 50 ms timeout against a 150 ms endpoint. A load generator sending 500 req/s with a 15-second timeout against a service that’s starting to buckle.

The Death Spiral

On the reproduction run — unlimited memory, k6 at 500 req/s constant arrival — a Postgres-side sampler captured the collapse in real time. One query, every 5 seconds:

SELECT sum(xact_commit) FROM pg_stat_database WHERE datname = 'space';

Deltas between samples:

5931 → 5255 → 2062 → 592 → 5

Commits per 5-second window. Healthy to flatline in 25 seconds.

The mechanism is a positive feedback loop:

1. Under sustained load, some responses slow past 15 seconds
2. k6 aborts those requests. The abort propagates as a Reactive Streams cancellation
3. The cancellation hits r2dbc-pool mid-query. The pool leaks that connection
4. Fewer available connections → remaining queries take longer → more cross 15 seconds
5. More cancellations → more leaks → fewer connections
6. After 8 leaks the pool is empty. Forever

Step 4 is the amplifier. One leaked connection raises latency for the other seven. Higher latency means more timeouts. More timeouts mean more cancellations. The system eats itself.

At 500 req/s with 8 pool slots, the steady state before the spiral is maybe 6–7 connections busy, response times climbing. The first leak is the crack. Within 25 seconds the entire pool is gone.

The Bug

r2dbc-pool 1.0.2.RELEASE. The current release, shipping in Spring Boot 4 and Micronaut 5.

This was partially fixed in 0.8.8.RELEASE — the original report from 2021 described the exact scenario: a cancelled request fails to release its connection, the pool fills up, new requests get sorry, too many clients already. The fix addressed the common case. Edge cases survived into the current release.

Issue #198 (open, “waiting-for-triage”) describes the remaining leak path: a concurrent query cancelled during a zip operation leaks a different connection’s slot. The reporter confirmed via pg_stat_activity — the connection sits idle on the database side while the pool considers it borrowed.

Issue #206 (open) is the downstream symptom: all connections show as acquired, empty idle deque, pending requests starving.

spring-framework #35774 (open) adds the Spring transaction angle — TransactionalOperator cleanup fails on cancel with Transaction synchronization is not active, leaking the connection through a different path.

Three open issues. One closed partial fix. The pool code is reactor-pool 1.2.5 under reactor-core 3.7.12.

Why Quarkus Is Immune

Quarkus reactive doesn’t use r2dbc-pool. It uses vertx-pg-client — Vert.x’s own SQL client with its own connection pool.

At native-128 on the same workload, Quarkus reactive hits 78.8 rps with 39% errors. At native-256, it does 258.7 rps with 0% errors. It doesn’t stall alive — it either works or it dies by OOM. The container gets killed, the orchestrator restarts it, traffic moves. A failure mode you can detect and recover from.

Same DAOs. Same SQL. Same Postgres. Three frameworks, three connection pools, three failure modes: r2dbc-pool wedges silently, vertx-pg-client dies loudly, and HikariCP (imperative) never sees cancellations because blocking threads don’t propagate them.

Your Load Balancer Does This

The k6 15-second timeout isn’t exotic. It’s standard. Your load balancer has a timeout. Your API gateway has a timeout. Your service mesh sidecar has a timeout. Your mobile client has a timeout.

Under normal operation, none of them fire. Under saturation — a traffic spike, a slow dependency, a GC pause that delays one response batch — some requests cross the threshold. The timeout fires. The connection cancels. The reactive pipeline cancels. r2dbc-pool leaks one connection.

One leaked connection makes the next request slightly slower. Slightly slower makes more timeouts slightly more likely. The spiral is patient. It doesn’t need a traffic surge — it needs one bad minute.

And when the pool is empty, the pod is still up. Health check passes. Readiness probe passes. The pod receives traffic and serves nothing. Your alerting sees a healthy fleet with rising p99. If you’re watching error rates, you see a gradual increase — not a cliff. If you’re watching saturation, the pod’s CPU is at 2% because it’s doing nothing.

The pod that dies gets restarted. The pod that stalls alive gets traffic forever.

Sources

• r2dbc-pool #198: Connection leaks when concurrent query is canceled — open, waiting-for-triage
• r2dbc-pool #140: Connection not released when request is cancelled — closed (partial fix in 0.8.8.RELEASE)
• r2dbc-pool #206: Pool runs out of connections — open
• spring-framework #35774: R2DBC pool leaks connections on request cancellation — open