Seven Fixes, Zero Errors

· 5 min read · java, micronaut, reactor, r2dbc, backpressure, jvm
The Memory Benchmark · Part 9
Contents

The fix for “JVM dies at 256 MB under load” was not a JVM flag.

Part 4 ended with Serial GC — the only GC that fits in 256 MB — serving 4,550 requests before the OOM killer showed up. 81% error rate. Same flags, same container, same load profile. Seven code changes later: 1,740,000 requests. Zero errors.

No GC tuning. No new JVM flags at all. The problem was never the runtime. It was the code asking the runtime to hold things it didn’t need to hold.

The Default Concurrency Problem

This line killed the container:

.flatMap(neo -> saveAsteroidWithApproaches(sessionId, neo))

flatMap without a second argument means concurrency 256. That’s the Reactor default. When NASA’s Near Earth Object feed returns 140 asteroids, this line launches 140 concurrent saveAsteroidWithApproaches operations — each doing multiple R2DBC inserts.

The R2DBC pool has 20 connections. 140 concurrent operations competing for 20 connections means 120 are queued. Queued in memory. Each holding its upstream context, its subscriber chain, its allocated buffers. All simultaneously.

The fix:

.flatMap(neo -> saveAsteroidWithApproaches(sessionId, neo), 4)

One argument. The most important second argument in Reactor.

Why 4? The app runs on 4 Netty event loop threads in this constrained configuration. There is no point saturating 256 concurrent operations when 4 threads process results. Match concurrency to actual parallelism, not to Reactor’s optimistic default.

Unbounded Collections

Three queries, all running concurrently via Mono.zip():

Mono<List<ApodEntry>> apods = apodRepository.findBySessionId(id).collectList();
Mono<List<NeoAsteroidRecord>> neos = neoRepository.findBySessionId(id).collectList();
Mono<List<IssPositionWithWeather>> iss = issRepository.findBySessionId(id).collectList();

collectList() means “buffer every element into a single list in memory.” Three of them, concurrently, each unbounded. Worst case: three full tables materialized simultaneously.

At 64 MB heap, “worst case” is about 2,000 records before the collector can’t keep up.

Mono<List<ApodEntry>> apods = apodRepository.findBySessionId(id).take(500).collectList();
Mono<List<NeoAsteroidRecord>> neos = neoRepository.findBySessionId(id).take(500).collectList();
Mono<List<IssPositionWithWeather>> iss = issRepository.findBySessionId(id).take(500).collectList();

.take(500) before .collectList(). Caps memory at a known ceiling. The API returns paginated results anyway — no caller needs all records at once.

Same pattern in the REST controller:

// Before
return asteroidRepository.findAll();

// After
return asteroidRepository.list(Pageable.from(page, Math.min(size, 500)));

Slice Over Page

The pagination uses Slice, not Page. The difference:

  • Page<T> executes two queries: the data query + SELECT COUNT(*) FROM ...
  • Slice<T> executes one: just the data, fetching N+1 rows to determine “has next”

Under a 64 MB heap with the connection pool reduced to 8 (down from 20 — each R2DBC connection costs 1-2 MB in Netty channel buffers), the COUNT query matters. Not because it returns much data — but because it holds a connection open for the duration of the count scan while the data query’s connection is also checked out.

Two connections per paginated request vs. one. At 8 pool slots, that’s the difference between serving 8 concurrent paginated requests and 4.

Fire-and-Forget Without Timeout

The scheduled collector job:

collectAllData()
    .subscribe(
        success -> log.info("Collection completed"),
        error -> log.error("Collection failed", error)
    );

.subscribe() without .timeout(). If the NASA API hangs — and it does, this is a free-tier government API — the subscription holds its resources indefinitely. The reactive chain stays allocated. The R2DBC connections it acquired stay checked out. The Netty channel buffers stay alive.

collectAllData()
    .timeout(Duration.ofMinutes(5))
    .subscribe(
        success -> log.info("Collection completed"),
        error -> log.error("Collection failed: {}", error.getMessage())
    );

Five minutes is generous. But “forever” was the previous timeout, and forever is longer than the OOM killer’s patience.

The Heap Paradox

After all code fixes, one counterintuitive finding: 48 MB heap survived better than 64 MB.

The logic:

  1. Smaller heap triggers Serial GC more frequently
  2. More frequent collections = shorter individual pauses (less garbage per cycle)
  3. Shorter pauses = less peak memory between collections
  4. Less peak = more headroom for non-heap growth (thread stacks, Netty buffers, metaspace)

At -Xmx64m, Serial GC waits longer between collections. Longer waits mean more live objects at collection time. More live objects mean longer pauses. During those longer pauses, non-heap memory continues growing unchecked.

At -Xmx48m, the GC runs more often but each pause is trivial. The heap never pressures the container limit because it’s constantly being swept.

This only works because the code fixes eliminated unbounded allocations. Before: frequent GC couldn’t help because the live set itself was unbounded (256 concurrent flatMap operations all holding references). After: the live set is bounded, so frequent collection actually reclaims memory.

The code fixes made the heap paradox possible. The heap paradox made the container survivable.

Five Patterns for Constrained Reactive Services

  1. .flatMap(fn, concurrency) — Match concurrency to thread count, not Reactor’s default 256
  2. .take(n) before .collectList() — Cap memory for any collection that could grow
  3. Slice over Page — One query per request, not two. Saves a connection slot.
  4. .timeout() on fire-and-forget — Bounded resource lifetime, even for background work
  5. Validate upstream bounds — If the external API doesn’t paginate, enforce limits client-side

None of these are exotic. All of them are in the Reactor documentation. Every one was a default behavior that’s fine at 4 GB and fatal at 256 MB.

The JVM at 256 MB doesn’t need a special GC. It needs code that doesn’t assume infinite memory — which is what every Reactor default assumes.

Sources