The Observability Tax: When Metrics Cost More Than Your App

· 6 min read · java, micronaut, graalvm, native-image, observability, prometheus
The Memory Benchmark · Part 3
Contents

Add observability to understand your app’s memory usage. Watch observability consume all the memory.

The JVM at 128 MB worked in Part 2. It no longer works.

What Changed

One dependency:

<dependency>
    <groupId>io.micronaut.micrometer</groupId>
    <artifactId>micronaut-micrometer-registry-prometheus</artifactId>
    <scope>runtime</scope>
</dependency>

Three lines of YAML:

micrometer:
  export:
    prometheus:
      enabled: true

Prometheus endpoint exposed, micrometer collecting JVM metrics — heap, non-heap, GC pauses, threads, CPU, R2DBC pool stats. The standard production observability setup.

The JVM at 128 MB now OOM-kills at startup. Before serving a single request.

The Budget

Here’s what’s inside the JVM at idle. 256 MB container, Serial GC, -Xms64m -Xmx64m:

Heap:
  Eden Space:              3.2 MB
  Survivor Space:          1.0 MB
  Tenured Gen:            23.6 MB
  ─────────────────────────────
  Total heap:             27.8 MB

Non-heap:
  Metaspace:              64.1 MB
  Compressed Class Space: 12.1 MB
  CodeHeap (profiled):     8.5 MB
  CodeHeap (non-profiled): 2.0 MB
  CodeHeap (non-nmethods): 1.5 MB
  ─────────────────────────────
  Total non-heap:         88.2 MB

Direct buffers:            0.8 MB

Prometheus reports 117 MB total. Docker reports 192 MiB. The container hasn’t served a single request.

Non-heap alone is 88 MB. At a 128 MB container limit, that leaves 40 MB for heap, buffers, thread stacks, JVM internals, and the actual application. Thread stacks at default 1 MB per thread, 58 live threads at this startup snapshot: 58 MB. That’s 146 MB before the heap allocates a single object. Dead.

The Invisible 325 MB

After 1.6 million requests through the JVM unlimited scenario:

Docker stats:          539 MiB
Prometheus reports:
  Heap:                 91 MB  (Eden 48 + Old 43)
  Non-heap:            122 MB  (Metaspace 74 + CompClass 13 + CodeCache 35)
  Direct buffers:        1 MB
  ────────────────────────────
  Total visible:       214 MB

Invisible:             325 MB

325 MB that micrometer cannot see.

Thread stacks account for the biggest chunk: 95 live threads at 1 MB default stack. JIT compiler working memory, classloader overhead beyond what Metaspace reports, memory-mapped JARs, G1GC bookkeeping — none of it appears in any micrometer gauge.

Your observability tool shows 214 MB. Your container uses 539 MiB. The tool is lying by omission, about the thing it exists to observe. Part 5 walks through /proc/PID/smaps to account for every byte.

Why Native Image Doesn’t Care

Same app. Same micrometer dependency. Same prometheus endpoint. 128 MB container:

Docker stats:          50 MiB
Prometheus reports:
  Heap:                 33 MB  (Eden 13 + Old 21)
  Non-heap:              0 MB
  ────────────────────────────
  Total visible:        33 MB

Invisible:              17 MB

Non-heap is zero. Not “low” — zero. No metaspace because there’s no runtime class loading. No code cache because there’s no JIT compiler. No compressed class space because classes are embedded in the binary.

The 17 MB gap: the AOT-compiled binary’s mapped pages and native thread stacks. Native image threads use smaller stacks — the substrate VM doesn’t need HotSpot’s deep frame chains.

Micrometer’s own overhead — metric registries, prometheus text formatter, counters and gauges and timers — lives in heap. On the JVM, that’s ~5-10 MB of heap plus the metaspace cost of loading micrometer’s classes: dozens of metric types, registry implementations, exposition format converters. On native image, those classes are already in the binary. The heap cost is identical. The metaspace cost is zero.

Adding micrometer to a native image costs heap. Adding micrometer to a JVM costs heap, metaspace, code cache, and compressed class space. The JVM pays four tolls. Native pays one.

96 Threads for an App That Needs 8

The benchmark host has 16 cores. Here’s what Micronaut creates by default:

Netty server worker threads: 2 x 16 = 32 Netty client event loop threads: 2 x 16 = 32 Micronaut scheduled executor pool: 2 x 16 = 32 (for one @Scheduled job)

96 threads before the application does anything (not all launch immediately — prometheus showed 58 at idle, but they all spin up under load). At default 1 MB per stack: 96 MB for thread stacks alone. The prometheus metric executor_pool_core_threads{name="scheduled"} reads 32 — for a single cron job.

The obvious fix: -Dio.netty.eventLoopThreads=4. The standard Netty system property.

It does nothing. Micronaut creates its own EventLoopGroup beans and ignores the Netty system property. The fix requires Micronaut-specific configuration:

-Dmicronaut.server.netty.worker.threads=4
-Dmicronaut.netty.event-loops.default.num-threads=4
-Dmicronaut.executors.scheduled.core-pool-size=2

64 Netty threads become 8. 32 scheduled threads become 2. At -Xss256k, the savings: (96 - 10) x 256 KB = 22 MB. That’s the margin between starting and OOM-killing in a 256 MB container.

All benchmarks below ran with these thread settings applied.

What k6 Found That ab Missed

Part 2 used Apache Bench: 1,000 requests, 20 concurrent, one endpoint at a time. A polite benchmark.

k6 runs 200 concurrent virtual users hitting 7 endpoints with weighted random selection for 8.5 minutes, then 500 req/s constant arrival for 2 minutes. An impolite one.

ScenarioRequestsAvg Latencyp99Errors
native-unlimited1,641,8630.81 ms8.27 ms0%
native-256mb1,642,2600.82 ms8.35 ms0%
native-128mb1,634,3380.94 ms11.19 ms0%
jvm-unlimited1,663,3410.43 ms1.70 ms0%
jvm-256mb24,3160.41 ms2.42 ms81.3%

The JVM unlimited result is the one worth staring at. Average latency 0.43 ms vs native’s 0.81 ms. The JVM is 2x faster. Peak CPU: 13.5% vs 21.6%. The JVM uses 40% less CPU at comparable throughput.

G1GC plus JIT beats Serial GC plus AOT when memory isn’t a problem. This wasn’t visible with ab’s 1,000-request bursts. It takes 1.6 million requests and 10 minutes of sustained load for the JIT to fully warm up and the advantage to stabilize.

Native image doesn’t care how much memory you give it. Three rows, three memory limits, all within 15% of each other on every metric. The JVM’s performance is a step function: unlimited or dead.

The JVM at 256 MB told this story in 60 seconds. Non-heap climbs from 89 MB at startup to 117 MB as the JIT compiles hot paths. At 17:57:36, the last prometheus scrape succeeds — 4,550 requests served. At 17:57:38, the scrape returns empty. Sixty seconds of load, then silence.

The Recursive Problem

You can’t benchmark a JVM app at 128 MB with production observability tooling. The observability itself exceeds the budget.

This isn’t a Micronaut problem. Any JVM framework with micrometer-prometheus will hit the same metaspace wall, the same code cache expansion, the same thread stack overhead. Micronaut’s compile-time DI doesn’t help here — micrometer registers metrics at runtime, in the same metaspace as everyone else.

Native image at 128 MB with full prometheus metrics: 50 MiB RSS, zero errors, 1.6 million requests. The observability adds ~5 MB of heap and nothing else, because AOT compilation means there’s no “else” to grow into.

The question from Part 2 was “which runtime is better at 128 MB?” The answer from Part 3: the JVM can’t answer that question anymore. The tooling needed to ask it properly is itself the reason it can’t fit.

Sources