107 MB of Invisible JVM RAM

Prometheus says 137 MB. Docker says 244 MiB. /proc/PID/smaps is the only witness that doesn’t lie.

The Discrepancy

JVM at 256 MB, Serial GC, after serving 1.66 million requests. Measured at idle:

Docker stats:         244.2 MiB / 256 MiB  (95.38%)
Prometheus reports:
  Heap used:           35 MB  (Eden 4.4 + Survivor 0.1 + Tenured 30.5)
  Non-heap:           101 MB  (Metaspace 71.9 + CompClass 12.7 + CodeCache 16.3)
  Direct buffers:     0.75 MB
  ────────────────────────────
  Total visible:      ~137 MB

Gap:                  107 MB

107 MB that no JMX MBean exposes. No micrometer gauge records it. No Grafana dashboard shows it. To know it exists, you exec into the container and parse kernel memory maps.

What Lives in the Gap

/proc/PID/smaps is the kernel’s per-mapping memory breakdown. Each virtual memory area gets an entry with RSS, PSS, private/shared dirty pages. Sum the RSS column and you get the number Docker sees.

Here’s where the 107 MB hides:

Top four rows: Prometheus territory. Bottom four: the void. The void totals ~89 MB. Add fragmentation and page-alignment waste, reach 107.

Thread Stacks: The Quietest Consumer

75 threads at 256 KB each. At default stack size (1 MB), this would be 75 MB — half the container. The -Xss256k flag from Part 4 saves 56 MB, but 19 MB still vanishes into a monitoring black hole.

Thread stacks don’t appear in any JMX memory pool. The jvm_threads_live_threads gauge tells you how many threads exist. Never how much memory they consume. Multiply the count by -Xss yourself. Hope you set -Xss explicitly — if you didn’t, the default is platform-dependent and undiscoverable from JMX.

At idle: 57 threads × 256 KB = 14 MB invisible. Under load: 75 threads × 256 KB = 19 MB invisible. At default stacks without tuning: 75 threads × 1 MB = 75 MB invisible.

Thread stacks alone can exceed the heap.

libjvm.so and Friends: The Binary Tax

The JVM itself is a shared library. It has to live somewhere in the process address space:

libjvm.so              ~17 MB  (text + rodata + bss)
CDS archive            ~ 8 MB  (class data sharing, preloaded metadata)
modules image          ~ 5 MB  (lib/modules, JDK's own classes)

~30 MB. Fixed cost. Doesn’t grow. Doesn’t shrink. Not reported by any memory pool because these are memory-mapped files — the kernel counts them as RSS, the JVM doesn’t count them as anything.

This is the entry fee. Before a single class loads, before the heap allocates one byte, 30 MB is spoken for.

GC Bookkeeping

Serial GC is the cheapest collector. Its bookkeeping is still non-zero:

• Card table: 1 byte per 512 bytes of heap = 128 KB for 64 MB
• Mark bitmap: 1 bit per 8 bytes of heap = 1 MB for 64 MB
• Symbol tables, string tables, internal allocation tracking

Plus NMT (Native Memory Tracking) overhead if you enable it to diagnose where memory went — which itself consumes memory to track the memory it’s tracking.

Total: ~20 MB for JVM-internal structures below the level of any monitoring API.

Netty’s Native Arenas

Netty allocates off-heap I/O buffers using jemalloc-style arena allocators. Direct ByteBuffers report to JMX (the 0.75 MB in our breakdown). Netty’s pooled allocators do not — they bypass ByteBuffer.allocateDirect() and call Unsafe.allocateMemory() directly.

Under sustained load, Netty’s arenas hold ~20 MB of native memory. Pooled, not leaked — Netty reuses it. The kernel counts it. Micrometer doesn’t.

The Ratio That Gets Worse

As you constrain the JVM, the danger grows:

At 256 MB, Prometheus shows 119 MB of apparent headroom. The truth is 12 MiB.

Your monitoring says:    137 MB used / 256 MB limit  →  "46% headroom"
Reality:                 244 MB used / 256 MB limit  →  "5% headroom"

Your dashboard is wrong by a factor of 9.

Native Image: Where the Gap Shrinks

Same app, GraalVM native image, 128 MB container:

Docker stats:          60 MiB / 128 MiB
Prometheus reports:
  Heap used:           33 MB
  Non-heap:             0 MB  (always — no metaspace, no JIT, no class space)
  ────────────────────────────
  Total visible:       33 MB

Gap:                   ~27 MB

27 MB vs 107 MB. The gap is the AOT binary itself (partially resident in memory), plus thread stacks at the substrate VM’s smaller defaults.

Non-heap is 0 MB. Not “low” — zero. Permanently. No class loading means no metaspace growth. No JIT means no code cache. The invisible portion doesn’t grow under load.

Native image’s invisible memory is:

• Fixed — doesn’t change with request count
• Smaller — 27 MB vs 107 MB
• Calculable — binary size + (threads × stack size)

You can estimate it from ls -la and a thread count. Try that on HotSpot.

Three Ways This Breaks Production

Kubernetes Limits Lie

resources:
  limits:
    memory: 256Mi

Grafana shows 137 MB. Apparent headroom: 119 MB. Actual headroom: 12 MB. The next connection burst allocates thread stacks and Netty arenas. The cgroup kills the process. Alerts never fired because the heap-based thresholds were never breached.

Autoscaling Never Triggers

HPA on jvm_memory_used_bytes / container_memory_limit:

137 MB / 256 MB = 53%  →  below 80% threshold  →  no scale-up

RSS is at 95%. One allocation spike from death, autoscaler satisfied. It triggers after the OOM kill restarts the pod — reactive, never proactive.

Cost Estimation Is Fiction

“Our service uses 150 MB per replica. At $0.04/GB-hour, 20 replicas: $2.88/day.”

Your service uses 244 MB per replica. Real cost: $4.69/day. You’re underestimating by 63%. Across a fleet, across a year, the invisible gap is an invisible line item.

The Fix (Such As It Is)

There is no fix that makes JMX report thread stacks or libjvm.so mappings. The JVM’s monitoring APIs were designed for managed memory. The gap is architectural.

Workarounds:

1. Monitor RSS, not JVM metrics — container_memory_usage_bytes from cAdvisor, not jvm_memory_used_bytes from micrometer
2. Alert on cgroup proximity — container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.85
3. Budget for the invisible — assume 40-60% overhead for JVM infrastructure
4. Or use native image — where the gap is 27 MB, fixed, and predictable

The cleanest solution is #4. Not because native image has no invisible memory — it does — but because the invisible portion doesn’t grow, doesn’t surprise you, and doesn’t depend on load patterns you can’t predict.

Sources

• Linux /proc filesystem documentation — smaps format and RSS accounting
• Oracle NMT Documentation — Native Memory Tracking categories
• JVM Options Reference — thread stack size, code cache reservation
• GraalVM Native Image Memory Management — native image memory model
• Part 3: The Observability Tax — 325 MB invisible gap at unlimited
• Part 4: Every GC Failed at 256 MB — Serial GC as only survivor, thread tuning