Your JVM Reports 151 MB. Your Container Uses 248.

· 5 min read · java, jvm, micronaut, graalvm, native-image, docker, memory
The Memory Benchmark · Part 6
Contents

151 MB. That’s what micrometer-prometheus reports for the JVM at 256 MB after serving 1.66 million requests. Heap 35 MB, metaspace 72 MB, compressed class space 13 MB, code cache 16 MB, direct buffers 0.75 MB. Add it up. 137 MB. Rounded generously: 151 MB with some measurement timing jitter.

Docker stats says 244.2 MiB. Out of 256 MiB. 95.38%.

The Visible Budget

After 1.66 million requests, Serial GC, -Xms64m -Xmx64m -Xss256k, here’s what the JVM admits to owning:

jvm_memory_used_bytes{area="heap",id="Tenured Gen"}        31,976,264    ~30.5 MB
jvm_memory_used_bytes{area="heap",id="Eden Space"}           4,654,464     ~4.4 MB
jvm_memory_used_bytes{area="heap",id="Survivor Space"}         132,720     ~0.1 MB
──────────────────────────────────────────────────────────────────────────
Heap total                                                                  ~35 MB

jvm_memory_used_bytes{area="nonheap",id="Metaspace"}        75,436,992    ~72 MB
jvm_memory_used_bytes{area="nonheap",id="Compressed Class"} 13,326,520    ~13 MB
jvm_memory_used_bytes{area="nonheap",id="CodeCache"}        17,058,560    ~16 MB
──────────────────────────────────────────────────────────────────────────
Non-heap total                                                             ~101 MB

jvm_buffer_memory_used_bytes{id="direct"}                      786,719     ~0.8 MB
──────────────────────────────────────────────────────────────────────────
JVM-visible total                                                         ~137 MB

137 MB. Docker says 244 MiB. The gap: 107 megabytes that micrometer cannot see, that Grafana will never chart, that your alerts will never fire on.

The Invisible Budget

Walk through /proc/PID/smaps on the running container. Every byte accounted for.

Thread stacks: ~19 MB. 75 threads at peak, each with 256 KB stacks (-Xss256k). No micrometer gauge for this. The JVM knows how many threads exist (jvm_threads_live_threads: 74) but not how much stack memory they consume. At the default 1 MB per thread, this would be 75 MB. The -Xss256k flag saved 56 MB and isn’t visible anywhere in the metrics.

libjvm.so + CDS archive + modules: ~30 MB. The JVM itself, memory-mapped into the process. Class Data Sharing archive. The java.base module and friends. Shared pages, but they count toward RSS because the kernel charged them to the cgroup.

JVM internals: ~20 MB. Symbol tables. String intern tables. Constant pool caches. The ServiceLoader resolution cache. GC bookkeeping for Serial GC (mark bitmaps, card tables). These are JVM-internal data structures that don’t map to any JMX MBean or micrometer gauge.

Netty arenas + native allocations: ~20 MB. Netty’s PooledByteBufAllocator uses arena-based allocation via sun.misc.Unsafe or java.lang.foreign. These allocations bypass the JVM’s direct buffer tracking. jvm_buffer_memory_used_bytes{id="direct"} shows 0.8 MB. That’s the NIO ByteBuffer.allocateDirect() path. Netty doesn’t use that path.

Visible to Prometheus:    137 MB    (57%)
Invisible to everything:  107 MB    (43%)
──────────────────────────────────────────
Docker RSS:               244 MB    (100%)

43% of the container’s memory consumption doesn’t exist in your monitoring stack.

Scale the Problem

Remove the constraints. JVM unlimited — no memory limit, default G1GC, default thread stacks, default everything:

JVM-visible (heap + non-heap + buffers):    186 MB
Docker RSS:                                 570 MB
Invisible:                                  384 MB    (67%)

384 megabytes. Two-thirds of the container’s actual memory consumption is invisible to JVM metrics.

The gap grows because every JVM feature adds invisible overhead. G1GC’s remembered sets, card tables, and concurrent marking bitmaps: ~25 MB that Serial GC doesn’t need. 94 threads at default 1 MB stacks: 94 MB vs the constrained scenario’s 19 MB. The JIT’s C2 compiler arenas grow larger with more available memory. The JVM expands to fill available space, and only reports the expansion it’s proud of.

The Native Image Control

Same app, same micrometer dependency, same prometheus endpoint. Native image at 128 MB container:

jvm_memory_used_bytes{area="heap",id="eden space"}         22,544,384    ~21 MB
jvm_memory_used_bytes{area="heap",id="old generation"}     22,429,760    ~21 MB
jvm_memory_used_bytes{area="nonheap",id="runtime code cache (code and data)"}  0      0 MB
jvm_memory_used_bytes{area="nonheap",id="runtime code cache (native metadata)"}  0    0 MB
──────────────────────────────────────────────────────────────────────────
Visible total                                                              ~42 MB

Docker RSS:                                                  60 MB
Invisible:                                                   18 MB    (30%)

Non-heap: zero. Not low — zero. No metaspace (no runtime class loading). No code cache (no JIT). No compressed class space (classes baked into the binary). The entire “non-heap” category that costs the JVM 101 MB doesn’t exist.

The 18 MB gap: the native binary’s mapped segments (~12 MB), resident stack pages for 72 threads (~4 MB — SubstrateVM allocates 256 KB per thread but most pages stay untouched), and a small malloc heap (~2 MB).

JVM 256 MBJVM UnlimitedNative 128 MB
Docker RSS244 MB570 MB60 MB
JVM-visible137 MB186 MB42 MB
Invisible107 MB (43%)384 MB (67%)18 MB (30%)

The JVM’s invisible overhead scales with its sophistication. More threads, bigger GC, active JIT — each feature pays a hidden memory tax that only /proc and docker stats can see.

Why This Kills Your Kubernetes Config

resources:
  limits:
    memory: 256Mi
  requests:
    memory: 256Mi

Your Grafana dashboard shows JVM heap at 35 MB out of 64 MB max. 45% utilization. Comfortable.

The container is at 244 MiB out of 256 MiB. 95.38% utilization. One allocation spike from the OOM killer.

Horizontal Pod Autoscaler watching jvm_memory_used_bytes? It sees 137 MB in a 256 MiB container. 53% utilization. Below the 70% scale-up threshold. No new pods.

Vertical Pod Autoscaler watching the same metric? Recommends downsizing. The JVM clearly only needs 150 MB.

The OOM killer watches RSS. RSS is 244 MiB. Your monitoring sees 137 MB. These are both correct. They measure different things. And every automated decision based on the wrong number will be wrong.

The Honest Metric

container_memory_working_set_bytes from cAdvisor. Or docker stats. Or cgroup memory.current. These report what the kernel actually charges the container.

For alerting: RSS, not JVM heap. For autoscaling: RSS, not JVM non-heap. For capacity planning: RSS plus a margin, because the 570 MB JVM unlimited scenario shows the JVM will grow into any space you give it.

If you must use JVM metrics, add 50-80% overhead factor:
  256 MB container → JVM should report ≤ 140 MB
  512 MB container → JVM should report ≤ 280 MB
  Unlimited        → JVM will use 3-4x what it reports

The native image doesn’t have this problem. Not because the gap is zero — it’s 18 MB — but because 30% overhead on a 60 MB footprint is 18 MB. 67% overhead on a 570 MB footprint is 384 MB. Same percentage, different cloud bill.

Sources