Measuring OpenTelemetry Overhead Without Fooling Yourself

Saturday 24 October, 17:05 UTC: the first cell of my telemetry overhead matrix starts. At 18:55 UTC (21:55 on the local clock that driver.log keeps) I stopped it, seven cells into thirty. Five of the seven had passed the gate, and two of those five had been OOM-killed 39 and 48 times.

The question was one number per setup: what OpenTelemetry costs a Spring Boot 4.1.2 service in memory and in CPU per order. The service is the order pipeline from part 1, in two flavours: plain uses the bare RabbitMQ, Mongo and JDBC clients, spring uses Spring Data and Spring AMQP. The setups are none, a Prometheus reference, Boot's own OpenTelemetry starter (mo-*), the OpenTelemetry project's starter (on-*) and the JVM-only OTel Java agent; -m is metrics only, -mt metrics and traces. The second attempt finished all 60 cells. Most of this post is what had to be true of a cell before its number counted.

TL;DR — one run per app, so a gap of about 0.1 ms per order is not something these numbers can settle.

• Native, 256 MB limit (the cleanest tier): metrics only added 7–12 MiB of average RSS and 0.17–0.29 ms of CPU per order; metrics plus traces at 100 % sampling added 8–24 MiB and 0.32–0.86 ms. Boot's starter with traces cost the most CPU in both apps.
• Agent vs OTel starter at equal span coverage (JVM): the agent used more memory (plain app at 512 MB: +73.6 vs +28.8 MiB); CPU per order was close without a memory limit (+0.40 vs +0.29 and +0.41 vs +0.38 ms).
• JVM at 512 MB: my 64 MB test heap was sized for an app without telemetry. Every agent cell in the spring app crashed out of it, and two of four hung in the plain app. That is my heap flag, not a verdict on whether the agent fits in 512 MB.

A baseline with no telemetry still had a meter registry#

The first design kept the actuator in none, on the belief that it pulled in micrometer-core without binding a registry. A review of the plan checked the jars: in Boot 4.1.2, spring-boot-starter-actuator depends on spring-boot-starter-micrometer-metrics, which brings in this:

@AutoConfiguration(before = CompositeMeterRegistryAutoConfiguration.class, after = MetricsAutoConfiguration.class)
@ConditionalOnBean(Clock.class)
@EnableConfigurationProperties(SimpleProperties.class)
@ConditionalOnMissingBean(MeterRegistry.class)
@ConditionalOnEnabledMetricsExport("simple")
public final class SimpleMetricsExportAutoConfiguration {

Remove the Prometheus registry and Boot falls back to an in-memory SimpleMeterRegistry, with the JVM, Tomcat and Hikari binders still recording into it. The baseline would have paid for most of the metrics collection, and every delta would have come out too small.

So none has no actuator at all. Its health endpoint is a plain controller, and its gate test asserts there is no MeterRegistry, ObservationRegistry, Micrometer Tracer or OpenTelemetry bean.

One build per mode, and a mode check at run time#

Each mode is its own build, because Spring AOT and native images settle @Conditional outcomes at build time and a runtime switch can quietly do nothing. Each build has a gate test of which beans exist. One of them went red on Boot 4.1.2, and chasing it turned up a tracing key that binds to nothing (part 2).

A JVM test can't see what a native image loads. In a smoke run, the native telemetry cells ran on library defaults (the OTel starter cells logged localhost:4318), because the image never got its config file (part 3). Now /health reports the mode the running app loaded, and the cell stops with exit 3 if it differs from the build. A mo-m cell saved this as mode.txt:

{"traces":"false","mode":"mo-m","status":"UP"}

Any OTEL_* variable that reaches the app container is exit 5: Boot and the OTel SDK both read them, and one stray variable would reconfigure a cell silently.

The gate that read spans and ignored the verdict#

Back to the seven cells. The old gate asked one question: did the expected spans arrive? Four of its answers:

Both crash loops passed: one had spans, the other was metrics-only, where zero spans is correct. Both FAILs were for the wrong reason: cells that emit no spans got "unknown" instead of zero.

The container was no help either. For the 39-kill cell, docker inspect said oom_killed=false exit_code=0 status=running, because the restart policy had already put a fresh container in place. Only the Docker event stream and RestartCount had seen the kills, and the verdict counts events. It also counts OutOfMemoryError lines in the app log, since the consumers catch Error to keep their RabbitMQ channels alive and a heap OOM doesn't have to kill the app.

The fixed gate reads the verdict first:

# A crashed / OOM-killed / hung / degraded app is never a clean cell, whatever its spans say.
if [ -n "$verdict" ] && [ "$verdict" != ok ]; then reasons="$reasons verdict=$verdict"; fi

No-span cells now read the collector's receiver counter and get a real 0; trace cells need at least one span per order.

The JVM tier that didn't fit before telemetry arrived#

The third row is a cell with no telemetry at all: plain app, 256 MB limit, the hand-tuned recipe from my earlier 256 MB JVM benchmark, -Xms64m -Xmx64m -Xss256k -XX:+UseSerialGC -XX:ReservedCodeCacheSize=32m. Three container OOM kills, no OutOfMemoryError, and a highest 2-second RSS sample of 272 MiB. A 64 MiB heap that never threw puts the excess outside the heap; that is my reading, since no Native Memory Tracking run split it up.

So the constrained JVM tier became 512 MB with the same flags, and native stayed at 256 MB. A 128 MB native tier fell to a rule written into the cell list in advance: both apps' no-telemetry peaks had to be at or under 100 MiB, and they came in at 126 and 161.

One asymmetry stays in every JVM comparison: the unlimited tier runs G1 with a heap sized from host RAM, the 512 MB tier the Serial recipe with a 64 MB heap. They differ in GC policy, not just in the limit.

Planting an OOM the gate has to catch#

An OOM detector that has never caught a planted fault is untested, so the plan had controls: the none cell with a heap too small for the load, which the gate must flag. The cell script passes extra flags last:

# EXTRA_JAVA_OPTS: appended last (later flags win), for controls such as the deliberate OOM run (-Xmx24m).
export JAVA_TOOL_OPTIONS="$JTO${EXTRA_JAVA_OPTS:+ $EXTRA_JAVA_OPTS}"

Later flags win for -Xmx, and the tier's -Xms64m stays. All three controls, -Xmx24m at 256 MB and then -Xmx40m and -Xmx48m at 512 MB, never started. Each left six 50-byte gc.log files, all that the log rotation keeps, each ending at a line like this one:

[0.003s][info][gc,init] CardTable entry size: 512

The matrix log said FAIL: app not healthy after 120s. The first control kept no app log; the two later ones saved a startup log with the JVM's own message in it 11 times. I reproduced that message in the app's runtime image:

$ docker run --rm eclipse-temurin:25.0.4.1_1-jre@sha256:fd4b1c02c64eef378212e23656fc0b0caa41a357876fa0b60e51ceda94a3db64 java -Xms64m -Xmx24m -Xlog:gc+init -version
[0.004s][info][gc,init] CardTable entry size: 512
Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size

Exit code 1, the same for -Xmx40m and -Xmx48m, while -Xms40m -Xmx40m prints the version. The first reading of the -Xmx24m failure was "too small to start", and the proposed fix, 40 and 48 MB, carried the clash forward.

The detectors did fire in the real runs: on the crash loops above and on the seven flagged cells below. A planted fault is a different test, and the fixed control, -Xms equal to -Xmx, counts only if the cell reaches load and the gate flags it. With the flags fixed, it took two more runs to plant one.

At 40 and then 48 MB the cell reached load and passed. After each full collection the app's live heap never went above 33 MiB, so 40 MB held it: Serial GC ran about 1,170 full collections over 90,003 orders, and not one request failed. Garbage doesn't cause an OutOfMemoryError; live data does.

At 24 MB, below that live set, the app started healthy and ran out of heap during the 50-orders-per-second warm-up: 27 orders created, 74 % errors. The main load never ran, because the load tool refused to start against an unhealthy app, and the drain gave up after 300 s with 106 messages still queued. The gate failed the cell with verdict=hung and counted 75 OutOfMemoryErrors. Docker, asked the same question, answered oom_killed=false, status=running, restart_count=0. The 40 and 48 MB sizes were the ones that fit inside the app. 24 MB, the size the plan named first, caught the fault once -Xms matched it, and the measured live set is what says why.

What "fair" means in the second attempt#

The other rules, each with the defect it prevents:

This study skipped the two-runs rule, and the recompute held a surprise.

Why there is no AOT-cache tier#

A Leyden AOT cache was the obvious rescue for a 256 MB JVM tier. I dropped it for bias: the upstream recipe for the agent with an AOT cache (PR #20001, agent 2.32.0 and later) creates the cache without an active -javaagent, and says "Virtual fields on cached classes loaded before premain fall back to weak maps". The agent would run differently from the starter cells beside it.

The harness had also reported that the warm-up didn't train. Its check grepped for a k6 http_reqs line the current k6 script no longer prints; the warm-up itself ran 1,501 orders at 0.000 % errors. So whether the cache fits 256 MB was never tested. How such a cache fails quietly is in the AOT cache tier post.

The numbers#

The recompute was a separate script, written from the raw CSVs and timelines using the window the report's text described: load plus drain. Of 519 headline values, 508 matched and 11 didn't, all for one reason. The report code averaged every sampled row, and the sampler ran on for 10–17 s after the drain, through the final export and the app stop. Over the code's window, 863 of 863 values matched exactly, and I changed the report text to describe that window. The text and the code had covered different windows.

Two reading rules. Without a memory limit the heap grows with host RAM, and average RSS moves by tens of MiB for other reasons: on the unlimited JVM, plain on-mt came out at +0.0 MiB and plain on-m at +25.1. So memory comes from the limited tiers and from peaks. And the JVM 512 MB tier is GC-bound, because its 64 MB heap was sized for no telemetry: plain agent-full-m ran 1,725 full GCs, 1.08 ms of GC per order against 0.02 for none. CPU deltas there are telemetry plus heap pressure.

One run. Baselines (none): native 256 MB plain 115.6 MiB / 1.58 ms, spring 156.1 / 2.51; JVM 512 MB plain 278.4 / 1.29, spring 364.1 / 2.25. JVM 512 MB is GC-bound under a 64 MB test heap, so its CPU bars include GC; the hung and crashed cells there are the test heap, not the agent.

Native at 256 MB has neither problem, so it carries the headline. Deltas against none, average / peak RSS in MiB, CPU in ms per order:

Metrics alone cost 7–12 MiB on average and 0.17–0.29 ms per order with either starter. Traces split them: Boot's starter added 0.79 and 0.86 ms per order, the OpenTelemetry starter 0.32 and 0.44, with Boot's starter exporting more spans per order (12.96 against 8.95 and 10.95).

At JVM 512 MB the starters added 28–44 MiB in the plain app and 36–43 MiB in the spring cells that passed. The agent is where the tier ran out. In the plain app, agent-default and agent-full-mt hung with 400 and 376 OutOfMemoryError lines, while agent-full-m and agent-matched-mt passed at +59.9 and +73.6 MiB. In the spring app all four agent configurations crashed, and so did Boot's starter with traces; both OTel starter cells passed. The crashed cells peaked at 378–394 MiB of process RSS, and no container was OOM-killed: they ran out of my 64 MB heap. In spring the agent needs more heap than that even metrics-only; how much more, I didn't measure. Those 7 cells are excluded from every delta, leaving 53 of 60.

At equal span coverage, comparing the agent trimmed to the OTel starter's instrumentation (agent-matched-mt) with the starter itself (on-mt): plain at 512 MB, +73.6 against +28.8 MiB average. Without a limit, peaks of +152.7 against +5.9 MiB (plain) and +129.4 against +57.4 (spring), and CPU of +0.40 against +0.29 and +0.41 against +0.38 ms per order. The agent costs more memory for the same spans; the CPU gap is within what one run can't separate from noise.

For part 1's question, going native while keeping the dashboards: the OTel starter with traces averaged 131 MiB native at 256 MB against 307 MiB on the JVM at 512 MB in the plain app, and 174 against 400 in spring. Different limits and a 64 MB JVM heap, so read it as the size of the gap, not a ratio to quote.

Of 67 cell starts, 66 passed the idle gate. The one that timed out, plain jvm-agent-full-mt-512mb, after 3,601 s at a load average of 1.11, hung anyway and is excluded. The gate records load, not idleness, and a sampler for other processes' CPU only ran from 14:26 UTC on 25 October, so the plain run has no record beyond the gate.

Two numbers this run can't give#

Run-to-run noise is the first. With one run per app, a CPU gap of a tenth of a millisecond per order, like the agent's +0.40 against the starter's +0.29, has nothing to be compared with. Measuring it means running the 60-cell matrix a second time, the run I cut.

The agent's heap floor is the second. Every spring agent cell died in a 64 MB heap, so all this run says is that the floor sits above 64 MB. Finding it takes a heap sweep over those four cells with the same gate, each step judged by verdict and live heap after full GC, not by whether spans arrived.

Filed under: java, spring-boot, opentelemetry, observability, benchmarks, native-image

Read next#

• Counts Match, Dashboards Don't: OTel Agent vs Starter: part 1, what each setup covers, the other half of every cost above.
• The Config I Measured Was Not the Config That Ran: part 3, the native image on library defaults, and why every cell now proves its mode.
• Every GC Failed at 256 MB (Except the Boring One): where the hand-tuned 64 MB heap recipe comes from.
• A Broken AOT Cache Still Serves 200: the AOT-cache tier this study left out, and how it fails quietly.

Sources#

• Spring Boot v4.1.2, SimpleMetricsExportAutoConfiguration: the fallback SimpleMeterRegistry and its conditions.
• Spring Boot v4.1.2, spring-boot-starter-actuator/build.gradle: the actuator starter's dependency on spring-boot-starter-micrometer-metrics.
• Spring Boot v4.1.2, spring-boot-starter-micrometer-metrics/build.gradle: that starter's dependency on the module holding the auto-configuration.
• opentelemetry-java-instrumentation PR #20001: agent JAR on the bootstrap class path, and the AOT-cache recipe without an active -javaagent.
• opentelemetry-java-instrumentation v2.32.0 release notes: the first release with #20001.