Counts Match, Dashboards Don't: OTel Agent vs Starter

· 17 min read
Contents

Twenty-four test cells, 284 row checks, 24 cross-checks, zero failures. I ran one Spring Boot order pipeline under the OpenTelemetry Java agent, OpenTelemetry's Spring Boot starter and Boot's own OpenTelemetry starter, on the JVM and, for the two starters, as a GraalVM native image. Every count with an exact expectation, 284 rows of them, equalled what the load generator had sent.

The question I'd been asked was whether a service on the agent today can go native and keep its Grafana dashboards and alerts. The agent doesn't run in a native image, so native means a starter, and a parity report that green reads like permission to swap. A panel reads something else: a metric name, a unit, label keys, label values and histogram buckets. Scored on those, OpenTelemetry's starter in native keeps 4 of the 26 things a panel in my two test apps can read, and all four are the job and instance labels. Boot's starter in native keeps 2, both of them job. A perfect score on the wrong exam.

TL;DR — moving a Spring Boot service from the OTel Java agent to a starter, which native forces:

  • Counts survive. 24 of 24 cells matched ground truth, JVM and native.
  • Most panels need edits. Of 26 things a panel can read here, the OTel starter keeps 6 on the JVM and 4 in native. Boot's starter in native keeps 2.
  • Native adds little. On one starter, JVM and native emit identical store metrics; native drops some runtime metrics and renames memory pools.
  • Worst hits: per-queue RabbitMQ metrics (the OTel starter has totals or nothing), Hikari pool labels, Boot's milliseconds histograms without buckets, and Boot's missing instance label.
  • My label values aren't yours. Run your own panel and alert queries against a starter build in staging.

Test setup: one order pipeline, three telemetry stacks#

The app is a Spring Boot 4.1.2 order pipeline on Java 25. POST /orders publishes to RabbitMQ, and four consumers take it from there: one calls a mock REST API and writes MongoDB, one inserts into Postgres, and one publishes a second hop that a fourth turns into an audit row. GET /orders/{id} reads the order back. It comes in two builds: plain clients (amqp-client, the Mongo sync driver, JDBC on HikariCP) and Spring-idiomatic (Spring AMQP, Spring Data MongoDB, Spring Data JPA).

The stacks: the OTel Java agent 2.33.0; opentelemetry-spring-boot-starter 2.33.0, with the SDK raised to the 1.67.0 that release targets (OpenTelemetry's docs name Boot 2.6+ and 3.1+ for it; on 4.1.2 it worked); and Boot's spring-boot-starter-opentelemetry 4.1.2, which is Micrometer exporting OTLP. JVM cells ran Temurin 25.0.4.1; native ones were built with GraalVM CE 25.4.4.1.1 on JDK 25.0.4.1.1. Each of the twelve cells per app took 200 orders/s for 100 s, cold, one run, at 100 % trace sampling in the traced cells, into a collector writing OTLP JSON to disk. A count is the last export minus the value at load start. A native build of this app once shipped without its telemetry config, which is part 3; the native cells here match their JVM twins store instrument for store instrument.

Counts and durations: no drift#

Each count with an exact expectation equals its own cell's ground truth, the orders and reads the load generator sent. In the plain app an order is 2 publishes, 4 deliveries, 2 Postgres inserts and 1 Mongo write. The traced agent cell with opt-ins took 20,001 orders and reported 40,002 publishes, 80,004 deliveries and 20,001 rows per table, and so did the OTel starter's native cell. Order counts move a little between cells, 19,990 to 20,001 in the Spring app, and every count follows its own. Where two setups emit the same measure with an exact expectation, they agree. Where they differ, one of them emits nothing.

Mean durations mostly agree. The agent and the OTel starter share 17 duration rows. Compared twice, full agent and matched agent against the same starter run, 30 of the 34 comparisons land within ±5 %. With one run per cell that band is tight: the same agent timing the same JDBC insert, metrics-only against traced, moved 8.58 %, and the traced cell was the faster one in all eight JDBC rows compared that way. Two of the four outliers are a 5 µs select and a 46 µs Mongo update. The other two are the POST.

POST time under the full agent#

On the starter's base, POST /orders took 9.7 % longer under the full agent in the plain app (0.996 against 0.908 ms) and 13.3 % in the Spring app (1.125 against 0.993 ms). The matched agent switches off the instrumentations the starter lacks, in the plain app only RabbitMQ, and the gap falls to 4.1 % and 2.25 %. The publish and its confirm run inside the request in every cell, so what the matched cell removes is the agent's own RabbitMQ work on the request thread: its spans and the trace header it injects. That's my reading of one cold run without per-span timing, not a profile.

Dashboard compatibility: 26 things a panel reads#

The collector files hold every export, so they answer the dashboard question too. Baseline: the agent with the semconv opt-ins, metrics only, on the JVM. Per app there are 13 things a panel can depend on: the HTTP, JDBC and Mongo instruments, four RabbitMQ instruments, Hikari's use-time histogram, the Hikari and JVM metric families, and the job, instance and target_info labels Prometheus builds from resource attributes. Seven outcomes, mildest first: keeps working; value change, where a label value is gone and only queries that filter or group on it break; label change, where a label key or a measure is gone and every query using it breaks; partly gone, where a metric family keeps most of its names and loses some; values meaningless, where a metric is still emitted but its numbers carry no information; rewrite, a new name, a new unit or no buckets; gone.

target (26 rows, both apps)keeps workingvalue changelabel changepartly gonevalues meaninglessrewritegone
OTel starter, JVM6640028
OTel starter, native4460228
Boot starter, native20400155

Store by store, with the strings that moved (OTel starter native as on the JVM unless noted):

storeagent with opt-ins (baseline)OTel starterBoot starter, native
HTTP serverhttp.server.request.duration [s]; 404s get http.route="/**"value change: 404s carry no routerewrite: http.server.requests [milliseconds], no buckets, Micrometer keys such as method, status, uri
Postgres (JDBC)db.client.operation.duration [s], db.namespace="space_observatory|space"value change: db.namespace="space_observatory"gone: no statement metric
MongoDBdb.client.operation.duration [s] with network.peer.address, network.peer.port, server.addresslabel change: those three keys gonerewrite: mongodb.driver.commands [milliseconds], no buckets
RabbitMQ (4 rows)per-destination messaging.*plain: 2 rewrite, 2 gone, client-wide rabbitmq.* totals; Spring AMQP: 4 goneplain: 2 rewrite, 2 gone, rabbitmq.* totals; Spring: 3 rewrite, 1 gone, spring.rabbitmq.listener [milliseconds] per queue
Hikari pool (2 rows)db.client.connection.* [s], pool space_observatoryplain, module added by hand: label change, pool HikariPool-1, db.namespace, db.system.name, server.address gone; Spring: gonerewrite: hikaricp.connections.*, timers in milliseconds
JVM runtime20 jvm.* namesJVM: keeps working; native: values meaningless, class counts read 0 and committed tracks used; also 5 of 20 names missing, pools renamedrewrite: Micrometer's jvm.* names; jvm.memory.max reads −1
Resource (3 rows)job, instance, 18 target_info keysJVM: 2 keep, 1 value change; native: 2 keep, 1 label change, service.version, process.command_args, process.executable.path gone1 keep, 2 label change: no instance, 15 target_info keys gone

HikariPool-1 is Hikari's generated default name. The starter has no pool metrics of its own, so the plain app got OTel's opentelemetry-hikaricp-3.0 module by hand; the Spring app ran the starter alone. For RabbitMQ OpenTelemetry has no library instrumentation at all: plain amqp-client falls back to its own client-wide counters, and Spring AMQP gets nothing.

The baseline matters too. On its defaults the agent emits HTTP, JVM and Hikari metrics, the last under the old db.client.connections.* names, and nothing for Postgres, Mongo or RabbitMQ. If that's your setup, your store panels are fed from somewhere else, and that source is outside this comparison.

Through Prometheus, names also carry unit and type. http.server.request.duration in seconds becomes http_server_request_duration_seconds_bucket, _sum and _count; a monotonic counter gains _total, so amqp-client's totals arrive as rabbitmq_published_total and rabbitmq_consumed_total. That's the default in Prometheus' OTLP receiver and in the collector's Prometheus exporter, so a unit change renames the series too. The exporter also stamps the instrumentation scope on every series as otel_scope_name, and that moves as well: io.opentelemetry.tomcat-10.0 under the agent, io.opentelemetry.spring-webmvc-6.0 under the OTel starter, empty under Boot's.

Native changes runtime metrics, not store metrics#

On the same starter, the JVM and native cells emit identical store instruments: names, units, types, label keys, label values, buckets. That's 11 of 11 rows for the OTel starter and 25 of 25 for Boot's. Native takes away runtime metrics: five jvm.* names under the OTel starter, jvm.gc.duration and jvm.memory.limit among them, and 13 under Boot's, among them jvm.gc.pause and jvm.compilation.time. The memory pools keep reporting under new values, so G1 Eden Space and G1 Old Gen turn into eden space and old generation space. Some survivors carry no information: Boot's jvm.memory.max reads −1 for every pool, jvm.memory.committed tracks jvm.memory.used, both starters' class counts read 0, and so do Boot's thread-state and GC-overhead gauges. The Java 25 post has the upstream reasons. Apart from those and a few resource attributes, everything in the tables above comes from leaving the agent, and the OTel starter charges most of it on the JVM as well.

Boot's starter: Micrometer names in milliseconds#

Boot's starter is Micrometer with an OTLP exporter, and the collector files show it. Here are 20,000 POSTs as the agent on the JVM and Boot's starter in native each reported them, from two separate runs, trimmed:

{"name":"http.server.request.duration","description":"Duration of HTTP server requests.","unit":"s","histogram":{"dataPoints":[…{"attributes":[…{"key":"http.route","value":{"stringValue":"/orders"}},…],…,"count":"20000","sum":19.78435938299999,"bucketCounts":["19971","12","7","4","5","1","0",…],"explicitBounds":[0.005,0.01,0.025,0.05,0.075,0.1,0.25,0.5,0.75,1,2.5,5,7.5,10],…}
{"name":"http.server.requests","unit":"milliseconds","histogram":{"dataPoints":[…{"attributes":[…{"key":"uri","value":{"stringValue":"/orders"}}],…,"count":"20000","sum":17928.088147},…

That's three breaks in two lines. The name is Micrometer's http.server.requests, so a panel that asks Prometheus for http_server_request_duration_seconds_count would now need http_server_requests_milliseconds_count. The unit is the string milliseconds, so a 0.5 s latency threshold becomes 500. And there are no bucketCounts and no explicitBounds: with Boot's defaults, which these cells ran, a timer exports a count and a sum, histogram_quantile has nothing to work with, and p95 panels and percentile alerts go blank. Boot has a property for that, management.metrics.distribution.percentiles-histogram, which I didn't test. Mongo, the pool and Spring AMQP follow the same pattern.

Two more breaks belong to Boot's starter alone:

  • No service.instance.id. Prometheus builds instance from it, so the label has no value: queries that select by instance find nothing, and replicas can't be told apart by it. target_info loses 15 keys, among them host.name, os.type, process.pid, container.id and service.version.
  • The traced build changes metric names. It switches on Spring AMQP observations, which the metrics-only build leaves off, and in the Spring app spring.rabbitmq.listener gives way to spring.rabbit.listener and spring.rabbit.template.

Starter wiring: Mongo, Hikari, RabbitMQ and two flags#

The starter cells weren't zero-config. In the plain app, Boot's starter got Mongo and RabbitMQ through Micrometer binders I wired, and OpenTelemetry's got Mongo, Hikari and RabbitMQ wired by hand. At v2.33.0 the OTel starter hooks Mongo into Boot's own client builder, which this app's client skips, so it got MongoTelemetry.create(openTelemetry).createCommandListener() (I didn't test it without). amqp-client got its own OpenTelemetryMetricsCollector, and the pool a static post-processor, so the module is attached before the pool starts:

/** Static, so the post-processor is registered before the DataSource is created; OpenTelemetry resolved lazily. */
@Bean
static BeanPostProcessor hikariTelemetry(ObjectProvider<OpenTelemetry> openTelemetry) {
    return new BeanPostProcessor() {
        @Override
        public Object postProcessBeforeInitialization(Object bean, String beanName) {
            if (bean instanceof HikariDataSource ds) {
                ds.setMetricsTrackerFactory(
                        HikariTelemetry.create(openTelemetry.getObject()).createMetricsTrackerFactory());
            }
            return bean;
        }
    };
}

GotchaNo database or messaging metrics without opt-in flags

Out of the box, neither the agent nor the OTel starter emits JDBC or Mongo metrics, and a wired Hikari pool reports under the old db.client.connections.* names in ms. The opt-in cells here got -Dotel.semconv-stability.opt-in=database and -Dotel.semconv-stability.preview=messaging as system properties. At v2.33.0 the library code behind the starter resolves both once: from global declarative config, which the starter doesn't register, then from a system property, then from the environment (OTEL_SEMCONV_STABILITY_OPT_IN, OTEL_SEMCONV_STABILITY_PREVIEW). I never tried them in application.yml, so "YAML is ignored" comes from the source; no test of it ran.

Spans per order measure coverage#

The traces hold the second number that misleads. Spans per order at 100 % sampling:

setupplain appSpring app
agent with opt-ins (on defaults: the same, 20.95 / 36.85)20.9536.86
agent, matched to the starter8.9510.95
OTel starter, JVM / native8.95 / 8.9510.94 / 10.95
Boot starter, JVM / native12.96 / 12.9612.96 / 12.96

Read as cost, the agent looks 2.3 times as heavy as the starter in the plain app and 3.4 times in the Spring app. The matched agent, the same agent with the instrumentations the starter lacks switched off, lands on 8.95 and 10.95: the starter's numbers. In the plain app the agent's extra 12.00 spans per order are exactly its RabbitMQ scope: 2 publishes, 2 confirm waits, 4 acks and 4 processed deliveries. In the Spring app the extra 25.91 are RabbitMQ (8.00), Spring AMQP (4.00), Hibernate (7.95) and Spring Data (5.96). The agent doesn't emit more spans for the same work. It sees more of the work.

What that coverage costs in memory and CPU belongs to part 4. The short version: at equal coverage the agent used more memory than the OTel starter, and without a memory limit CPU per order came out close. That is one run per app, so read it as a direction rather than a margin.

Two findings that aren't telemetry#

Spring Data JPA issued 1.503 times the SQL statements of plain JDBC for the same orders, 119,470 against 79,504 in the agent cells. The entities carry app-assigned ids and no @Version, and the Spring Data JPA docs say what follows: "If the identifier property is null, then the entity is assumed to be new. Otherwise, it is assumed to be not new." Not new means merge, and Hibernate selects before it inserts. The agent and the OTel starter count the same statements (Boot's starter has no statement metric), so the bill belongs to the app.

An insert into the same table measures 2.845 ms through the plain DAO and 0.134 ms through JPA, and the two aren't the same statement: the plain DAO sends INSERT … ON CONFLICT (id) DO NOTHING, JPA a plain INSERT. My reading, not isolated: the plain DAO also runs in autocommit, HikariCP's default, so its statement time likely includes a commit that waits for WAL (Postgres defaults synchronous_commit to on), while JPA commits outside the statement. Either way, db.client.operation.duration doesn't compare across the two styles.

Before you switch#

Any starter, in native: rebuild GC-timing and heap-limit panels around the missing jvm.* names, and per-pool panels around the renamed pools. Heap used survives; committed tracks used, and Boot's jvm.memory.max reads −1, so used-over-max ratios have to go. Panels and alerts on class counts, and on Boot's thread states and GC overhead, sit at 0 in native; a GC-overhead alert never fires.

Any starter, JVM or native: set service.name explicitly. job survived every swap here only because every setup set it (the agent through -Dotel.service.name, the starters in their config); how each one picks a default name, I didn't test.

The OTel starter:

  • Pass the semconv opt-ins as system properties or environment variables.
  • Drop filters on http.route="/**". 404s now arrive with no route.
  • Re-check the db.namespace values your Postgres panels filter on.
  • Take network.peer.address, network.peer.port and server.address out of Mongo queries.
  • Add opentelemetry-hikaricp-3.0, then change the pool name in queries and drop db.namespace, db.system.name and server.address from them.
  • Rebuild RabbitMQ panels without per-queue rates: client-wide _total counters with amqp-client, nothing with Spring AMQP.
  • In native, stop joining target_info on service.version.

Boot's starter:

  • Move every query to Micrometer names, and multiply latency thresholds by 1000.
  • Expect no buckets: enable percentiles-histogram and check the output, or give up percentiles.
  • Set service.instance.id yourself (untested), or lose instance.
  • Find another source for Postgres statement panels.
  • Build the panels against the tracing mode you'll run.

NoteWhat this table can't see

Label values here come from this app's traffic only: two routes under load, two tables, one collection, four queues, one pool. Your routes, schemas and pool names will break in their own ways. Before switching, point your key panel and alert queries at a starter build in staging and look for empty results.

The parity check asked whether every setup saw the same orders, and 284 rows said yes. Grafana asks for a name, a unit and a label, and draws an empty panel when any of them moves.

Filed under: java, spring-boot, opentelemetry, observability, prometheus, native-image

Sources#