The Config I Measured Was Not the Config That Ran

· 11 min read
Contents

optional: is the most forgiving word in a Spring Boot config file. Put it in front of an import and Boot loads the file if it's there and carries on if it isn't. A missing file can never keep the app from starting. My benchmark app imported its telemetry settings that way. In the native image the import kept that promise to the letter. The file wasn't there, Boot carried on without it, and not one byte of telemetry reached the collector.

The app is the Spring Boot 4.1.2 order pipeline from part 1: POST an order, and RabbitMQ carries it through four consumers into Mongo and Postgres. Each telemetry mode is its own build, because Spring AOT and native images settle bean conditions at build time. mo-m and mo-mt use Boot's own spring-boot-starter-opentelemetry; on-m and on-mt use the OpenTelemetry project's Spring Boot starter. -m is metrics only, -mt is metrics and traces. A mode's settings live in one small YAML file that the Maven profile adds to the classpath. application.yml pulls it in with spring.config.import: optional:classpath:telemetry.yml.

Native smoke cells: healthy, and exporting to localhost#

At 00:45 on 24 October I started a smoke pass over five native builds: one 30-second step at 100 orders/s per mode. The pass exists to check that each mode delivers what it claims to the OpenTelemetry collector; nothing in it is a measurement. The same pass on the JVM, earlier that night, had filled the collector's metrics files, and the traces files of the traced modes.

Natively, three of the four telemetry cells looked perfect. App healthy after 2s (3 s for on-mt). verdict=ok. About 5,800 requests each, failure rate zero. And in all four cells, metrics.jsonl and traces.jsonl at the collector stayed at 0 bytes. The fourth cell, mo-mt, never passed its health check inside 120 seconds. No app log survived that run, and after the fix the same mode came up in two seconds, so whether the missing file caused that, I can't say.

The on-m app's error output said where the data had gone, and on-mt printed the same lines:

app-1  | java.net.ConnectException: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4318
app-1  | 	Suppressed: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:4318
app-1  | 	Caused by: java.net.ConnectException: Connection refused (connect failed)
app-1  | Caused by: java.net.ConnectException: Connection refused (connect failed)

Compose had pointed the app at http://collector:4318, through an environment variable that the mode file reads. With the mode file gone, nothing read it. http://localhost:4318 is the default OTLP/HTTP endpoint in the OpenTelemetry spec, where an exporter sends data when nobody has told it otherwise. Inside the app's container, nothing listens there.

The metrics-only mo-m cell was vaguer: a bare java.net.ConnectException out of the JDK's HTTP client, with no host and no port. A build that was meant to export only metrics was dialling somewhere that didn't answer. By Boot 4.1.2's source, spans are the unlikely half: as part 2 found, Boot builds an OTLP span exporter only when management.opentelemetry.tracing.export.otlp.endpoint is set (in this app, which has no Docker Compose or Testcontainers connection to the collector), and the mode file never set it. That leaves the metrics export, whose collector address also lived in the missing file. The log doesn't name the signal; that last step comes from the source alone.

Part 1 asks whether dashboards survive the move to native. This is what losing them looks like from inside the app: health green, requests served, and the exporters talking to a port nobody listens on.

Native resource hints only cover application* files#

On the JVM, classpath:telemetry.yml resolves because Maven put the file on the classpath. In a native image, resources are whatever the build copied in, and the build copies a resource only when a hint registered at build time matches it. For config files, Spring Boot registers its hint in ConfigDataLocationRuntimeHints, which takes its file names from StandardConfigDataLocationResolver: static final String[] DEFAULT_CONFIG_NAMES = { "application" };. Spring Framework's FilePatternResourceHintsRegistrar turns names into patterns with one line, includes.add(location + filePrefix + "*" + fileExtension);.

The locations are the classpath root and config/. The extensions come from Boot's two property-source loaders: .properties, .xml, .yml and .yaml. So the image gets application*.yml and its siblings, and telemetry.yml matches none of them. application-telemetry.yml matches application*.yml, the same way every application-<profile>.yml does. That is why profile files work in a native image and my file didn't.

That expansion is read from the source, at the tags this build used (Boot 4.1.2, Framework 7.0.10). I never printed the resource configuration the broken images were built with, and none of those builds survived.

optional: turns a missing file into a TRACE line#

Without the prefix, this bug would have been loud. Boot's reference, under Optional Locations: "By default, when a specified config data location does not exist, Spring Boot will throw a ConfigDataLocationNotFoundException and your application will not start." By that sentence, the native app would have refused to start on its first run, in plain sight.

With the prefix, the location never gets that far. An optional reference counts as skippable, and StandardConfigDataLocationResolver checks for the file before anything else touches it:

private List<StandardConfigDataResource> resolveNonPattern(StandardConfigDataReference reference) {
	Resource resource = this.resourceLoader.getResource(reference.getResourceLocation());
	if (!resource.exists() && reference.isSkippable()) {
		logSkippingResource(reference);
		return Collections.emptyList();
	}
	return Collections.singletonList(createConfigResourceLocation(reference, resource));
}

private void logSkippingResource(StandardConfigDataReference reference) {
	this.logger.trace(LogMessage.format("Skipping missing resource %s", reference));
}

One line, at TRACE, a level a stock logging setup never prints. I had asked for a quiet skip and got exactly that.

The JVM gate test could not see the image#

Every mode build already had a guard: ModeGateTest, the per-build test from part 2. It checks which telemetry beans exist for the mode, and it reads the mode name the file sets, through @Value("${telemetry.mode}"). It passed for every mode. It runs on the JVM, under surefire, where the mode directory is a plain Maven resource:

<resource>
  <directory>src/telemetry/${telemetry.mode}</directory>
</resource>

On that classpath the file exists under any name. The test asked the right question, which mode did you load, of the wrong runtime. The test and the native image were built from the same mode file. What differed was whether the file ended up in the artifact, and only a check that runs inside the image can see that.

This is the second time a Spring native build skipped a file of mine. The first time, process-aot wrote its own native-image.properties over the hand-written one at build time. This time nothing was overwritten. The file was never packed.

Fix: an application- name, then make the app say its mode#

The fix is a rename. Six mode files moved to application-telemetry.yml with their content unchanged, and the import followed:

-# Telemetry mode settings, baked in at build time (-Pmode-<x> adds src/telemetry/<mode>/telemetry.yml).
-spring.config.import: optional:classpath:telemetry.yml
+# Telemetry mode settings, baked in at build time (-Pmode-<x> adds src/telemetry/<mode>/application-telemetry.yml).
+# The name must start with `application`: Spring Boot's native resource hints only include application*.yml, so a
+# telemetry.yml was silently missing from native images (optional import) and they ran on defaults.
+spring.config.import: optional:classpath:application-telemetry.yml

The rename fixes this file. The guard is for the next one. HealthController now returns the mode the running app loaded, read with @Value("${telemetry.mode:unset}"). No build is called unset, so a missing file can never pass for a real mode. Right after the app reports healthy, before any load, the harness saves /health and compares:

curl -s --connect-timeout 2 --max-time 5 http://localhost:8181/health > "$SCENARIO_RESULTS_DIR/mode.txt" 2>/dev/null || true
if [ -n "${OTEL_BUILD_MODE:-}" ] && ! grep -q "\"mode\":\"${OTEL_BUILD_MODE}\"" "$SCENARIO_RESULTS_DIR/mode.txt"; then
  log "ERROR: app reports $(cat "$SCENARIO_RESULTS_DIR/mode.txt") but the build mode is ${OTEL_BUILD_MODE}; stopping"
  docker compose "${COMPOSE_ARGS[@]}" down -v >/dev/null 2>&1 || true
  exit 3
fi

A mismatch tears the stack down, and the cell ends with exit 3 and no numbers. After the rename, native mo-mt answered {"mode":"mo-mt","status":"UP","traces":"true"} and delivered 57,756 spans to the collector. The JVM smoke of the same mode had delivered 57,768: 12 spans apart, in two single runs.

The guard has two limits. It checks the mode name, not each property, so a file that loads with a wrong endpoint in it would pass. And up to 25 October no real run had tripped it: it was built in response to this one smoke pass.

docker compose run: 48 consumers, the backlog of 16#

On the evening of 23 October, a couple of hours before the native smoke, the same lesson had arrived by another road. It was a capacity test on the native build at 500 orders/s for a minute. With 16 enrich consumers, the enrich queue peaked at 11,637 messages. So I gave the app 48, through an extra override file that sets one variable, PIPELINE_RABBIT_ENRICH_CONSUMERS: "48". I started the stack with it and ran k6 through docker compose run. The queue peaked at 11,423. A second try: 11,948. Forty-eight consumers, the backlog of sixteen.

The answer was printed above the k6 banner, in the first six lines of the run's output:

 Container otelsb-mongo-1 Running 
 Container otelsb-mock-1 Running 
 Container otelsb-postgres-1 Running 
 Container otelsb-rabbitmq-1 Running 
 Container otelsb-app-1 Recreate 
 Container otelsb-app-1 Recreated

docker compose run starts more than a k6 container: it also brings k6's dependencies in line with the files it was given. I had given it the base files, without the override. The running app didn't match that config, so compose recreated it with 16 consumers, and k6 load-tested the replacement. Both "48" runs logged the same two lines.

The run reference says it starts linked services that are stopped, and offers --no-deps to leave them alone. It doesn't mention recreating one that is running. The up reference does describe recreating containers whose configuration changed. So the recreate under run is something I observed twice; I didn't find it written down.

The third run kept compose away from the app. Its output opens with the k6 container being created and nothing else: no dependency lines, no Recreate. The queue peaked at 129. I didn't save that command line. The missing dependency lines look like --no-deps. Passing run the same -f list the stack was started with would also have left the app alone: the 16-consumer baseline run printed every dependency as Running and kept its app, though its command line wasn't saved either.

No guard caught this one. The harness scripts already hand their full -f list to every compose run, and these two tests were run by hand, outside them. What caught it was a number that refused to move. The fix was the re-run and a note in the branch docs.

Cell names are claims#

native-on-m-unlimited and peak-native-48 were both named after the config I meant, and neither ran it. The native cells now carry proof from inside the running app, and a cell that can't show its mode produces no numbers. The hand-run capacity tests have no such check; the bad ones were caught because a change that should have moved a number didn't. That leaves one check for everything without a guard: when a knob you turned changes nothing, make sure the thing you measured is the thing you turned the knob on.

Filed under: java, spring-boot, graalvm, native-image, opentelemetry, docker

Sources#