The AOT Hang That Wasn't

· 5 min read · java, micronaut, aot, debugging, native-image
The Memory Benchmark · Part 8
Contents

One line in CLAUDE.md: “Micronaut AOT hangs native image — keep both false. AOT disabled.”

Five sessions. Five different engineers (all me, different days). Nobody questioned it. The line accreted authority through repetition, the way all project folklore does. Somebody hit a problem, wrote the workaround, moved on. The workaround became canon. Canon became architecture.

Time to look under the coat.

Three Bugs, One Trenchcoat

What I’d been calling “the AOT hang” was three separate failures happening in sequence. Fix one, hit the next. Each one looked identical from the outside: app starts, prints banner, nothing happens. The diagnosis “AOT breaks everything” was reasonable. It was also wrong three different ways.

Failure 1: The Actual Hang

property-source-loader.generate.enabled=true

This one earns the name. The Micronaut AOT property source loader does something reasonable at build time: bakes application.yml into generated Java source. No YAML parsing at startup. Faster boot.

The problem: application.yml contains this:

r2dbc:
  datasources:
    default:
      url: ${DATASOURCE_URL}

At build time, there are no environment variables. The baked source contains the literal string ${DATASOURCE_URL}. Not the value. The placeholder text.

Liquibase gets this literal as its JDBC URL. Tries to resolve ${DATASOURCE_URL} as a hostname via DNS. DNS doesn’t NXDOMAIN it fast enough (some resolvers retry). The app hangs. Forever. No error. No timeout. No log line.

# What the operator sees:
$ docker logs myapp
 __  __ _                                  _
|  \/  (_) ___ _ __ ___  _ __   __ _ _   _| |_
| |\/| | |/ __| '__/ _ \| '_ \ / _` | | | | __|
| |  | | | (__| | | (_) | | | | (_| | |_| | |_
|_|  |_|_|\___|_|  \___/|_| |_|\__,_|\__,_|\__|

# ...forever

The banner prints because it’s before the application context wires datasources. Everything after that is silence.

Fix: property-source-loader.generate.enabled=false. The dividing line is simple: any AOT optimization that bakes values at build time will destroy any config that uses ${} environment variable substitution.

Failure 2: The Zombie

Fixed the property source loader. Rebuilt. Deployed. Same symptom — nothing on port 8181.

Except this time it wasn’t the app hanging. It was the app starting fine, trying to bind port 8181, and failing silently because the port was already taken.

By what? The previous container. docker compose down had been called, but the container’s process was still holding the socket. Docker’s cleanup is eventually consistent, and “eventually” was longer than the 3 seconds between down and up.

$ ss -tlnp | grep 8181
LISTEN  0  128  *:8181  *:*  users:(("java",pid=48291,fd=7))
$ docker ps -a | grep myapp
# ...exited 47 seconds ago, still in the process table

The app logged a bind exception. At WARN level. Which brings us to:

Failure 3: The Silence

Logback configuration:

<logger name="io.micronaut" level="WARN" />

Clean. Aggressive. Production-appropriate, in theory. Except Micronaut’s startup confirmation — the line that says “Startup completed in 832ms. Server Running: http://localhost:8181” — is logged at INFO by io.micronaut.runtime.Micronaut.

The WARN threshold swallowed it. The app was running. Serving requests. Logging absolutely nothing. The operator sees the banner, then silence, and concludes: hang.

<!-- Fix: let the runtime package through before the catch-all -->
<logger name="io.micronaut.runtime" level="INFO" />
<logger name="io.micronaut" level="WARN" />

Order matters. Logback evaluates most-specific first.

The Flags That Actually Work

After untangling three failures from their trenchcoat, here’s the actual AOT inventory for Micronaut 5.0.0 with R2DBC + Liquibase:

Safe (enable these):

cached.environment.enabled=true
serviceloading.jit.enabled=true
scan.reactive.types.enabled=true
deduce.environment.enabled=true
known.missing.types.enabled=true
logback.xml.to.java.enabled=true

Must be false (runtime env var substitution):

property-source-loader.generate.enabled=false
precompute.environment.properties.enabled=false
sealed.property-source.enabled=false

The dividing line: if the optimization bakes config values into generated source at compile time, and your config uses ${} placeholders for runtime environment variables, that optimization will embed the placeholder as a literal string. Everything else — service loading, type scanning, environment deduction — operates on structural metadata that doesn’t change between build and runtime.

The Twist

On the perf-testing-aot branch, with the safe AOT flags enabled and one other change — OpenAPI annotations and swagger-ui commented out:

ConfigRequestsErrorsOutcome
No AOT, with OpenAPI (Part 4)4,55081.3%Dead at ~60s
AOT enabled, no OpenAPI1,662,5910%Survived full benchmark

Same code. Same endpoints. Same JVM flags. Same 256 MB container limit. Same k6 script.

The OpenAPI annotation processors load reflection metadata into metaspace for every DTO. swagger-ui adds static resource classloader overhead. Together: ~18 MB of non-heap that serves exactly zero production requests.

Non-heap dropped from 119 MB to 101 MB. That 18 MB of headroom was the difference between a JVM that metaspace-OOMs under load and one that runs indefinitely at 0% error rate.

The AOT flags added their own savings on top — cached environment, pre-resolved service loading. But the kill shot was removing the library that existed purely for developer convenience in production containers.

The Documentation Was Right (About the Wrong Thing)

“AOT hangs native image” was true. One specific flag, in one specific configuration, with one specific trigger (environment variable placeholders in datasource URLs). The documentation just… didn’t decompose it. Couldn’t. The person who wrote it experienced three sequential failures and attributed them all to the last thing they’d changed.

Five sessions of avoidance. Eighteen megabytes of swagger. Three bugs in a trenchcoat. One line in CLAUDE.md that nobody checked under.

Sources