No Manual Intervention Required

· 9 min read

A retry test asserted three requests and read zero. Three is not a guess: the stub upstream answers 500, the client is configured for two retries, and the same assertion had been green on Quarkus 3.33.2.1. Zero means the stub saw nothing at all, and the stub was running.

That was the first result of moving six benchmark branches from the 3.33 LTS to the 3.40 LTS, a change to one property, quarkus.platform.version. The 3.40 migration guide expects nothing of the sort: "Quarkus 3.40 LTS is the direct continuation of 3.39 so we don’t expect any changes requiring manual intervention in 3.40."

The finished commit changed no file under src/main/java. It changed how two test classes keep their counters (a 46-line diff each) and added a Dev Services key that does nothing in production. Both traps sat entirely in the test harness, and neither is in the 3.34–3.39 migration guides.

Five retry tests counted zero requests#

./mvnw clean verify on 3.40.1 ran 48 tests and failed 5:

[ERROR]   ApodRangeFetchTest.aRangeFetchDeserialisesTheArrayAndRetriesAFailedAttempt:53 the first attempt failed, so the retry must have issued a second request ==> expected: <2> but was: <0>
[ERROR]   RetryTriggerTest.aDroppedConnectionIsRetried:60 a transport failure must be retried twice ==> expected: <3> but was: <0>
[ERROR]   RetryTriggerTest.aRedirectIsNotFollowed:90 a 302 must not cost a second request ==> expected: <1> but was: <0>
[ERROR]   RetryTriggerTest.aStatusOutsideTheEnumIsStillClassifiedCorrectly:73 an unlisted 4xx must stop the loop ==> expected: <1> but was: <0>
[ERROR]   RetryTriggerTest.serverErrorsAreRetriedAndClientErrorsAreNot:44 a 5xx must be retried twice ==> expected: <3> but was: <0>

RetryTriggerTest is the stub test that counted eighteen requests for one redirect on Micronaut. On Quarkus 3.33 it passed. Zero is a strange answer from it: even a client that never retries makes one request. The test is three statics and a profile:

@QuarkusTest
@TestProfile(RetryTriggerTest.StubUpstream.class)
class RetryTriggerTest {

    static final AtomicInteger REQUESTS = new AtomicInteger();
    static final AtomicInteger STATUS = new AtomicInteger(500);
    static final HttpServer SERVER = start();
    // ...
    public static class StubUpstream implements QuarkusTestProfile {
        @Override
        public Map<String, String> getConfigOverrides() {
            return Map.of("quarkus.rest-client.nasa-apod.url",
                    "http://127.0.0.1:" + SERVER.getAddress().getPort());
        }
    }
}

The stub's handler increments REQUESTS. Every callsFor(status) sets the status, resets the counter, makes one client call and reads the counter back; serverErrorsAreRetriedAndClientErrorsAreNot does that five times. Two debug prints settled it: one in start() and one just before the read, each printing the class loader and the stub's port.

      4 DBG callsFor cl=QuarkusClassLoader:Quarkus Runtime ClassLoader: TEST for JUnitQuarkusTest-dev.tsvinc.space.client.RetryTriggerTest$StubUpstream (QuarkusTest) restart no:0@43982337 port=42317
      1 DBG start port=34805 cl=io.quarkus.test.junit.classloading.ParentLastURLClassLoader@657c8ad9
      1 DBG start port=42317 cl=QuarkusClassLoader:Quarkus Runtime ClassLoader: TEST for JUnitQuarkusTest-dev.tsvinc.space.client.RetryTriggerTest$StubUpstream (QuarkusTest) restart no:0@43982337
      1 [ERROR] Tests run: 4, Failures: 4, Errors: 0, Skipped: 0
      1 [ERROR] Tests run: 4, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 9.175 s <<< FAILURE! -- in dev.tsvinc.space.client.RetryTriggerTest

There are two RetryTriggerTest classes, each with its own statics, so two stub servers start. The test methods run in the Quarkus runtime class loader and read the counter of the stub on 42317. The other copy lives in the JUnit-side ParentLastURLClassLoader and owns a stub on 34805. The profile's config override is the only thing that tells the REST client where to go, and 42317's counter stayed at zero. So the client must have been talking to 34805, whose counter sits in a copy of the class no assertion reads. I never printed that counter; the run only shows that 42317's stayed at zero.

On 3.33.2.1 the same two test classes pass; I checked that in a throwaway worktree of the old commit. ParentLastURLClassLoader is byte-identical in both releases, so the loader isn't new. Since the tests passed there, the profile and the test methods must have shared one copy of the class on 3.33; I didn't print the loaders on 3.33, so which side moved on 3.40 is an inference I can't settle, and I didn't bisect the six releases in between. The closest upstream changes I found are #52575 (3.34.0.CR1), which stops getConfigOverrides() from passing its values through system properties and writes a generated application.properties instead, and #52919 (3.35.0.CR1), which does the same for Dev Services and test resources. Neither says the profile is now evaluated in a different class loader.

The commit messages on five of the six branches say the tests were "silently reading 0". They weren't silent. Five assertions failed by name. The sixth branch's message got it right, and the other five are pushed history now.

Moving the counters into system properties#

Both copies of the class share one JVM, so they share System.getProperties(). The first attempt stored AtomicInteger objects in there. Properties is a Hashtable<Object,Object> and takes them without complaint. Narayana's build step does not. Two lines from a grep of that test log:

135:Caused by: java.lang.NullPointerException
[...]
149:Caused by: java.lang.RuntimeException: unable to load properties from jar:file:///home/user/.m2/repository/org/jboss/narayana/jta/narayana-jta/7.3.4.Final/narayana-jta-7.3.4.Final.jar!/jbossts-properties.xml

Narayana copies every system property into its own configuration with System.getProperty(name). For a value that isn't a String that call returns null, and Properties.setProperty(name, null) is the NPE. So the version that shipped stores a String:

static final class SharedInt {
    private final String key;

    SharedInt(String name, int initial) {
        key = RetryTriggerTest.class.getName() + "." + name;
        System.getProperties().putIfAbsent(key, Integer.toString(initial));
    }

    int get() {
        return Integer.parseInt(System.getProperty(key));
    }

    void set(int value) {
        System.setProperty(key, Integer.toString(value));
    }

    int incrementAndGet() {
        synchronized (System.getProperties()) {
            set(get() + 1);
            return get();
        }
    }
    // getAndDecrement() follows the same pattern
}

The key is the class's fully qualified name, which is the same in both loaders, so both copies land on the same entry. Quarkus spent two releases moving its test configuration out of system properties, and my fix moved the test counters in.

One thing is still wrong. static final HttpServer SERVER = start() still runs once per loader, so every test class starts two stubs and stops neither. Only one of them ever gets traffic. It's a test-only leak and it stays in the backlog.

The second branch inherited the first branch's database#

The second trap showed up on the next branch in line. Quarkus started, and Liquibase refused the schema:

Caused by: java.lang.IllegalStateException: Error starting Liquibase
[...]
Caused by: liquibase.exception.CommandExecutionException: liquibase.exception.ValidationFailedException: Validation Failed:
     2 changesets check sum
          db/changelog/001-create-schema.yaml::1-create-tables::tsvinc was: 9:5d83e942beedca05c91b370bc415747a but is now: 9:db6b0257a417664bfed770c477d0e049
          db/changelog/001-create-schema.yaml::1-indexes::tsvinc was: 9:87fb84e057688b7d0cc1a6e8ae8ecdce but is now: 9:872ccc18baf6b86a5f9d6149956acf90

"Was" is what the database had recorded. "Is now" is this branch's changelog. The six Quarkus branches carry three different versions of 001-create-schema.yaml, so a mismatch means the Dev Services Postgres had already been migrated by another branch. docker ps showed two Dev Services Postgres containers:

f831cba7bd84 dazzling_chaum postgres:18 About a minute ago datasource=default,io.quarkus.devservice.launch-mode=TEST,org.testcontainers.copied_files.hash=1,org.testcontainers.hash=2d9e4e2a696a7e4555b7e3019e85ae5c00bcc95f,org.testcontainers.lang=java,org.testcontainers.version=2.0.5,org.testcontainers=true
47d89150377c interesting_satoshi postgres:18 5 minutes ago datasource=default,io.quarkus.devservice.launch-mode=TEST,io.quarkus.devservice.process-uuid=8f2ea83b-c5f0-453d-aa3d-f9ea7f8e9838,org.testcontainers.copied_files.hash=1,org.testcontainers.hash=653ddb05231eb87fd4a3f7552ae26a8df67128f4,org.testcontainers.lang=java,org.testcontainers.version=2.0.4,org.testcontainers=true

The newer one came from a 3.40.1 run: Testcontainers 2.0.5, an org.testcontainers.hash label, no process-uuid. The older one, 47d89150377c, was started about five minutes earlier and not by any of my runs; it is a 3.33-era container and carries a process-uuid. And this box opted into reuse long ago:

$ cat ~/.testcontainers.properties
#Modified by Testcontainers
#Tue Feb 10 15:43:33 EET 2026
docker.api.version=1.44
docker.client.strategy=org.testcontainers.dockerclient.UnixSocketClientProviderStrategy
ryuk.container.privileged=true
testcontainers.reuse.enable=true
ryuk.disabled=true

Container reuse had never worked here until 3.40#

quarkus.datasource.devservices.reuse defaults to true. That is true on 3.33 too: the 3.33 Dev Services guide and the current one word the property the same way. Testcontainers reuses a container when the create command hashes to the same value. In 3.33.2.1, every Dev Services container got this label unconditionally:

container.withLabel(QUARKUS_PROCESS_UUID, RunningDevServicesRegistry.APPLICATION_UUID);

That's a fresh UUID per JVM, so every run produced a new hash and nothing ever matched. Under 3.33, every test run on this host asked for reuse and none got it. With ryuk.disabled=true nothing removed them either, which is how a container like 47d89150377c ends up still running, unclaimed. In 3.40.1 the label is conditional:

public static boolean shouldConfigureProcessUuidLabel(GenericContainer<?> container) {
    return !(TestcontainersConfiguration.getInstance().environmentSupportsReuse()
            && container.isShouldBeReused());
}

That is the upstream fix for #53312, "Testcontainers are not reused between builds and test executions with testcontainers.reuse.enable=true", milestone 3.38.0.CR1. It is a correct fix for a real bug, and it was also backported: the ConfigureUtil.java at tag 3.33.4 has the same method, and 3.33.3 does not. So even a patch bump on the old LTS would have set off the same trap.

Once the hash is stable, it doesn't depend on the branch either. All six declare the same datasource: same default image, same username, same password, same default database name. Every Quarkus branch on this host now matched one long-lived Postgres, and whichever branch ran Liquibase first owned the checksums.

One build-time key#

$ /usr/bin/git show origin/quarkus/imperative-jdbc-synth -- src/main/resources/application.properties
[...]
+# Dev Services must not keep its Postgres for the next run. With testcontainers.reuse.enable=true
+# on the host, Quarkus 3.40 hands a later test run the same container -- also from a sibling
+# branch with a different changelog, which then fails Liquibase's checksum validation.
+quarkus.datasource.devservices.reuse=false
[...]

Dev Services reads it from the build-time datasource config, and in a prod or native build there are no Dev Services, so the key does nothing there. It lives in src/main/resources without a profile prefix, which works but hides the intent. %test.quarkus.datasource.devservices.reuse=false would say outright that this is a test-harness setting. Every run now gets its own container, and with the key in place all six branches pass: 48, 38, 30, 43, 35 and 27 tests, zero failures.

For the next bump, I'll check the host files the tests depend on, not only the migration guide. ~/.testcontainers.properties decided this one, and it isn't in any repository.

Filed under: java, quarkus, testing, debugging

Sources#