Should execution order depend on TestEngine service discovery order?

[trancexpress]

Hi,

I have a question regarding test execution order, and its dependence to how the org.junit.platform.engine.TestEngine services are discovered in the JVM.

Should this code order test engines explicitly?

org.junit.platform.launcher.core.LauncherFactory.collectTestEngines(LauncherConfig)

	private static Set<TestEngine> collectTestEngines(LauncherConfig config) {
		Set<TestEngine> engines = new LinkedHashSet<>();
		if (config.isTestEngineAutoRegistrationEnabled()) {
			new ServiceLoaderTestEngineRegistry().loadTestEngines().forEach(engines::add);
		}
		engines.addAll(config.getAdditionalTestEngines());
		return engines;
	}

It seems that class loading can lead to different test engine order, which leads to different test execution order. This is from my POV not great.

Here is a small example of some test suites:

jep4194_discovery_call.zip

The file structure is:

$ tree
.
├── junit
│   ├── junit-4.12.jar
│   ├── junit-jupiter-api-5.12.2.jar
│   ├── junit-jupiter-engine-5.12.2.jar
│   ├── junit-jupiter-engine-5.12.2-sources.jar
│   ├── junit-platform-commons-1.12.2.jar
│   ├── junit-platform-commons-1.12.2-sources.jar
│   ├── junit-platform-console-standalone-1.12.2.jar
│   ├── junit-platform-engine-1.12.2.jar
│   ├── junit-platform-engine-1.12.2-sources.jar
│   ├── junit-platform-launcher-1.12.2.jar
│   ├── junit-platform-suite-1.12.2.jar
│   ├── junit-platform-suite-api-1.12.2.jar
│   ├── junit-vintage-engine-5.12.2.jar
│   └── opentest4j-1.3.0.jar
├── test
│   ├── InnerTestSuite1.java
│   ├── InnerTestSuite2.java
│   ├── InnerTestSuite3.java
│   ├── InnerTestSuite4InnerTestSuite1.java
│   ├── InnerTestSuite4.java
│   ├── MyTest1.java
│   ├── MyTest2.java
│   ├── MyTest3.java
│   ├── MyTest41.java
│   ├── MyTest42.java
│   ├── MyTest4.java
│   ├── MyTest5.java
│   ├── MyTest6.java
│   ├── MyTest7.java
│   ├── NoTest1.java
│   └── OuterTestSuite.java
└── TestRunner.java

2 directories, 31 files

I can run the tests with:

$ /usr/lib/jvm/java-21/bin/javac -cp "junit/*" TestRunner.java test/*.java &&  /usr/lib/jvm/java-21/bin/java -cp ".:junit/opentest4j-1.3.0.jar:junit/junit-vintage-engine-5.12.2.jar:junit/junit-platform-suite-api-1.12.2.jar:junit/junit-platform-suite-1.12.2.jar:junit/junit-platform-launcher-1.12.2.jar:junit/junit-platform-engine-1.12.2.jar:junit/junit-platform-commons-1.12.2.jar:junit/junit-jupiter-engine-5.12.2.jar:junit/junit-4.12.jar:junit/junit-platform-console-standalone-1.12.2.jar" TestRunner
MyTest41.test()
MyTest7.test()
MyTest42.test()
MyTest4.test()
MyTest1.test()
MyTest3.test()
MyTest2.test()
MyTest5.test()
MyTest6.test()

And I can run the tests with:

$ /usr/lib/jvm/java-21/bin/javac -cp "junit/*" TestRunner.java test/*.java &&  /usr/lib/jvm/java-21/bin/java -cp ".:junit/opentest4j-1.3.0.jar:junit/junit-platform-suite-api-1.12.2.jar:junit/junit-platform-suite-1.12.2.jar:junit/junit-platform-launcher-1.12.2.jar:junit/junit-platform-engine-1.12.2.jar:junit/junit-platform-commons-1.12.2.jar:junit/junit-jupiter-engine-5.12.2.jar:junit/junit-4.12.jar:junit/junit-platform-console-standalone-1.12.2.jar:junit/junit-vintage-engine-5.12.2.jar" TestRunner
MyTest5.test()
MyTest6.test()
MyTest42.test()
MyTest41.test()
MyTest7.test()
MyTest4.test()
MyTest1.test()
MyTest3.test()
MyTest2.test()

Specifying a different -cp resulting in a different test order comes as a bit of a surprise to me. My impression was that the execution order of tests was stable. In the Eclipse case we have a class loader that seems to cause different order of test engines (I'm still debugging that); its not a question of having a different command line that leads to different test order.

Background:

I'm looking into an issue when running Eclipse tests with mvn and tycho-surefire: eclipse-jdt/eclipse.jdt.ui#2423

One part of the issue is instable test order - we had a test that didn't properly handle resource clean-up. However, only some CI jobs would run into failures. This is due to instable test order - sometimes the "bad" test would run "first" and result in som following failing. But more often it would run last, resulting in no failures.

From debugging, it looks like tycho code will discover services in an unstable order, resulting in an unstable order in LauncherFactory.collectTestEngines(). This means the Eclipse JDT UI mixture of JUnit 4, JUnit 5 and test suites results in sometimes running test suites first, sometimes running JUnit 4 tests first, sometimes running JUnit 5 tests first - depending on the order of how engines were collected (depending on how services were discovered).

Best regards and thanks,
Simeon

[marcphilipp]

From debugging, it looks like tycho code will discover services in an unstable order

That sounds like a serious issue with Tycho that might cause other issues as well. I'd definitely recommend you getting to the bottom of that.

It seems that class loading can lead to different test engine order, which leads to different test execution order. This is from my POV not great.

The ServiceLoader mechanism that is used under the hood does not support ordering and documents that the order depends on the classpath order.

However, for some ServiceLoader-registered interfaces such as TestExecutionListener we already support filtering so I can see us adding support for sorting as well.

For example, we could introduce the following interface and allow setting it via a system property:

interface ServiceLoaderOrderer {
	Optional<Comparator<? super T>> provideComparator(Class<T> service);
}