Concurrent Testing

Finally had some time at JavaOne to sit down and whack away at something that I have never had the time to explore thoroughly: parallel JUnit testing using the Maven Surefire plugin.

The executive overview is that you can run your JUnit tests in parallel in a number of different ways, and it’s worth understanding them all thoroughly so you don’t inadvertently structure your tests in such a way that parallelism is impossible.

To start with, let’s look at the general architecture of a normal, non-exotic JUnit test as run by Surefire.

Unless you’ve done something unusual, you likely have a class or two lying around in your src/test/java tree named TestCaseSomethingOrOther.java. And in that test class you likely have various methods annotated with @Test.

If you run this with Surefire 2.16 out of the box, you’ll note that the following things happen in order:

1. The Maven JVM forks exactly one additional JVM. Its settings are taken from the maven-surefire-plugin:test goal’s argLine property (or defaulted).
2. The Surefire JVM just forked loads your test class and instructs JUnit to run it.
3. JUnit runs any @BeforeClass-annotated methods in your class.
4. For each test method:
   1. JUnit creates a new instance of your test class.
   2. JUnit runs any @Rule-annotated public fields in your test class.
   3. JUnit runs any @Before-annotated methods in your test class.
   4. JUnit executes your test method.
   5. JUnit runs any @After-annotated methods in your test class.
5. JUnit runs any @AfterClass-annotated methods in your class.

…and that’s it.

It’s important to note that a new instance of your test class is created for each test method that is run. File that away for a moment.

Now, as regards parallelism, we can control the number of threads that are dedicated to running JUnit test methods, and we can control the number of processes that can run these threads.

First, let’s look at the processes. We can control these in Surefire using the forkCount and reuseForks properties.

I don’t know about you, but these terms were somewhat confusing. forkCount is the maximum number of forked JVMs that can be running in parallel at any time. It says nothing about the total number of these forked JVMs that might exist over time. reuseForks makes it so that the number of total operating system processes spawned over time is either governed (and equal to the forkCount) or ungoverned. So if you want to ensure that only two processes, period, are created by Surefire, then you want <forkCount>2</forkCount> and <reuseForks>true</reuseForks>. If, on the other hand, you don’t really care how many processes Surefire ends up spawning and killing, but you want to make sure that no more than two are running at the same time, then you want <forkCount>2</forkCount> and <reuseForks>false</reuseForks>.

Normally you want to reuse forks. The only time I could think of where you wouldn’t want that is if your tests exercise some kind of static singleton or something that has state inside it that you can’t reset. If that’s true, then if you reuse forks it is possible that the state of this singleton could pollute subsequent tests.

Next, let’s look at methods.

You can parallelize test methods at the thread level, but not at the process level.

A JUnit test method, as we’ve seen, is conceptually equivalent to a constructor invocation, some setup work, the method invocation and some teardown work. There’s no way to instruct Surefire to somehow create a new JVM process as well for this; process parallelism stops at the class level. (Creating a separate process for each test method invocation would be a little nuts; if you really need that you can create JUnit test classes that take great care to have only one test method in them. You could probably do something else too with JUnit suites.)

So we’re looking at threads when we’re looking at parallelizing JUnit test methods. You can indicate that you want Surefire to run your test methods in parallel by using the aptly-named parallel property. Together with the threadCount, perCoreThreadCount and useUnlimitedThreads properties, you can control how many threads are spawned to run test methods.

Note that if you have more than one thread rummaging around in your test case, the static information (if any) your test might have needs to be thread safe. It is best of course if you can avoid it to not have any such static information that can change.

Recall as well that a test method invocation is also semantically a constructor invocation, so oddly enough while your static information must be thread safe, your instance information does not have to be thread safe.

Regarding things like database connections and whatnot, you want to make sure that your test methods are as isolated as humanly possible—or at least that (if they are not) they take great care to lock on shared resources.

Advertisement