Running Automated Tests in Parallel with TestNG

Running automated tests in parallel can be a time saver.  There are 2 aspects I will be discussing: the first one refers to using data providers, while the second one implies that you are grouping and running tests using XML suite files.

Using data providers

Whenever a test method receives parameter values through a data provider method, we annotate the provider method with the ‘@DataProvider’ annotation. By default, we don’t specify any attribute to this annotation. The test method is invoked once, with one of the parameter values at a time. 

If in turn, we would like for the values to be passed to the test method in parallel, so that the same method but with different values runs in parallel, we can update the ‘@DataProvider’ annotation. For that, we only need to add the ‘parallel’ attribute, and assign it the ‘true’ value, as a String:

@DataProvider(parallel = true)

By default, in TestNG, the maximum number of tests that will run in parallel that are using this data provider is 10.

Choosing a Global Software Development Partner to Accelerate Your Digital Strategy

To be successful and outpace the competition, you need a software development partner that excels in exactly the type of digital projects you are now faced with accelerating, and in the most cost effective and optimized way possible.

Get the Guide

Suite-level parallelism configuration

Whenever we have XML configuration suites files, whether or not any test in the suite is using a @DataProvider, we can apply parallelism to the tests in the suite. Looking at a simple XML configuration suite is made up of: one ‘suite’ tag; one or more ‘test’ tags inside the ‘suite’ tag; one or more ‘package’ or ‘class’ tags inside each ‘test’ tag. A simple example of a suite file using packages would be:

<suite name="Parallel Suite">

<test name="First test tag">

<packages>

<package name="firstPackage"/>

</packages>

</test>

<test name="Second test tag">

<packages>

<package name="secondPackage"/>

</packages>

</test>

</suite>

Now, in order to assign parallelism to the tests, we can either do that on a ‘suite’ tag level or a ‘test’ tag level:

• on a suite tag level, we can have parallelism so that:

We run each ‘test’ tag in parallel – that means that everything inside the test tag will run sequentially, but the test tags themselves will run in parallel. To achieve this, at the ‘suite’ tag level, we need to add the attribute ‘parallel’ with value ‘tests’:

<suite name="Parallel Suite" parallel="tests">

• • we run classes in parallel – this means the ‘test’ tags will run sequentially, but whenever the current ‘test’ tag runs, the classes inside the ‘test’ tag will run in parallel. To achieve this, at the ‘suite’ tag level, we need to add the attribute ‘parallel’ with value ‘classes’:

<suite name="Parallel Suite" parallel="classes">

• • we run methods in parallel – this means the ‘test’ tags will run sequentially, but whenever the current ‘test’ tag runs, the methods inside the ‘test’ tag will run in parallel. To achieve this, at the ‘suite’ tag level, we need to add the attribute ‘parallel’ with value ‘methods’:

<suite name="Parallel Suite" parallel="methods">

• on a test tag level: We can add, for each test tag, parallelism for either classes or methods. This is achieved using the same attribute and values as before. Either:

<test name=“First test tag” parallel=“classes”>

or

<test name=“First test tag” parallel=“methods”>

However, too much parallelism can lead to unexpected results and unpredictability. Especially if you have dependencies between tests, possibly data used by each test, where each test updates this data. So, use this mechanism carefully.

Conclusion

As we have seen in this article, our tests can easily be configured to run in parallel with TestNG. We can try different settings to find the best way to configure them so that they run efficiently and predictably.