TestNG Listeners

There are several interfaces that allow you to modify TestNG’s behavior. These interfaces are broadly called "TestNG Listeners". Here are a few listeners:

Listener list

Listener name	User Documentation	API Documentation
IAlterSuiteListener	docs	{javadocs-base-url}/org/testng/IAlterSuiteListener.html[javadocs]
IAnnotationTransformer	docs	{javadocs-base-url}/org/testng/IAnnotationTransformer.html[javadocs]
IConfigurationListener	docs	{javadocs-base-url}/org/testng/IConfigurationListener.html[javadocs]
IDataProviderListener	docs	{javadocs-base-url}/org/testng/IDataProviderListener.html[javadocs]
IExecutionListener	docs	{javadocs-base-url}/org/testng/IExecutionListener.html[javadocs]
IExecutionVisualiser	docs	{javadocs-base-url}/org/testng/IExecutionVisualiser.html[javadocs]
IHookable	docs	{javadocs-base-url}/org/testng/IHookable.html[javadocs]
IConfigurable	docs	{javadocs-base-url}/org/testng/IConfigurable.html[javadocs]
IInvokedMethodListener	docs	{javadocs-base-url}/org/testng/IInvokedMethodListener.html[javadocs]
IClassListener	docs	{javadocs-base-url}/org/testng/IClassListener.html[javadocs]
IMethodInterceptor	docs	{javadocs-base-url}/org/testng/IMethodInterceptor.html[javadocs]
IDataProviderInterceptor	docs.	{javadocs-base-url}/org/testng/IDataProviderInterceptor.html[javadocs]
IReporter	docs	{javadocs-base-url}/org/testng/IReporter.html[javadocs]
ISuiteListener	docs	{javadocs-base-url}/org/testng/ISuiteListener.html[javadocs]
ITestListener	docs	{javadocs-base-url}/org/testng/ITestListener.html[javadocs]

Listener name

User Documentation

API Documentation

IAlterSuiteListener

docs

{javadocs-base-url}/org/testng/IAlterSuiteListener.html[javadocs]

IAnnotationTransformer

docs

{javadocs-base-url}/org/testng/IAnnotationTransformer.html[javadocs]

IConfigurationListener

docs

{javadocs-base-url}/org/testng/IConfigurationListener.html[javadocs]

IDataProviderListener

docs

{javadocs-base-url}/org/testng/IDataProviderListener.html[javadocs]

IExecutionListener

docs

{javadocs-base-url}/org/testng/IExecutionListener.html[javadocs]

IExecutionVisualiser

docs

{javadocs-base-url}/org/testng/IExecutionVisualiser.html[javadocs]

IHookable

docs

{javadocs-base-url}/org/testng/IHookable.html[javadocs]

IConfigurable

docs

{javadocs-base-url}/org/testng/IConfigurable.html[javadocs]

IInvokedMethodListener

docs

{javadocs-base-url}/org/testng/IInvokedMethodListener.html[javadocs]

IClassListener

docs

{javadocs-base-url}/org/testng/IClassListener.html[javadocs]

IMethodInterceptor

docs

{javadocs-base-url}/org/testng/IMethodInterceptor.html[javadocs]

IDataProviderInterceptor

docs.

{javadocs-base-url}/org/testng/IDataProviderInterceptor.html[javadocs]

IReporter

docs

{javadocs-base-url}/org/testng/IReporter.html[javadocs]

ISuiteListener

docs

{javadocs-base-url}/org/testng/ISuiteListener.html[javadocs]

ITestListener

docs

{javadocs-base-url}/org/testng/ITestListener.html[javadocs]

Specifying listeners with testng.xml or in Java

Here is how you can define listeners in your testng.xml file:

<suite>
  <listeners>
    <listener class-name="com.example.MyListener" />
    <listener class-name="com.example.MyMethodInterceptor" />
  </listeners>
<!-- rest of the contents omitted for brevity -->
</suite>

Or if you prefer to define these listeners in Java:

@Listeners({ com.example.MyListener.class, com.example.MyMethodInterceptor.class })
public class MyTest {
  // ...
}

The @Listeners annotation can contain any class that extends org.testng.ITestNGListener except IAnnotationTransformer. The reason is that these listeners need to be known very early in the process so that TestNG can use them to rewrite your annotations, therefore you need to specify these listeners in your testng.xml file.

Note that the @Listeners annotation will apply to your entire suite file, just as if you had specified it in a testng.xml file. If you want to restrict its scope (for example, only running on the current class), the code in your listener could first check the test method that’s about to run and decide what to do then. Here’s how it can be done.

First define a new custom annotation that can be used to specify this restriction:

@Retention(RetentionPolicy.RUNTIME)
@Target ({ElementType.TYPE})
public @interface DisableListener {}

Add an edit check as below within your regular listeners:

public void beforeInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
  ConstructorOrMethod consOrMethod =iInvokedMethod.getTestMethod().getConstructorOrMethod();
  DisableListener disable = consOrMethod.getMethod().getDeclaringClass().getAnnotation(DisableListener.class);
  if (disable != null) {
    return;
  }
  // else resume your normal operations
}

Annotate test classes wherein the listener is not to be invoked:

@DisableListener
@Listeners({ com.example.MyListener.class, com.example.MyMethodInterceptor.class })
public class MyTest {
  // ...
}

Specifying listeners with ServiceLoader

Finally, the JDK offers a very elegant mechanism to specify implementations of interfaces on the class path via the ServiceLoader class.

With ServiceLoader, all you need to do is create a jar file that contains your listener(s) and a few configuration files, put that jar file on the classpath when you run TestNG and TestNG will automatically find them.

Here is a concrete example of how it works.

Let’s start by creating a listener (any TestNG listener should work):

package test.tmp;

public class TmpSuiteListener implements ISuiteListener {
  @Override
  public void onFinish(ISuite suite) {
    System.out.println("Finishing");
  }

  @Override
  public void onStart(ISuite suite) {
    System.out.println("Starting");
  }
}

Compile this file, then create a file at the location META-INF/services/org.testng.ITestNGListener, which will name the implementation(s) you want for this interface.

You should end up with the following directory structure, with only two files:

$ tree
|____META-INF
| |____services
| | |____org.testng.ITestNGListener
|____test
| |____tmp
| | |____TmpSuiteListener.class

$ cat META-INF/services/org.testng.ITestNGListener
test.tmp.TmpSuiteListener

Create a jar of this directory:

$ jar cvf ../sl.jar .
added manifest
ignoring entry META-INF/
adding: META-INF/services/(in = 0) (out= 0)(stored 0%)
adding: META-INF/services/org.testng.ITestNGListener(in = 26) (out= 28)(deflated -7%)
adding: test/(in = 0) (out= 0)(stored 0%)
adding: test/tmp/(in = 0) (out= 0)(stored 0%)
adding: test/tmp/TmpSuiteListener.class(in = 849) (out= 470)(deflated 44%)

Next, put this jar file on your classpath when you invoke TestNG:

$ java -classpath sl.jar:testng.jar org.testng.TestNG testng-single.yaml
Starting
f2 11 2
PASSED: f2("2")
Finishing

This mechanism allows you to apply the same set of listeners to an entire organization just by adding a jar file to the classpath, instead of asking every single developer to remember to specify these listeners in their testng.xml file.

Ordering listeners in TestNG

TestNG now allows you to control the order in which listeners are executed.

This is particularly useful when you have multiple listeners that are altering the test result states and so you would like to ensure that they do so in some deterministic order.

This feature is ONLY available from TestNG version 7.10.0

Following is how we can get this done.

To start with, you would need to build an implementation of the interface org.testng.ListenerComparator
Now this implementation needs to be plugged into TestNG via the configuration parameter -listenercomparator

TestNG orders only user’s listeners and which are not part of the exclusion list that can be specified via the JVM argument -Dtestng.preferential.listeners.package (Multiple fully qualified class names can be specified as comma separated values). If nothing is specified, TestNG by default excludes all the IntelliJ IDEA listeners under the package com.intellij.rt.*.

Let’s see an example.

Let’s create a custom annotation that will help us define the order in which our listeners should be invoked.

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({ TYPE})

public @interface RunOrder {
  int value();
}

Now let’s create an implementation of org.testng.ListenerComparator which honours this custom annotation that we just now created.

import org.testng.ITestNGListener;
import org.testng.ListenerComparator;
import java.util.Optional;

public class AnnotationBackedListenerComparator implements ListenerComparator {

  @Override
  public int compare(ITestNGListener l1, ITestNGListener l2) {
    int first = getRunOrder(l1);
    int second = getRunOrder(l2);
    return Integer.compare(first, second);
  }

  private static int getRunOrder(ITestNGListener listener) {
    RunOrder runOrder = listener.getClass().getAnnotation(RunOrder.class);
    return Optional.ofNullable(runOrder)
        .map(RunOrder::value)
        .orElse(Integer.MAX_VALUE); //If annotation was not found then return a max value so that
    //the listener can be plugged in to the end.
  }
}

Lets say we have the below listeners as samples.

import org.testng.IExecutionListener;

public class ExecutionListenerHolder {

  public static final String PREFIX = ExecutionListenerHolder.class.getName() + "$";

  public abstract static class KungFuWarrior implements IExecutionListener {
    @Override
    public void onExecutionStart() {
      System.err.println(getClass().getSimpleName() + ".onExecutionStart");
    }

    @Override
    public void onExecutionFinish() {
      System.err.println(getClass().getSimpleName() + ".onExecutionFinish");
    }
  }

  @RunOrder(1)
  public static class MasterOogway extends KungFuWarrior { }

  @RunOrder(2)
  public static class MasterShifu extends KungFuWarrior { }

  public static class DragonWarrior extends KungFuWarrior { }
}

A sample code snippet that uses the TestNG APIs to run the test could look like below:

TestNG testng = create(NormalSampleTestCase.class);
testng.addListener(new ExecutionListenerHolder.DragonWarrior());
testng.addListener(new ExecutionListenerHolder.MasterShifu());
testng.addListener(new ExecutionListenerHolder.MasterOogway());
testng.setListenerComparator(new AnnotationBackedListenerComparator());
testng.run();

Here’s a variant of the same above sample, which invokes the main() method:

String prefix = ExecutionListenerHolder.PREFIX;
String[] args = new String[] {
    "-listener",
    prefix + "DragonWarrior,"+
    prefix + "MasterShifu,"+
    prefix + "MasterOogway",
    "-testclass",
    NormalSampleTestCase.class.getName(),
    "-listenercomparator",
    AnnotationBackedListenerComparator.class.getName()
};
TestNG.main(args);

Here’s how the trimmed version of execution output would look like:

MasterOogway.onExecutionStart
MasterShifu.onExecutionStart
DragonWarrior.onExecutionStart

===============================================
Command line suite
Total tests run: 2, Passes: 2, Failures: 0, Skips: 0
===============================================

DragonWarrior.onExecutionFinish
MasterShifu.onExecutionFinish
MasterOogway.onExecutionFinish

As seen from the above output, we wanted to have the listener MasterOogway be invoked first, followed by the listener MasterShifu and finally DragonWarrior (because this class does not have any @RunOrder annotation and hence it should be ONLY added at the end)

Also it should be noted that the teardown methods get executed in a symmetrical order. So if onExecutionStart() of the listener DragonWarrior got executed as the last listener, then its corresponding onExecutionFinish() would be called first.