Step-by-step guide to TestNG

TestNG is a flexible testing framework developed by Cédric Beust and Alexandru Popescu. It is a much more powerful alternative to JUnit that fixes and improves many of its shortcomings.

In this introductory guide I will show how to setup and integrate TestNG with IDEA, hoping to highlight specific gems of TestNG in the process.

TestNG at a Glance

Here's a short list of my favorite features:

• Ability to run existing JUnit test without any problem, providing a smooth transition path from JUnit to TestNG;
• Ability to specify individual method thread pools;
• Ability to specify certain test methods as dependent on the successful completion of others;
• Separation of Java code from the way tests are run -run all tests, some tests, or only a few test methods of a given class. No need to recompile classes to run a different set of tests or suites;
• Seamless integration with IDEA and Eclipse;
  

IDEA Setup

• Download the latest TestNG library from http://testng.org/doc/download.html. My examples here are based on version 5.0.
• Unzip the contents of the archive onto a directory of your choice.
• Launch IDEA and create a new project of Java Module Type. I created one labeled testng.
• Go to File->Settings, or press CTRL+Alt+S, to launch the IDE Settings panel.
• From within the IDE Settings panel select the Plugins tab on the left hand side panel.

Figure 1 - Download and install the TestNG plug-in for IDEA [click on image to enlarge]

• Download and install the TestNG-J plug-in from the right hand side panel. This will integrate the library with IDEA. You may need to restart the IDE for the changes to take effect.
• Right-click on the newly created project and choose Module Settings.

Figure 2 - Module Settings of project [click on image to enlarge]

• On the Module Settings panel, pick the Libraries (Classpath) tab, and then choose Add Jar/Directory to add the TestNG's jar to your project's classpath. I'm using JDK5.0, so I chose the testng-5.0-jdk15.jar archive.

Figure 3 - Add TestNG jar to project's classpath [click on image to enlarge]

This completes the integration of the testing library with IDEA, and we are now ready to write and test some code.

Exploring the Next Generation of Testing

Simply put, a TestNG unit test class is a POJO with annotated methods. There is no requirement to extend a specific class or implement a specific interface, just tag methods with Java annotations. A very simple Test class looks like this:

package testng.basic;

import org.testng.annotations.Test;

public class FirstTest {
  public FirstTest() {
  }
  @Test
  public void isOneEqualsTwo(){
          assert(1 == 2);
  }
}

I wrote a simple Java class and will use it to illustrate some key features of TestNG, namely the ability to specify individual method thread pools, the ability to specify certain test methods as dependent on the successful completion of others, and the seamless integration with IDEA.

Create a new class named ThreadUnsafe on IDEA and copy and paste the code below:

import org.testng.annotations.Test;
public class ThreadUnsafe {
      private static int[] accounts = new int[] {0, 0};
      private static int MAX = 1000;

      @Test(threadPoolSize = 1000, invocationCount = 1000, timeOut = 2000)
      public void swap(){
              int amount = (int) (MAX * Math.random());
              accounts[1] += amount;
              accounts[0] -= amount;
              System.out.println("Account[0]: " + accounts[0]);
              System.out.println("Account[1]: " + accounts[1]);
              int balance = checkBalance();
              assert(balance == 0);
      }
      public int checkBalance(){
              int sum = 0;
              for (int i = 0; i < accounts.length; i++){
                     sum += accounts[i];
              }
              System.out.println("Balance : " + sum);
              return sum;
      }
      public static void main(String[] args){
              ThreadUnsafe tu = new ThreadUnsafe();
              tu.swap();
      }
}

ThreadUnsafe is a very simple class that swaps random values from one account to another, ensuring the balance remains zero. That is, when we add an amount to one of the accounts, we subtract the same amount from the other.

Compile and run the code. It should exit printing Balance : 0.

Specifying thread pools

The only different and interesting thing on the code above is the annotation @Test on method swap():

@Test(threadPoolSize = 1000, invocationCount = 1000, timeOut = 2000)

This annotation is saying "give me a pool of 1000 threads, invoke this 1000 times, and exit if an invocation takes longer than 2000 milliseconds to return".
If a method takes longer than the specified timeout, TestNG will interrupt the method and mark it as unsuccessful.

To debug this sample project select the TestNG tab from the Debug panel. From there we can specify the desired granularity of the test, which can range from the swap() method to the whole package. I chose to test the ThreadUnsafe class itself.

Figure 4 - Debug granularity of TestNG [click on image to enlarge]

The output of running the project in debug mode can be seen below (values will vary):

Figure 5 - TestNG output [click on image to enlarge]

On the output console shown above we can see that after a few successful runs the value of Balance is 0 as expected. However, later on some weird values start showing up. Thus, multiple threads interfered with each other and the test shows the code to be thread unsafe.

This brute force thread safety testing can be useful to confirm a bug report due to improper synchronization, though it can be tricky to come up with a representative number of threads and repetitions.

Specifying method dependencies

The ability to specify certain test methods as dependent on the successful completion of others is a very useful feature. Let's move the initialization into a method of its own. Note changes on the accounts variable declaration and init() method.

import org.testng.annotations.Test;

public class ThreadUnsafe {
     private static int[] accounts;
     private static int MAX = 1000;
     @Test
     public void init() {
             accounts = new int[]{0, 0};
     }
     @Test(threadPoolSize = 1000, invocationCount = 1000, timeOut = 2000)
     public void swap(){
             int amount = (int) (MAX * Math.random());
             accounts[1] += amount;
             accounts[0] -= amount;
             System.out.println("Account[0]: " + accounts[0]);
             System.out.println("Account[1]: " + accounts[1]);
             int balance = checkBalance();
             assert(balance == 0);
     }
     public int checkBalance(){
             int sum = 0;
             for (int i = 0; i < accounts.length; i++){
                    sum += accounts[i];
             }
             System.out.println("Balance : " + sum);
             return sum;
     }
     public static void main(String[] args){
             ThreadUnsafe tu = new ThreadUnsafe();
             tu.init();
             tu.swap();
     }
}

The difference from the previous code is that initialization is now done on the init() method, which is itself a @Test annotated method that will be included in the testing report.

Now running the code in Debug TestNG mode results in an error because the array accounts used by the swap() method has not been initialized.

Figure 6 - NullPointerException caused by non-initialized dependency [click on image to enlarge]

What we want here is the ability to tell that a certain test method, swap(), depends on the successful completion of a previous test method, init().
We want to guarantee that certain methods or groups of methods are always invoked before others.

TestNG let's us do that with the dependsOnMethods annotation.
To specify swap() as being dependent on the successful execution of init(), we use the dependsOnMethods annotation as shown below:

@Test(dependsOnMethods = {"init"}, threadPoolSize = 1000, invocationCount = 1000, timeOut = 2000)
public void swap() {
     ...
}

Running the code in debug TestNG mode now results in a successful execution:

Figure 7 - Method dependency with TestNG [click on image to enlarge]

For unreliable systems TestNG introduces the notion of partial failure:

@Test(timeOut = 10000, invocationCount = 1000, successPercentage = 98)
public void waitForAnswer() {
         while (!success){
                 Thread.sleep(1000);
}
}

The example above instructs TestNG to invoke the method a thousand times, but to consider the overall test passed even if only 98% of them succeed.

All the above are simple yet very powerful examples that are either very hard or impossible to do with JUnit.

Resources

• Website: http://testng.org
• IBM DeveloperWorks article: http://www-128.ibm.com/developerworks/java/library/j-testng/
• JavaWorld article: http://www.javaworld.com/javaworld/jw-04-2005/jw-0404-testng.html
  
• Key features: http://www.beust.com/weblog/archives/000176.html
• Method dependency:http://www.beust.com/weblog/archives/000170.html
• Annotation inheritance: http://beust.com/weblog/archives/000170.html
• Statistical Testing: http://beust.com/weblog/archives/000369.html