Tuesday, August 22

Step-by-step guide to TestNG

TestNG is a flexible testing framework developed by Cédric Beust and Alexandru Popescu. It is a much more powerful alternative to JUnit that fixes and improves many of its shortcomings.

In this introductory guide I will show how to setup and integrate TestNG with IDEA, hoping to highlight specific gems of TestNG in the process.

TestNG at a Glance

Here's a short list of my favorite features:

  • Ability to run existing JUnit test without any problem, providing a smooth transition path from JUnit to TestNG;
  • Ability to specify individual method thread pools;
  • Ability to specify certain test methods as dependent on the successful completion of others;
  • Separation of Java code from the way tests are run -run all tests, some tests, or only a few test methods of a given class. No need to recompile classes to run a different set of tests or suites;
  • Seamless integration with IDEA and Eclipse;

IDEA Setup

  • Download the latest TestNG library from http://testng.org/doc/download.html. My examples here are based on version 5.0.
  • Unzip the contents of the archive onto a directory of your choice.
  • Launch IDEA and create a new project of Java Module Type. I created one labeled testng.
  • Go to File->Settings, or press CTRL+Alt+S, to launch the IDE Settings panel.
  • From within the IDE Settings panel select the Plugins tab on the left hand side panel.

Figure 1 - Download and install the TestNG plug-in for IDEA [click on image to enlarge]

  • Download and install the TestNG-J plug-in from the right hand side panel. This will integrate the library with IDEA. You may need to restart the IDE for the changes to take effect.
  • Right-click on the newly created project and choose Module Settings.

Figure 2 - Module Settings of project [click on image to enlarge]

  • On the Module Settings panel, pick the Libraries (Classpath) tab, and then choose Add Jar/Directory to add the TestNG's jar to your project's classpath. I'm using JDK5.0, so I chose the testng-5.0-jdk15.jar archive.

Figure 3 - Add TestNG jar to project's classpath [click on image to enlarge]

This completes the integration of the testing library with IDEA, and we are now ready to write and test some code.


Exploring the Next Generation of Testing

Simply put, a TestNG unit test class is a POJO with annotated methods. There is no requirement to extend a specific class or implement a specific interface, just tag methods with Java annotations. A very simple Test class looks like this:

package testng.basic;

import org.testng.annotations.Test;

public class FirstTest {
public FirstTest() {
}
@Test
public void isOneEqualsTwo(){
assert(1 == 2);
}
}

I wrote a simple Java class and will use it to illustrate some key features of TestNG, namely the ability to specify individual method thread pools, the ability to specify certain test methods as dependent on the successful completion of others, and the seamless integration with IDEA.

Create a new class named ThreadUnsafe on IDEA and copy and paste the code below:


import org.testng.annotations.Test;
public class ThreadUnsafe {
private static int[] accounts = new int[] {0, 0};
private static int MAX = 1000;

@Test(threadPoolSize = 1000, invocationCount = 1000, timeOut = 2000)
public void swap(){
int amount = (int) (MAX * Math.random());
accounts[1] += amount;
accounts[0] -= amount;
System.out.println("Account[0]: " + accounts[0]);
System.out.println("Account[1]: " + accounts[1]);
int balance = checkBalance();
assert(balance == 0);
}
public int checkBalance(){
int sum = 0;
for (int i = 0; i < accounts.length; i++){
sum += accounts[i];
}
System.out.println("Balance : " + sum);
return sum;
}
public static void main(String[] args){
ThreadUnsafe tu = new ThreadUnsafe();
tu.swap();
}
}


ThreadUnsafe
is a very simple class that swaps random values from one account to another, ensuring the balance remains zero. That is, when we add an amount to one of the accounts, we subtract the same amount from the other.

Compile and run the code. It should exit printing Balance : 0.

Specifying thread pools

The only different and interesting thing on the code above is the annotation @Test on method swap():
@Test(threadPoolSize = 1000, invocationCount = 1000, timeOut = 2000)

This annotation is saying "give me a pool of 1000 threads, invoke this 1000 times, and exit if an invocation takes longer than 2000 milliseconds to return".
If a method takes longer than the specified timeout, TestNG will interrupt the method and mark it as unsuccessful.

To debug this sample project select the TestNG tab from the Debug panel. From there we can specify the desired granularity of the test, which can range from the swap() method to the whole package. I chose to test the ThreadUnsafe class itself.

Figure 4 - Debug granularity of TestNG [click on image to enlarge]

The output of running the project in debug mode can be seen below (values will vary):

Figure 5 - TestNG output [click on image to enlarge]

On the output console shown above we can see that after a few successful runs the value of Balance is 0 as expected. However, later on some weird values start showing up. Thus, multiple threads interfered with each other and the test shows the code to be thread unsafe.

This brute force thread safety testing can be useful to confirm a bug report due to improper synchronization, though it can be tricky to come up with a representative number of threads and repetitions.

Specifying method dependencies

The ability to specify certain test methods as dependent on the successful completion of others is a very useful feature. Let's move the initialization into a method of its own. Note changes on the accounts variable declaration and init() method.


import org.testng.annotations.Test;

public class ThreadUnsafe {
private static int[] accounts;
private static int MAX = 1000;
@Test
public void init() {
accounts = new int[]{0, 0};
}
@Test(threadPoolSize = 1000, invocationCount = 1000, timeOut = 2000)
public void swap(){
int amount = (int) (MAX * Math.random());
accounts[1] += amount;
accounts[0] -= amount;
System.out.println("Account[0]: " + accounts[0]);
System.out.println("Account[1]: " + accounts[1]);
int balance = checkBalance();
assert(balance == 0);
}
public int checkBalance(){
int sum = 0;
for (int i = 0; i < accounts.length; i++){
sum += accounts[i];
}
System.out.println("Balance : " + sum);
return sum;
}
public static void main(String[] args){
ThreadUnsafe tu = new ThreadUnsafe();
tu.init();
tu.swap();
}
}

The difference from the previous code is that initialization is now done on the init() method, which is itself a @Test annotated method that will be included in the testing report.

Now running the code in Debug TestNG mode results in an error because the array accounts used by the swap() method has not been initialized.

Figure 6 - NullPointerException caused by non-initialized dependency [click on image to enlarge]


What we want here is the ability to tell that a certain test method, swap(), depends on the successful completion of a previous test method, init().
We want to guarantee that certain methods or groups of methods are always invoked before others.

TestNG let's us do that with the dependsOnMethods annotation.
To specify swap() as being dependent on the successful execution of init(), we use the dependsOnMethods annotation as shown below:


@Test(dependsOnMethods = {"init"}, threadPoolSize = 1000, invocationCount = 1000, timeOut = 2000)
public void swap() {
...
}

Running the code in debug TestNG mode now results in a successful execution:

Figure 7 - Method dependency with TestNG [click on image to enlarge]

For unreliable systems TestNG introduces the notion of partial failure:
@Test(timeOut = 10000, invocationCount = 1000, successPercentage = 98)
public void waitForAnswer() {
while (!success){
Thread.sleep(1000);
}
}

The example above instructs TestNG to invoke the method a thousand times, but to consider the overall test passed even if only 98% of them succeed.

All the above are simple yet very powerful examples that are either very hard or impossible to do with JUnit.

Resources

Thursday, August 17

Business need: Digg for Consumer Electronics

I'm on the prowl for new gadgets (smart-phone, wide-screen TV, games console), and I think a market like Digg for consumer electronics would help prune the search space.
With the number and combination of brands, specs, makers, designs, features, personal preferences and whatnot involved, such an application would tap into the wisdom of crowds and transform many diverse opinions into a single collective judgement.
This is the central thesis of the Wisdom of Crowds:

"With most things, the average is mediocrity. With decision making, it's often
excellence. We've been programmed to be collectively smart
".


I hope this is what Jay Adelson [Digg's current CEO] means when he says:
"The best I can tell you is Digg as a concept can be applied to other content aside from news. Just wait and see what we’re going to apply it to."


In fact, Amazon is also in a good place to experiment with crowdsorting their results pages (and when you're at it Amazon, please let us also do column sorting so that we don't need to navigate through pages of results that we don't care about to find what we want).


[Update]:
Product Clash's clash and compare feature almost nails it:
"Now you can clash and compare your best consumer products for low price offers.
Product Clash features cheap cameras, cheap cell phones, cheap computers, cool
gadgets, cheap home entertainment, cheap peripherals and portable media
comparison clashes! Clashing is fun for movies, music, games and DVDs."



Useful Links:
Venture Voice Show #37 - Jay Adelson of Digg
The Wisdom of Crowds
Independent Individuals and Wise Crowds
The Observer: Here's Hoping This Group-Think Effort Is Full Of Wisdom
http://en.wikipedia.org/wiki/Crowdsourcing

The road ahead for OpenOffice

I agree 100% with these arguments, being an active but often frustrated OO user for over 5 years.
Decoupling the document from the vendor suite (with the Open Document format) was a tremendous achievement, but the road ahead for OO does not look so promising:

  • Instead of redefining and redesigning an Office suite tailored to user needs it is developing into a pale, MS Office-wannabe sibling. It confuses the (real) user need of a better Office suite with that of an open alternative to MS Office.
  • It has most of the same flaws of (older) MS Office with less of its (newer) functionality. The trade off is to its disadvantage, since it's now playing to catch-up with competition that is evolving.
  • It is still very buggy in critical areas such as the spell-checker and word count, which were also introduced late as proper features.
  • It is not modular, forcing one huge download and installation of the whole suite. I'd like to, in true open market fashion, be able to choose and use the best components from each suite.
  • It has a very slow and clumsy release cycle. The early 2.0 beta was an embarrassment.
  • It doesn't clean-up properly after an uninstall. There are no excuses for this.
  • It lacks a much needed auto-update feature. For the average computer user, upgrading a release is a pain.
  • Last but not least, it lacks truthful, constructive, and objective criticism -criticism is often regarded as a nod towards Microsoft. Most users and adopters of OO tend to be people that are somewhat partial in their reviews. The views within some parts of the open-source community mix blind, anti-corporate bashing with the open source ideal.
I tend to regard the less-than-100% compatibility issue as an overall successful struggle to reverse-engineer a proprietary document format.

As it stands, OO is an alternative to MS Office, which is a good thing to have. In fact, it is the alternative to MS Office.
But mostly for monetary or ideologic reasons, not based on the quality of the offering.