Debug TestNG data driven tests when you only want to debug one row of data

Update Sunday, 15th of April 2018, 05:34:43 PM: added versioning information at the bottom of the post.

TestNG has a relatively straightforward mechanism for data driven tests that lets you write a method (annotated with @DataProvider) that generates data that will be used to invoke a test method (annotated by @Test). Here is an example where one of the tests will fail with a NullPointerException.

import static org.junit.Assert.assertTrue;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class TestString{

   @DataProvider(name = "dataForTestStringLength")
   public Object[][] dataForTestStringLength() {
      return new Object[][] { //
            new Object[] { "string" }, //
            new Object[] { null }, //
            new Object[] { "longer string" }, //
            new Object[] { "an even longer string" }, //
      };
   }

   @Test(dataProvider = "dataForTestStringLength")
   public void testStringLength(final String stringToTest) {
      assertTrue(stringToTest.length() > 3);
   }

}

This is what the result looks like in Eclipse.

This is fine and expected, but the problem starts when you want to run just that one failing test again, or to debug it.

Even though you right click on just the one data set, TestNG will re-run all of them (all four in this case). This makes it hard to debug tests, because if you set a debug point, you have to keep skipping until you get the actual test case you want. This becomes much more frustrating when you have a large data set. It might be easy to deal with just just four data sets, but not so easy when you have twenty or thirty or more. Plus, more complex tests won't have just one column of data to test, but four, five or more - it will be hard to even identify which test case failed when each of the combinations look similar.

Here is a technique I use to make it much easier to debug specific test cases.

import static org.junit.Assert.assertTrue;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class TestString {

   @DataProvider(name = "dataForTestStringLength")
   public Object[][] dataForTestStringLength() {
      int count = 0;
      final Object[][] data = new Object[][] { //
            new Object[] { count++, "string" }, //
            new Object[] { count++, null }, //
            new Object[] { count++, "longer string" }, //
            new Object[] { count++, "an even longer string" }, //
      };
      // Use this to test specific cases only.
      final Object[][] debuggingData = new Object[][] { /* data[1] */ };
      if (debuggingData.length > 0) {
         return debuggingData;
      }
      return data;
   }

   @Test(dataProvider = "dataForTestStringLength")
   public void testStringLength(final int index, final String stringToTest) {
      assertTrue(stringToTest.length() > 3);
   }

}

I have made two important changes here.

1. I have modified the @DataProvider method so that it doesn't automatically return every test case. Modify the debuggingData line to specify the index of the test case you want to test.
2. How do we know which test case to debug? That's the reason for the extra parameter added to the data returned from the @DataProvider method: an int count that gets incremented for each test case and received in the @Test as final int index.

Now look what Eclipse shows when a test fails:

The index of the failed test is now clearly visible, so it's easy for me to adjust the @DataProvider method to only return the data set I actually want to debug.

final Object[][] debuggingData = new Object[][] { data[1] };

Versions used in this post.

• Eclipse is Spring Tool Suite 3.9.2.RELEASE (build on Eclipse Oxygen.2 (4.7.2))
• TestNG 6.14 plugin
• TestNG dependency in my pom.xml:

  
    org.testng
    testng
    6.8
    test