Monday, April 16, 2018

Data driven tests in JUnit 5 with exception handling

My previous post on data driven tests with TestNG (Debug TestNG data driven tests when you only want to debug one row of data) showed how handy parameterized tests are and how to get around some limitations of TestNG in Eclipse.

JUnit is in my opinion a better test engine, and JUnit 5 has some great new features and offers a far superior set of tooling and mechanisms for running data driven tests, some of which I shall show here.

Data driven testing in JUnit

In this example, I am testing the ability to use regular expressions in Java: specifically the ability to find the first capturing group.

public String searchString(final String regex, final String stringToMatchAgainst) {
   final Pattern pattern = Pattern.compile(regex);
   final Matcher matcher = pattern.matcher(stringToMatchAgainst);
   if (matcher.matches()) {
      return matcher.group(1);
   }
   return null;
}

A bit about this method.

  1. This method accepts two parameters:
    1. The regular expression, named regex.
    2. The string we will apply the regular expression to, in oder to find a match. It is called stringToMatchAgainst.
  2. The method returns a string, specifically the first capturing group.
    • Quick example of regex groups. Groups are bracketed portions of a regular expression that can be referred to later as backreferences. If you were to apply the regular expression "([a-z]+) ([0-9]+)" against the string "letters 334", the first group would be "letters" and the second group would be "334".

This is the test data I am using.

Test label Regular expression String to apply regex to Expected first group
Any string matches any string. (.*) xyz xyz
Pick year digits out of a date. [0-9]{2}/[0-9]{2}/([0-9]{4}) 31/12/2032 2032
No digits found. [0-9]+ not a digit null
First two words. (?<firstTwoWords>\\w+ \\w+) .* abc xyz one two abc xyz
Bad regex. ([0-9+) 123 PatternSyntaxException: (?s)Unclosed character class.*

(This test case will throw an exception because the regular expression is invalid.)

As described earlier, the regular expressions here use capturing groups. In the fourth data set, I have also used named capturing groups - a cool regex feature added to Java in JDK 8.

Data driven testing (a.k.a. parameterized tests) is about being able to call a unit test method multiple times, giving it the actual data used to run the tests. This is as opposed to unit tests where the data is hard-coded within the test. JUnit 5's parameterized tests gives you a lot of flexibility around how to generate your test data.

  • @ValueSource - an array initialised within the annotation.
  • @EnumSource - an enum
  • @MethodSource - a factory method
  • >@CsvFileSource - a CSV file

The @MethodSource is arguably the most flexible because you can do so many things within a method to generate the returned data, including generating the data from an array, an enum, a CSV file, even doing a database lookup or calling some other service to get the data. My example will use @MethodSource and will generate a stream of JUnit 5 arguments.

private static Stream<Arguments> dataForTestSearchString() {
   return Stream.of(//
         Arguments.of("Any string matches any string.", "(.*)", "xyz", "xyz") //
         , Arguments.of("Pick year digits out of a date.", "[0-9]{2}/[0-9]{2}/([0-9]{4})", "31/12/2032", "2032") //
         , Arguments.of("No digits found.", "[0-9]+", "not a digit", null) //
         , Arguments.of("First two words.", "(?<firstTwoWords>\\w+ \\w+) .*", "abc xyz one two", "abc xyz") //
         , Arguments.of("Bad regex.", "([0-9+)", "123", "PatternSyntaxException: (?s)Unclosed character class.*") //
   );
}

Notes about this method:

  • It has no annotations, but it returns a stream of JUnit specific objects - org.junit.jupiter.params.provider.Arguments
  • Use the Java single line comment // by itself to ensure a line of code gets broken off when you use automatic formatting.
  • When creating a Java list (or array, or any collection really), putting the comma at the start of each line (after the first) makes it easy to copy and paste a line to form the next list element without having to worry about removing the comma from the end of the last element.

Here is the @Test method that consumes this test data.

@ParameterizedTest(name = "#{index} - [{0}]")
@MethodSource("dataForTestSearchString")
public void testFilesFromDirectoriesAndPattern(final String label, final String regex,
      final String stringToMatchAgainst, final String expected) {
   final String actual = searchString(regex, stringToMatchAgainst);
   assertEquals(expected, actual);
}

Notes about this method.

  • The @ParameterizedTest annotation lets you specify the "name" of each test data, which affects how each call to your test method is labelled in Eclipse. I have used the pattern #{index} - [{0}], which means each test case will be labelled with the index of that test data and whatever the first argument to the test method is (so the first argument becomes the test case label). See the screenshot below to see how wonderful this is.
  • The @MethodSource annotation allows you to name the method (or give an array naming multiple methods) that will generate data to be consumed by this test.
    • In the background, JUnit will use reflection to call the method, which has a side effect in Eclipse: if nothing else calls that method, Eclipse will give you a warning that your data generating method is unused (because it cannot work out from this annotation that you are really using it).

This is what you see in Eclipse when running this class as a JUnit test.

One of the most important advantages of JUnit in Eclipse over TestNG is that you can run individual test cases by right-clicking on the one you want to re-run and select "Run" or "Debug" (TestNG would re-run the whole lot).

Here is version 1 the full test case, incorporating the method being tested.

import static org.junit.jupiter.api.Assertions.assertEquals;

import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Stream;

import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;

public class JunitDataDrivenTest {

   /**
    * @return test data. Each set of arguments should consist of
    *         <ol>
    *         <li>test label</li>
    *         <li>regex</li>
    *         <li>string we apply the regex to</li>
    *         <li>first group matched by the regex</li>
    *         </ol>
    */
   @SuppressWarnings({ "unused" }) // Eclipse thinks this method is not used.
   private static Stream<Arguments> dataForTestSearchString() {
      return Stream.of(//
            Arguments.of("Any string matches any string.", "(.*)", "xyz", "xyz") //
            , Arguments.of("Pick year digits out of a date.", "[0-9]{2}/[0-9]{2}/([0-9]{4})", "31/12/2032", "2032") //
            , Arguments.of("No digits found.", "[0-9]+", "not a digit", null) //
            , Arguments.of("First two words.", "(?<firstTwoWords>\\w+ \\w+) .*", "abc xyz one two", "abc xyz") //
            , Arguments.of("Bad regex.", "([0-9+)", "123", "PatternSyntaxException: (?s)Unclosed character class.*") //
      );
   }

   /**
    * @param label
    *           description of the current test. Data is used for reporting only.
    * @param regex
    *           regular expression.
    * @param stringToMatchAgainst
    *           string that we will apply the regex to
    * @param expected
    *           first group matched by the regex. Will be null if no match is expected.
    */
   @ParameterizedTest(name = "#{index} - [{0}]")
   @MethodSource("dataForTestSearchString")
   public void testFilesFromDirectoriesAndPattern(final String label, final String regex,
         final String stringToMatchAgainst, final String expected) {
      final String actual = searchString(regex, stringToMatchAgainst);
      assertEquals(expected, actual);
   }

   /**
    * A method being tested.
    *
    * @param regex
    *           regular expression.
    * @param stringToMatchAgainst
    *           string that we will apply the regex to
    * @return first group matched by the regex. Will be null if no match is expected.
    */
   public String searchString(final String regex, final String stringToMatchAgainst) {
      final Pattern pattern = Pattern.compile(regex);
      final Matcher matcher = pattern.matcher(stringToMatchAgainst);
      if (matcher.matches()) {
         return matcher.group(1);
      }
      return null;
   }

}

Testing for exceptions

Another new aspect of JUnit 5 is the set of assertions that use lambdas from JDK 8, particularly the assertThrows method which I will demonstrate next.

As mentioned above, the last test case throws an exception because of an invalid regular expression ("([0-9+)" doesn't close the square brackets: it should be "([0-9]+)"). A normal part of unit testing should be ensuring that code throws appropriate exceptions when something goes wrong. Here is the unit test adjusted so that if our parameterized data indicates that an exception is expected, we test for it. We test for the expected exception type and have a regular expression to test the exception's message.

@ParameterizedTest(name = "#{index} - [{0}]")
@MethodSource("dataForTestSearchString")
public void testFilesFromDirectoriesAndPattern(final String label, final String regex,
      final String stringToMatchAgainst, final String expected) {

   // Types of exceptions we test for.
   final String patternSyntaxException = "PatternSyntaxException: ";

   // PatternSyntaxException
   if (expected != null && expected.startsWith(patternSyntaxException)) {
      final PatternSyntaxException pse = assertThrows(PatternSyntaxException.class, () -> {
         searchString(regex, stringToMatchAgainst);
      });
      final Pattern exceptionPattern = Pattern.compile(expected.substring(patternSyntaxException.length()));
      final Matcher exceptionMatcher = exceptionPattern.matcher(pse.getMessage());
      assertTrue(exceptionMatcher.matches());
   } else {
      // Normal operation.
      final String actual = searchString(regex, stringToMatchAgainst);
      assertEquals(expected, actual);
   }

}

Notes about this method.

  • We use one of the method parameters, a string named expected to tell the method what result we should expect under normal (successful) operation.
  • I have extended the use of that string to also tell me what to expect in exceptional circumstances. In my code, I know what exceptions I am testing for and have decided that for test cases that should throw an exception, the expected string should start with the class name of the exception and the rest of the string will be a regular expression that should match the exception's message (getMessage()).
    • To do this I need some string processing: a test to see if the string starts with an exception class name and then a substring operation to pull out the regex.
    • I then need code to create the Pattern, the Matcher and then test if the regex matches the exception message.

The unit test now passes in all cases.

Here is the final version of the full test case, incorporating exception testing.

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;
import java.util.stream.Stream;

import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;

public class JunitDataDrivenTest {

   /**
    * @return test data. Each set of arguments should consist of
    *         <ol>
    *         <li>test label</li>
    *         <li>regex</li>
    *         <li>string we apply the regex to</li>
    *         <li>first group matched by the regex</li>
    *         </ol>
    */
   @SuppressWarnings({ "unused" }) // Eclipse thinks this method is not used.
   private static Stream<Arguments> dataForTestSearchString() {
      return Stream.of(//
            Arguments.of("Any string matches any string.", "(.*)", "xyz", "xyz") //
            , Arguments.of("Pick year digits out of a date.", "[0-9]{2}/[0-9]{2}/([0-9]{4})", "31/12/2032", "2032") //
            , Arguments.of("No digits found.", "[0-9]+", "not a digit", null) //
            , Arguments.of("First two words.", "(?<firstTwoWords>\\w+ \\w+) .*", "abc xyz one two", "abc xyz") //
            , Arguments.of("Bad regex.", "([0-9+)", "123", "PatternSyntaxException: (?s)Unclosed character class.*") //
      );
   }

   /**
    * @param label
    *           description of the current test. Data is used for reporting only.
    * @param regex
    *           regular expression.
    * @param stringToMatchAgainst
    *           string that we will apply the regex to
    * @param expected
    *           first group matched by the regex. Will be null if no match is expected.
    */
   @ParameterizedTest(name = "#{index} - [{0}]")
   @MethodSource("dataForTestSearchString")
   public void testFilesFromDirectoriesAndPattern(final String label, final String regex,
         final String stringToMatchAgainst, final String expected) {

      // Types of exceptions we test for.
      final String patternSyntaxException = "PatternSyntaxException: ";

      // PatternSyntaxException
      if (expected != null && expected.startsWith(patternSyntaxException)) {
         final PatternSyntaxException pse = assertThrows(PatternSyntaxException.class, () -> {
            searchString(regex, stringToMatchAgainst);
         });
         final Pattern exceptionPattern = Pattern.compile(expected.substring(patternSyntaxException.length()));
         final Matcher exceptionMatcher = exceptionPattern.matcher(pse.getMessage());
         assertTrue(exceptionMatcher.matches());
      } else {
         // Normal operation.
         final String actual = searchString(regex, stringToMatchAgainst);
         assertEquals(expected, actual);
      }

   }

   /**
    * A method being tested.
    *
    * @param regex
    *           regular expression.
    * @param stringToMatchAgainst
    *           string that we will apply the regex to
    * @return first group matched by the regex. Will be null if no match is expected.
    */
   public String searchString(final String regex, final String stringToMatchAgainst) {
      final Pattern pattern = Pattern.compile(regex);
      final Matcher matcher = pattern.matcher(stringToMatchAgainst);
      if (matcher.matches()) {
         return matcher.group(1);
      }
      return null;
   }

}

Versions

Software used in this post.

  • JDK 1.8 (1.8.0_40)
  • Eclipse is Spring Tool Suite 3.9.2.RELEASE (build on Eclipse Oxygen.2 (4.7.2))
  • Junit plugin is built in
  • JUnit dependencies in my pom.xml:
    
    <!-- JUnit Jupiter is the API we write tests against. -->
    <dependency>
       <groupId>org.junit.jupiter</groupId>
       <artifactId>junit-jupiter-engine</artifactId>
       <version>5.1.0</version>
       <scope>test</scope>
    </dependency>
    <!-- Parameterised tests is separate download because it is currently an experimental feature. -->
    <dependency>
       <groupId>org.junit.jupiter</groupId>
       <artifactId>junit-jupiter-params</artifactId>
       <version>5.1.0</version>
       <scope>test</scope>
    </dependency>
    <!-- The engined used by the IDE to run tests. JUnit is built in to Eclipse, but apparently my version of Eclipse can't run JUnit 5 tests. -->
    <dependency>
       <groupId>org.junit.platform</groupId>
       <artifactId>junit-platform-launcher</artifactId>
       <version>1.1.0</version>
       <scope>test</scope>
    </dependency>
    

Saturday, April 14, 2018

Debug TestNG data driven tests when you only want to debug one row of data

Update Sunday, 15th of April 2018, 05:34:43 PM: added versioning information at the bottom of the post.

TestNG has a relatively straightforward mechanism for data driven tests that lets you write a method (annotated with @DataProvider) that generates data that will be used to invoke a test method (annotated by @Test). Here is an example where one of the tests will fail with a NullPointerException.

import static org.junit.Assert.assertTrue;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class TestString{

   @DataProvider(name = "dataForTestStringLength")
   public Object[][] dataForTestStringLength() {
      return new Object[][] { //
            new Object[] { "string" }, //
            new Object[] { null }, //
            new Object[] { "longer string" }, //
            new Object[] { "an even longer string" }, //
      };
   }

   @Test(dataProvider = "dataForTestStringLength")
   public void testStringLength(final String stringToTest) {
      assertTrue(stringToTest.length() > 3);
   }

}

This is what the result looks like in Eclipse.

This is fine and expected, but the problem starts when you want to run just that one failing test again, or to debug it.

Even though you right click on just the one data set, TestNG will re-run all of them (all four in this case). This makes it hard to debug tests, because if you set a debug point, you have to keep skipping until you get the actual test case you want. This becomes much more frustrating when you have a large data set. It might be easy to deal with just just four data sets, but not so easy when you have twenty or thirty or more. Plus, more complex tests won't have just one column of data to test, but four, five or more - it will be hard to even identify which test case failed when each of the combinations look similar.

Here is a technique I use to make it much easier to debug specific test cases.

import static org.junit.Assert.assertTrue;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class TestString {

   @DataProvider(name = "dataForTestStringLength")
   public Object[][] dataForTestStringLength() {
      int count = 0;
      final Object[][] data = new Object[][] { //
            new Object[] { count++, "string" }, //
            new Object[] { count++, null }, //
            new Object[] { count++, "longer string" }, //
            new Object[] { count++, "an even longer string" }, //
      };
      // Use this to test specific cases only.
      final Object[][] debuggingData = new Object[][] { /* data[1] */ };
      if (debuggingData.length > 0) {
         return debuggingData;
      }
      return data;
   }

   @Test(dataProvider = "dataForTestStringLength")
   public void testStringLength(final int index, final String stringToTest) {
      assertTrue(stringToTest.length() > 3);
   }

}

I have made two important changes here.

  1. I have modified the @DataProvider method so that it doesn't automatically return every test case. Modify the debuggingData line to specify the index of the test case you want to test.
  2. How do we know which test case to debug? That's the reason for the extra parameter added to the data returned from the @DataProvider method: an int count that gets incremented for each test case and received in the @Test as final int index.

Now look what Eclipse shows when a test fails:

The index of the failed test is now clearly visible, so it's easy for me to adjust the @DataProvider method to only return the data set I actually want to debug.

final Object[][] debuggingData = new Object[][] { data[1] };

Versions used in this post.

  • Eclipse is Spring Tool Suite 3.9.2.RELEASE (build on Eclipse Oxygen.2 (4.7.2))
  • TestNG 6.14 plugin
  • TestNG dependency in my pom.xml:
    
      org.testng
      testng
      6.8
      test