Labels

Data driven tests in JUnit 5 with exception handling

My previous post on data driven tests with TestNG (Debug TestNG data driven tests when you only want to debug one row of data) showed how handy parameterized tests are and how to get around some limitations of TestNG in Eclipse.

JUnit is in my opinion a better test engine, and JUnit 5 has some great new features and offers a far superior set of tooling and mechanisms for running data driven tests, some of which I shall show here.

Data driven testing in JUnit

In this example, I am testing the ability to use regular expressions in Java: specifically the ability to find the first capturing group.

// Method under test that applies a regex to a string, returning the first group
public String searchString(final String regex, final String stringToMatchAgainst) {
   final Pattern pattern = Pattern.compile(regex);
   final Matcher matcher = pattern.matcher(stringToMatchAgainst);
   if (matcher.matches()) {
      return matcher.group(1);
   }
   return null;
}

A bit about this method.

  1. This method accepts two parameters:
    1. The regular expression, named regex.
    2. The string we will apply the regular expression to, in oder to find a match. It is called stringToMatchAgainst.
  2. The method returns a string, specifically the first capturing group.
    • Quick example of regex groups. Groups are bracketed portions of a regular expression that can be referred to later as backreferences. If you were to apply the regular expression "([a-z]+) ([0-9]+)" against the string "letters 334", the first group would be "letters" and the second group would be "334".

This is the test data I am using.

Test label Regular expression String to apply regex to Expected first group
Any string matches any string. (.*) xyz xyz
Pick year digits out of a date. [0-9]{2}/[0-9]{2}/([0-9]{4}) 31/12/2032 2032
No digits found. [0-9]+ not a digit null
First two words. (?<firstTwoWords>\\w+ \\w+) .* abc xyz one two abc xyz
Bad regex. ([0-9+) 123 PatternSyntaxException: (?s)Unclosed character class.*

(This test case will throw an exception because the regular expression is invalid.)

As described earlier, the regular expressions here use capturing groups. In the fourth data set, I have also used named capturing groups - a cool regex feature added to Java in JDK 8.

Data driven testing (a.k.a. parameterized tests) is about being able to call a unit test method multiple times, giving it the actual data used to run the tests. This is as opposed to unit tests where the data is hard-coded within the test. JUnit 5's parameterized tests gives you a lot of flexibility around how to generate your test data.

  • @ValueSource - an array initialised within the annotation.
  • @EnumSource - an enum
  • @MethodSource - a factory method
  • @CsvFileSource - a CSV file

The @MethodSource is arguably the most flexible because you can do so many things within a method to generate the returned data, including generating the data from an array, an enum, a CSV file, even doing a database lookup or calling some other service to get the data. My example will use @MethodSource and will generate a stream of JUnit 5 arguments.

// A method that returns a stream of JUnit Arguments
private static Stream<Arguments> dataForTestSearchString() {
   return Stream.of(//
         Arguments.of("Any string matches any string.", "(.*)", "xyz", "xyz") //
         , Arguments.of("Pick year digits out of a date.", "[0-9]{2}/[0-9]{2}/([0-9]{4})", "31/12/2032", "2032") //
         , Arguments.of("No digits found.", "[0-9]+", "not a digit", null) //
         , Arguments.of("First two words.", "(?<firstTwoWords>\\w+ \\w+) .*", "abc xyz one two", "abc xyz") //
         , Arguments.of("Bad regex.", "([0-9+)", "123", "PatternSyntaxException: (?s)Unclosed character class.*") //
   );
}

Notes about this method:

  • It has no annotations, but it returns a stream of JUnit specific objects - org.junit.jupiter.params.provider.Arguments
  • Use the Java single line comment // by itself to ensure a line of code gets broken off when you use automatic formatting.
  • When creating a Java list (or array, or any collection really), putting the comma at the start of each line (after the first) makes it easy to copy and paste a line to form the next list element without having to worry about removing the comma from the end of the last element.

Here is the @Test method that consumes this test data.

// JUnit5 @Test method whose data comes from dataForTestSearchString
@ParameterizedTest(name = "#{index} - [{0}]")
@MethodSource("dataForTestSearchString")
public void testFilesFromDirectoriesAndPattern(final String label, final String regex,
      final String stringToMatchAgainst, final String expected) {
   final String actual = searchString(regex, stringToMatchAgainst);
   assertEquals(expected, actual);
}

Notes about this method.

  • The @ParameterizedTest annotation lets you specify the "name" of each test data, which affects how each call to your test method is labelled in Eclipse. I have used the pattern #{index} - [{0}], which means each test case will be labelled with the index of that test data and whatever the first argument to the test method is (so the first argument becomes the test case label). See the screenshot below to see how wonderful this is.
  • The @MethodSource annotation allows you to name the method (or give an array naming multiple methods) that will generate data to be consumed by this test.
    • In the background, JUnit will use reflection to call the method, which has a side effect in Eclipse: if nothing else calls that method, Eclipse will give you a warning that your data generating method is unused (because it cannot work out from this annotation that you are really using it).

This is what you see in Eclipse when running this class as a JUnit test.

One of the most important advantages of JUnit in Eclipse over TestNG is that you can run individual test cases by right-clicking on the one you want to re-run and select "Run" or "Debug" (TestNG would re-run the whole lot).

Here is version 1 the full test case, incorporating the method being tested.

// First version of the data driven test, including the method being tested
import static org.junit.jupiter.api.Assertions.assertEquals;

import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Stream;

import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;

public class JunitDataDrivenTest {

   /**
    * @return test data. Each set of arguments should consist of
    *         <ol>
    *            <li>test label</li>
    *            <li>regex</li>
    *            <li>string we apply the regex to</li>
    *            <li>first group matched by the regex</li>
    *         </ol>
    */
   @SuppressWarnings({ "unused" }) // Eclipse thinks this method is not used.
   private static Stream<Arguments> dataForTestSearchString() {
      return Stream.of(//
            Arguments.of("Any string matches any string.", "(.*)", "xyz", "xyz") //
            , Arguments.of("Pick year digits out of a date.", "[0-9]{2}/[0-9]{2}/([0-9]{4})", "31/12/2032", "2032") //
            , Arguments.of("No digits found.", "[0-9]+", "not a digit", null) //
            , Arguments.of("First two words.", "(?<firstTwoWords>\\w+ \\w+) .*", "abc xyz one two", "abc xyz") //
            , Arguments.of("Bad regex.", "([0-9+)", "123", "PatternSyntaxException: (?s)Unclosed character class.*") //
      );
   }

   /**
    * @param label
    *           description of the current test. Data is used for reporting only.
    * @param regex
    *           regular expression.
    * @param stringToMatchAgainst
    *           string that we will apply the regex to
    * @param expected
    *           first group matched by the regex. Will be null if no match is expected.
    */
   @ParameterizedTest(name = "#{index} - [{0}]")
   @MethodSource("dataForTestSearchString")
   public void testFilesFromDirectoriesAndPattern(final String label, final String regex,
         final String stringToMatchAgainst, final String expected) {
      final String actual = searchString(regex, stringToMatchAgainst);
      assertEquals(expected, actual);
   }

   /**
    * A method being tested.
    *
    * @param regex
    *           regular expression.
    * @param stringToMatchAgainst
    *           string that we will apply the regex to
    * @return first group matched by the regex. Will be null if no match is expected.
    */
   public String searchString(final String regex, final String stringToMatchAgainst) {
      final Pattern pattern = Pattern.compile(regex);
      final Matcher matcher = pattern.matcher(stringToMatchAgainst);
      if (matcher.matches()) {
         return matcher.group(1);
      }
      return null;
   }
}

Testing for exceptions

Another new aspect of JUnit 5 is the set of assertions that use lambdas from JDK 8, particularly the assertThrows method which I will demonstrate next.

As mentioned above, the last test case throws an exception because of an invalid regular expression ("([0-9+)" doesn't close the square brackets: it should be "([0-9]+)"). A normal part of unit testing should be ensuring that code throws appropriate exceptions when something goes wrong. Here is the unit test adjusted so that if our parameterized data indicates that an exception is expected, we test for it. We test for the expected exception type and have a regular expression to test the exception's message.

// JUnit test that checks if code throws an exception if our parameterized data indicates one is expected
@ParameterizedTest(name = "#{index} - [{0}]")
@MethodSource("dataForTestSearchString")
public void testFilesFromDirectoriesAndPattern(final String label, final String regex,
      final String stringToMatchAgainst, final String expected) {

   // Types of exceptions we test for.
   final String patternSyntaxException = "PatternSyntaxException: ";

   // PatternSyntaxException
   if (expected != null && expected.startsWith(patternSyntaxException)) {
      final PatternSyntaxException pse = assertThrows(PatternSyntaxException.class, () -> {
         searchString(regex, stringToMatchAgainst);
      });
      final Pattern exceptionPattern = Pattern.compile(expected.substring(patternSyntaxException.length()));
      final Matcher exceptionMatcher = exceptionPattern.matcher(pse.getMessage());
      assertTrue(exceptionMatcher.matches());
   } else {
      // Normal operation.
      final String actual = searchString(regex, stringToMatchAgainst);
      assertEquals(expected, actual);
   }

}

Notes about this method.

  • We use one of the method parameters, a string named expected to tell the method what result we should expect under normal (successful) operation.
  • I have extended the use of that string to also tell me what to expect in exceptional circumstances. In my code, I know what exceptions I am testing for and have decided that for test cases that should throw an exception, the expected string should start with the class name of the exception and the rest of the string will be a regular expression that should match the exception's message (getMessage()).
    • To do this I need some string processing: a test to see if the string starts with an exception class name and then a substring operation to pull out the regex.
    • I then need code to create the Pattern, the Matcher and then test if the regex matches the exception message.

The unit test now passes in all cases.

Here is the final version of the full test case, incorporating exception testing.

// Final version of the data driven test including exception testing
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;
import java.util.stream.Stream;

import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;

public class JunitDataDrivenTest {

   /**
    * @return test data. Each set of arguments should consist of
    *         <ol>
    *            <li>test label</li>
    *            <li>regex</li>
    *            <li>string we apply the regex to</li>
    *            <li>first group matched by the regex</li>
    *         </ol>
    */
   @SuppressWarnings({ "unused" }) // Eclipse thinks this method is not used.
   private static Stream<Arguments> dataForTestSearchString() {
      return Stream.of(//
            Arguments.of("Any string matches any string.", "(.*)", "xyz", "xyz") //
            , Arguments.of("Pick year digits out of a date.", "[0-9]{2}/[0-9]{2}/([0-9]{4})", "31/12/2032", "2032") //
            , Arguments.of("No digits found.", "[0-9]+", "not a digit", null) //
            , Arguments.of("First two words.", "(?<firstTwoWords>\\w+ \\w+) .*", "abc xyz one two", "abc xyz") //
            , Arguments.of("Bad regex.", "([0-9+)", "123", "PatternSyntaxException: (?s)Unclosed character class.*") //
      );
   }

   /**
    * @param label
    *           description of the current test. Data is used for reporting only.
    * @param regex
    *           regular expression.
    * @param stringToMatchAgainst
    *           string that we will apply the regex to
    * @param expected
    *           first group matched by the regex. Will be null if no match is expected.
    */
   @ParameterizedTest(name = "#{index} - [{0}]")
   @MethodSource("dataForTestSearchString")
   public void testFilesFromDirectoriesAndPattern(final String label, final String regex,
         final String stringToMatchAgainst, final String expected) {

      // Types of exceptions we test for.
      final String patternSyntaxException = "PatternSyntaxException: ";

      // PatternSyntaxException
      if (expected != null && expected.startsWith(patternSyntaxException)) {
         final PatternSyntaxException pse = assertThrows(PatternSyntaxException.class, () -> {
            searchString(regex, stringToMatchAgainst);
         });
         final Pattern exceptionPattern = Pattern.compile(expected.substring(patternSyntaxException.length()));
         final Matcher exceptionMatcher = exceptionPattern.matcher(pse.getMessage());
         assertTrue(exceptionMatcher.matches());
      } else {
         // Normal operation.
         final String actual = searchString(regex, stringToMatchAgainst);
         assertEquals(expected, actual);
      }
   }

   /**
    * A method being tested.
    *
    * @param regex
    *           regular expression.
    * @param stringToMatchAgainst
    *           string that we will apply the regex to
    * @return first group matched by the regex. Will be null if no match is expected.
    */
   public String searchString(final String regex, final String stringToMatchAgainst) {
      final Pattern pattern = Pattern.compile(regex);
      final Matcher matcher = pattern.matcher(stringToMatchAgainst);
      if (matcher.matches()) {
         return matcher.group(1);
      }
      return null;
   }
}

Versions

Software used in this post.

  • JDK 1.8 (1.8.0_40)
  • Eclipse is Spring Tool Suite 3.9.2.RELEASE (build on Eclipse Oxygen.2 (4.7.2))
  • Junit plugin is built in
  • JUnit dependencies in my pom.xml: 
    
  <!-- JUnit Maven dependencies -->
  <!-- JUnit Jupiter is the API we write tests against. -->
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.1.0</version>
    <scope>test</scope>
  </dependency>
  <!-- Parameterised tests is separate download because it is currently an experimental feature. -->
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-params</artifactId>
    <version>5.1.0</version>
    <scope>test</scope>
  </dependency>
  <!-- The engined used by the IDE to run tests. JUnit is built in to Eclipse, but apparently my version of Eclipse can't run JUnit 5 tests. -->
  <dependency>
    <groupId>org.junit.platform</groupId>
    <artifactId>junit-platform-launcher</artifactId>
    <version>1.1.0</version>
    <scope>test</scope>
  </dependency>
Labels:

Popular Posts