Randomised Testing - How we find the needles in the haystack

An impossible search for something relatively tiny, lost or hidden in something that is relatively enormous - the first use of this expression, and its likely origin, is by the writer Miguel de Cervantes, in his story Don Quixote de la Mancha written from 1605-1615. According to Bartlett's, the expression 'As well look for as needle in a bottle of hay' (translated from the original Spanish) appears in part III, chapter 10. 'Bottle' is an old word for a bundle of hay, taken from the French word botte, meaning bundle. Brewer (1870-94 dictionary and revisions) lists the full expression - 'looking for a needle in a bottle of hay' which tells us that the term was first used in this form, and was later adapted during the 1900's into the modern form

In my last post I endeavoured to explain the what, when and why of randomised testing. In this post I will detail how we have implemented randomised testing for our Java codebases.

At Brandwatch we use a few tools to assist us with unit testing. We hang our unit tests off of the JUnit4 testing framework so it made sense to continue to use this for our randomised test cases. Maven has done a great job of building and running our tests so there was no reason to replace it, while Jenkins quite happily handles scheduling and running tests as needed.

However, we didn't have anything in place to handle the actual randomisation and as is usually the case with shiny new things the temptation to write our own framework was strong. However having seen this brilliant talk on randomised testing by Dawid Weiss at Berlin Buzzwords 2014 I was already aware of a solution which matched our needs exactly - Carrot Search Labs' RandomizedTesting infrastructure.

RandomizedTesting framework

RandomizedTesting grew out of the Lucene project's randomised testing architecture which has been used with much success by their team. It provides a framework which produces pseudo random (e.g., repeatable randomised) JUnit tests along with Maven integration and a suite of tools for generating various random parameters.

The tests are made repeatable via a combination of a seed recovery facility and annotation which can be used to rerun a given test or class in the configuration provided by a given seed

java.lang.AssertionError  
  at __randomizedtesting.SeedInfo.seed([AF567B2B9F8A8F1C:7A6253D06C8B11FB]:0)
  at org.junit.Assert.fail(Assert.java:92)
@Seed("7A6253D06C8B11FB")
@Test(expected = PasswordChangeException.class)
public void validatePassword_throwsPasswordChangeException()  
        throws PasswordChangeException {

A JUnit test runner called RandomizedRunner is provided to access the seed recovery facility and a class called RandomizedTest can be extended to provide random generators which work with the seed recovery

@RunWith(RandomizedRunner.class)
public class PasswordStrengthRuleRandomisedTest extends RandomizedTest {  

As a bonus RandomizedRunner randomises the test method order each run which can help to eliminate hidden dependency in tests and/or the code they are exercising.

Because RandomizedTesting does most of the randomisation work for you it's simple to take existing tests and transform them into randomised tests. Likewise new tests can be quickly implemented with randomisation in place from the get go.

For complete implementation details and examples see the RandomizedTesting github repository.

Separating responsibilities

One of the key differences between regular testing and randomised testing is that we expect our tests to occasionally fail (in fact if your tests don't fail they're probably not tough enough). This will obviously be an issue for code which is being deployed a lot during development and time critical live deploys as we usually just want to be informed of critical failures.

We solved this issue by implementing separate Maven profiles for normal deploys and randomised testing by setting up an interface we could use in order to mark a class as belonging to the random tests profile using JUnit's @Category

package com.brandwatch.group;

public interface RandomisedTests {}  
@Category(RandomisedTests.class)
@RunWith(RandomizedRunner.class)
public class PasswordStrengthRuleRandomisedTest extends RandomizedTest {  

Any class annotated with RandomisedTests was then excluded or included in Maven builds as necessary using appropriate profiles alongside Apache's surefire plugin configurations

  <profiles>
    <!--Default profile used when no groups are specified, e.g., mvn clean install-->
    <profile>
      <id>default</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
       <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            ...
            <configuration>
              <excludedGroups>com.brandwatch.group.RandomisedTests</excludedGroups>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>  
    <!--Profile used for running just the randomised tests when the random group is specified, e.g., mvn clean test -Prandom-->
    <profile>
      <id>random</id>
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            ...
            <configuration>
              ...
              <groups>com.brandwatch.group.RandomisedTests</groups>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>

Running it every which way

The reason we set up randomised testing in the first place was to try and cover as much of the test space for specific tests as possible. To do this we needed some way of running these tests as much as we realistically could - which is where Jenkins entered the fray.

Jenkins plays nicely with maven out of the box so it was trivial to set up a nightly job which runs the randomised testing maven profile continuously (resources permitting). Jenkins is also friendly with git so we could ensure the tests and code under test were up to date with our latest changes every time they were run.

Having set all of this up we only needed to write our randomised tests and then sit back and wait for the needles to roll in.

These are actually pins