Are your tests good enough? What makes a test good? How do you know?

Mutation Testing

What is mutation testing?

Mutation testing (or mutation analysis or program mutation) is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways.[1] Each mutated version is called a mutant and tests detect and reject mutants by causing the behaviour of the original version to differ from the mutant. This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill. New tests can be designed to kill additional mutants. Mutants are based on well-defined mutation operators that either mimic typical programming errors (such as using the wrong operator or variable name) or force the creation of valuable tests (such as dividing each expression by zero). The purpose is to help the tester develop effective tests or locate weaknesses in the test data used for the program or in sections of the code that are seldom or never accessed during execution. Mutation testing is a form of white-box testing.
– Wikipedia entry on mutation testing

In other words: testing your tests.

How do you know if your tests are good? Mutation testing helps you find out!

If the application logic is modified - creating a mutant - and the tests fail - killing a mutant - then your tests are good; if the tests pass - the mutant survives - then your tests are less than good.

But Wait, What About Code Coverage

So you’ve got 90% test coverage?

Test coverage isn’t everything. If your tests aren’t testing precise conditions, they’re not good enough, because mutants survive. That means that a coder - maybe you? - could modify the logic and the change would not be caught by tests.

Stryker

What is Stryker? It’s my mutation testing tool of choice, named after X-Men villain William Stryker, hater of mutants.

It’s an easy to use tool, and works with .NET, js, and Scala.

Show Me the Tests

Let’s give an example, shall we?

Let’s say I wanted to make a calculator that added all the values it was given together.

Calculator

The interface:

namespace Calculator
{
    public interface ICalculator
    {
        int AddNumbers(List<int> numbers);
    }
}

The implementation:

namespace Calculator
{
    public class Calculator : ICalculator
    {
        public int AddNumbers(List<int> numbers)
        {
            return numbers.Sum();
        }
    }
}

Simple. It adds things.

Now, what about the tests?

Calculator Tests

I wrote this incredible test.

using FluentAssertions;
using NUnit.Framework;

namespace Calculator.Tests
{
    [TestFixture]
    public class CalculatorTests
    {
        private ICalculator _calculator = new Calculator();

        [Test]
        public void Add_given1and2_should_begreaterthan0()
        {
            var list = new List<int>() { 1, 2 };
            var result = _calculator.AddNumbers(list);
            result.Should().BeGreaterThan(0);
        }
    }
}

Surely this is the only test I need, right? It passes, what more do I need?

Enter Stryker

When you run Stryker from the command line, you get a lovely output something like this:

Uh oh. A mutant survived. Could it be my test is bad, or I am not testing enough to protect against being aware that the code has changed?

It could.

As you can see from the above, Stryker has given me a lovely report.

Let’s have a peek, shall we?

Hmm. My test let a mutant live. How could that be? Let’s find out.

Stryker will show you what the mutant was that survived, which will help you to identify if you need more tests for the method, or how to fix the test you have to catch the mutant and kill it dead.

Stryker mutated the code and sneakily changed Sum to Max. Max will return the maximum value in a collection.

In this case, that would be 2. My test is checking to see if the result is greater than 0, and 2 is indeed greater than zero.

Drat. What could I do to catch this dastardly mutant?

Catching a Mutant

I think what I need is a better test. Let’s have a go:

[Test]
[TestCase(new []{1, 2}, 3)]
public void Add_giveninputs_should_beequaltoexpectedResult(int[] numbersToAdd, int expectedResult)
{
    var result = _calculator.AddNumbers(numbersToAdd.ToList());
    result.Should().Be(expectedResult);
}

Looks better to me. 1 + 2 should equal 3. If I did Max on this, I’d get 2, so that mutant should be caught and put to death.

Let’s run Stryker again and see what happens.

Dead! They’re all dead! We win!

Wrap Up

Mutation testing is a great way to test your tests, and Stryker is a great tool for doing it.

Give it a shot, and enjoy putting those pesky mutants to death!