Mutation Testing

Code coverage tells you what percentage of your code runs during testing. It doesn’t, however, reveal whether your tests actually catch bugs. Mutation testing fills that gap by injecting small changes into the source code and checking whether tests detect them. If a mutation survives the test run, the test clearly isn’t thorough enough.

In a Nutshell

• Mutation testing injects artificial faults into source code.
• The goal is to measure test quality, not just coverage.
• Mutation Score = killed mutants / total mutants.
• High computational cost is the biggest challenge.

Compact Technical Definition

Mutation testing generates mutants—slightly modified versions of production code. A mutant might change an arithmetic operator, flip a condition, or alter a return value. The test suite runs against each mutant. If at least one test fails, the mutant is killed. If all tests pass, the mutant survives, indicating weak tests.

Mutation Score

The mutation score is calculated as follows:

Mutation Score = (Killed Mutants / Total Mutants) * 100

A high score means your tests reliably catch bugs. A low score suggests tests run but don’t effectively validate behavior.

Types of Mutations

Arithmetic Operators

int result = a + b;  // Mutant: a - b

Relational Operators

if (x > 10) { ... }  // Mutant: x < 10

Return Values

return true;  // Mutant: return false;

Statement Deletion

System.out.println("log");  // Mutant: line removed

Conditions

if (a && b)  // Mutant: if (a || b)

Key Tools

• PIT: Mutation testing for Java and JVM languages.
• Stryker: Mutation testing for JavaScript, TypeScript, and C#.
• MutPy: Mutation testing for Python.
• Cosmic Ray: Mutation testing for Python.
• Infection: Mutation testing for PHP.

Practical Example: Simple Mutation in Java

public class DiscountCalculator {
    public int calculate(int amount) {
        if (amount > 100) {
            return amount - 10;
        }
        return amount;
    }
}

@Test
public void testDiscountForLargeAmount() {
    DiscountCalculator calc = new DiscountCalculator();
    assertEquals(90, calc.calculate(100));  // Problem: boundary value 100 instead of 101
}

A mutant changes amount > 100 to amount >= 100. For the input value 100, both produce 90. The test survives the mutant even though the condition differs. This reveals the test doesn’t properly validate the boundary condition.

Strengths and Limitations

Strengths

• Better test quality: Measures whether tests actually validate behavior.
• Beyond false coverage: Coverage alone doesn’t prove quality.
• Reveals weak spots: Inadequate tests become visible.
• Forces meaningful assertions: Developers write more specific test cases.

Limitations

• Computational overhead: Many mutants require many test runs.
• Equivalent mutants: Some mutations don’t change behavior but can’t be automatically detected.
• Long execution time: Large projects can take hours for mutation testing.
• Maintenance burden: Mutants must regenerate when code changes.

Key Exam Points

• Definition and purpose of mutation testing.
• Mutation score and its calculation.
• Types of mutations.
• Tools like PIT, Stryker, MutPy.
• Difference between coverage and mutation testing.
• The problem of equivalent mutants.

Typical Exam Questions (with Brief Answers)

1. What is mutation testing? A method that injects artificial faults into code to measure test quality.

2. What does the mutation score indicate? The percentage of mutants that tests detect and kill.

3. What is an equivalent mutant? A mutation that doesn’t change program behavior but still survives the test suite.

4. Name a mutation testing tool. PIT, Stryker, MutPy, or Infection.

5. What’s the difference between coverage and mutation testing? Coverage measures which code executes. Mutation testing verifies whether tests catch actual defects.

Key Resources

1. https://pitest.org
2. https://stryker-mutator.io
3. https://en.wikipedia.org/wiki/Mutation_testing

Next in the Software Testing Learning Path

The next article in the Software Testing Learning Path covers Shift-Left Testing—how tests are placed earlier in the development process.