# How to measure test quality

<figure><img src="/files/WMp2aI6bIoCzXoQ063fA" alt=""><figcaption></figcaption></figure>

Traditional test coverage (i.e line, statement, branch, etc.) measures what code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code.

Mutation testing fills this gap. Faults (or **mutations**) are seeded into your code, then your tests are run. If your tests fail then the mutation is **killed**, if your tests pass then the mutation **lived**. The quality of your tests can be gauged from the percentage of mutations killed.

Source code is changed or mutated in three ways – [value](#value-mutations), [decision](#decision-mutations) and [statement](#statement-mutations).

#### Value mutations

Change a small value to a larger value or vice versa.

```c
/* Initial code */

int mod = 1000000007;
int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;

/* Changed code */

int mod = 1007;
int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;
```

#### Decision mutations

Change logical or arithmetic operators to detect errors.

```c
/* Initial code */

if (a < b) {
  c = 10;
} else {
  c = 20;
}

/* Changed code */

if (a > b) {
  c = 10;
} else {
  c = 20;
}
```

#### Statement mutations

Delete a statement or replace it with some other statement

```c
/* Initial code */

if (a < b) {
  c = 10;
} else {
  c = 20;
}

/* Changed code */

if (a < b) {
  d = 10;
} else {
  d = 20;
}
```

### Advantages of mutation testing

* Brings a good level of error detection in the program
* Discovers ambiguities in the source code
* Helps testers write better tests and measure effectiveness of tests written
* Makes source code more efficient

### Mutation Testing Plugins

* [PIT](https://pitest.org/) *(recommended)*
* [Judy](http://www.mutationtesting.org/judy2/)
* [Jester](https://jester.sourceforge.net/)
* [Jumble](https://jumble.sourceforge.net/)

***

### Setting up PIT on your project

#### Maven

[Documentation](https://pitest.org/quickstart/maven/) | [Plugin](https://central.sonatype.com/artifact/org.pitest/pitest-maven)

Add this plugin to your `pom.xml`:

```xml
<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.17.4</version>
</plugin>
```

Then run: `mvn test-compile org.pitest:pitest-maven:mutationCoverage`, which will output an HTML report to `target/pit-reports/<YYYYMMDDHHMI>`.

For multi-module projects, please see [Aggregating tests across modules](https://pitest.org/aggregating_tests_across_modules/).

#### Gradle

[Documentation](https://gradle-pitest-plugin.solidsoft.info/) | [Plugin](https://central.sonatype.com/artifact/info.solidsoft.gradle.pitest/gradle-pitest-plugin)

Add this plugin to your `build.gradle`:

```gradle
plugins {
    id 'info.solidsoft.pitest' version '1.15.0'
}

// To aggregate Pitest results for a multi-module projects
apply plugin: 'info.solidsoft.pitest.aggregator'

// For multi-module projects
subprojects {
    apply plugin: 'java'
    apply plugin: 'info.solidsoft.pitest'

    pitest {
        threads = 4
        // Export mutations.xml and line coverage for aggregation
        outputFormats = ["XML"]
        exportLineCoverage = true
        timestampedReports = false

        if (project.name in ['module-without-any-test']) {
            failWhenNoMutations = false
        }

        reportAggregator {
            // aggregation configuration (optional)
        }
    }
}
```

Run `gradle pitest pitestReportAggregate` task to have an aggregated report be produced at `${PROJECT_DIR}/build/reports/pitest`.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://cover-docs-preview.diffblue.com/features/tutorials/how-to-measure-test-quality.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.