Foundry Mutation Testing

An overview of integrating mutation testing into Foundry based off the Gambit framework.

Command

The suggestion is to use subcommand forge mutate --sample-seed ..<...>. Another suggestion is to use a flag forge test --mutate but that's not recommended because it would introduce the mutation test flags into test and could become confusing for developers.

pub struct MutationTestArgs {
    /// Output mutation test results in JSON format.
    #[clap(long, short, help_heading = "Display options")]
    json: bool,

    #[clap(flatten)]
    filter: MutationTestFilterArgs,

    /// Stop running mutation tests after the first failure
    #[clap(long)]
    pub fail_fast: bool,

    /// List mutation tests instead of running them
    #[clap(long, short, help_heading = "Display options")]
    list: bool,

    #[clap(flatten)]
    evm_opts: EvmArgs,

    #[clap(flatten)]
    opts: CoreBuildArgs,

    #[clap(value_enum, default_value = "function")]
    test_mode: TestMode,
}

Configuration

Filter

We aim to support 8 types of mutation filters

• Random Sample
• Function pattern
• Function pattern inverse
• Contract pattern
• Contract pattern inverse
• Path pattern
• Path pattern inverse
• Mutation rules

pub struct MutationTestFilterArgs {
    /// Only run randomly chosen mutation based on this seed
    #[clap(long = "sample-seed", visible_alias = "ms", value_name = "number")]
    pub sample_seed: Option<u64>,
    
    /// Only run mutations on functions matching the specified regex pattern.
    #[clap(long = "match-function", visible_alias = "mf", value_name = "REGEX")]
    pub function_pattern: Option<regex::Regex>,

    /// Only run mutations on functions that do not match the specified regex pattern.
    #[clap(
        long = "no-match-function",
        visible_alias = "nmf",
        value_name = "REGEX"
    )]
    pub function_pattern_inverse: Option<regex::Regex>,

    /// Only run mutations on functions in contracts matching the specified regex pattern.
    #[clap(long = "match-contract", visible_alias = "mc", value_name = "REGEX")]
    pub contract_pattern: Option<regex::Regex>,

    /// Only run mutations in contracts that do not match the specified regex pattern.
    #[clap(
        long = "no-match-contract",
        visible_alias = "nmc",
        value_name = "REGEX"
    )]
    pub contract_pattern_inverse: Option<regex::Regex>,

    /// Only run mutations on source files matching the specified glob pattern.
    #[clap(long = "match-path", visible_alias = "mp", value_name = "GLOB")]
    pub path_pattern: Option<GlobMatcher>,

    /// Only run mutations on source files that do not match the specified glob pattern.
    #[clap(
        name = "no-match-path",
        long = "no-match-path",
        visible_alias = "nmp",
        value_name = "GLOB"
    )]
    pub path_pattern_inverse: Option<GlobMatcher>,
    
    /// Only use this select mutation rules
    #[clap(long = "mutation-rules", visible_alias = "mtr", value_name = "String")]
    pub mutation_rules: Vec<String>
}

TestMode

Mutation testing can take quite a lot of time, this aims to support different testing mode
that enable faster development and better developer UX.

pub enum TestMode {
    /// Only run tests matching the contract file name
    /// e.g. for Counter.sol mutations run tests in Counter.t.sol 
    File,
    
    /// Only run tests matching similar function names
    /// e.g. for Counter.sol:addNumber mutation run tests in test suite
    /// matching a regex test_addNumber, testAddNumber, test_AddNumber, 
    /// testFuzz_[a|A]ddNumber, testFail_addNumber
    Function,
    
    /// Run the entire test suite. This should be used in a CI 
    /// environment as it would take quite sometime 
    Full
}

Future Feature Configurations

• Support loading of custom mutation rules

Implementation

Mutation Rules

We support similar mutation rules as in Gambit

pub enum MutationRules {
    AssignmentMutation,
    BinaryOpMutation,
    DeleteExpressionMutation,
    ElimDelegateMutation,
    FunctionCallMutation,
    IfStatementMutation,
    RequireMutation,
    SwapArgumentsFunctionMutation,
    SwapArgumentsOperatorMutation,
    UnaryOperatorMutation,
}

Mutation Test Output

Output of running mutation tests

CLI Summary Output

| File               | Killed    |  Survived  | Equivalent     | 
|--------------------|-----------|------------|----------------|
| src/Counter.sol    | 100       | 50         | 3              |
| Total              | 100       | 50         | 4              |

CLI Detailed Output

| File               | Mutation         |    Diff                     | Result         |
|--------------------|------------------|-----------------------------|----------------|
| src/Counter.sol:10 | BinaryOpMutation |   number -= 1               | Killed         |

Json Output

This is similar to Gambit JSON output with the addition of "result" property.

[
    {
        "id": "1",
        "name": "BinaryOpMutation.sol",
        "description": "BinaryOpMutation",
        "diff": "--- original\n+++ mutant\n@@ -4,7 +4,8 @@\n \n contract BinaryOpMutation {\n     function myAddition(uint256 x, uint256 y) public pure returns (uint256) {\n-\treturn x + y;\n+\t/// BinaryOpMutation(`+` |==> `-`) of: `return x + y;`\n+\treturn x-y;\n     }\n \n     function mySubtraction(uint256 x, uint256 y) public pure returns (uint256) {\n",        
        "original": "BinaryOpMutation/BinaryOpMutation.sol",
        "result": "killed"
   }
]