Foundry Mutation Test - Next Steps

Progress so far

There is now an alpha implementation of mutation testing in Foundry. To check this out follow the below steps

git clone https://github.com/samparsky/foundry.git mutate-foundry
cd mutate-foundry
cargo build

# init a sample foundry project or any valid foundry project
forge init sample-project
cd sample-project
# Run the below command to check the help 
/<path to the clone directory>/mutate-foundry/target/debug/forge mutate --help
# To run the mutation test
/<path to the clone directory>/mutate-foundry/target/debug/forge mutate
# To run mutation test on just a specific function
/<path to the clone directory>/mutate-foundry/target/debug/forge mutate --matching-function "addNumber"

At the moment there are huge performance bottlenecks so I recommend using the below command instead of running a plain forge mutate

forge mutate --match-contract <Contract Name> --match-test-contract <Test Contract Name> --summary

Challenges

We have made progress on integrating Gambit framework into Foundry. However, there are several performance challenge with the current prototype. These challenges are discussed below

1) Performance
Gambit recompiles the solidity files here. This introduces additional overhead into the mutation testing pipeline. This additional compilation step is unecessary as Foundry has excellent logic to handle this.

2) Custom AST Parser
Gambit implements a custom AST parser. This is redundant as Foundry has an excellent AST parser. It introduces additional performance overhead.

3) Mutation Inference
Gambit provides Mutant information on a file level e.g. Mutants for Counter.sol. While this is useful it's more useful if we could provide Mutation information on a Function level too. This would provide the best analysis for developers.

4) Incremental Mutation Testing
Generating and Testing mutants takes quite a lot of time. It necessary we don't re-generate mutants on every run and re-test. Gambit does not support incremental mutation testing. Incremental mutation testing enables us to cache already generated and tested mutants and improve mutation testing speed.

5) Mutant Equivalence
A simple approach for equivalent mutant is doing bytecode comparison of the original source and mutant. Gambit doesn't check for equivalent mutants.

6) Compilation Time
Compiling takes majority of the time spent in the mutation test pipeline. There is a need to improve compilation times in foundry-compilers. Either compile_sparse also compile inheriting contracts or compile() performance is improved.

Next Steps / Solution

The next step is we can either fork the gambit codebase and make distinct design decisions that fit the Foundry codebase or make changes to the Gambit codebase to make it Foundry compatible.

Another improvement is improving compilation times in foundry-compilers.

There is a deadlock condition when the number of mutants exceed 100. That's also being investigated.