# Foundry Mutation Test - Next Steps


## Progress so far

There is now an alpha implementation of mutation testing in Foundry. To check this out follow the below steps

git clone https://github.com/samparsky/foundry.git mutate-foundry
cd mutate-foundry
cargo build

# init a sample foundry project or any valid foundry project
forge init sample-project
cd sample-project
# Run the below command to check the help 
/<path to the clone directory>/mutate-foundry/target/debug/forge mutate --help
# To run the mutation test
/<path to the clone directory>/mutate-foundry/target/debug/forge mutate
# To run mutation test on just a specific function
/<path to the clone directory>/mutate-foundry/target/debug/forge mutate --matching-function "addNumber"

At the moment there are huge performance bottlenecks so I recommend using the below command instead of running a plain `forge mutate`

forge mutate --match-contract <Contract Name> --match-test-contract <Test Contract Name> --summary

## Challenges

We have made progress on integrating Gambit framework into Foundry. However, there are several performance challenge with the current prototype. These challenges are discussed below

**1) Performance**
Gambit recompiles the solidity files [here](https://github.com/Certora/gambit/blob/master/src/compile.rs#L97-L150). This introduces additional overhead into the mutation testing pipeline. This additional compilation step is unecessary as Foundry has excellent logic to handle this.

**2) Custom AST Parser**
Gambit implements a custom AST parser. This is redundant as Foundry has an excellent AST parser. It introduces additional performance overhead.

**3) Mutation Inference**
Gambit provides Mutant information on a file level e.g. Mutants for Counter.sol. While this is useful it's more useful if we could provide Mutation information on a Function level too. This would provide the best analysis for developers.

**4) Incremental Mutation Testing**
Generating and Testing mutants takes quite a lot of time. It necessary we don't re-generate mutants on every run and re-test. Gambit does not support incremental mutation testing. Incremental mutation testing enables us to cache already generated and tested mutants and improve mutation testing speed.

**5) Mutant Equivalence**
A simple approach for equivalent mutant is doing bytecode comparison of the original source and mutant. Gambit doesn't check for equivalent mutants.

**6) Compilation Time**
Compiling takes majority of the time spent in the mutation test pipeline. There is a need to improve compilation times in `foundry-compilers`. Either `compile_sparse` also compile inheriting contracts or `compile()` performance is improved.

## Next Steps / Solution

The next step is we can either fork the gambit codebase and make distinct design decisions that fit the Foundry codebase or make changes to the Gambit codebase to make it Foundry compatible.

Another improvement is improving compilation times in `foundry-compilers`.

There is a deadlock condition when the number of mutants exceed 100. That's also being investigated.