# Running EOF `diff_fuzz`

*As of 29 Mar 2025*

## Repositories

* [ethereum/execution-spec-tests](https://github.com/ethereum/execution-spec-tests) - `main` branch
* [shemnon/execution-spec-tests](https://github.com/shemnon/execution-spec-tests/tree/shemnon/eof-fuzz) - `eof-fuzz` branch
* [holiman/goevmlab](https://github.com/holiman/goevmlab) - `main` branch

## Required Tools

The following tooling is needed, and we rely on the reader knowing how to install these tools

* Python 3.10, 3.11, or 3.12.
* UltraViolet (uv)
* Go Language 1.23.0 or higher
* Tooling needed to build or install other Ethereum clients

## Build GoEVMLab

We will need the `runtest` program and may benefit from the `diffview` program

* Install all the goevmlab binaries into your `$GOROOT` (`$HOME/go/bin` if `$GOROOT` is not set, and the current golang recommendation is to not set the root variable).

git clone [e-mail]:holiman/goevmlab.git
cd goevmlab
go install ./...

## Setup EEST

Ethereum Execution Spec Tests (EEST) is a python repository managed by ultraviolet.

Follow the [EEST Installation](https://github.com/ethereum/execution-spec-tests?tab=readme-ov-file#installation) guide. The most impactful line is `uv sync --all-extras`.

## Generate Corpus Files

`runtest` in GoEVMLab runs off of the state_test file format from EEST (fixtures), and only works reliably with one test per state test. (This is because clients may parallelize evaluation and report answers out of order. This includes the stack trace, and the test is only identified at the end of the segment.)

Also, EEST will only generate fixtures for tests that validate against the assertions in the generation. So if a client creating the corpus has a bug then those tests will not be generated as corpus files.

Here is the script I use to generate corpus files
* the `debug` directory includes tooling output that debugs failed test filling.
* the `fuzzing` directory is where the fixtures are generated
* Besu is filling the tests, changing the `--evm-bin` is the only essential update to get other clients with `t8n` support to fill tests.
* `--single-fixture-per-file` is the most critical EEST option
* Execution time is several hours, regardless of client used.

Execute this in the main branch of `ethereum/execution-spec-tests`

rm -rf debug/besu_fuzzing
mkdir debug/besu_fuzzing

if [[ $1 =~ 'tests/' ]]
then
  TESTS=${1}
  shift
else
  TESTS=tests/osaka/eip7692_eof_v1/
fi

uv run fill \
  --fork=Osaka \
  --t8n-dump-dir  debug/besu_fuzzing \
  -v \
  $TESTS \
  --evm-bin /Users/shemnon/git/shemnon/besu/build/install/besu/bin/evmtool  \
  --evm-dump-dir  debug/besu_fuzzing \
  --html=debug/besu_fuzzing/report-fuzz.html \
  --output fuzzing \
  --flat-output \
  --single-fixture-per-file \
  $* \
  > debug/besu_fuzzing/res.txt \
  2> debug/besu_fuzzing/err.txt

grep -E ' in \d+\.\d+s' debug/besu_fuzzing/res.txt

## Build the clients under test

How the clients are built is out of scope for this document. Clients known to work required binaries are

* Besu / `evmtool`
* Erigon / `evm`
* Geth / `evm`
* Nethermind / `nethtest`
* Revm (Reth) / `revme`

Clients that should work (or coulde be made to work) but whose status has not been validated

* EELS
* EthereumJS
* nimbus
* evmone

Most issues in getting a client set up relate to conformance to [EIP-7765](https://eips.ethereum.org/EIPS/eip-7756), specifically for lines representing execution in an EOF container
* presence of `functionDepth` field
* presence of `section` field
* Counting pc=0 from the beginning of the container

## Execute the Fuzzer

The fuzzer currently live in the `eof-fuzz` branch of the `shemnon/execution-spec-tests` repository. Ensure that the repo has been initted with ultraviolet via the `uv sync --all-extras` command.

A typical command looks like:

uv run diff_fuzz \
  -w /tmp/diff_fuzz\
  -c fuzzing/state_tests \
  --cleanup-tests True \
  -r $HOME/go/bin/runtest \
  --client geth $HOME/git/github.com/ethereum/go-ethereum/build/bin/evm \
  --client erigon $HOME/git/github.com/racytech/erigon/build/bin/evm \
  --client besubatch $HOME/git/hyperledger/besu/build/install/besu/bin/evmtool \
  --skip-trace False \
  --step-count 1000 --step-num 1 \
  --max-gas 100000000

The `-w` option sets where results will be stored, as well as a temporary working directory for mutations. The `-c` option is where the corps files generated in [a prior step](#Generate-Corpus-Files) exist. `-r` points to GoEVMLab's `runtest` binary, `--cleanup-tests` deletes successful tests after execution, `--step-count` and `--step-num` control the names of result directories.

The `--client` option takes two arguments, the first is the name of a client supported by `runtest` and the second is the location of the relevant binary for that option.

`--skip-trace` is set to False. The best result come from full trace comparisons.

`--max-gas` sets the maximum gas available for the test transaction. This value should not exceed 2 billion as clients start handling out of gas situations differently.

:::info
The only values mutated are in the `pre` section of the state tests. No post data is updated. The tests will almost always have an incorrect state root, and the transaction expressed in `txbytes` may not reflect the transaction the fuzzer expects the tool to test.
:::