Running EOF diff_fuzz

As of 29 Mar 2025

Repositories

• ethereum/execution-spec-tests - main branch
• shemnon/execution-spec-tests - eof-fuzz branch
• holiman/goevmlab - main branch

Required Tools

The following tooling is needed, and we rely on the reader knowing how to install these tools

• Python 3.10, 3.11, or 3.12.
• UltraViolet (uv)
• Go Language 1.23.0 or higher
• Tooling needed to build or install other Ethereum clients

Build GoEVMLab

We will need the runtest program and may benefit from the diffview program

• Install all the goevmlab binaries into your $GOROOT ($HOME/go/bin if $GOROOT is not set, and the current golang recommendation is to not set the root variable).

git clone [e-mail]:holiman/goevmlab.git
cd goevmlab
go install ./...

Setup EEST

Ethereum Execution Spec Tests (EEST) is a python repository managed by ultraviolet.

Follow the EEST Installation guide. The most impactful line is uv sync --all-extras.

Generate Corpus Files

runtest in GoEVMLab runs off of the state_test file format from EEST (fixtures), and only works reliably with one test per state test. (This is because clients may parallelize evaluation and report answers out of order. This includes the stack trace, and the test is only identified at the end of the segment.)

Also, EEST will only generate fixtures for tests that validate against the assertions in the generation. So if a client creating the corpus has a bug then those tests will not be generated as corpus files.

Here is the script I use to generate corpus files

• the debug directory includes tooling output that debugs failed test filling.
• the fuzzing directory is where the fixtures are generated
• Besu is filling the tests, changing the --evm-bin is the only essential update to get other clients with t8n support to fill tests.
• --single-fixture-per-file is the most critical EEST option
• Execution time is several hours, regardless of client used.

Execute this in the main branch of ethereum/execution-spec-tests

rm -rf debug/besu_fuzzing
mkdir debug/besu_fuzzing

if [[ $1 =~ 'tests/' ]]
then
  TESTS=${1}
  shift
else
  TESTS=tests/osaka/eip7692_eof_v1/
fi

uv run fill \
  --fork=Osaka \
  --t8n-dump-dir  debug/besu_fuzzing \
  -v \
  $TESTS \
  --evm-bin /Users/shemnon/git/shemnon/besu/build/install/besu/bin/evmtool  \
  --evm-dump-dir  debug/besu_fuzzing \
  --html=debug/besu_fuzzing/report-fuzz.html \
  --output fuzzing \
  --flat-output \
  --single-fixture-per-file \
  $* \
  > debug/besu_fuzzing/res.txt \
  2> debug/besu_fuzzing/err.txt

grep -E ' in \d+\.\d+s' debug/besu_fuzzing/res.txt

Build the clients under test

How the clients are built is out of scope for this document. Clients known to work required binaries are

• Besu / evmtool
• Erigon / evm
• Geth / evm
• Nethermind / nethtest
• Revm (Reth) / revme

Clients that should work (or coulde be made to work) but whose status has not been validated

• EELS
• EthereumJS
• nimbus
• evmone

Most issues in getting a client set up relate to conformance to EIP-7765, specifically for lines representing execution in an EOF container

• presence of functionDepth field
• presence of section field
• Counting pc=0 from the beginning of the container

Execute the Fuzzer

The fuzzer currently live in the eof-fuzz branch of the shemnon/execution-spec-tests repository. Ensure that the repo has been initted with ultraviolet via the uv sync --all-extras command.

A typical command looks like:

uv run diff_fuzz \
  -w /tmp/diff_fuzz\
  -c fuzzing/state_tests \
  --cleanup-tests True \
  -r $HOME/go/bin/runtest \
  --client geth $HOME/git/github.com/ethereum/go-ethereum/build/bin/evm \
  --client erigon $HOME/git/github.com/racytech/erigon/build/bin/evm \
  --client besubatch $HOME/git/hyperledger/besu/build/install/besu/bin/evmtool \
  --skip-trace False \
  --step-count 1000 --step-num 1 \
  --max-gas 100000000

The -w option sets where results will be stored, as well as a temporary working directory for mutations. The -c option is where the corps files generated in a prior step exist. -r points to GoEVMLab's runtest binary, --cleanup-tests deletes successful tests after execution, --step-count and --step-num control the names of result directories.

The --client option takes two arguments, the first is the name of a client supported by runtest and the second is the location of the relevant binary for that option.

--skip-trace is set to False. The best result come from full trace comparisons.

--max-gas sets the maximum gas available for the test transaction. This value should not exceed 2 billion as clients start handling out of gas situations differently.

The only values mutated are in the pre section of the state tests. No post data is updated. The tests will almost always have an incorrect state root, and the transaction expressed in txbytes may not reflect the transaction the fuzzer expects the tool to test.