# Blockchain Integration Test Harness ## Background In Prysm, we have two main kinds of tests: e2e and unit tests. For the most part, unit tests have a very complex setup especially in critical files such as the state transition, forkchoice, and the sync service. Even these complex examples tend to mock a lot of the blockchain's functionality by populating the database themselves or using mock versions of services for simplicity. Most recently, we had a situation where we had a bug at runtime in one of the caches used during block processing [here](https://github.com/prysmaticlabs/prysm/pull/11567), and the resolution was pretty simple but actually almost impossible to test with our current testing suite. This triggered a discussion about _why_ it was hard to test this code and if we could do anything about it. It turns out we don't have test harnesses powerful enough to give us a meaningful test aside from our end-to-end suite, and that immediately triggered some concerns. This document designs a new test harness for testing chain functionality in Prysm with minimal mocking, and helps us test different behaviors of block processing that are hard to set up otherwise outside of e2e. ## Existing blockchain test harnesses When testing blockchain behavior, we care about the following situations: 1. Running state transition functions and checking the results of applying a block 2. Making assertions about the chain state and chain metadata, such as the head block, validator balances, and consistency with the specification 3. Asserting results of forkchoice in complex fork scenarios Each package in the Prysm repo has different responsibilities. Let's see how we accomplish these assertions today in our Go unit tests and from there, figure out what we're missing: ### Anatomy of Prysm Chain Tests **State Transition Assertions** The state transition function is pure, so we typically initialize a genesis state, and process a series of pre-formed blocks through a pipeline to get some assertions about post-states. This is pretty straightforward, but somewhat repetitive in the codebase: ```go func TestExecuteStateTransition_FullProcess(t *testing.T) { // Set up a deterministic genesis state. beaconState, privKeys := util.DeterministicGenesisState(t, 100) // Update the state slot and retireve some metadata. beaconState.SetSlot(params.BeaconConfig().SlotsPerEpoch-1)) oldMix := beaconState.RandaoMixAtIndex(1) beaconState.SetSlot(beaconState.Slot()1)) epoch := time.CurrentEpoch(beaconState) randaoReveal err := util.RandaoReveal(beaconState, epoch, privKeys) beaconState.SetSlot(beaconState.Slot()-1) // Advance the chain some empty slots. nextSlotState := transition.ProcessSlots(beaconState.Copy(), beaconState.Slot()+1) parentRoot := nextSlotState.LatestBlockHeader().HashTreeRoot() proposerIdx := helpers.BeaconProposerIndex(nextSlotState) // Create a new block to advance the chain with. block := util.NewBeaconBlock() block.Block.ProposerIndex = proposerIdx block.Block.Slot = beaconState.Slot() + 1 block.Block.ParentRoot = parentRoot[:] block.Block.Body.RandaoReveal = randaoReveal // Compute state root and signature. stateRoot := transition.CalculateStateRoot(beaconState, block) block.Block.StateRoot = stateRoot[:] sig := util.BlockSignature(beaconState, block.Block) require.NoError(t, err) block.Signature = sig.Marshal() // Process block through state transition. beaconState = transition.ExecuteStateTransition(beaconState, block) // Assert results from the post-state, namely RANDAO data. assert.Equal(t, params.BeaconConfig().SlotsPerEpoch, beaconState.Slot(), "Unexpected Slot number") mix := beaconState.RandaoMixAtIndex(1) assert.DeepNotEqual(t, oldMix, mix, "Did not expect new and old randao mix to equal") } ``` We perform a lot of manual building of blocks, advancing a chain artificially, and performing assertions. Although repetitive, it gives us fine-grained control over the system in a test environment. **Forkchoice Testing** For forkchoice testing, we tend to create complex block trees using fake block hashes []byte{1}, []byte{2}, []byte{3} as examples for easy assertions. We tend to use the forkchoice store itself to do this and then verify values such as weights, the current head, and are able to also add votes to our current store computation. Here's what this looks like: ```go func TestFFGUpdates_TwoBranches(t *testing.T) { balances := []uint64{1, 1} f := setup(0, 0) ctx := context.Background() r, err := f.Head(context.Background(), balances) require.NoError(t, err) assert.Equal(t, params.BeaconConfig().ZeroHash, r, "Incorrect head with genesis") // Define the following tree: // 0 // / \ // justified: 0, finalized: 0 -> 1 2 <- justified: 0, finalized: 0 // | | // justified: 1, finalized: 0 -> 3 4 <- justified: 0, finalized: 0 // | | // justified: 1, finalized: 0 -> 5 6 <- justified: 0, finalized: 0 // | | // justified: 1, finalized: 0 -> 7 8 <- justified: 1, finalized: 0 // | | // justified: 2, finalized: 0 -> 9 10 <- justified: 2, finalized: 0 // Left branch. state, blkRoot, err := prepareForkchoiceState(context.Background(), 1, indexToHash(1), params.BeaconConfig().ZeroHash, params.BeaconConfig().ZeroHash, 0, 0) require.NoError(t, err) require.NoError(t, f.InsertNode(ctx, state, blkRoot)) state, blkRoot, err = prepareForkchoiceState(context.Background(), 2, indexToHash(3), indexToHash(1), params.BeaconConfig().ZeroHash, 1, 0) require.NoError(t, err) require.NoError(t, f.InsertNode(ctx, state, blkRoot)) state, blkRoot, err = prepareForkchoiceState(context.Background(), 3, indexToHash(5), indexToHash(3), params.BeaconConfig().ZeroHash, 1, 0) require.NoError(t, err) require.NoError(t, f.InsertNode(ctx, state, blkRoot)) state, blkRoot, err = prepareForkchoiceState(context.Background(), 4, indexToHash(7), indexToHash(5), params.BeaconConfig().ZeroHash, 1, 0) require.NoError(t, err) require.NoError(t, f.InsertNode(ctx, state, blkRoot)) state, blkRoot, err = prepareForkchoiceState(context.Background(), 4, indexToHash(9), indexToHash(7), params.BeaconConfig().ZeroHash, 2, 0) require.NoError(t, err) require.NoError(t, f.InsertNode(ctx, state, blkRoot)) // Right branch. state, blkRoot, err = prepareForkchoiceState(context.Background(), 1, indexToHash(2), params.BeaconConfig().ZeroHash, params.BeaconConfig().ZeroHash, 0, 0) require.NoError(t, err) require.NoError(t, f.InsertNode(ctx, state, blkRoot)) state, blkRoot, err = prepareForkchoiceState(context.Background(), 2, indexToHash(4), indexToHash(2), params.BeaconConfig().ZeroHash, 0, 0) require.NoError(t, err) require.NoError(t, f.InsertNode(ctx, state, blkRoot)) state, blkRoot, err = prepareForkchoiceState(context.Background(), 3, indexToHash(6), indexToHash(4), params.BeaconConfig().ZeroHash, 0, 0) require.NoError(t, err) require.NoError(t, f.InsertNode(ctx, state, blkRoot)) state, blkRoot, err = prepareForkchoiceState(context.Background(), 4, indexToHash(8), indexToHash(6), params.BeaconConfig().ZeroHash, 1, 0) require.NoError(t, err) require.NoError(t, f.InsertNode(ctx, state, blkRoot)) state, blkRoot, err = prepareForkchoiceState(context.Background(), 4, indexToHash(10), indexToHash(8), params.BeaconConfig().ZeroHash, 2, 0) require.NoError(t, err) require.NoError(t, f.InsertNode(ctx, state, blkRoot)) // With start at 0, the head should be 10: // 0 <-- start // / \ // 1 2 // | | // 3 4 // | | // 5 6 // | | // 7 8 // | | // 9 10 <-- head r, err = f.Head(context.Background(), balances) require.NoError(t, err) assert.Equal(t, indexToHash(10), r, "Incorrect head with justified epoch at 0") ``` The set up is complex, and requires a lot of manual plumbing in order to create a forked chain, but gives the developer full control of the system and it is self-contained to the forkchoice package. **Asserting Chain State / Metadata** When asserting chain state, in the blockchain service, we _do_ have the capabilities of initializing a real blockchain service, real forkchoice store, and database in order to process a pipeline of blocks. ```go func TestStore_OnBlockBatch_NotifyNewPayload(t *testing.T) { ctx := context.Background() // Set up a real database beaconDB := testDB.SetupDB(t) // Set up a real forkchoice store fc := doublylinkedtree.New() // Set up a real blockchain service opts := []Option{ WithDatabase(beaconDB), WithStateGen(stategen.New(beaconDB, fc)), WithForkChoiceStore(fc), } service, err := NewService(ctx, opts...) require.NoError(t, err) // Initialize genesis data st, keys := util.DeterministicGenesisState(t, 64) require.NoError(t, service.saveGenesisData(ctx, st)) bState := st.Copy() // Initialize 4 blocks, run them through state transition functions var blks []interfaces.SignedBeaconBlock var blkRoots [][32]byte blkCount := 4 for i := 0; i <= blkCount; i++ { b := util.GenerateFullBlock(bState, keys, util.DefaultBlockGenConfig(), types.Slot(i)) bState = transition.ExecuteStateTransition(ctx, bState, b) root := b.Block.HashTreeRoot() service.saveInitSyncBlock(ctx, root, b) blks = append(blks, b) blkRoots = append(blkRoots, root) } err = service.onBlockBatch(ctx, blks, blkRoots) require.NoError(t, err) } ``` However, we do not perform as many assertions aside from happy path checks. We lack coverage in many of these block processing functions because the complexity of testing the unhappy paths is too much. We do not make as many assertions on the results of block processing, and have a hard time building up negative tests. ## What our test harnesses are missing Our current test harnesses do a decent job at **separating concerns**. Namely, forkchoice test setups don't require us setting up a genesis state or real attestations, as forkchoice just operates on an abstract notion of "votes". The state transition tests don't require us to setup a blockchain, because they are pure, and blockchain tests tend to be more of a "black box". However, our test harnesses fail to capture the _timing_ of events and the synchronous, distributed system nature of a node. For example, we should be able to: 1. Control a set of validators submitting messages to a node at different intervals in a slot and control their latency 2. Observe the status of caches and inner functioning of the blockchain service at different points in time For this, we rely on our end-to-end test suite, which is error-prone, takes ages to run, and is a lot more of a black box than our unit tests. Can we do better? Let's think about why the [cache bug](https://github.com/prysmaticlabs/prysm/pull/11567) discovered by Terence was hard to test. First of all, the bug relies on skip slots happening in the chain, and on validators submitting messages at different intervals within a slot or at epoch boundaries. The bug Terence discovered would lead to a cache miss and give the proposer less time to prepare a block as they would not have a cached payload ahead of time and would have to go through normal processing. ## Brainstorming a better test harness Thinking out loud, it would be really nice if we had a testutil we can initialize and have a lot of granular control over a blockchain with minimal mocking. This is similar to how we call `testutil.SetupDB(t)` and you get a real, boltDB instance that is full-featured and works without issues. Here are some features that would be nice to see in this harness: - Have a chain running with slots and time advancing in the background - Ability to feed blocks into this chain at specific times, slot intervals, and simulate some latency (although adding non-determinism is a bad move in unit tests, by making it such that we can select intervals of a slot for submission rather than timestamps, we could alleviate this risk) - Ability to create forked chains with very easy configurations, perhaps in a way that is easy to read - Ability to have fake validator "agents" running in a goroutine and submitting messages to the chain, and make it easy to control their behavior, latency, and timings - Ability to run state transitions and advance a chain from blocks - Ability to trigger skip slots at various intervals or granular points - Ability to inspect the chain state, metadata, and perform assertions at specific times, for example, check caches, check validator balances at each N slot, etc. - Ability to control the execution client attached to the system. This will need to be mocked A problem with these ideas is that if we want to really tick slots, our test will take a long time to complete, even if we go down to two seconds per slot. Can we abstract the concept of time and slot intervals? How could we get this to work in a unit test? ## Design Overview ```go= { chain := testutil.NewSimulatedChain(t) .UpToEpoch(5) .SkipEveryNSlots(2) // Skip every 2nd slot .NumValidators(64) // Add granular control of when blocks are submitted // and which proposers submit them late. .LateBlocks(...) // Add the ability to control specific actions of proposers // such as what intervals they submit their slots in, whether // or not they should submit slashable offenses, etc. .ProposerBehaviors(...) // Similar to e2e, each slot check X things about the chain, // caches, fork choice. .Evaluators(...) .Populate() defer chain.Cleanup() // Could be done as part of test cleanup chain.Start() // Runs a chain in the background, non-blocking } // Evaluator API func proposerEvaluator(t *testing.T, chain *ChainHarness) { proposer := chain.ProposerAt(chain.HeadSlot()) assert.Equal(t, true, proposer.EffectiveBalance == 32) } ``` We can take a lot of inspiration for how our e2e suite uses evaluators at different times in the lifecycle of a chain, and create assertions based on this. Questions: what else should this harness have? How can we really test these timing scenarios effectively in an integration test? How can we find a middle-ground between e2e and unit tests? Perhaps these can run separately in CI than normal unit tests. Can we set this up with Bazel? ## References - https://github.com/prysmaticlabs/prysm/pull/11567
Last changed by  
 
Prysmatic Labs
eth2 development team
321

Read more

Migrating Prysm to Slog

Contextual logging:

12/6/2023
Running BOLD Challenges on Sepolia

This guide will set up

11/17/2023
Generalized History Commitment

Today in BOLD, challenges can have subchallenges, and we have a total of 3 levels. We do this because at each challenge, validators have to commit to N state roots of a block’s history, which can be very expensive to compute. For example, computing 2048 machine hashes for the execution of a block and collecting them takes around 5 minutes on a modern MacBook Pro.

8/16/2023
If I Had a Reset Button...

If you knew what you know now, and you were starting a new CL from scratch, what structures would you use? Let me start myself to set the tone, you can make it language specific (for example Go and avoid templated methods) but I am not really interested in the small details but rather the main components of the critical parts of code. Typical questions I’m thinking about areWould you use a custom allocator for the beacon-state/validators/attestations structures?Would you use a struct like object for them or would you use a functional style object. The typical situation is this:We have a buffer with 8192 roots in the BeaconState, but these roots are sequential, if state for N had up to root r, the state for N+1 will share essentially all of the roots as that for N except at one point where it would have changed one. If you store vectors and then you want to share the common part then you are lead to structures like the multi-value slice or similar structures, where you have a base slice and then somewhere else you store the diffs (or some mathematically equivalent statement). Another way to store the object is as a linked list (with some additions to make it into a round buffer) and when you change one element you just change a single node on that list, someone that holds the pointer to the current node will resolve to the current full round buffer, while someone that holds the pointer to the previous-to-modified node, will resolve (by traversing the links from it) to the previous full round buffer. If you have a fork this still works, you just need to add a few pointers on either side of the fork. The point is that from the point of view of the structure itself, there’s no “base state” and some diffs, and rather you can just delete the pointer and let GC in Go, or the Destructuror in C++ deal with the rest. No need to be pruning yourself from a multi-value slice for example. On the other side, the multi-value slice would incur in way less memory fragmentation if you are dealing with a single state in the happy case.There are variations of this for all structures in the Beacon State and there are benefits and cons for each one of them.What sort of thread management would you use? (I think here there won’t be a disagreement and just relying on the Go runtime is fine)What sort of fork management would you use? I originally thought about templating, something like having the Fork being the template parameter for a BeaconBlock structure. In C++ at least, the best structure seems to be vertical polymorphism in the structures themselves like having class AltairBlock : public Phase0Block but not use methods of these structures, but rather pure functions like attestations(block Block) []Attestations and just let AltairBlock and Phase0Block satisfy the concept Block, which in this case would mean they both have a member attestations so you can just simply implement that function as return block.attestations. In Go this seems to be quite complicated, templates as far as I could read do not allow us to do this, and interfaces themselves we have them, but we still are forced to repeat the underlying structures completely for AltairBlock and Phase0Block instead of invisibly inheriting and obscuring the changed ones.Would you trade security and correctness for performance? I think this is a funny one: if you’re coding for 0.1% of the market, then you can do lots of shit like only keeping the last two states and break if there’s a depth 3 reorg. That’s an extreme case, but that’s probably not so far away from what Erigon’s CL may be, I would bet that they just follow blindly whatever is voted. But besides the extreme cases, how about Optimistic sync? I am pretty sure we are the only ones that considered a recursive system in which a bunch of branches turned out to be invalid. I am convinced that you can simplify a lot the optimistic status dealing if you just ignore correctness in a bunch of cases, with no real danger to the network.How would you deal with the engine? Would you change the interaction with the engine if you knew that ALL your users are using MEV-Boost? would you optimize the builder code in detriment of the local code? for example, it could be perfectly valid to not wait for local blocks and not request them in advance until it’s late in the block in case we know we are not resource-heavy. Not wait for the result of FCU is also good, since we are guaranteed already from a VALID response from NewPayload (this is something we could do right now on Prysm).

8/2/2023
Read more from Prysmatic Labs
Published on HackMD