# Issues in Block Proposal - GetBlock() in particular

It was reported by users that the blocks are not proposed leading to "missed proposals". It was also reported that the blocks are proposed late (after 4 seconds in the slot) leading to less propagation of the block acoss the network and therefore lesser attestations. This was observed more if the proposal duty was on the first slot of the epoch.

On the outside the code looked to be more synchronous in proposing blocks and thus this [issue](https://github.com/prysmaticlabs/prysm/issues/8943) was created.

## The code path

1) Validator pick up the "block proposer" duty during the slot and calls the beacon node for the block to propose in ProposeBlock->validatorClient.GetBlock().
2) Beacon node calls the heavy function GetBlock() which does the following
- Get the head state and its root
- Process the slots
- Collect the eth1Data to add in the block
- Pack ETH1 deposits which have not been included in the beacon chain
- Pack aggregated attestations which have not been included in the beacon chain
- Create the block using the data obtained in the above steps
- Process the new block and run it across the new state
3) Validator gets the new block and signs it. It thens send it back to beacon node to gossip the block.
4) Beacon node gossips this new block in the beacon_block channel.

## The Problem

The heavy function in the code path was 2) GetBlock() which does so many things to create the block. On looking at the functionand doing some debugging it was clear that there are two types of problems.

1) Preliminary fixes: some functions can be parallized or omitted to achieve speed.
2) Performance issues: some functions had inherent performance issues and were taking long to execute.

## Prilminary fix

The preliminary fixes are addressed in this [PR GetBlock() Optimizations](https://github.com/prysmaticlabs/prysm/pull/9541)
Two things were done there
- Omit a redundant BlockByTimeStamp for earliestValidTime as it is not used in the code for now.
- Parallize deposits() and packAttestations() as hey are independent

## Performance Issues

There were quite a number of performance issues that were found using traces. The major ones are listed below.

### Setup
- Two VMs, one for running the node and other for jagger.
- both VMs have same config
- VM Config
- 2 vCPU
- 4 GB RAM
- 80GB SSD disk
- good bandwidth, i dont know how much :-)

### beacon node options

- beacon-chain --http-web3provider=https://mainnet.infura.io/v3/XXXXX --tracing-endpoint=http://45.76.153.183:14268/api/traces --trace-sample-fraction=1 --subscribe-all-subnets --enable-tracing --enable-next-slot-state-cache --log-file=~/logfile.log

### Issue 1 - eth1DataMajorityVote
- **BlockByTimeStamp()**: This is a search function which tries to minimize the access rate by using a header cache during normal operation. This function somehow misses the headerCache and proceeds to fetch the header from the eth1POW chain (external call) which is inside the search iteration. Even if the call takes 200ms every time, it adds up and crosses a second easily.
- **scenario 1**
- **scenario 2**)

- **DepositsNumberAndRootAtHeight()** - This clearly the **biggest bang for the buck**. Even though the trace shows up here in the trace, the log shows that the OnBlock()->insertFinalizedDeposits() is the actual culprit.
- **scenario 1**
- **scenario 2**
- **scenario 3**

### Issue 2 - stategen->StateBySlot() / StateByRoot()
- This was observed few times that the loadStateByRoot() takes several seconds and the proposal will be lost. Bolt might be the culpreit here as it might take time to get state that is not in hoStateCache or the epochCache. The **second major time saving** is in this fix.


### Issue 3 - computeStateRoot->ProcessSlots()
- **ProcessEpochPreCompute()** - This is the culprit for taking time in epoch boundraries. In several occacions this was seen taking more than a second.
- **scenario 1**
- **scenario 2**

### Issue 4 - computeStateRoot->CalculateStateRoot->ProcessSlots()
- **skipSlotCache.Get()** - This is the most strangest one. skipSlotCache is a simle cache and its get takes more than a second. The only reason i can think is that the lock inside Get() is held by someone for a far more long time.
- **scenario 1**
- **scenario 2**

### Issue 5 - HashTreeRoot
It is observed that the hashing algorithm is taking some time to calculate the root. The VM had support for AVX2 and AVX512. May be the hypervisor didnt allow SIMD instruction set, i am not sure of that. But this happens in this setup. This is also captured by potus and Nishant has created a issue for this [here](https://github.com/prysmaticlabs/prysm/issues/9389)
- **scenario 1**
- **scenario 2**

### Issue 6 - packAttestations

I could not get the trace for this, but the logs shows that packing attestations also takes about a second some times. This should be kept for future
- **scenario 1**
- **scenario 2**