# Week 14 Update: Odinson

It was a not so productive week as I was having my midsems, which are still going on. But after my last week's update, got some reviews from Michael on Monday, and then Dapplion was back and he also went through the updates, checked in on my PR and left some reviews which we have been discussing in the group, will be talking about it below. Apart from that, made a PR on an issue regarding adding a [max delay](https://github.com/sigp/lighthouse/pull/8067) to column reconstruction. Was unable to join the office hours as I had my college, but watched at a later time and it was a great session!
Lastly, had some talks with Dave from libp2p and regarding work on the universal connector.

## Work for the week

Michael looked at the logs I provided. As I mentioned [here](https://hackmd.io/vWFGJxwjQ3KIvNeCghzBtw?stext=3884%3A161%3A0%3A1758728304%3AbGQ--w&both=), one issue was that the `state-cache-size` was very small, around 2 to 3. So one thing he wanted to check was how it performs if we increase the limit, as the subtrees MemorySize calculation was set up in a way that it consumed a lot of memory for the first state, and it decreased after that. Another thing was to look into the fact that when I was spamming HTTP queries, it timed out after some time, so if that was my branch specific or a generalized problem.

Apart from that, dapplion also came back, and left a few comments, mainly regarding the use of timer instead of stopping the metric explicitly, adding a histogram to track per-state measurement times, moving the recomputation off the main cache path to avoid blocking, making the batch clamp values constants, and simplifying the pruning to a single bounded cull per cycle rather than looping with repeated measurements.

I am yet to look into the comments of dapplion and test those out, but other than that, i did run the lighthouse logs in `unstable` branch, finding that the timeout for HTTP queries was a generalised stuff.

odinson lighthouse$curl -s http://127.0.0.1:5054/metrics | egrep -i 'store_beacon_state_cache_memory_size|beacon_state_memory_size_calculation_time|store_beacon_state_cache_size'

# HELP store_beacon_state_cache_size Current count of items in beacon store state cache
# TYPE store_beacon_state_cache_size gauge
store_beacon_state_cache_size 128
odinson lighthouse$HEAD=$(curl -s localhost:5052/eth/v1/beacon/headers/head | jq -r '.data.header.message.slot')
LOW=$((HEAD-1024))

seq -f %.0f "$LOW" "$HEAD" | xargs -P 8 -n1 -I{} \
  curl -sS -m 30 -o /dev/null \
  "http://127.0.0.1:5052/eth/v1/beacon/states/{}/committees"

seq -f %.0f "$LOW" "$HEAD" | xargs -P 8 -n1 -I{} \
  curl -sS -m 30 -o /dev/null \
  "http://127.0.0.1:5052/eth/v1/beacon/blob_sidecars/{}"

curl: (28) Operation timed out after 30006 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30006 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30001 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30004 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30005 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30006 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30006 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30006 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30005 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30006 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30005 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30006 milliseconds with 0 bytes received
curl: (28) Operation timed out after 30011 milliseconds with 0 bytes received

And set the `--state-cache-memory-size` to around 2GB, 3GB and got to find what we expected, that gradually a lot of the `state-cache-size` was stored as we increased the memory size.
Here are the logs for 2GB

odinson lighthouse$curl -s http://127.0.0.1:5054/metrics | egrep -i 'store_beacon_state_cache_memory_size|beacon_state_memory_size_calculation_time|store_beacon_state_cache_size'

# HELP store_beacon_state_cache_memory_size Memory consumed by items in the beacon store state cache
# TYPE store_beacon_state_cache_memory_size gauge
store_beacon_state_cache_memory_size 2127643883
# HELP store_beacon_state_cache_size Current count of items in beacon store state cache
# TYPE store_beacon_state_cache_size gauge
store_beacon_state_cache_size 50
odinson lighthouse$curl -s http://127.0.0.1:5054/metrics | egrep -i 'store_beacon_state_cache_memory_size|beacon_state_memory_size_calculation_time|store_beacon_state_cache_size'

# HELP store_beacon_state_cache_memory_size Memory consumed by items in the beacon store state cache
# TYPE store_beacon_state_cache_memory_size gauge
store_beacon_state_cache_memory_size 2111360549
# HELP store_beacon_state_cache_size Current count of items in beacon store state cache
# TYPE store_beacon_state_cache_size gauge
store_beacon_state_cache_size 53
odinson lighthouse$curl -s http://127.0.0.1:5054/metrics | egrep -i 'store_beacon_state_cache_memory_size|beacon_state_memory_size_calculation_time|store_beacon_state_cache_size'

# HELP store_beacon_state_cache_memory_size Memory consumed by items in the beacon store state cache
# TYPE store_beacon_state_cache_memory_size gauge
store_beacon_state_cache_memory_size 2071193673
# HELP store_beacon_state_cache_size Current count of items in beacon store state cache
# TYPE store_beacon_state_cache_size gauge
store_beacon_state_cache_size 50

Next, for 3GB, where we see all the 128 states are stored in the cache within 3GB

odinson lighthouse$curl -s http://127.0.0.1:5054/metrics | egrep -i 'store_beacon_state_cache_memory_size|beacon_state_memory_size_calculation_time|store_beacon_state_cache_size'

# HELP store_beacon_state_cache_memory_size Memory consumed by items in the beacon store state cache
# TYPE store_beacon_state_cache_memory_size gauge
store_beacon_state_cache_memory_size 2713055514
# HELP store_beacon_state_cache_size Current count of items in beacon store state cache
# TYPE store_beacon_state_cache_size gauge
store_beacon_state_cache_size 128
odinson lighthouse$curl -s http://127.0.0.1:5054/metrics | egrep -i 'store_beacon_state_cache_memory_size|beacon_state_memory_size_calculation_time|store_beacon_state_cache_size'

# HELP store_beacon_state_cache_memory_size Memory consumed by items in the beacon store state cache
# TYPE store_beacon_state_cache_memory_size gauge
store_beacon_state_cache_memory_size 2720084050
# HELP store_beacon_state_cache_size Current count of items in beacon store state cache
# TYPE store_beacon_state_cache_size gauge
store_beacon_state_cache_size 128

Other than that, created a smol PR regarding adding a generalized Reconstruction deadline to column reconstruction, compared to the fixed 3 seconds delay that was previously present, to make it a function of slot duration. This is the [PR](https://github.com/sigp/lighthouse/pull/8067), where I modified the `RECONSTRUCTION_DEADLINE` to a 1/4 fraction, and then converted the slot_duration to millis and made the delay one-fourth of the slot duration.

## Resources

1. Project [PR](https://github.com/sigp/lighthouse/pull/7803)
2. Max Reconstruction delay as a function of slot time [PR](https://github.com/sigp/lighthouse/pull/8067)

## Conclusion

Next week it will be a little challenging to put time to work as my exams still continue, and there is this festive season, called Durga Puja, but I am looking forward to joining the office hours having Kolby ML from Ream, and addressing the comments Dapplion mentioned and me and Michael discussed about