# Feb 2nd Standup

### v5 updates
#### Database changes
* `uncle_root` => `uncles_hash`
* `node_id` => `node_ids`
* `state_cids` and `storage_cids` refactored to index only leaf nodes
* `node_type` changed to `removed` flag
* PK uses `leaf_key` instead of `path`
* No longer need to find the latest `state_cids.state_path` corresponding to a `state_cids.state_leaf_key` in order to JOIN on `storage_cids`
* `state_accounts` table removed; relevant fields moved into `state_cids` table
* since all `state_cids` records now map 1-to-1 with a state account object
* `eth_probe` tables adapted and added to the ipld-eth-db schema
* Remove unused/broken `known_gaps` table
* https://github.com/cerc-io/go-ethereum/compare/v1.10.26-statediff-4.3.3-alpha...v1.10.26-statediff-4.3.4-alpha

#### Geth changes
* Removed log trie materialization and indexing
* Removed processing of intermediate nodes into `state_cids` and `storage_cids` (their IPLD objects are still inserted into `public.blocks`)
* Record headers from different sources
* Fix internalized leaf node issue
* This was greatly simplified by the decision to no longer index intermediate nodes.
* Rewrite tests
* Problem with our statdiff builders tests is that there is no other tool/code to generate a "standard" statediff object for which to test against
* So you have to build small MMPTs with a ChainMaker and write the expected corresponding statediff models out manually
* See https://github.com/cerc-io/go-ethereum/blob/v1.10.26-statediff-v4/statediff/builder_test.go
* Remove some unused IPLD code that will need to be replaced with go-ipld-prime down the line

#### TODO
1. I still have a statediff builder test that is failing in a unreliable manner
2. Finish propagating the changes to the rest of the FIN stack
1. ipld-eth-server
2. ipld-eth-state-snapshot
3. ipld-eth-db-validator
5. eth-statediff-service
6. eth-ipfs-state-validator
3. Integration testing
4. Benchmarking
5. Ensure Vulture/Carrion is ready for deployment

#### Discuss
* Renaming tables (remove/replace "cids" suffix)
* eth_probe
* A little reluctant to be including the changes to support eth_probe in this upgrade, as we are not currently in a place to use or even test these changes
* Probability that breaking changes will need to be made is high
* Transforming v4 data (from csvs or from pg) into v5 data instead of going levelDB => v5
* The database schema changes are relatively minor
* The schema changes don't involve data that is not already present in v4
* But the transformation for state_cids and storage_cids is still not simple
1. Drop all records that are not node_type = "leaf"
2. Drop the removed columns
3. For all state_cids leaf_records, find the corresponding state_accounts record and combine records
4. Main issue is finding the internalized leaf nodes
5. Find all branch nodes
6. Fetch the ipld block linked to from those records
7. RLP decode the ipld block
8. Iterate over all 16 children in the branch node, checking if any of them are actually internalized leaf nodes

#### Other updates
* Feedback on our filecoin indexing grant; grant has been ammended to fit within the new budget
* Marko and Bez have invied myself and Roy to join the storage working groups on Wednesdays, will be attending this going forward. Good way to keep track of potentially breaking upstream changes and also help maintain good relations with the ICF/cosmos community.
* Roy is back today, given the point I am at in the dev cycle instead of bringing him on to help with this I am goign to have him begin work on ipld-eth-db-validator and other anti-fragility stuff