EPF6 - Week 7 Updates

# EPF6 - Week 7 Updates ## tl;dr - After discussing with mentors, I decided to start implementing the second option, ["Marshal everything, and treat as string of bytes"](https://hackmd.io/@junsong/Byr3_lfPxl#Option-2-Marshal-everything-and-treat-as-string-of-bytes--Byte-Parsing-Engine). - Proof-of-Concept ([PR](https://github.com/syjn99/prysm/pull/1)) is done. Another [write-up](https://hackmd.io/@junsong/B1v3x4wwle) that explains technical aspects of my implementation. - Go struct tag is crucial to determine its SSZ type. I studied in an empirical way with SSZ projects written in Go, and realized `ssz-size` and `ssz-max` tags are the keys for it. One more [write-up](https://hackmd.io/@junsong/H101DKnwxl). ## Details This week was also another solid week for me. We have a [great proposal](https://github.com/eth-protocol-fellows/cohort-six/blob/master/projects/ssz-ql-with-merkle-proofs.md) now, and need some details in **implementation side**. ### Lay out a big picture ![image](https://hackmd.io/_uploads/HJ4O0p3wgg.png) > [Link](https://discord.com/channels/476244492043812875/1387734369527136297/1399373429203009597) to this message. [Last week](https://hackmd.io/@junsong/SyqSBVQDgl) I suggested two options for implementation. Mentors were aligned with me: they also prefer Option 2 which deals with a **serialized** SSZ object. ![image](https://hackmd.io/_uploads/ByV6Apnvgg.png) > [Link](https://discord.com/channels/476244492043812875/1387734369527136297/1399418641698525316) to this message. Bastin questioned about the feasibility of Option 2, so I planned to do a PoC. PoC would demonstrate one round-trip test: constructing a SSZ object, query with actual path, and compare the result with the object. ### Proof-of-Concept completed (and to be continued...) ![image](https://hackmd.io/_uploads/H1q5yR2wex.png) > [Link](https://discord.com/channels/476244492043812875/1387734369527136297/1400046343619346434) to this message. I finished the [PoC](https://github.com/syjn99/prysm/pull/1) and wrote an [doc](https://hackmd.io/@junsong/B1v3x4wwle) about the implementation details. I put all details in the write-up, so my weekly update can be neat. By the way, **I prefer organizing my thoughts as an write-up because almost all communications are asynchronous**. Write-ups are helpful to provide the missing context for readers. Radek likes my PoC and write-up and he raised several questions about unmarshalling the result. > Extract only needed part from bytes slice, and then unmarshal it if it is needed. In PoC, I just compared two byte slices (assert.DeepEqual), but it is trivial to unmarshal a bytes slice into wanted struct. I wrote unmarshalling is "trivial", but he wanted me to clarify it. I needed to [add more fields](https://github.com/syjn99/prysm/blob/750d0a46cf579460703f74165fd7c06fd5a22430/ssz-query/ssz_info.go#L68) in `sszInfo` struct for unmarshalling, and reinforced my round-trip test with it. ### Empirical study with great projects PoC only supports querying the fixed size types. To expand the scope, it is inevitable to handle the various sized types like `List` types. It seems Go struct tags like `ssz-size` and `ssz-max` are the key, so I studied how other (legendary) projects do. There are several projects that provide `sszgen` feature: - [ferranbt/fastssz](https://github.com/ferranbt/fastssz) - [karalabe/ssz](https://github.com/karalabe/ssz) - [OffchainLabs/methodical-ssz](https://github.com/OffchainLabs/methodical-ssz) And I guess (Please correct me if I'm wrong 😢) - `fastssz` was the first project with `sszgen`. - Thus `fastssz` starts to use tags with prefix `ssz-`. - Other projects that kicked off later follow the convention of `fastssz`. - At this moment, using `ssz-size` and `ssz-max` tags are a de facto standard. Anyway, I learned a lot by reading those projects, and figured out that the points that I learnt should be applied to my future work. Also, I was curious about Prysm's automated scripts (and it's related to my PoC), which finally [organized as a part of article](https://hackmd.io/@junsong/H101DKnwxl#Background-How-are-pbgo-files-generated-And-How-does-sszgen-works). ## What's next - The current [response schema](https://hackmd.io/@junsong/rkAN9lIIxx#Query-Result) doesn't return the actual JSON object: it only represents the data as 32-byte hash value. Don't we need to add more fields that contains the actual object? - Based on my latest [write-up](https://hackmd.io/@junsong/H101DKnwxl), I have to handle various sized types in my PoC. ## Resources - My write-ups this week. - [SSZ-QL: Proof-of-Concept](https://hackmd.io/@junsong/B1v3x4wwle) - [Struct Tags used in `sszgen`](https://hackmd.io/@junsong/H101DKnwxl)