Week 16 - HackMD

# Week 16 ## Recap This week was mainly spent on switching SSZ libraries to the up-to-date and continually maintained Teku internal SSZ library. I was able to learn the general design of the library in addition to almost all of the necessary internal parts to design decoding schemas for every packet. I then began adding in the new decoding logic and was able to switch everything to the new library. This proved successful for all but one packet as all others were able to pass every Portal Wire unit test vector that I had implemented in the previous week. Which was also relieving news and confirmed my suspicion that the issues I was experiencing were the result of an outdated library. The only packet that I couldn't get working was the CONTENT packet. The CONTENT packet is the only packet to differ in SSZ storage type from the specified Portal Wire packets and uses the SSZ Union type. When using the Teku Union design I continue running into unit test inconsitencies that show something isn't quite right with the 4-byte offset being produced. I have tried a few ways to fix this but I believe it is more so a matter of just finding the correct solution instead of anything library related. If all else fails I can also just directly parse the packet into three different SSZ Container types based on the Union selector, though it would be prefferable to use the built in Teku SSZ Union type to handle everything. ## Learning My learning this week was almost purely just sifting through and reading the Teku internal SSZ library to understand how it works. A large portion of this learning was just trying to understand how the systems in the library could be used, as not all of them are actually used in the Teku client. ### Concepts #### Teku Internal SSZ I was able to look through the code in the Teku internal SSZ library and attain a fairly solid understanding of how it works. This included how I could initialize/use the various SSZ types that it offered as well as how to utilize its recursive encode/decode design. The library is designed in such a way that all types both primitive and complex follow a schema. This schema then allows each type to be read recursively so that types contained within types are properly encoded/decoded. I had to find a way to adapt these schemas and build on top of them to create my own schemas. I found that most of the packets could be designed within the Container schemas. The only packet I have troubles with is the CONTENT packet as it is specified to be contained in the SSZ Union type. However this type does not behave in the same way as containers and there is no example of its usage in the Teku client outside of tests, so it is something I am still figuring out. ##### Resources 1. https://github.com/Consensys/teku 2. https://github.com/Consensys/teku/tree/master/infrastructure/ssz #### Portal Wire Specifications Through my shift in SSZ libraries I continued to refer to the Portal Wire Specifications to ensure that everything matched up properly and was working as intended. The vector tests were once again useful in making sure that generated byte offsets were of the correct value. ##### Resources 1. https://github.com/ethereum/portal-network-specs/blob/master/portal-wire-protocol.md 2. https://github.com/ethereum/portal-network-specs/blob/master/portal-wire-test-vectors.md ## Tasks The implementation work I did this week was almost exclusively switching the SSZ libraries for the Tuweni SSZ library to the Teku internal library. I was able to successfully switch everything over and set up the appropriate infrastructure for every packet. In doing so I was able to get all but one packet to pass every serialization and deserialization unit test. This confirmed that the issues I was experiencing last week were likely based on an expired library. The only downside to switching to the Teku SSZ library is that its use was drastically more complicated. Requiring the design of an individual SSZ schema for each packet type which adds an additional layer of complexity to the client design. Because the library was designed primarily for use with Teku, it also is missing some quality of life additions in regards to certain SSZData types. Currently I have designed client side SSZ processing tools that circumvent this, though adding them straight to the Teku library might be something to consider as a side project in the future. When adapting to the new library, I also took the time to add a more encapsulated design to the Portal Wire Packets, ensuring that the data the client will be using is presented exactly how it will be used. Opposed to storing most values as their raw byte form. This change then required an update in the unit tests to reflect the values in the way the client would actually use them. The only packet that I could not quite get working is the CONTENT packet as it is the only packet to use the SSZ Union type. The Union schema behaves a lot differently than the Container schema and so far I have not been able to get it to produce the correct byte offset values that the Portal Wire specs need. https://github.com/meldsun0/samba/commit/6b624540f8d625550e33d3ef2cc3c226397d275f https://github.com/meldsun0/samba/commit/98a653732c74e85b6f8386ba592ba82934353b67 ## Week 17 Plan Going into Week 17 I am looking forward to finally moving on from serialization. Outside of fixing the one problem I was experiencing spefically with the CONTENT SSZ union type from the Teku internal library I can move on to actually putting the serialization/deserialization to use. Throughout this portion I will be working closely with Meldsun to ensure our work can merge together without conflicts and the sepparate systems we are working on can interact when complete. The tasks I am looking to tackle this week are: - Finish the CONTENT serialization in regards to the Teku SSZ library Union type, either by way of determining what is causing my offset issue or just directly parsing the packet for the appropriate type to filter into the respective schema containers - Merge testing and library transfer feature branch and account for any experienced conflicts in the code from reorganization - Configure the PING/PONG handlers to work directly with the PortalWireMessageHandler and the corresponding outgoing request system - Begin work on the FINDNODES handler in collaboration with the DHT system being designed by Meldsun Unlike this week, the coming week should be much less linked to the ambiguity of learning a new library and should follow much more closely as just implementation work. At this point the design for most systems have been at least considered if not already realized, it is more so a matter of implementing them and linking them with existing parts of the client. I am also looking to shift some focus onto the design/use of uTP as we approach node information transfer, as it is an important part of the Portal Wire Protocol and how information is shared across clients.