Development updates 12-14

Week 12

This week, I've been working on improving the documentation over on HackMD about debcrafter. I wanted to make sure Debcrafter options were well-documented and easy to understand, before diving into using it. However, I realized that the initial documentation wasn't up to par, so I took the initiative to rewrite and clarify the options. You can check out the updated documentation here: HackMD Debcrafter SPS Options.

What I found interesting is that the format generation by AI system was actually pretty good.
(Disclaimer: I was using AI for generating sentences, and sentence structures, most of the work was collecting, organizing and understanding as much as possible what should go into the documentation)
The problem I have encountered was my own understanding of Debcrafter options, which at many case were incorrect. The documentation still need further improvement, I will come back to it after having packages ready.

Week 13

Unfortunately, this week wasn't as productive as I'd hoped. I've been under the weather due to what seems like a case of food poisoning. As a result, I couldn't make as much progress on the tasks as I would have liked.

On the bright side, I did manage to find some time to work on the Docusaurus documentation. You can check out the progress on the Docusaurus docs here.

Week 14

This week, my main focus has been on creating the package structure using Debcrafter. The progress I've made in shaping the initial package structure, and you can view it on the GitHub repository at eth-deb.

While working on this, I encountered a few challenges related to the usage of Debcrafter, some essential naming conventions that I should document, and virtual package documentation.

The packaging structure I've developed for the eth-deb repository closely aligns with the proposed naming conventions. Here's a quick overview:

The installed package is named eth-node.
Each client will be named eth-node-${client-name}-cli, eth-node-${client-name}-config-$(variant), and eth-node-${client-name}-service, where client-name represents each client (such as Besu or Geth), and variant signifies a configuration choice for each package. (I will provide some examples on next week, of how this will look in practice.)

When you install the eth-node package, you will receive all eth-node-execution-client and all eth-node-consensus-client as suggested dependencies or packages that satisfy those dependencies. You can pick a requirement on your needs.

Configuration options for each package can be supplied by the config packages. Config packages can be supplied by community, and you can use the one which you prefer. This will allow to have multiple config options shipped and shared for each client.

One noteworthy challenge this week was related to the absence or issues with makefiles in some repositories when building from source. I took the time to investigate and learn about makefiles in order to address these issues effectively.

Interestingly, this detour into Makefiles turned out to be quite helpful, especially in conjunction with Debcrafter. Here's an overview of the process for building a client from commandline:

• Create .sss and .sps files for each client using templates.
• Utilize Debcrafter to generate the Debian directory.
• Locate the package source on the client's GitHub repository.
• Download the source and rename it to comply with Debian's naming convention. This renamed tar.gz file is referred to as the "original tar" during package building.
• Untar the source, renaming it to match the required structure.
• Copy the Debcrafter-generated /debian source files into the source directory.
• Attempt to build the package and address any issues that arise during the build process, particularly those related to source compatibility with Debian package building. This phase can be quite challenging.
• Repeat this process for each client, starting from scratch for each one to avoid modifying the source folder other than the /debian folder provided by Debcrafter.
• Verify that the built packages are installable on a different machine.
• Check if the builds are consistent across different architectures.
• Finally, check the source into the eth-deb repository under the built-system for others to reproduce the building process without the need to perform these steps manually.

By providing a Makefile, it helps the process, making it more efficient and manageable. This not only enhances the development workflow but also paves the way for future improvements, such as CI integration, local packaging, automation, and so on. It's kind of do it once, and reap the benefit 100 times.

One of the key issues I've encountered in this manual process is:

• Versioning: When new versions are released, it's important to decide which steps to repeat. While it's not overly complex to do so, there are multiple steps involved, which could lead to errors if not executed correctly.
• Patching: If patching is required in some versions, the build system should handle this automatically. It should provide default options and utilize agreed-upon paths to simplify the patching process.
• Repetition: Performing this process manually can be quite tedious, and any errors mean starting over from scratch. Wherever possible, automation should be embraced, and that's why build systems were created in the first place.

As mentioned in previous updates, the official methods for packaging Debian packages often rely on conventions rather than strict standardization. The proposal aims to simplify and standardize the process, allowing developers to focus on software development, agree on different configuration options, and enhance security, (all of this from packaging perspective). The process can be intricate, but the intention to encapsulate this complexity in a "sandwich complexity" manner, making it more accessible and manageable for everyone involved.
Makefiles offer to hide the complexity of these steps. Again, I was not originally planning to add this, this is something I have discovered while reading upon makefiles, as the previously mentioned as problems.

One more advantage of having a build system, which the eth-deb directory will be, is that different organizations can fork the repository and package the software to meet their specific requirements. This eliminates the need for manual package creation or seeking permission to modify the original repository, though that possibility should be always open.

These organizations can maintain their own apt.sources.list, build and provide packages, all while inheriting the security benefits and best practices embedded in this process. It's a way to ensure that people will readily embrace this repository, knowing that it offers a robust and secure packaging solution for their needs.
This can be helpful for organization where they offer hosted nodes, and have to adhere strict security conditions and while having the ability to update to the latest clients as possible.