# Tezos `lib_context` replay benchmarks - Plan for Q1/23 Rough plan on how to proceede with the `lib_context` replay benchmarks. ## Context `lib_context` is the part of the Tezos Octez implementation that uses Irmin. The Irmin team has created a shim around lib_context that records all `lib_context` API calls and records them in a trace. This trace (after some pre-processing) is used to replay `lib_context` API calls with different versions of Irmin in order to benchmark performance of changes to Irmin. These benchmarks are central to the contractual obligations between Tarides and Tezos and usually need to be produced as part of a milestone. In Q1/23 the Irmin will begin development of improvemtns for Tezos stores with unlimited history (Archive nodes). ## Goals & How to get there ### Client (Tezos/Nomadic Labs) should be able to create traces independantly Minimize the number of patches on the `tezos` repository that contain only the recording shim (1-2 patches). This will allow client to apply them wherever they want and create a trace independantly. #### Potential problems: - Recording shim introduces two new dependencies: `bentov` and `printbox`. It would be easier if no new dependencies are introduced. - [ ] check if dependencies are really needed and can be removed > [name=metanivek] I'm definitely in favor of upstreaming our patches if the octez maintainers will accept them. Even if they don't create the traces for us, it would be nice to not have to maintain our patches and continually rebase with their changes. ### Allow client to send us trace and store This serves as input for benchmarks run by the Irmin team. - Use [git-annex](https://git-annex.branchable.com/) whit some external storage (e.g. publically readable S3). Client checks in replay trace and store into Git repository that is shared with Irmin team. > [name=metanivek] I don't have a strong preference either way, but I've always used git lfs (https://git-lfs.com/) for this sort of "out of repo" storage. It works well. Have you used git-annex previously? #### Todos - [ ] Ask infrastructure team to provide suitable storage - 2023-01-13: Asked @mtelvers: "I don’t think we have anything like that." > [name=metanivek] If we control the storage, I guess that we would also control the repository. Managing access rights so that a client has write permissions could be a bit of a pain compared to us controlling it all internally. ### Make replay code maintainable Currently replay code are piled on the tezos repository. - Move replay code to a seperate repository `tezos-context-replay`. - Code in this repository depends on `lib_context` and (transatively) on `irmin`. We have the [tooling](https://github.com/adatario/guix-tarides#creating-a-development-environment-with-package-from-custom-branches) required to build the replay code with specific versions of irmin and `lib_context`. > [name=metanivek] If we can do this, I am *very* much in favor of it! :clap: Does this need our patches upstreamed to octez? (Also, we may want to not use guix since not everybody is on linux :disappointed:) ### Add more Tezos specific statistics to the `lib_context` trace Current `lib_context` trace does not contain Tezos specific performance metrics that might be good to have (e.g. Transactions Per Second (TPS)). Should this be added to the trace somehow? Potentially this needs to be done later, as it may mean that information needs to go in the trace from outside of `lib_context`. A tracing architecture closer to how OCaml 5 eventrings (https://github.com/ocaml/ocaml/pull/11474) work would make this much easier than the current `lib_context` shim. > [name=metanivek] I'm not sure I understand this section. We have some tezos specific metrics (like transactions per second) in the summary (see https://hackmd.io/@metanivek/BkPwZwQYj). Is this a part of the replay code? #### Todos - [ ] Ask the client ## See also - [mirage/irmin #1486 - Replayable Actions Trace (V0 → V1)](https://github.com/mirage/irmin/issues/1486): An (old) issue that tracks similar goals.