# 2022-02 Monthly Report
*week 05-08: Irmin and Tooling reports*
# Overview
## Irmin
### Improve the Performance
- **Improve `irmin-pack` I/O Performance.** We have been working for a few months on improving the IO performance of `irmin-pack` by implementing the `structured keys` feature. `irmin-pack`, the Irmin backend used in the Tezos storage layer, writes data in an append-only file called a pack file. In order to efficiently retrieve the data, it uses the `index` library, which maps hashes of objects to their location in the pack file. The `structured keys` feature consists of adding more information in the keys (and not just the hash) to avoid accessing the `index` for every read. This new method leads to fewer stored objects in the `index` at the cost of some potential duplication of objects on disk.
With structured keys, we can choose between two indexing strategies: the `minimal indexing` strategy only adds the commits to `index` and duplicates all nodes and contents under the commits; or the `always indexing` strategy, similar to the current implementation, where all objects (commits, nodes, and contents) are added to `index`.
With the `minimal indexing` strategy, the numbers of IO syscalls is drastically reduced, resulting in doubling the TPS for the replaying of the bootstrap trace. We are still benchmarking this new feature, and we are writing a blog post that will show the final benchmarks.
This month we released Irmin.3.0, which contains the structured keys features. It's in the Tezos master branch with the `always indexing` strategy for now. In March, we'll deploy the `minimal indexing` strategy as well. It will be available in Octez v13.
Relevant links: [mirage/irmin#1769](https://github.com/mirage/irmin/pull/1769), [tezos/tezos#4477](https://gitlab.com/tezos/tezos/-/merge_requests/4477).
- **Integration of Layered Store/GC.** The layered store feature consists of organising the context of a Tezos node in two layers: one read-write layer that contains the latest cycles of the blockchain, which can still be reorganised and are not final, and a read-only layer containing older data that's frozen. The feature initially targets the rolling nodes, which will only need to keep the read-write layer. Every cycle calls a GC operation that's in charge of discarding the oldest cycle and keeping only the more recent ones in the context.
The advantages of a layered store for the rolling nodes is that it uses less disk space and has constant performance throughout the run of a node, without the need to do a snapshot export and import when the disk's context size gets too big. For archive nodes, the benefit is also better performance while bootstrapping, as only the read-write layer is needed for validating recent blocks.
This month we finished writing a design document for the layered store and implemented an initial prototype. The GC operation must perform a traversal of the oldest block in the live blocks' window to make sure that all necessary objects are kept in the context. We analysed the expense of this traversal, in terms of runtime and objects visited, for a Hangzhou context. We tested our initial prototype and are currently debugging some issues that occurred during this initial phase of testing.
In March, we plan to benchmark this prototype and start working on a proper integration in Irmin, followed by its integration in Tezos. We envisage it to be part of a Octez release before the end of Q2 this year.
Relevant links: [mirage/irmin/2022_layered](https://github.com/mirage/irmin/tree/2022_layered).
- **Improve Snapshot Import/Export.** Importing and exporting a snapshot from a Tezos node started to consume a lot of memory (up to 8Gb) in Hangzgou. This is due to the fact that after the [context flatenning](https://gitlab.com/tezos/tezos/-/merge_requests/2771), the very large directories of a Tezos context (containing a few million entries) were loaded in memory during snapshot import and export. We proposed a temporary fix for importing in December and worked since then on a proper fix. It consists of exposing the internal nodes used in Irmin to represent these large directories in an optimised way. This allows the node to charge the large directories in memory by chunks and therefore considerably reduce the memory usage. The snapshot's format is changed, but the import will be backward compatible with older formats. The export, however, will only use this new format.
We have an initial proposition for this which showed promising benchmarks. As this feature comes after Irmin 3.0, more work is needed to properly integrate it with the structured keys feature. We also worked on integrating the import/export operation in our benchmarks, so we can run them every month and ensure there is no regression.
The improvements to the snapshot will be part of a Irmin 3.2 release, expected in March, and should be part of Octez v13.
Relevant links: [mirage/irmin#1757](https://github.com/mirage/irmin/pull/1757), [mirage/irmin#1770](https://github.com/mirage/irmin/pull/1770).
- **Record/Replay of the Tezos Bootstrap Trace.** Benchmarking Irmin consistently (monthly, at each release, and on each pull request) is a top priority, to ensure that no regression is introduced. Our benchmarks consists of recording interaction traces between Irmin and Octez during the bootstrap of a node, which are then replayed for benchmarking and gathering stats. We are working on improving them on several fronts:
- Benchmarking `lib_context` instead of `irmin`, to catch any regression occuring in either `irmin` or lib_context. We'll also maintain the existing benchmarks in Irmin for users that want to bench Irmin code directly and for other usecases.
- Starting the benchmark from a snapshot instead of the empty store, as the chain's performance over the last few months is more informative that its performance has been since the beginning
- Allowing for different parameters, e.g., different `ocaml-gc` strategies as well as different configuration options for the context
- Adding more stats, including `ocaml-gc` stats and OS stats (e.g., related to page caching)
These benchmarks are very important for Irmin developers, but they're also important for anyone that contributes to `lib_context` and for any storage alternative. For instance, it can be used by the Rust implementation of the node [tezedge](https://github.com/tezedge/tezedge). It's enough to plug a new storage in `lib_context` to run these benchmarks.
We have a WIP branch on which we have made a lot of progress. We're using the improved benchmarks to compare Irmin 3.0 and the Irmin 2.10, but this required a bit of extra work when importing the benchmarks to the Irmin 2.10 branch.
We will start integrating this progressively in the master branch of Octez starting next month.
Relevant links: [Replaying benchmark in irmin](https://tarides.com/blog/2021-10-04-the-new-replaying-benchmark-in-irmin).
- **Publish Irmin/Tezos Performance Benchmarks.** We have been running monthly benchmarks for Irmin over the past year. However, we changed the process during this time in order to automate it more. Our goal is to have a year of monthly benchmarks that are all run on the same machine with the same configurations for GC/OS caching and, more importantly, that have reproducible dependencies. For this reason, we are creating Opam lock files and using `opam-monorepo` to install them. We are also writing scripts to produce graphs in order to ease readability.
For the future monthly benchmarks, we want to automate the process to tag every month’s first day on each benchmarked repo, generate lockfile every month, deploy a bare-metal machine on Equinix (using Equinoxe library), run the benchmarks, and store the result on a backup machine. The results will then be sent to `ocurrent-bench` GUI to expose them on a public webpage.
- **Continuous Benchmarking Infrastructure.** The `current-bench` infrastructure has had improvements to the frontend and backend.
As GUI improvements, the graphs are now aligned vertically and expand to the full width of the display. The UI now has a link to show line numbers captured in the database for each metric. The error page now has an easy navigation back to the main page.
The `Alcotest` testing framework has been integrated with the project, as well as code coverage using `bisect-ppx`. The production reports and logs are now sent to Slack to be notified of any unexpected failures. The `Cmdliner` version 1.1.0 has been updated in current-bench. We fixed some bugs related to the schemas used between GraphQL and PostgreSQL.
Two new features of the CB we are working on consists of firstly, adding a new API end-point where results from external benchmark runs can be added to current-bench GUI. The HTTP codes have also been updated to be more specific and useful to the user. Secondly, a Raspberry Pi 4 has been setup to run benchmarks in production, along with necessary changes to current-bench worker to support ARM builds.
Relevant links: [ocurrent/current-bench#294](https://github.com/ocurrent/current-bench/pull/294), [ocurrent/current-bench#287](https://github.com/ocurrent/current-bench/pull/287), [ocurrent/current-bench#301](https://github.com/ocurrent/current-bench/pull/301), [ocurrent/current-bench#320](https://github.com/ocurrent/current-bench/pull/320).
### Add Merkle Proofs
- **Optimise Merkle Proofs.** Merkle proofs are a light way to share partial chain states between parties. These can be used to implement efficient proxies, light clients (i.e., clients without storage components) or more complex [Layer-2 Rollups 2](https://tezos.gitlab.io/alpha/transaction_rollups.html).
We proposed an initial implementation of Merkle proofs, which was had improved by the release of Irmin 2.10.2. We then focused on properly integrating the Merkle Proofs in Irmin 3.0. Since this changes the structure of the keys, it has an important impact on how the system implements the proofs.
This month we worked with Nomadic Labs, DaiLamba, and Trilitech on integrating the Merkle proofs API from Irmin to Tezos. The hashing schema used in the proof needs to be shared with context because nodes in the chain must verify proofs generated by the rollup nodes. Two types of hashing schema are allowed: the one used by for the context hash (proof use trees with 32 children) and a binary context (proofs use binary tree), which is optimised for generating smaller proofs.
We plan to write a blog post next month explaining the Merkle proofs API. We also opened a few issues containing improvements to the proofs that can be done in Irmin, such as tests and code refactoring. These are however non-blocking for using the API by Tezos developers.
Relevant links: [tezos/tezos#4086](https://gitlab.com/tezos/tezos/-/merge_requests/4086), [tezos/tezos#2418](https://gitlab.com/tezos/tezos/-/issues/2418), [mirage/irmin#1751](https://github.com/mirage/irmin/pull/1751), [mirage/irmin#1729](https://github.com/mirage/irmin/pull/1729), [tezos/tezos#4307](https://gitlab.com/tezos/tezos/-/merge_requests/4307), [tezos/tezos#4465](https://gitlab.com/tezos/tezos/-/merge_requests/4465).
### Provide Support to Tezos
- **Respond to and Track Issues Reported by Tezos Maintainers.** While running Ithacanet, users observed an increase in memory usage. In collaboration with Nomadic Labs, we found that the issue was related to an LRU cache located in Irmin. We bencharmked different LRU sizes to check for cache misses as well as check the connection between object types and the LRU memory usage. We agreed on decreasing the LRU size by half, which doesn't negatively impact the node's performance but improves the memory footprint.
For future work, we plan to change the LRU so that it's bound by its containing elements' size instead of by the number of elements, as it is now. This is a safer option because the elements' size can grow a lot and lead to a memory increase, even if the LRU is bound by number of elements. We plan to work on this improvement in the next few months.
Relevant links: [tezos/tezos#2376](https://gitlab.com/tezos/tezos/-/issues/2376).
### Improve Tezos Interoperability
- **Rust and Python Bindings for Irmin.** The C binding for Irmin are now part of the Irmin repo, and the Python and Rust bindings have been released and are available at [mirage/irmin-py](https://github.com/mirage/irmin-py) and [mirage/irmin-rs](https://github.com/mirage/irmin-rs). As examples, the following code snippets show how to get a commit from the command line and list all paths containing contents in a Tezos context in python:
```python=
commit_hash = Hash.of_string(repo, sys.argv[2])
commit = Commit.of_hash(repo, commit_hash)
# Open the `master` branch
store = Store(repo, branch=commit)
def list_path(store, path):
"""
Prints all content paths
"""
for k in store.list(path):
p = path.append(k)
if p in store:
print(p)
else:
list_path(store, p)
```
and similarly in rust:
```rust=
// Resolve commit
let hash = Hash::of_string(&repo, &args[2])?;
let commit = Commit::of_hash(&repo, &hash)?.expect("Commit not found");
// Open the store
let store = Store::of_commit(&repo, &commit)?;
fn list_path<T: Contents>(store: &Store<T>, path: Path) -> Result<(), Error> {
for k in store.list(&path)? {
let p = path.append_path(&k)?;
// If the store has contents at `p` then print the path
if store.mem(&p) {
println!("{}", p.to_string()?);
} else {
list_path(store, p)?;
}
Ok(())
}
```
The scripts are available at [mirage/irmin-py/examples/tezos.py](https://github.com/mirage/irmin-py/blob/main/examples/tezos.py) and [mirage/irmin-rs/examples/tezos.rs](https://github.com/mirage/irmin-rs/blob/master/examples/tezos.rs).
Relevant links: [mirage/irmin#1713](https://github.com/mirage/irmin/pull/1713#pullrequestreview-885697018), [mirage/irmin-py](https://github.com/mirage/irmin-py), [mirage/irmin-rs#4](https://github.com/mirage/irmin-rs/pull/4)
- **Storage Appliance.** Using an Irmin daemon instead of using it as a library, like Tezos uses it now, potentially reduces the memory usage of the node. Instead of two processes interacting with Irmin (and caching things in memory), only one process is used for all interactions with the Irmin context. It also simplifies the semantics of interactions between read-write and read-only instances (as we don't have to use the filesystem as an inefficient inter-process communication mechanism).
The server can also be used to access a context that is remote, e.g., in case not enough storage is available locally.
The implementation for `irmin-server` is well tested, and our next work consists in integrating a storage alternative in Tezos. We start working on a branch of Tezos that integrates the `irmin-server`. Our strategy is to add `irmin-server/client` as backend in Irmin. This requires minimal changes to the Tezos code to integrate the Irmin daemon, while also making the backend available to other Irmin users. To figure out the difficulty of this, we plan to first try the `irmin-http` backend in Tezos. We refactored the code to remove a dependency on Unix, and we're working towards abstracting the I/O. A dependency in Tezos needs this.
As this isn't high priority, the integration in Tezos will be ready for users to try it out later this the year.
Relevant links: [mirage/irmin-server#29](https://github.com/mirage/irmin-server/pull/29).
- **Irmin in the Browser.** An internship project started this month that consists of combining [irmin-indexeddb](https://github.com/talex5/irmin-indexeddb) and [irmin-server](https://github.com/mirage/irmin-server) to produce an offline-first application with fast and efficient synchronisation. Offline-first applications ensure that apps work just as well offline as they do online. Thanks to Irmin’s mergeable replicated data-types, it becomes much easier to build applications that can transform state offline and re-synchronise the state later.
We use `irmin-indexddb`, an Irmin backend that utilises the browser’s indexed-db storage, so Irmin compiles to the browser and persists the data. Re-synchronising Then Irmin stores are re-synchronised with `irmin-server`, which implements a specialised wire protocol for efficient communication.
We have experimented with `irmin-server`, which only provides a CLI to interact with. We have decided to instead try `irmin-graphql`. Our short term goal is to build a simple `mini-github` project to display details of a GitHub repository as a sample project on how to combine `irmin-graphql + dream` on the server side and `jsoo + dream` for the client.
### Maintain Tezos's MirageOS Dependencies
- **General Irmin Maintenance.** The changes in Irmin 3.0 affected other backends than `irmin-pack`. We fixed the GraphQL backend and the `sync` function in the `irmin-git` backend. We're also reworking the GraphQL test suite.
In collaboration with Nomadic Labs, we ported `irmin-pack.mem` to compile with `js_of_ocaml` to make parts of Octez usable inside browsers. This required splitting the Unix specific part out of `irmin-pack`, and we released Irmin 3.1 with this change.
We experimented a little with IPFS, an online distributed content-addressing file system, as it has a similar storage model to Irmin.
Also, we have a lockfile for Irmin in a draft PR. It relies on a patch of `ocaml-yaml` while waiting for Dune 3.0 to be released.
Relevant links: [mirage/irmin#1618](https://github.com/mirage/irmin/pull/1618), [tezos/tezos#4285](https://gitlab.com/tezos/tezos/-/merge_requests/4285).
## Compiler
### Ensure OCaml compiler quality
- **Verify existing bits of the stack using GOSPEL.** We are developing the Gospel specification language to verify and test parts of the existing software stack. This month we continued reworking the Gospel AST. This redesign aims at easing upcoming Gospel extensions such as higher-order functions, and providing a clearer semantics for the language.
In a Gospel specification, a user can refer to a previous value of an expression using the `old` primitive. We continued our work on formalizing Gospel's handling of `old` to make sure it is working as intended and performs well.
Gospel is attracting renewed academic and industrial interest. We spent some time discussing aspects of the language design, the meaning of different constructs, e.g., in the presence of aliases, and planning how we can extend the specification language with more advanced features without sacrificing the simplicity of the current core language.
### Provide support to Tezos
- **Assist with compiler related issues as needed.** The Tezos protocol compiler was suffering from a stack overflow which was triggered when the resulting code would be passed to the OCaml compiler. The problem was fixed by adjusting the compilation scheme for modules.
## CI
### Improving CI
Work continues to transform the prototype Octez pipeline into a robust service. We now build all the merge requests and branches from the main GitLab Tezos project on the cluster, with results available on https://tezos.ci.dev but not yet posted to GitLab. We added further cluster capacity with one new ARM64 worker alongside the existing two x86_64 workers, allowing us to build the full set of architectures from the Octez pipeline.
We're onboarding a new developer for the GitLab `ocaml-ci` application with the goal to provide a full `ocaml-ci` service for selected Tezos OCaml projects.
### OCaml Projects Are Continuously Tested
The cluster has gained additional resources with four Dell 256-thread x86_64 machines added to the Linux AMD64 pool, two Raptor 176-thread POWER9 machines added to the Linux PPC64 pool, and four Mt. Snow 80 core ARM64 machines added to the Linux ARM64 pool. This completes deployment of additional machines for the cluster.
The work on the macOS implementation of OBuilder rolled out to the cluster and has entered a maintenance phase with existing services using Mac workers. Windows implementation of OBuilder continues work on stabilizing the use of `lwt` and providing upstream `lwt` fixes and functionality. New work has started on extracting the Opam solver to run as a service on the cluster, providing both additional resources for running solves and freeing up dedicated solver machines to add back into the cluster.
## Tooling
### Merlin
The main focus in Merlin continues to be the new mechanism to handle definitions. This relies on new calculus API (shapes) based on the compiler that will be part of OCaml 4.14. This API can be used directly in Merlin to compute where a value is defined or declared. The next step of this feature is to use it in reverse and see where a particular value is used. This requires building some tooling and updating an index as part of the build system. This part is being prototyped.
### `opam-monorepo`
`opam-monorepo` is a tool to assemble code with its dependencies into a self-contained Dune workspace. This is an alternative to installing dependencies into an Opam switch.
This improves reproducibility and caching, and it gives more granularity in the build plan.
Work continues to support building Octez as a monorepo. In particular there has been a focus on testing by making the `lock` step itself more reproducible. This helps write feature tests for `opam-monorepo`. In terms of packages, we've been finding solutions to some tricky parts of the Octez dependency graph by helping finalise the port of ctypes to Dune and try an alternative strategy to port `hacl-star-raw` to Dune. This is supported by a new CI mechanism that assists in porting dependencies to Dune.
## Appendix A: Benchmarks for March
https://hackmd.io/ro0GRKt8ST6vu_Tsa5J6SA?view#1343496-commits-up-to-Edo1
## Irmin
### Add Merkle Proofs
- Optimise Merkle proofs (Irm68)
- Add Merkle proofs to the protocol [tezos/tezos#4086](https://gitlab.com/tezos/tezos/-/merge_requests/4086).
- Reply to reviews on the Merkle proofs MR: [tezos/tezos#4086](https://gitlab.com/tezos/tezos/-/merge_requests/4086)
- Env: Context.produce_tree_proof throws exception [tezos/tezos#2418](https://gitlab.com/tezos/tezos/-/issues/2418).
- 2.10
- [2.10] Fix large stable inodes in streamed proofs [mirage/irmin#1741](https://github.com/mirage/irmin/pull/1741)
- Release 2.10.2 [mirage/irmin#1751](https://github.com/mirage/irmin/pull/1751)
- [new release] Irmin (2.10.2) [ocaml/opam-repository#20628](https://github.com/ocaml/opam-repository/pull/20628)
- 3.0
- Irmin: Trees carry a placeholder proof env [mirage/irmin#1749](https://github.com/mirage/irmin/pull/1749)
- Irmin: Portable in tree [mirage/irmin#1755](https://github.com/mirage/irmin/pull/1755)
- Helping with shipping the dedicated Merkle proof API to Tezos, and forward-porting this to 3.0, including making a 2.10.2 release containing bugfixes for Merkle proofs.
- Irmin: Improve cache usage in Tree [mirage/irmin#1745](https://github.com/mirage/irmin/pull/1745#pullrequestreview-869799850)
- Irmin: Improve error handling in Tree [mirage/irmin#1744](https://github.com/mirage/irmin/pull/1744#pullrequestreview-869803865)
- Irmin: Add `with_handler` and `head` to `Node_portable` [mirage/irmin#1746](https://github.com/mirage/irmin/pull/1746#pullrequestreview-870611807)
- `irmin-pack`: Fix proofs in inodes [mirage/irmin#1747](https://github.com/mirage/irmin/pull/1747#pullrequestreview-870807186)
- New hash_exn function [mirage/irmin#1748](https://github.com/mirage/irmin/pull/1748#pullrequestreview-870808218)
- Support Irmin.Generic_key.S in `irmin-graphql` [mirage/irmin#1618](https://github.com/mirage/irmin/pull/1618#pullrequestreview-871079670)
- Interactions on adding Merkle proofs to the protocol [tezos/tezos#4086](https://gitlab.com/tezos/tezos/-/merge_requests/4086).
- Prepare the code for the MR following !4086 [here](https://github.com/tarides/tezos/compare/e495ea153f15619f12f97d802770f8f6ea358517...Ngoguey42:binary-contexts2)
- Merkle proofs 3.0 [mirage/irmin#1729](https://github.com/mirage/irmin/pull/1729).
- Open a few Irmin issues regarding what's left to do on Merkle proofs
- A complex case in tree proof extenders should be covered by UT feature/merkle-proofs [mirage/irmin#1766](https://github.com/mirage/irmin/issue/1766).
- Tree proof's findv is not covered by UT feature/merkle-proofs [mirage/irmin#1765](https://github.com/mirage/irmin/issue/1765).
- Proof production should handle less functions feature/merkle-proofs good-first-issue [mirage/irmin#1764](https://github.com/mirage/irmin/issue/1764).
- Rework env sharing in tree feature/merkle-proofs [mirage/irmin#1763](https://github.com/mirage/irmin/issue/1763).
- Verification of tree proofs could check minimality feature/merkle-proofs good-first-issue [mirage/irmin#1762](https://github.com/mirage/irmin/issue/1762).
- More reviewing of the work in forward-porting the Merkle proofs feature:
- Irmin: Add the `Store.Tree.kinded_hash` function [mirage/irmin#1767](https://github.com/mirage/irmin/pull/1767#pullrequestreview-880161231)
- Context Merkle proof encoding [tezos/tezos#4307](https://gitlab.com/tezos/tezos/-/merge_requests/4307).
- Introduce v1 contexts [tezos/tezos#4465](https://gitlab.com/tezos/tezos/-/merge_requests/4465).
- Verify proof functions dispatch according to version [tezos/tezos#4536](https://gitlab.com/tezos/tezos/-/merge_requests/4536).
### Improve Tezos Interoperability
- Storage appliance (Irm53)
- Made a new branch to abstract I/O in order to remove `conduit-lwt-unix` dependency [zshipko/irmin-server/io](https://github.com/zshipko/irmin-server/tree/io).
- More work on abstracting I/O
- Finished adding examples [mirage/irmin-server#29](https://github.com/mirage/irmin-server/pull/29).
- Updates needed to work with Tezos
- Started Tezos branch with `irmin-server`
- Worked on Tezos branch with `irmin-server`
- Started working on making `irmin-client` work as an Irmin backend to simplify integration with Tezos.
- Rust bindings for Irmin (Irm75)
- Added error handling to `IrminRepo` instead of using a global variable and updated `irmin-rs/irmin-py` [mirage/irmin#1713](https://github.com/mirage/irmin/pull/1713).
- Still trying to debug ARM64 issue [mirage/irmin#1713](https://github.com/mirage/irmin/pull/1713)
- Minor cleanup, work on documentation
- Add a C bindings library to Irmin: [mirage/irmin#1713](https://github.com/mirage/irmin/pull/1713#pullrequestreview-885697018)
- Moved Python bidings to [mirage/irmin-py](https://github.com/mirage/irmin-py).
- Updated `irmin-rs` to use `libirmin` [mirage/irmin-rs#4](https://github.com/mirage/irmin-rs/pull/4).
- Worked on documentation and CI for Python and Rust bindings
- Will make beta release for Python and Rust bindings next week
- Cleaned up bindings to prepare for release
- Made `irmin-rs` release: [crates.io/irmin](https://crates.io/crates/irmin); [docs.rs/irmin](https://docs.rs/irmin/0.3.1-beta.0/irmin/) and [mirage/irmin-rs#5](https://github.com/mirage/irmin-rs/pull/5)
- Made `irmin-py` release: [pypi.org/irmin](https://pypi.org/project/irmin) and [mirage/irmin-py#6](https://github.com/mirage/irmin-py/pull/6).
- Irmin in the browser (Irm89)
- Went through `irmin-server` code and ran the examples
- Tried setting up `js_of_ocaml` to test out `irmin-server`
- Discovered we need to find out a way to interact with the `irmin-server` aside the CLI currently built in (still not clear)
- Went through some articles on offline-first applications for the second time (https://martin.kleppmann.com/papers/local-first.pdf)
- Went though Irmin and ran some of the examples
- Tried setting up a chat app to test out `irmin-server` but failed
- Pair programming explaining Irmin and `irmin-graphql`.
- Discussions about making request to the `irmin-server` which meant a work around on the use of `conduit-lwt-unix`. We need to provide a fix to support both `cli` and `browser`. In the mean time, we are trying out `irmin-graphql` which already supports browser request.
- Setup a simple `irmin-graphql` project using `irmin-git` as store and made some query from the client using `js_of_ocaml`
- Helping with setting up a dream with `irmin-server`.
- Combined `irmin-graphql` with `dream`
- Served `irmin-graphql` schema to `dream` request
- Used `irmin-git` for data storage.
- Created [simple_mini_github](https://github.com/dinakajoy/simple_mini_github) which is still a work in progress
### Improve the Performance
- Publish Irmin/Tezos performance benchmarks (Irm11)
- Run February benchmark and transfer the result to a remote server
- Date selection, pretty branch names and rolling history
- Run produce benchmark results and share with the team
- Write a new automatic script and regenerate december, January and February Irmin benchmarks on the same machine to avoid biais
- Regenerate benchmarks for December, January, February, and Irmin 3.0
- Write a graph script to generate diagram from the data
- Improve the graph script to generate diagram from the data
- Produce benchmark summary and share with the team
- Improve the script for Irmin replay to support cache dropping if you are root
- Regenerate benchmarks for `july-21-patch` (a version that includes a patch for a bug fix in the benchmarks on `index`) and 3.0
- Produce benchmark summary and share with the team. Produce a new graph which looks like exactly what we expected.
- Introduce lockfile for monthly benches [tarides/tezos-storage-bench#1](https://github.com/tarides/tezos-storage-bench/pull/1)
- Lock files from January 2021 to February 2022 are available and allow us to execute benchmark in a reproductible way.
- Start moving the script to Python 3.6 (easier to maintain in time) and change it to download the right lock file. We need to see if it can help him with the automation of the monthly benchmark.
- Equinoxe:
- Mock an API for a test purpose [maiste/equinoxe#70](https://github.com/maiste/equinoxe/pull/70).
- Merge [maiste/equinoxe#74](https://github.com/maiste/equinoxe/pull/74) and release `equinoxe.0.2.0`, `equinoxe-hlc.0.2.0` and `equinoxe-cohttp.0.2.0`
- Correct and merge [maiste/equinoxe#71](https://github.com/maiste/equinoxe/pull/71): it introduces a statically type module to handle the API and remove the old one with `Ezjsonm.value`.
- Rebase after Equinoxe API change [maiste/equinoxe#70](https://github.com/maiste/equinoxe/pull/70).
- Improve `irmin-pack` I/O performance (Irm72)
- Review several PRs related to Irmin 3.0
- Help finishing Irmin 3.0
- Prefix the prehash for default contents, nodes and commits [mirage/irmin#1715](https://github.com/mirage/irmin/pull/1715).
- Update CHANGES to account for 2.{8,9,10} releases [mirage/irmin#1768](https://github.com/mirage/irmin/pull/1768)
- Cut a 3.0 release of Irmin, containing the structured keys feature for `irmin-pack`:
- Release 3.0.0 [mirage/irmin#1769](https://github.com/mirage/irmin/pull/1769)
- [new release] index and index-bench (1.6.0) [ocaml/opam-repository#20729](https://github.com/ocaml/opam-repository/pull/20729)
- Worked on releasing Irmin 3.0 to `opam-repository` and to Tezos [ocaml/opam-repository#20734](https://github.com/ocaml/opam-repository/pull/20734).
- `lib_context`: update to use Irmin 3.0 [tezos/tezos#4477](https://gitlab.com/tezos/tezos/-/merge_requests/4477).
- Plots for future blog post
- Continuous Benchmarking Infrastructure (Irm41)
- Add unit tests [ocurrent/current-bench#294](https://github.com/ocurrent/current-bench/pull/294
- Endpoint PR [ocurrent/current-bench#287](https://github.com/ocurrent/current-bench/pull/287)
- Return better HTTP codes [ocurrent/current-bench#292](https://github.com/ocurrent/current-bench/pull/292)
- Fix UNIQUE constraints and NULL values in benchmark_name column [ocurrent/current-bench#292](https://github.com/ocurrent/current-bench/pull/292).
- Updating Cmdliner to newest version [ocurrent/current-bench#304](https://github.com/ocurrent/current-bench/pull/304).
- Worker support for Raspberry ARM [ocurrent/current-bench#301](https://github.com/ocurrent/current-bench/pull/301)
- Fix GitHub status update and wait for benchmark to terminate [ocurrent/current-bench#299](https://github.com/ocurrent/current-bench/pull/299)
- Show all commits for all metrics and vertically align graphs [ocurrent/current-bench#320](https://github.com/ocurrent/current-bench/pull/320)
- Record/replay of the Tezos bootstrap trace (Irm76)
- Work on[Ngoguey42/tezos#1](https://github.com/Ngoguey42/tezos/pull/1)
- Store new raw traces on remote server
- Correct differents bugs on `raw to replayable` and start running replays on `Equinix`
- Work on [Ngoguey42/tezos#1](https://github.com/Ngoguey42/tezos/pull/1)
- WIP new script to run `lib_context` replays with different parameters easily
- Meeting to discuss about the future of replays
- Introduce a new strategy to test with replay [mirage/irmin#1761](https://github.com/mirage/irmin/pull/1761).
- Made plans on what to do next about the `lib_context` replay project
- WIP new script to run `lib_context` replays with different parameters easily.
- Write and test the script for replay with `lib_context` on Irmin 3.0`
- Start working on the integration of Irmin 2.10` from the replay/record branch.
- `irmin-bench`: Improve public API for replay [mirage/irmin#1781](https://github.com/mirage/irmin/pull/1781).
- Start working on the integration of Irmin 2.10.
- Improve the script to download the `context` and the `replay trace` if it's not found on the server. It also performs a `tezos` install if it's not found. We need to find a way to easily move between version (git patch, branch, flags).
- `irmin-bench`: Improve public API for replay [mirage/irmin#1781](https://github.com/mirage/irmin/pull/1781).
- `irmin-bench`: Enforce certain configurations for the replays [mirage/irmin#1760](https://github.com/mirage/irmin/pull/1760#pullrequestreview-877469518)
- Improve snapshot export/import (Irm81)
- Expose a type for snapshots in inodes [mirage/irmin#1757](https://github.com/mirage/irmin/pull/1757).
- Use inodes in snapshots [tarides/tezos#19](https://github.com/tarides/tezos/pull/19).
- Improving the memory footprint of tree traversals, in preparation for moving the Tezos snapshot export logic into Irmin itself:
- Irmin: use a more compact accumulator for unique tree traversals [mirage/irmin#1770](https://github.com/mirage/irmin/pull/1770)
- Benchmarks snapshots import and export using the replay branch
- Discussions on the snapshots improvements
- Started implementing the improvements
- Inodes are not integrity checked on read [mirage/irmin#1773](https://github.com/mirage/irmin/issues/1773)
- Start working on making `lib_context` stats work with snapshot export
- Start working on snapshot export in order to make it work with snapshot export
- Working on the implementation for snapshots
### Improve the Space Usage
- Integration of Layered store/GC (Irm49)
- Researching object graph traversal; commenting inodes; reading [deserialization code](https://github.com/Ngoguey42/irmin/blob/looking-for-corruption/src/irmin-pack/brute_force_integrity_report.ml)
- Design document draft (posted to slack)
- Design document improvements; posting to `irmin-dev` slack; posting to nomadic-tarides slack
- Polishing prototype [tomjridge/sparse-file](https://github.com/tomjridge/sparse-file/tree/master/src).
- Start implementing design in Irmin proper on [branch](https://github.com/mirage/irmin/tree/2022_layered).
- Start working on producing stats from a Hangzhou pack store. These stats should help making decisions on the layered store design
- Implementing layered design in Irmin proper [mirage/irmin/2022_layered](https://github.com/mirage/irmin/tree/2022_layered):
- branch https://github.com/mirage/irmin/tree/2022_layered
- implement obj store
- implement suffix file
- implement pread/pwrite
- implement external sorting + other external routines
- add extent calculation code
- implement sparse file
- implement worker
- fully switch from `obj_store` to `sparse_file`
- use `out_channel` rather than `mmap` in some situations
- add tracing to low-level I/O, controlled by envvar
- get Hangzhou context
- use Repo.iter to iterate over a commit in Hangzhou context
- create a trace of all reads from Hangzhou store during Repo.iter (758MB)
- add external (on-disk) binary search
- add external "less-than-or-equal" binary search
- add hook to trace_replay.ml to trigger GC every n commits
- separate out "Pre_io" to break cycle between I/O and worker
- extend `pack_store_IO.ml` to handle worker fork
- add various binaries for debugging and for calculating store reachability
- add .exe to convert from old pack store format to layered pack store format
- add global hook `trigger_gc`, which when set to "Some commit_hash", triggers GC for that hash
- fix some problems caused by caching of objects which are not live wrt. commit A, stored on disk before commit A, but then magically come alive for a later commit B (added a global "clear caches" function... which will need to be replaced with something better eventually)
- handle worker termination, switch to new sparse+suffix after it has been created by the worker
- current status: during replay it seems to work, except that a region (off,len) that was recorded as live wrt. an earlier commit, is then read "wrongly." The trace replay attempts to read (off,len+n) bytes, and so the last n bytes are not recorded in the sparse file and something goes wrong; need to understand why Repo.iter is only partially reading some of the values it traverses
### Maintain Tezos' MirageOS Dependencies
- General Irmin maintenance (Irm60)
- Work on the Irmin page for Notion to centralize expose information to other teams.
- Clean some Irmin issues and update the project board with `Unschedule` issue. Also move issues from `Irmin - 2021` (now `Irmin - DEPRECATED`) to `Irmin - 2022` (now ` Irmin - Roadmap`) as we can rely on one roadmap only.
- [2.10] Opam: fix git lower bound [PR#1743](https://github.com/mirage/irmin/pull/1743)
- Merged generic key PR for GraphQL [mirage/irmin#1618](https://github.com/mirage/irmin/pull/1618).
- Worked on GraphQL test suite rewrite, still adding tests but should be ready to review next week after I add a few more tests [mirage/irmin#1750](https://github.com/mirage/irmin/pull/1750).
- Opened draft PR for GraphQL test suite rewrite [mirage/irmin#1750](https://github.com/mirage/irmin/pull/1750).
- More GraphQL tests
- Tezos MR to add `tezos-node storage head-commit` was merged [tezos/tezos#4285](https://gitlab.com/tezos/tezos/-/merge_requests/4285).
- Experiments using IPFS for sharing data between stores.
- Started reviewing `irmin-git` before Irmin 3.0 release
- Finished updating irmin.org tutorial for latest release [github.com/mirage/irmin.org#89](https://github.com/mirage/irmin.org/pull/89).
- Fixed an issue with command line parsing for push, pull, fetch, and clone [mirage/irmin#1776](https://github.com/mirage/irmin/pull/1776).
- Updated the default Mimic context when using `Irmin_unix.Git` [mirage/irmin#1778](https://github.com/mirage/irmin/pull/1778).
- Split the Unix-specific part out of `irmin-pack` [mirage/irmin#1783](https://github.com/mirage/irmin/pull/1783)
- Talk with Mirage about domain use cases for Irmin
- Work on releasing Irmin 3.1 to Tezos.
- Split the Unix-specific part out of `irmin-pack` [mirage/irmin#1783](https://github.com/mirage/irmin/pull/1783#pullrequestreview-893703240)
- Release 3.1.0 [mirage/irmin#1785](https://github.com/mirage/irmin/pull/1785)
- [new release] Irmin project (3.1.0) [ocaml/opam-repository#20803](https://github.com/ocaml/opam-repository/pull/20803)
- We have a lockfile for Irmin 3.0. It relies on a patch of `ocaml-yaml` while waiting for Dune 3.0 to be released. We will be able to have lockfiles in Irmin once the true correction is merged.
- Take a look at `opam-monorepo` lock file, but there seems to be an issue with the solver. It needs more investigations.
### Provide Support to Tezos
- Respond to and track issues reported by Tezos maintainers (TZ2)
- Irmin's LRU cache is not bounded in memory but by number of elements [tezos/tezos#2376](https://gitlab.com/tezos/tezos/-/issues/2376).
- Benchmarked Irmin 2.10 with the LRU changes (limiting the LRU in number of elements, or in the size of the elements).
- Analysed cache misses with the LRU changes.
- With Nomadic Labs, we decided to stick with a smaller LRU and not change the implementation of the LRU for now.
## Compiler
### Ensure OCaml compiler quality
- Verify existing bits of the stack using GOSPEL (Irm70)
- PR: Draft proposition for a second version of the AST [ocaml-gospel/gospel#138](https://github.com/ocaml-gospel/gospel/pull/138)
- Added some notes on the [version 2 of the Gospel AST](https://github.com/n-osborne/gospel/blob/v2/src2/changes.md):
what is different, why and what to expect.
- gospel: discussion, planning, and review of the AST proposal
- gospel: investigated spec attachment
- gospel: maintenance on the documentation
- gospel: reading about [`inferno`](https://gitlab.inria.fr/fpottier/inferno) and type inference (substitution vs constraints) for the Gospel typechecker
- gospel: a lot of discussions around design questions and semantics and feature planning
- PR: remove dead code [ocaml-gospel/gospel#139](https://github.com/ocaml-gospel/gospel/pull/139)
- PR: update cmdliner to 1.1.0 [ocaml-gospel/gospel#148](https://github.com/ocaml-gospel/gospel/pull/148)
- PR: redefine ghost to not confuse merlin [ocaml-gospel/gospel#155](https://github.com/ocaml-gospel/gospel/pull/155)
- Issue: Update Cmdliner [ocaml-gospel/gospel#142](https://github.com/ocaml-gospel/gospel/issues/142)
- ortac: experiments with `repr` for type representatives
- ortac: experiments with models verification
- ortac: formalizing the correctness of the `old` handling. Coq's inductions are nasty.
- ortac: more work on the formalisation and coq proofs
- cachecache: started preparing a bibliography for a coming internship
### Provide support to Tezos
- Assist with compiler related issues as needed (Comp94)
- Changed the protocol compilation scheme to prevent a stack overflow
[tezos/tezos#4340](https://gitlab.com/tezos/tezos/-/merge_requests/4340)
## Tooling
### Merlin
- Links between definition, declaration and usage (Plat106)
- Fixed the behavior of jumping from values of a functor arguments. It is indeed more useful to jump to the declaration's of that value in the argument's signature.
- Started working on getting documentation by using UIDs instead of the current heuristic. This is now working quite well. However we are still forced to use the legacy approach when the doc is being locked for in the active buffer since Merlin's pure parser does not implement the compiler's doc-association algorithm.
- We rebased the 414 and shapes branches. A new test introduced on master led us to discover an usability issue when a project is build with Dune: the new locate algorithm does not systematically traverse module aliases which leads to unexpected behavior when interacting with Dune generated aliasing. We have some ideas to solve this but it's a bit saddening to have to introduce back some ad-hoc behavior.
- Implemented some logic in the new locate code to automatically go through aliases when their are Dune-generated (that is, part of a `.ml-gen` file). This makes the new shape-based locate behave as expected in Dune projects.
- This also led to a fix for issue [#1421](https://github.com/ocaml/merlin/issues/1421) related to top-level module documentation queries.
- The PR was rebased and cleaned once again.
- Occurrences and rename (Mer7)
- We started working on the project-wide occurences feature. We drafted a first design that involves a new binary called by Dune to generate indexes of UIDs and their locations. Started working on a first prototype.
- Wrote a small binary named `ocaml-uideps` that builds indexes of uids and their locations from the `cmt`files of a project. It uses shapes to compute the definitions of all the values appearing in the typedtree. Next steps involve changes to Dune to call that new binary and to Merlin to use the results.
- Progressed on `ocaml-uideps` which is now generating indexes for Irmin without any errors. Next steps involve having Dune call that executable and Merlin use the resulting indexes. The binary in itself is also a cleaner and able to optionaly log information during the run for debugging purposes.
- We opened a PR on the compiler to enable a [small hack](https://github.com/ocaml/ocaml/pull/11020) that speedup shape reduction (which is very heavily used by the indexing prosess) by reusing the memoization table accros several calls.
- Merlin is now able to iterate on the typedtree to search occurrences in the current file and to query an `workspace.uideps` file for external occurrences. It's still experimental, and the next step will be to orchestrate `uideps` generation with Dune and communicate to Merlin the paths to these files.
### `opam-monorepo`
- General `opam-monorepo` maintenance (Plat162)
- PR: Add debug log to list locally detected packages [ocamllabs/opam-monorepo#252](https://github.com/ocamllabs/opam-monorepo/pull/252)
- PR: Fix explicit repo solver [ocamllabs/opam-monorepo#253](https://github.com/ocamllabs/opam-monorepo/pull/253)
- PR: Generate base packages rather than copying them in each test [ocamllabs/opam-monorepo#254](https://github.com/ocamllabs/opam-monorepo/pull/254)
- PR: Add a flag to select cross-compiling packages when available [ocamllabs/opam-monorepo#256](https://github.com/ocamllabs/opam-monorepo/pull/256)
- PR: Add cross-compile flag to Zarith and fix version number [dune-universe/mirage-opam-overlays#3](https://github.com/dune-universe/mirage-opam-overlays/pull/3)
- `opam-monorepo`: improve error messages (Plat161)
- Rebasing, fixing, and adding tests to
[ocamllabs/opam-monorepo#248](https://github.com/ocamllabs/opam-monorepo/pull/248)
- Issue: `grep -z` causes files to diff as binary [ocaml/dune#5417](https://github.com/ocaml/dune/issues/5417)
- PR: Test for binary grep output in Cram tests [ocaml/dune#5416](https://github.com/ocaml/dune/pull/5416)
- Finished and merged the better error message on packages that have newer
- [Patch Tezos repository for `opam-monorepo`](https://gitlab.com/tezos/tezos/-/merge_requests/4497)
- [Freshen up hacl-star port](https://github.com/dune-universe/hacl-star-raw/tree/dune-universe-v0.4.5)
- `opam-overlays` CI (Plat165)
- PR: Allow running multiple checks before exiting
[dune-universe/duniverse-lint#7](https://github.com/dune-universe/duniverse-lint/pull/7)
- Issue: Allow configurable version suffix
[dune-universe/duniverse-lint#8](https://github.com/dune-universe/duniverse-lint/issues/8)
- Issue: Allow checking for specific tags
[dune-universe/duniverse-lint#9](https://github.com/dune-universe/duniverse-lint/issues/9)
- APPROVED Add cross compile flag to Zarith and fix version number
[dune-universe/mirage-opam-overlays#3](https://github.com/dune-universe/mirage-opam-overlays/pull/3#pullrequestreview-881342989)
- `opammonorepo`: hybrid mode (Plat160)
- Rebasing and fixing
[ocamllabs/opam-monorepo#234](https://github.com/ocamllabs/opam-monorepo/pull/234)
- Hybrid mode PR: rebase on current redesign logic in how it picks packages
[ocamllabs/opam-monorepo#234](https://github.com/ocamllabs/opam-monorepo/pull/234)
- Solve the dependecies a second time to get a co-installable set of dependencies
[ocamllabs/opam-monorepo#234](https://github.com/ocamllabs/opam-monorepo/pull/234)
## CI
### Improve CI
- Support `ocaml-ci` on Tezos (Plat145)
- OCaml GitLabs maintenance and Opam release (0.1.2).
- Build fixes for `tezos-ci` UI library [tarides/current-web-pipelines#2](https://github.com/tarides/current-web-pipelines/pull/2)
- Prototype Octez (Tezos) GitLab CI Pipelines with an OCurrent-based system (Plat154)
- Fixed missing web hook events from gitlab.com/tezos/tezos (tarides/tezos-ci#13)
- Tezos cluster maintenance, restarting stuck workers and edits to Grafana dashboard.
- Set-up OCurrent-based CI for Nomadic Labs projects (Plat166)
- Added ARM capacity to Tezos cluster. The cluster now has 2 x86_64 workers and 1 ARM64 worker.
- New engineer starting on Tezos-CI: setting test deployment
and beginning with low-hanging fruit (updating dependencies, etc.)
- Fix issue with `tezos-ci` with parsing GitLab refs that included non-ASCII symbols (tarides/tezos-ci#11)
- Review Gaps between OCurrent prototype and Octez GitLab Pipelines documented. Meeting with Nomadic Labs.
### OCaml Projects Are Continuously Tested
- General maintenance of OCurrent projects (Plat105)
- Finalised v3.ocaml.org ansible setup, ready for launch. [ocamllabs/infrastructure#135](https://github.com/ocamllabs/infrastructure/pull/135)
- Add Opam 2.1 builds [ocurrent/ocaml-ci#413](https://github.com/ocurrent/ocaml-ci/pull/413)
- Various `ocaml-ci` updates:
- Remove deprecated Centos8 support [ocurrent/ocaml-ci#415](https://github.com/ocurrent/ocaml-ci/pull/415)
- Add Fedora 35 support
- Updating opam.ocaml.org Docker image with newer Debian version and deployable via `ocurrent-deployer`.
- Updated the dependencies of `opam-repo-ci` to ease Windows port.
- Extracting solver into a service to run on OCluster [ocurrent/ocluster#157](https://github.com/ocurrent/ocluster/pull/157)
- Optimsed `opam-repo-ci` performance when tracking jobs [ocurrent/opam-repo-ci#150](https://github.com/ocurrent/opam-repo-ci/pull/150)
- Rebased and updated `docker-base-images` changes and merged PRs (avsm/ocaml-dockerfile#85, etc.)
- Various cmdliner 1.1.0 pins and updates to the new API.
- (ocurrent/docker-base-images#159) (ocurrent/ocurrent#310) (ocurrent/opam-repo-ci#157) (ocurrent/ocurrent-deployer#99)
- [ocaml-opam/opam-0install-solver#39](https://github.com/ocaml-opam/opam-0install-solver/pull/39)
- [mirage/ocaml-git#558](https://github.com/mirage/ocaml-git/pull/558)
- [ocurrent/obuilder#100](https://github.com/ocurrent/obuilder/pull/100)
- [ocurrent/ocluster#160](https://github.com/ocurrent/ocluster/pull/160)
- [ocurrent/ocurrent#311](https://github.com/ocurrent/ocurrent/pull/311)
- [stedolan/crowbar#64](https://github.com/stedolan/crowbar/pull/64)
- [dinosaure/hxd#12](https://github.com/dinosaure/hxd/pull/12)
- [mirage/capnp-rpc#249](https://github.com/mirage/capnp-rpc/pull/249)
- [MagnusS/okra#103](https://github.com/MagnusS/okra/pull/103)
- [mirage/ocaml-matrix#18](https://github.com/mirage/ocaml-matrix/pull/18)
- [mirage/alcotest#339](https://github.com/mirage/alcotest/pull/339)
- [mirage/index#382](https://github.com/mirage/index/pull/382)
- [mirage/irmin#1771](https://github.com/mirage/irmin/pull/1771)
- Reported bug in cmdliner with rendering duplicate help text for flags (dbuenzli/cmdliner#135)
- Fixes for `ocurrent-deployer` pipline for v3.ocaml.org (ocamllabs/infrastructure#137) (ocurrent/ocurrent-deployer#97)
- New Docker image and deploy of opam-3.ocaml.org to replace existing opam.ocaml.org.
- Testing fix for missing `opam-dev` PGP key (ocaml/opam.ocaml.org#25) and (ocaml/opam.ocaml.org#23)
- Maintenance of OCaml GitHub library (mirage/ocaml-github#264) and (mirage/ocaml-github#265)
- Maintenance of OCurrent libraries (ocurrent/current_incr#2), (ocurrent/ocurrent#312)
- Issue with HTTPS redirect for deploy.ocamllabs.io and ci.ocamllabs.io (ocamllabs/infrastructure#141)
- Maintenance of `ocaml-ci` issues
- Issue: Log Rules UI Improvements (ocurrent/ocaml-ci#422)
- Issue: Add support for building GitLab hosted projects. (ocurrent/ocaml-ci#423)
- Issue: Expose ignored projects in Web UI (ocurrent/ocaml-ci#424)
- Issue: Jobs UI Improvements. (ocurrent/ocaml-ci#425)
- Update vendored dependencies in `ocaml-ci` and tezos-ci.
- Install Cap'n Proto binaries in Windows GHA and run tests [mirage/capnp-rpc#248](https://github.com/mirage/capnp-rpc/pull/248)
- Maintenance and release of Capnp 3.5.0 library. Remove compiler flags for flambda.
- Remove compiler flags for Flambda [capnproto/capnp-ocaml#83](https://github.com/capnproto/capnp-ocaml/pull/83)
- Release Capnp 3.5.0 [ocaml/opam-repository#20778](https://github.com/ocaml/opam-repository/pull/20778)
The previous release was still using Dune 1.0, and didn't work with Dune 3.0.
- Update docker-base images to Cmdliner 1.1.0 [ocurrent/docker-base-images#160](https://github.com/ocurrent/docker-base-images/pull/160)
- Patch and then restore `opam-lint` in `ocaml-ci` as `opam lint` rather than `dune-opam-lint` tool
(ocurrent/ocaml-ci#427) (ocurrent/opam-dune-lint#44)
- Improving the `opam-dune-lint` installation to re-solve with the constraint of `opam-dune-lint` added to the package that's being tested
Dune 3.0 removed internal library that `opam-dune-lint` was using, need Dune help to restore functionality.
- Work on getting the Windows workers running again in the cluster
- Fixes for `opam-repo-ci-web` network and Docker issues. Updated deployment setup with webhook secret and NGINX proxy fix. (ocamllabs/infrastructure#144)
- `ocaml-ci`, `opam-repo-ci`, and `opam-health-check` support Linux, macOS and Windows (Plat49)
- Released Extunix (0.4.0) [ocaml/opam-repository#20649](https://github.com/ocaml/opam-repository/pull/20649)
- "Typical" `Makefile` fix for Windows, will be removed when Mirage 4.0 is released: On Windows, run `PKG_CONFIG_PATH` through `cygpath` ([mirage/mirage-crypto#150](https://github.com/mirage/mirage-crypto/pull/150))
- General maintenance of OSX machines in the cluster
- Debugging issue with Window worker deployment for new machines and fixing the deployer script
- Rebased Windows LWT patches to fix file descriptor leak
- Rebase Windows LWT CreateProcess patch on OCaml. Found the problem that on Windows, Unix.kill, and Unix.waitpid only work on processes spawned by the current process and not on arbitrary processes.
- Issue raised for Win32Unix: cannot kill and wait for arbitrary processes
[ocaml/ocaml#11021](https://github.com/ocaml/ocaml/issues/11021)
- General maintenance updating base images for Mac workers
- MacOS worker maintenance
- Setup Windows debugging infrastructure in LWT to track down Windows worker issues. Porting LWT test suite to Alcotest and submitted patches
to Alcotest.
- Document filter parameter [mirage/alcotest#341](https://github.com/mirage/alcotest/pull/341)
- Correctly lower-bound on OCaml 4.05 [mirage/alcotest#342](https://github.com/mirage/alcotest/pull/342)
- Port `lwttester` to Alcotest [ocsigen/lwt#935](https://github.com/ocsigen/lwt/pull/935)
- Testing Dune 3.0 on Windows and reporting bugs.
- Dune 3.0 `dune exec --` on Windows *always* rebuilds the executable [ocaml/dune#5466](https://github.com/ocaml/dune/issues/5466)
- Dune 3.0 all generated or promoted files on Windows are set executable [ocaml/dune#5467](https://github.com/ocaml/dune/issues/5467)
- Dune 3.0 cannot delete files named `NUL` on Windows [ocaml/dune#5485](https://github.com/ocaml/dune/issues/5485)
Sign in with Wallet
Connect another wallet