Next generation rust-optimizer

Up until the 0.9 series, rust-optimizer was simple. It got exactly one simple cargo project and built it into a single contract.wasm and hash.txt. This contract is single crate that can be published to crates.io or any other package repository to allow reproducible builds. The build result was always exactly one file found in target/wasm32-unknown-unknown/release/*.wasm.

With the rise of workspace based repositories in cosmwasm-plus, we get a better development experience when working with multiple contracts and their dependencies. At the same time, the build and verification process gets more complicated. Some of the problems are

1. Cargo has not-that-great support for working with workspaces
2. Hard to find the contract dirs
3. Hard to find build results
4. No contract-spefific Cargo.locks

Cargo has not-that-great support for working with workspaces

While Cargo's workspaces are nice to avoid re-compiling dependencies multiple times, its support for working with them is pretty much insufficient for our use case.

While you can run simple things like cargo fmt --all and cargo test --all to perform an action in all workspace members, cargo build leads to unexpected results with respect to feature configuration. Running a command on all workspace members is incomplete and inconsistent (cargo fmt --all vs. cargo build --workspace).

Cargo misses a way to iterate over workspace members and execute a command there, similar to lerna exec -- custom_command_here "\$LERNA_PACKAGE_NAME". As a consequence, we need to re-implement workspace member detection if we want to do that.

Hard to find the contract dirs

A naive way to find all contract repositories is to look for Cargo.tomls, which in practice is not trivial since they can appear in unexpected locations, e.g.:

$ cd cosmwasm-examples
$ find . -name Cargo.toml
./erc20/Cargo.toml
./voting/Cargo.toml
./nameservice/Cargo.toml
./nameservice/target/package/cw-nameservice-0.5.1/Cargo.toml
./nameservice/target/package/cw-nameservice-0.5.0/Cargo.toml
./nameservice/target/package/cw-nameservice-0.5.2/Cargo.toml
./nameservice/target/package/cw-nameservice-0.3.1/Cargo.toml
./nameservice/target/package/cw-nameservice-0.4.0/Cargo.toml
./mask/Cargo.toml
./mask/target/package/cw-mask-0.3.0/Cargo.toml
./mask/target/package/cw-mask-0.3.1/Cargo.toml
./mask/target/package/cw-mask-0.1.1/Cargo.toml
./mask/target/package/cw-mask-0.3.2/Cargo.toml
./mask/target/package/cw-mask-0.2.0/Cargo.toml
./escrow/Cargo.toml
./escrow/target/package/cw-escrow-0.5.0/Cargo.toml
./escrow/target/package/cw-escrow-0.5.1/Cargo.toml
./escrow/target/package/cw-escrow-0.3.1/Cargo.toml
./escrow/target/package/cw-escrow-0.5.2/Cargo.toml

Even when filtering out the package results, you cannot differentiate between contracts and other cargo packages:

$ cd cosmwasm-plus
$ find . -name Cargo.toml
./Cargo.toml
./contracts/cw20-base/Cargo.toml
./contracts/cw20-escrow/Cargo.toml
./contracts/cw1-whitelist/Cargo.toml
./contracts/cw1-subkeys/Cargo.toml
./packages/cw1/Cargo.toml
./packages/cw20/Cargo.toml
./packages/cw2/Cargo.toml

A more reliable way would be to parse workspace.members from the root Cargo.toml and loop over them. However, this still does not tell you which package is a contract and which is not. Also if we start parsing tomls, we should move from bash to a proper scripting language with a rich standard library like Python.

Hard to find build results

With workspaces, the build results will be somewhere in the repo root's target/wasm32-unknown-unknown/release, which is different from simple projects where ./target lives next to Cargo.toml. Supporting both targets folders at the same time is hard:

# find the target directories and build results, optimize and write to artifacts
while IFS= read -r -d '' target_dir; do
  for wasm in "$target_dir"/release/*.wasm; do
    echo "Found build result $wasm"
    name=$(basename "$wasm")
    wasm-opt -Os "$wasm" -o "$artifacts_dir/$name"
    echo "Created artifact $name"
  done
done <  <(find . -name wasm32-unknown-unknown -type d -print0)

Also having multiple build results can be a problem, especially when caching comes in. Then you never know if all the *.wasm files found really belong to the current build. The problem is the build and collect approach, where the build result is not connected to the execution of the build.

This problem can probably be solved by using the (night-only) --out-dir flag:

$ cargo +nightly -Z unstable-options build --out-dir ./contract_artifacts
$ ls ./contract_artifacts 
cw1_whitelist.wasm    libcw1_whitelist.rlib

In this case you don't even need to care where the target folder is.

No contract-spefific Cargo.locks

Cargo.lock is needed for reproducible builds. Without it, a rebuild of a contract can fetch a different version of a dependency and create a different result.

When working with a workspace repo, it is unclear what a single contract's source code is. It turns out that is cannot be the sub-folder of a certain package if there is no Cargo.lock included. As a consequence, every single contract must consider the entire workspace as its source code. This makes source code large and harder to review.