1. Software prerequisites 1. Related to biological data analysis 1. Exception(s): CRAN R packages go to conda-forge (but bioconductor -> bioconda)! 2. Publicly available, ideally on github/gitlab (no strictly necessary) 1. If Github/Gitlab with a proper release with tar ball! 3. Check dependencies available on [bioconda](https://bioconda.github.io/)/[conda-forge](https://conda-forge.org/feedstock-outputs/) 2. Environment prerequisites 1. [conda](https://docs.conda.io/en/latest/miniconda.html) installed 2. conda-build installed in (an) conda environment ```bash conda install conda-build ``` 3. Fork bioconda-recipes 4. Clone locally 5. Make a new branch 6. Make new directory of tool name (all lowercase) under `recipes/` 7. Make a `meta.yaml` file 1. ℹ️ Copy as much as possible from other tools! e.g. for `authentict` look at `autogenes` or `endorspy` 2. Make sure to download the `.tar.gz` of the tool and find the **`sha256`** of the tar ball ```bash wget https://github.com/StephanePeyregne/AuthentiCT/archive/refs/tags/v1.0.0.tar.gz sha256 v1.0.0.tar.gz ``` 3. **build** number: corresponds to build of recipe, i.e. internal version corresponding to build attempt of that particular build of a tool 4. for **requirements** `host` is build environment requirements for building, `run` is dependencies for the tool to perform calculations (when a user is actually using it) 5. **test** has different modes, easiest is `command`, check that e.g. `--help` works - should run with exit code 0. Not for full/heavy CI-like tests! 6. **about** license from fix list (will get linting error), and nice to specify the location of the licence file 7. **extras** bunch of optional useful stuff e.g. maintainer, DOI identifier 8. Possibly make `build.sh` 1. e.g. for C++, rust Java tools - copy as much as possible from other tools! 2. e.g. rust: alven-fry 3. Also see [guidelines](https://bioconda.github.io/contributor/guidelines.html) for suggestions 9. (Optionally) Run build locally with `conda build` (or better: mamba-build); or can let bioconda CI tests do it on Azure -> even slower though. ```bash conda build recipes/authentict ``` To locally load the build env and test ```bash cd /<path>/<to>/miniconda3/envs/bioconda-build/conda-bld/linux-64 conda create -n concoct-dev -c ./ concoct ## using the folder containing repodata.json as a 'channel' ## can confirm it's using teh right build if in the 'OK to install packages says 'concoct conda-bld/linux-64::concoct-1.1.0-py312h9bf148f_6 `` ``` Or even better(!), use the exact bioconda-build environment used on Azure! ```bash bioconda-utils build --docker --mulled-test --packages authentict ``` ℹ️ Can generally ignore warnings ⚠️ Can take some time! AuthentiCT: ~16m 10. Once working open pull request to bioconda repo! 11. Once Azure CI tests pass, add review label by being polite to the bot 12. Once review is in, reviewer (from bioconda team) will add another label for bot to merge 13. Once merged, docker container will automatically uploaded to biocontainers quay.io account 1. In some cases _new_ recipes (not updates) will not have quay.io repo updated correctly, need to ask on biocontainers gitter 14. 24h later, should then be converted to singulairty container on galaxy singularity depot Other tips and tricks: - bioconda scans github repos for new releases, if finds a new release will attempt to auto-build new recipe (does not pick up dependency changes though!) - greyskull for building recipes automagically for you via e.g. of python or CRAN tools ## Debugging bioconda-utils build 1. Access build env in docker, use command as reported during the command above but replace the script with just `bash` and `-t` to `-it` ```bash docker run -it --net host --rm -v /tmp/tmpqy75e4vf/build_script.bash:/opt/build_script.bash -v /home/james/bin/miniconda3/envs/bioconda-build/conda-bld/:/opt/host-conda-bld -v /home/james/git/jfy133/bioconda-recipes/recipes/phynder:/opt/recipe -e LC_ADDRESS=en_GB.UTF-8 -e LC_NAME=en_GB.UTF-8 -e LC_MONETARY=en_GB.UTF-8 -e LC_PAPER=en_GB.UTF-8 -e LANG=en_GB.UTF-8 -e LC_IDENTIFICATION=en_GB.UTF-8 -e LC_TELEPHONE=en_GB.UTF-8 -e LC_MEASUREMENT=en_GB.UTF-8 -e LC_TIME=en_GB.UTF-8 -e LC_NUMERIC=en_GB.UTF-8 -e HOST_USER_ID=1000 quay.io/bioconda/bioconda-utils-build-env-cos7:2.11.1 bash ``` Note that basic stuff like `vim` is not in there! So you will have to exit the docker image to edit the files each time 2. The recipe directory is stored in ```bash /opt/recipe/ ``` The build_script is ```bash /opt/build_script.sh ``` 3. Try to run one by one the commands in `build_script.sh` command (`cat build_script.sh`, and opt and paste), but WITHOUT running the `set -eo pipefail` at the beginning, as this will cause you to exit docker if something fails! In particular the following command is the one that is main building command, and from which you can debug from ```bash conda mambabuild -c file:///opt/host-conda-bld --override-channels --no-anaconda-upload -c conda-forge -c bioconda -c defaults -e /opt/host-conda-bld/conda_build_config_0_-e_conda_build_config.yaml -e /opt/host-conda-bld/conda_build_config_1_-e_bioconda_utils-conda_build_config.yaml /opt/recipe/meta.yaml 2>&1 ``` If this fails during the `build.sh` steps, and you want to run the `build.sh` commands manually, make sure to activate the conda environment defined in the `meta.yml` first, eg.. ```bash conda activate /opt/conda/conda-bld/phynder_1709250817719/_build_env ``` You may also need to set the `PREFIX` bash environment variable when dealing with `build.sh`, this is the long `_placeholder` directory that gets reported during teh typical build log, you see this with: ```bash 14:25:48 BIOCONDA INFO (OUT) export PREFIX=/opt/conda/conda-bld/phynder_1709265213923/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place ``` The building itsel happens in a `work` directory, e.g. ```bash /opt/conda/conda-bld/phynder_1709250044471/work ``` 4. To leave docker, type `exit`