changed 2 years ago
Linked with GitHub

AIRaster on Rackham/Snowy

Conda

On Rackham

module load conda
source conda_init.sh
export CONDA_ENVS_PATH=/crex/proj/.../nobackup/CONDA_ENVS
mkdir -p $CONDA_ENVS_PATH

# use maba to instal the env, conda takes forever to solve it.
(base) mamba env create -f AIRaster.yaml
(base) conda activate AIRaster

AIRaster.yml

# conda env create -f AIRaster.yaml
name: AIRaster
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.8
  - gdal
  - fiona
  - imagecodecs
  - matplotlib
  - pip
  - pip:
    - requests
    - tqdm
    - exifread
    - -r file:///FULL_PATH/project/requirements-38.txt
#    - git+https://github.com/blaze/dask.git

Handy reference https://carpentries-incubator.github.io/introduction-to-conda-for-data-scientists/04-sharing-environments/index.html

If you need an environment file that can produce environments that are reproducibile across Mac OS, Windows, and Linux, then you are better off just including those packages into the environment file that your have specifically installed.

$ conda env export --name machine-learning-env --from-history --file environment.yml

In short: to make sure others can reproduce your environment independent of the operating system they use, make sure to add the ``from-history` argument to the conda env export command.

Kind of important when mixing conda and pip.
https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#using-pip-in-an-environment

Singularity/Docker

Comments:

  • the docker file copy files COPY ./ ./ that are in AIRaster-dataprocessing I see - in the zip file
  • there are programs installed in the container that are probably missing in the conda environment but are available on your computer i.e. RUN apt-get install ffmpeg libsm6 libxext6

To build docker container then convert to Singularity container

cd project
docker build -t mdpx .
sudo singularity build mdpx.sif docker-daemon://mdpx:latest

Contacts:

tags: UPPMAX
Select a repo