---
# System prepended metadata

title: Installing ADI on Cumulus

---

Here are the steps I took to create an ADI environment on Cumulus:

Step 1: Log into Cumulus...

Step 2: Add to your .bashrc the ADI libraries. 

```
module use --append /sw/cirrus/ums/cli120/modulefiles/tcl/linux-rhel8-x86_64
module load py-adi-3.4.1-gcc-8.3.1-6n6sbr2
module load libarmutils-1.14.4-gcc-8.3.1-qk27exz
module load libcds3-1.20.2-gcc-8.3.1-63coejn
module load libdbconn-1.12.3-gcc-8.3.1-ag52wxy
module load libdsdb3-1.12.1-gcc-8.3.1-5kfxqx2
module load libdsproc3-2.55.0-gcc-8.3.1-3z2zwtx
```


Step 2: Create a new environment:

For this, I highly recommend against using the default Conda installation on Cumulus. It is 2 years out of date and has no support for the mamba solver. The mamba solver greatly speeds up the creation of new environments by using a C-based environment solver. There is a Miniconda distribution that contains the mamba solver called Mambaforge, available for x86 Linux [here](https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh).

Once you have Mamabforge installed in your local directory, then copy the following text into an `environment.yml` file:
```
name: my_vap_env
channels:
  - arm-doe
  - conda-forge
dependencies:
  - python
  - scipy
  - adi_py
  - numpy
  - pandas
  - xarray
  - act-atmos
```
* Then, create the new environment with `mamba env create -f environment.yml`
* Activate the new environment: `mamba activate my_vap_env`

Step 3: Define the base environment variables for ADI.

ADI requires that you specify where the logs are output, as well as the location of the datastreams, your data collection, and the quicklook output directory in four enviroment variables as below:

```
export LOGS_DATA=/ccsopen/home/rjackson/adi_logs
export DATASTREAM_DATA=/gpfs/wolf2/arm/atm124/proj-shared
export COLLECTION_DATA=/gpfs/wolf2/arm/atm124/proj-shared
export QUICKLOOK_DATA=/gpfs/wolf2/arm/atm124/proj-shared/cmac_quicklooks
```
Change the above environent variables to directories where you want these to be stored. In addition, you will need a `.db_connect` file containing the connection information to the database in your home directory.

Add this to your `.db_connect` file in your home directory (/ccsopen/home/user)


Step 4: Install and run your ingest!
You'll need to clone the ingest from code.arm.gov and then install in this environment.

I would recommend the following options when running your ingest in a slurm job:

`--asynchronous --disable-db-updates `

In addition, if your task takes a long time to process a single day, I recommend running the task in parallel with a node for each day using a SLURM job array. An example of such a slurm script for cumulus is given below:

```
#!/bin/bash
#SBATCH -N 1
#SBATCH -A atm124
#SBATCH --time=6:00:00
#SBATCH --mail-user=rjackson@anl.gov

mamba activate my_vap_env
export LOGS_DATA=/ccsopen/home/rjackson/adi_logs
export DATASTREAM_DATA=/gpfs/wolf/atm124/proj-shared
export COLLECTION_DATA=/gpfs/wolf/atm124/proj-shared
export QUICKLOOK_DATA=/gpfs/wolf/atm124/proj-shared/cmac_quicklooks
# Use the array task ID to specify date
if [$SLURM_ARRAY_TASK_ID -lt 10]
then
        _date=2022090
else
        _date=202209
fi

_date+=$SLURM_ARRAY_TASK_ID

# If you want each task to write to separate logs specify a log directory for each array job
export LOGS_DATA=/ccsopen/home/rjackson/adi_logs/array_$SLURM_ARRAY_TASK_ID
mkdir $LOGS_DATA

cmaccsapr2 --asynchronous -s hou -f S2 -b $_date.000000 -e $_date.235959 -D 1 -R --disable-db-updates 

```

`$SLURM_ARRAY_TASK_ID` is a variable that is specified by the user that gives an ID to a specific job in the array. These array jobs can be started on cumulus with the above as `sbatch --array=x-y sbatch_file`.