Run CESM (v1.2.1) on SNOW cluster (Feb, 2022)

Log in to the SNOW cluster

ssh -X [yourNetID]@head.arcc.albany.edu

Setup environment variables

Edit the .bashrc file in the home directory.

• Set CESM root path - export the CESM(v1.2.1) built by Brian Rose.

export CCSMROOT="/network/rit/home/br546577/roselab_rit/cesm/cesm1_2_1"

• Set Intel compiler path

export PATH="$PATH:/network/rit/lab/snowclus/2101/bin:/network/daes/linuxapps/intel17/bin"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/network/rit/lab/snowclus/2101/lib:/network/daes/linuxapps/intel17/lib/intel64"
export LM_LICENSE_FILE="/network/daes/linuxapps/intel17/bin/licenses/intel.lic"

• Set NetCDF library and linker flags

export NETCDF="/network/rit/lab/snowclus/2101"
export LDFLAGS="-L/network/rit/lab/snowclus/2101/lib"

Run CESM

• Create Case (set case name, resolution, component sets)

/network/rit/lab/roselab_rit/cesm/cesm1_2_1/scripts/create_newcase -mach snow -res [resolution] -compset [compset_name] -case [casename]

• Modify env_run.xml if needed. For example, modify the run time to 3 years and do not archive the data output.

./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=3
./xmlchange DOUT_S=FALSE

• Setup

./cesm_setup

• Build (on a compute node)

srun -p snow `pwd`/[casename].build

• Edit [casename].run
  For MPI4.0, an error (initialization failed) should be fixed. See https://www.open-mpi.org/faq/?category=openfabrics#ofa-device-error. In line #186, add -mca btl ^openib -mca pml ucx after mpirun.

#SBATCH --ntasks-per-node=32
#SBATCH --mem-per-cpu=3584M
mpirun -mca btl ^openib -mca pml ucx

• Submit

./[casename].submit

• Look up the log file

less +F cesm.log.xxxxxx

Slurm

• See what is running

squeue
sinfo

• See specific node info

scontrol show node [snow-xx]

• Kill the running case

scancel [JOBID]