# Running MITgcm on Sherlock

[MTIgcm](https://mitgcm.readthedocs.io/en/latest/getting_started/getting_started.html#) is a software package written largely in Fortran 90 that has been used for decades in Atmospheric and Oceanic science to run idealized forward models of weather or climate phenomena. The documentation is quite good, but seeing as I don't know what I'm doing on Sherlock or with Fortran, I figure that I'll make this HackMD file as I go.

## Step 1: Compile

Here is a script written by Rob King that is saved as build_mitgcm.sh. Fortran is a language that requires the code is first compiled, and then it is run. As such, it is a two step process. This code accomplishes the first step. It does not use MPI, but Niall may change this when necessary. Additionally, it assumes that you have downloaded MITgcm to your home directory. If you have not, change line 35 to reflect where you have downloaded MITgcm.

#!/bin/bash
#SBATCH -p serc
#SBATCH -c 16
#SBATCH --mem 8G
#SBATCH --time 00:30:00
#SBATCH --job-name buildmitgcm

## DOES NOT USE MPI CURRENTLY (easy to mod it allow this though)

# Load Modules
module purge
ml icc
ml ifort
ml impi
ml netcdf


if [ -z "${MITGCM_ROOT}" ]; then
   MITGCM_ROOT=$HOME/MITgcm
fi


if [ -d ./build ]; then
    cd build
else
    echo "No build directory present, assuming I can build where I am"
fi 

if [ -d ../code ]; then
     CODE_DIR="-mods ../code"
else
     CODE_DIR=""
fi
echo "Begin Build"
${MITGCM_ROOT}/tools/genmake2 $CODE_DIR -optfile ${MITGCM_ROOT}/tools/build_options/linux_amd64_ifort+impi_sherlock
make depend
make clean
make 
make clean
echo "done"

Additionally, there is a required modification to the compiler. As such, the following file also generated by Rob King called *linux_amd64_ifort+impi_sherlock* should be placed in `MITGCM_ROOT/tools/build_options/`:

#!/bin/bash
#
#  Build options for ifort with impi on Linux AMD64 platform
#
#  Tested on sherlock (yeet)
#
#-------

if test "x$MPI" = xtrue ; then
  CC=${MPICC:=mpiicc}
  FC=${MPIFC:=mpiifort}
  F90C=${MPIF90C:=mpiifort}
  LINK="$F90C -shared-intel -no-ipo"
else
  CC=icc
  FC=ifort
  F90C=ifort
  LINK="$F90C -shared-intel"
fi

DEFINES='-DWORDLENGTH=4'
F90FIXEDFORMAT='-fixed -Tf'
EXTENDED_SRC_FLAG='-132'
GET_FC_VERSION="--version"
OMPFLAG='-openmp'

NOOPTFLAGS='-O0 -g'
NOOPTFILES=''

#- for setting specific options, check compiler version:
fcVers=`$FC $GET_FC_VERSION | head -n 1 | awk '{print $NF}'`
if ! [[ $fcVers =~ ^[0-9]+$ ]] ; then
  echo "    un-recognized Compiler-release '$fcVers' ; ignored (-> set to 0)" ; fcVers=0 ;
else echo "    get Compiler-release: '$fcVers'" ; fi
if [ $fcVers -ge 20160301 ] ; then
    OMPFLAG='-qopenmp'
fi

if test "x$GENERIC" != x ; then
    PROCF=-axSSE4.2,SSE4.1,SSSE3,SSE3,SSE2
else
    PROCF=-xHost
fi

CFLAGS="-O0 -ip -m64 $PROCF"
FFLAGS="$FFLAGS -m64 -convert big_endian -assume byterecl"
#- for big setups, compile & link with "-fPIC" or set memory-model to "medium":
#CFLAGS="$CFLAGS -fPIC"
#FFLAGS="$FFLAGS -fPIC"
#-  with FC 19, need to use this without -fPIC (which cancels -mcmodel option):
 CFLAGS="$CFLAGS -mcmodel=medium"
 FFLAGS="$FFLAGS -mcmodel=medium"
#- might want to use '-r8' for fizhi pkg:
#FFLAGS="$FFLAGS -r8"

if test "x$IEEE" = x ; then     #- with optimisation:
    FOPTIM="-O2 -align -ip -fp-model source $PROCF"
else
  if test "x$DEVEL" = x ; then  #- no optimisation + IEEE :
    FOPTIM="-O0 -fp-model source -noalign $PROCF"
  else                          #- development/check options:
   #FFLAGS="$FFLAGS -debug all -debug-parameters all -fp-model strict"
    FOPTIM="-O0 -noalign -g -traceback $PROCF"
    NOOPTFLAGS=$FOPTIM
    NOOPTFILES='adread_adwrite.F'
    FOPTIM="$FOPTIM -warn all -warn nounused"
    FOPTIM="$FOPTIM -fpe0 -ftz -fp-stack-check -check all -ftrapuv"
  fi
fi

F90FLAGS=$FFLAGS
F90OPTIM=$FOPTIM

INCLUDEDIRS=''
INCLUDES=''
LIBS=''

if [ "x$NETCDF_ROOT" != x ] ; then
    INCLUDEDIRS="${NETCDF_ROOT}/include"
    INCLUDES="-I${NETCDF_ROOT}/include"
    LIBS="-L${NETCDF_ROOT}/lib64 -L${NETCDF_ROOT}/lib"
elif [ "x$NETCDF_HOME" != x ]; then
    INCLUDEDIRS="${NETCDF_HOME}/include"
    INCLUDES="-I${NETCDF_HOME}/include"
    LIBS="-L${NETCDF_ROOT}/lib64 -L${NETCDF_HOME}/lib"
elif [ "x$NETCDF_INC" != x -a "x$NETCDF_LIB" != x ]; then
    NETCDF_INC=`echo $NETCDF_INC | sed 's/-I//g'`
    NETCDF_LIB=`echo $NETCDF_LIB | sed 's/-L//g'`
    INCLUDEDIRS="${NETCDF_INC}"
    INCLUDES="-I${NETCDF_INC}"
    LIBS="-L${NETCDF_LIB}"
elif [ "x$NETCDF_INCDIR" != x -a "x$NETCDF_LIBDIR" != x ]; then
    INCLUDEDIRS="${NETCDF_INCDIR}"
    INCLUDES="-I${NETCDF_INCDIR}"
    LIBS="-L${NETCDF_LIBDIR}"
elif test -d /usr/include/netcdf-3 ; then
    INCLUDEDIRS='/usr/include/netcdf-3'
    INCLUDES='-I/usr/include/netcdf-3'
    LIBS='-L/usr/lib64/netcdf-3 -L/usr/lib/netcdf-3'
elif test -d /usr/local/pkg/netcdf ; then
    INCLUDEDIRS='/usr/local/pkg/netcdf/include'
    INCLUDES='-I/usr/local/pkg/netcdf/include'
    LIBS='-L/usr/local/pkg-x86_64/netcdf/lib64 -L/usr/local/pkg-x86_64/netcdf/lib -L/usr/local/pkg/netcdf/lib64 -L/usr/local/pkg/netcdf/lib'
elif test -d /usr/include/netcdf ; then
    INCLUDEDIRS='/usr/include/netcdf'
    INCLUDES='-I/usr/include/netcdf'
elif test -d /usr/local/netcdf ; then
    INCLUDEDIRS='/usr/include/netcdf/include'
    INCLUDES='-I/usr/local/netcdf/include'
    LIBS='-L/usr/local/netcdf/lib64 -L/usr/local/netcdf/lib'
elif test -f /usr/local/include/netcdf.inc ; then
    INCLUDEDIRS='/usr/local/include'
    INCLUDES='-I/usr/local/include'
    LIBS='-L/usr/local/lib64 -L/usr/local/lib'
fi

if [ -n "$I_MPI_ROOT" -a -z "$MPI_INC_DIR" ]; then
    MPI_INC_DIR="$I_MPI_ROOT/include"
fi

if [ -n "$MPI_INC_DIR" -a "x$MPI" = xtrue ] ; then
    INCLUDES="$INCLUDES -I$MPI_INC_DIR"
    INCLUDEDIRS="$INCLUDEDIRS $MPI_INC_DIR"
    #- used for parallel (MPI) DIVA
    MPIINCLUDEDIR="$MPI_INC_DIR"
   #MPI_HEADER_FILES='mpif.h mpiof.h'
fi

## Step 2: Run

You can put this in a .sh script, or you can run the following commands and save the code output (other than files generated) on `output.txt`:

cd ../run
ln -s ../input/* .
cp ../build/mitgcmuv .
module purge
ml icc
ml ifort
ml impi
ml netcdf
./mitgcmuv > output.txt

So fun. You now get a bunch of .data and .meta files. What do they mean?

### Step 2.1: Using MPI for Computationally Expensive Runs

## Step 3: Analyze Output

# Jupyter Access on Open OnDemand

## Symbolic Link for Folders that Live Outside of Home

Shoutout [Caltech](https://www.hpc.caltech.edu/documentation/software-and-modules/jupyter-notebook#:~:text=Launch%20Jupyter%20via%20OOD%20and,the%20%25pwd%20command%20to%20verify.) for this one. Suppose you want to access a folder that isn't your home directory while running Jupyter Lab on Sherlock. You can use use `mkdir -p /new/path/to/folder/` to create a folder that will have a parent directory. Next, symbolically link with `ln -s /new/path/to/folder/ ./`. Works like a charm.

## Modules to Load Prior to Using OnDemand

Note that you will have to load all of the modules you want before starting an active kernel. 

<!-- To save some time, there is an additional modules to load button when launching the server ([see here](https://www.sherlock.stanford.edu/docs/user-guide/ondemand/#jupyter-notebooks)), and a link to the list of software you can pull from. To save some time with python, load `py-scipystack/1.0_py36`, use python 3.6, and then see if you need anything else [here](https://www.sherlock.stanford.edu/docs/software/list/).

Update: they deprecated this. -->

## Pip Install Software in Python

[Sherlock](https://www.sherlock.stanford.edu/docs/software/) won't make the software widely available unless it is useful across many fields like numpy, scipy, pandas, tensorflow, etc.

So if you want to use MITgcm's Python software [MITgcmutils](https://mitgcm.readthedocs.io/en/latest/utilities/utilities.html#), you can use `pip install --user MITgcmutils` to install locally to your home directory.
> Since your $HOME directory is shared across nodes on Sherlock, you'll only need to install your Python packages once, and they'll be ready to be used on every single node in the cluster.

However, the software is available to nobody else. If you want a group shared software installation, read up on [Alternative Installation Path](https://www.sherlock.stanford.edu/docs/software/using/python/#installing-packages).