---
title: Setting Up and Running MITgcm at GFDL
tags: MITgcm, Gaea, GFDL
description: A guide to getting MITgcm running on Gaea using GitHub.
---
# Setting Up and Running MITgcm at GFDL
This page: https://hackmd.io/@marionalberty/mitgcmongaea
This document is a guide to getting [MITgcm](http://mitgcm.org/) downloaded and running on Gaea using Git and GitHub. Much of this document is based on the guidelines given in the [MITgcm documentation](https://mitgcm.readthedocs.io/en/latest/contributing/contributing.html) but with additional instructions for running and compiling the code on Gaea at GFDL.
Note any lines that are to be entered into your shell start with a dollar sign ($); the dollar sign is not to by typed into the terminal. Also replace any instances of `First.Last` with the first and last name associated with your actual username.
## Git and GitHub
[Git](https://en.wikipedia.org/wiki/Git) is a version control program that allows the user to track changes to code, develop new modules, and compare differences across versions of your code. It is extremely useful when collaborating on model development or data analysis and can even be used to version control documents written in [LaTeX](https://www.latex-project.org/) (though you should have Git ignore .PDFs, but here we digress). The bottom line is that Git is a very useful program that is generally included on Linux based systems, including the GFDL systems (Gaea, Public, etc).
If you are unfamiliar with Git, Software Carpentry has a [good tutorial](https://swcarpentry.github.io/git-novice/), though you should feel free to find whatever tutorial works for you.
[GitHub](https://github.com/) is a website that serves as an public, online respository for code and facilitates collaboration between individuals. The [MITgcm source code and its documentation](https://github.com/MITgcm/MITgcm) are hosted in a single repository on GitHub. **In order to contribute to MITgcm you will need to make an account with GitHub.** This is free and likely something that will be useful in an academic or scientific career.
## Getting Started on Gaea
### SSH'ing Into Gaea
Begin by ssh'ing onto Gaea. If you're on your work station, open up a terminal and execute:
```
$ ssh gaea
```
You will then need to enter your own 6-8 character passcode plus the 6 digit number from your RSA fob.
You can also remotely sign into Gaea from a work laptop by running the following command from a terminal (replacing `First.Last` with your actual username):
```
$ ssh First.Last@gaea.gfdl.noaa.gov
```
Alternatively you can add Gaea to your work laptop's ssh configuration file by appending the following lines to the configuration file `~/.ssh/config`:
```
Host gaea
Hostname gaea.gfdl.noaa.gov
User First.Last
```
Then execute `$ ssh gaea` from your terminal and enter your own 6-8 character passcode plus the 6 digit number from your RSA fob.
### Setting Git Configurations
Now that you are on Gaea and before you start working with Git, run the following commands from your terminal to identify yourself:
```
$ git config --global user.email <your email>
$ git config --global user.name '<Your Name>'
```
Eventually we will communicate with GitHub from the terminal and you will need to enter your GitHub username and password. This is something you will probably only want to do once so we will want the computer to securely remember your username and password. To do so run the following command:
```
$ git config --global credential.helper cache
```
This works for a Linux system like Gaea. If you plan to run use Git and GitHub on OSX you can execute the following command instead:
```
$ git config --global credential.helper osxkeychain
```
You may also want to set a default editor for working with Git; find more information about that [here](http://swcarpentry.github.io/git-novice/02-setup/index.html).
## Downloading MITgcm
While logged into GitHub, go to the main MITgcm repository, https://github.com/MITgcm/MITgcm, and click the **Fork** button
This will create a copy of the MITgcm repository in your GitHub repository, which we will refer to as 'origin', where all of your edits will be hosted on GitHub. Read more about forking repositories on GitHub [here](https://guides.github.com/activities/forking/).
Using terminal, we will now clone your copy of MITgcm from your GitHub page to your home directory on Gaea (`/ncrc/home1/First.Last/`) by executing the following commands:
```
$ git clone https://github.com/YOURUSERNAME/MITgcm.git
```
Move into the local clone repository on Gaea
```
$ cd MITgcm
```
We now add an additional remote that points back to the main MITgcm repository, which we will call 'upstream':
```
$ git remote add upstream https://github.com/MITgcm/MITgcm.git
```
We now have setup two remotes on Gaea; 'origin' which points Gaea to our personal copy of MITgcm on GitHub and 'upstream' which points back to the main MITgcm repository. To ensure that you have both properly setup run:
```
$ git remote -v
```
You should get an output that looks like:
```
origin https://github.com/YOURUSERNAME/MITgcm.git (fetch)
origin https://github.com/YOURUSERNAME/MITgcm.git (push)
upstream https://github.com/MITgcm/MITgcm.git (fetch)
upstream https://github.com/MITgcm/MITgcm.git (push)
```
## Creating an Unmodified Reference Branch
[Branches](https://help.github.com/en/articles/about-branches) are a fundamental part of using Git. Branches allow you to maintain an unmodified and *functioning* version of the code, the **master** branch, while you develop new code in parallel on other branches. At this point, the MITgcm that you have on Gaea is the **master** branch. To check which branch you are working on (at this point there should only be one) enter `$ git branch` in your shell. Is is good practice before we make any changes to **master** to also make a branch which contains the unmodified version of the code for reference. We will not add or change *anything* in this version of the code. To do this we will create a new branch using the following lines:
```
$ git fetch upstream
$ git checkout -b unmodified upstream/master
```
We will use short but descriptive branch names to ensure that we know the purpose of each branch. The first command `$ git fetch upstream` ensures that the new branch uses the most up-to-date version of MITgcm from the main 'upstream' repository. The second line creates the new branch **unmodified** and moves us to that branch.
To ensure that we have switched branched enter `$ git branch` again. Now the output should look like this:
```
* unmodified
master
```
We would like our GitHub page to reflect that we have created this second branch, **unmodified**, so it's a good time to push the changes we have made on Gaea to our online repository. This is a relatively small change but it is good to get in the practices of pushing our changes to GitHub often:
```
$ git push -u origin unmodified
```
GitHub will ask you to enter your GitHub username and password at this point. After entering, you may get the following error:
```
remote: Support for password authentication was removed on August 13, 2021. Please use a personal access token instead.
remote: Please see https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/ for more information.
fatal: Authentication failed for 'https://github.com/hdrake/MITgcm.git/'
```
If this is the case, follow the structions for creating a one-time-use "personal access token" and use the generated token as your password instead.
Either way, after entering your credentials you should get the following output:
```
Total 0 (delta 0), reused 0 (delta 0)
remote:
remote: Create a pull request for 'unmodified' on GitHub by visiting:
remote: https://github.com/marionalberty/MITgcm/pull/new/unmodified
remote:
To https://github.com/marionalberty/MITgcm.git
* [new branch] unmodified -> unmodified
Branch unmodified set up to track remote branch unmodified from origin.
```
We won't do anything else with the branch **unmodified**, but rather leave it as a reference for the future in the event that we need it.
However let's go to our own GitHub page and see what pushing the **unmodified** branch did. Navigate to the MITgcm repository on your GitHub page (https://github.com/YOURUSERNAME/MITgcm), find and click the 'Branch' button.
There you should see two branches: **master** and **unmodified** (the above example has an additional branch but we'll get there eventually). You can switch back and forth between the branches so look at the different versions of the code on GitHub. At present the two branches are identical but we'll be making changes to **master** shortly.
## Adding the Build Option for Gaea
Using your terminal which is logged into Gaea, switch back to the **master** branch:
```
$ git checkout master
```
Now move into the `build_options` of MITgcm in your own directory:
```
$ cd tools/build_options/
```
And we'll make a copy of a Gaea build options file (which includes NetCDF libraries) that lives in Marion Alberty's directory and put that copy in our own build options:
```
$ cp /ncrc/home1/Marion.Alberty/MITgcm/tools/build_options/gaea_c3 .
```
A quick side note, if you want the gaea build options file that outputes binary files rather than NetCDF, use the following command to copy the coresponding file from Marion Alberty's directory to your own.
```
$ cp /ncrc/home1/Marion.Alberty/MITgcm/tools/build_options/gaea_c2 .
```
Now we've added a new file to MITgcm on the **master** branch. Let's enter the command `$ git status` in the shell to see what that has done. Your output should look something like this:
```
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
gaea_c3
nothing added to commit but untracked files present (use "git add" to track)
```
We see that Git knows that there is a new file 'gaea_c3', but Git is not tracking that file yet. To make Git track this new build option file we execute the following lines:
```
$ git add gaea_c3
$ git commit
```
The first line `git add` tells Git to start paying attention to and tracking the specified file `gaea_c3`. The second command `git commit` effectively makes photo copy of the branch **master** at this point in time. It also changes the shell window and asks you to enter a comment that will tell you or anyone else tracking the development of this repository what changes you are committing right now. Strike the key `i` to change to insert mode (if default editor is vim) and add a short but descriptive comment like the following:
```
Added Gaea build option
```
To save your commit, hit `Esc` then enter `:wq` to write your changes and quit git commit. The way you exit `git commit` will depend on your default git editor, but `:wq` will work for the default editor (vim) on Gaea if you haven't changed it.
Now our local copy of **master** is ahead of 'origin' on GitHub, so let's rectify that. We'll push our changes of branch **master** to 'origin' on GitHub:
```
$ git push origin master
```
If you go to your GitHub page at this point you'll see that your most recent commit is "Added Gaea build option".
## Set MITgcm to Output NetCDF Instead of Binary files
Now that we've added the build option for Gaea, we will also add the [MNC package](https://mitgcm.readthedocs.io/en/latest/outp_pkgs/outp_pkgs.html#netcdf-i-o-pkg-mnc) to our code that allows us to output the model data to NetCDF files rather than binary files. In the next step, we're going to run `exp2` so we need to move to it's `code` repository to add the package:
```
$ cd ../../verification/exp2/code/
```
Open the file `packages.conf` and add the line `mnc` to the end of the file. The file contents should now look like this:
```
#-- list of packages (or group of packages) to compile for this experiment:
gfd
cd_code
mnc
```
Remember that we'd like Git to keep track of all these changes so execute:
```
$ git add packages.conf
```
We have two more changes to make before we've successfully added the package so we'll wait to commit all the changes together. Next move into the `input` directory,
```
$ cd ../input/
```
and open the file `data.pkg` and add the line `useMNC=.TRUE.,` under the package heading, so it looks like so:
```
# Packages
&PACKAGES
useMNC=.TRUE.,
&
```
Again adding `data.pkg` to git:
```
$ git add data.pkg
```
Finally, we add the file `data.mnc` to the directory `input` by copying the file from Marion Alberty's directory:
```
$ cp /ncrc/home1/Marion.Alberty/MITgcm/verification/exp2/input/data.mnc .
$ git add data.mnc
$ git commit -m "Added MNC to exp2 packages"
$ git push origin master
```
We also add the file to Git, make a Git commit, and push all the changes to our GitHub page. (The -m keyword option for git commit lets you add directly add a descriptive comment).
## Compiling and Running MITgcm on Gaea
Let's run the verification experiment `exp2` at this point to make sure we can build and run MITgcm on Gaea.
To efficiently utilize space on Gaea, we will work with MITgcm across three different directories:
1. The MITgcm code will live on `/ncrc/home1/FIRST.LAST/`, just as we have already set things up.
1. The code will be complied on `/lustre/f2/dev/FIRST.LAST/`. In the future, this is also where you should keep the binary `.bin` input files for initializing your run since those can be large and `dev` is less restrictive space-wise.
1. The model will be run in `/lustre/f2/scratch/FIRST.LAST/`.
### Compiling MITgcm in `dev`
Let's move to where we'll be compiling the code and setup a system of directories there to organize all the experiments that we will eventually build:
```
$ cd /lustre/f2/dev/$USER/
$ mkdir -p MITgcm/verification/exp2/build
$ cd MITgcm/verification/exp2/build/
```
This directory structure is used to emulate the organization of the MITgcm source code.
Now that there is an organized place to compile the verification run, it's time to compile the code:
```
$ module unload PrgEnv-intel
$ module load PrgEnv-pgi
$ module load cray-netcdf
$ ~/MITgcm/tools/genmake2 -rootdir ~/MITgcm -mpi -of ~/MITgcm/tools/build_options/gaea_c3 -mods ~/MITgcm/verification/exp2/code
```
First we unload and load the needed environments then a module that will write our output to NetCDF instead of binary. The next line runs `genmake2` which generates a Makefile, allowing us to compile the code. Here we've used the shorthand `~` instead of writing out the path to our home directory, where we have put the MITgcm code. We've also specified where the MITgcm code lives, pointed `genmake2` to our Gaea specific build option, and the experiment specific modifications. For a more in-depth discussion on `genmake2` see the [MITgcm documentaion](https://mitgcm.readthedocs.io/en/latest/getting_started/getting_started.html#generating-a-makefile-using-genmake2).
Finally we run `make depend` which specifies the dependencies of our Makefile and `make` which actually compiles the code:
```
$ make depend
$ make
```
These steps should result in producing the executable `mitgcmuv`, which we will use to run MITgcm.
### Running MITgcm in `scratch`
We then create a directory in `scratch` where the different runs of experiment will be done:
```
$ mkdir -p /lustre/f2/scratch/$USER/exp2
```
Now, we need to create a run script, for this example we'll call it `run_exp2_test`. The run script will exist in the `build` directory where the `mitgcmuv` executable was created. Start with creating the file and opening up vim (or whichever editor you like) with the following command:
```
$ vim run_exp2_test
```
In the editor, copy and paste the following lines:
```
#!/bin/csh
#SBATCH -A gfdl_o
#SBATCH -J eady
#SBATCH --clusters=c3
#SBATCH --nodes=1
#SBATCH -t 1:00:00
module unload PrgEnv-intel
module load PrgEnv-pgi
module load cray-netcdf
cd /lustre/f2/scratch/$USER/exp2/
mkdir -p run_test
cd run_test
cp /lustre/f2/dev/$USER/MITgcm/verification/exp2/build/mitgcmuv .
ln -s ~/MITgcm/verification/exp2/input/* .
srun --verbose --export=ALL --ntasks=1 --cpus-per-task=1 ./mitgcmuv
```
The first set of commands tells the shell how to read this file and then specifies the SLURM `#SBATCH` options.
An important option when working with MITgcm is the number of nodes required for the simulation. On Gaea, one node is equal to 32 tasks or processors. For this tutorial, MITgcm only needs one processor or task, which is set in the `SIZE.h`. The total number of processors or tasks required for a simmulation is `nPx` times `nPy` and we tell slurm how many processors/tasks are required in the final line of the run file with `--ntasks=1`. Thus the number of nodes require is `ntasks` divided by 32 and rounded up because you can't use a partial node, so for this case 1 node. If your simulation requires 32 task then you request 1 node also, but 2 nodes if you require 33 tasks.
The final `#SBATCH` option sets the amount of wall clock time your simulation requires. Here we request 1 hour, which means the job will stop once it has run for an hour, regardless if MITgcm has completed all the iterations that you specified. The order that SLURM schedules of GFDL's depends on the number of nodes requested and the length of wall clock time. Shorter jobs with fewer nodes are scheduled first.
The next set of commands make sure that the required modules are loaded. The third set move us into the `scratch` directory and make a new directory just for this particular run. The executable `mitgcmuv` is copied there and a link is made to the input files in our home directory. In the future you will likely want to split up your input files, putting the binary files in `dev` and keeping the text files in your `home` directory where Git can easily track them. If that is the case, you'll need to add the following line so you also link the binary files:
```
ln -s /lustre/f2/dev/$USER/MITgcm/verification/<YOUREXPERIMENT>/input/* .
```
The final line in the run script actually runs the model, specifying the requested number of tasks and cpus per task. Find an introduction and guide to SLURM [here](https://rdhpcs-common-docs.rdhpcs.noaa.gov/wikis/rdhpcs-common-docs/doku.php?id=start).
Once you have saved and closed the run script, you're ready to submit the job using SLURM to Gaea, which you do with the following command.
```
$ sbatch run_exp2_test
```
You can check you're position in the queue with the following command:
```
$ squeue -u $USER
```
To check if the model ran properly, `$ cd /lustre/f2/scratch/$USER/exp2/run_test/`, and look through the file `output.txt`. Also make sure the `MNC` package created a folder `mnc_test_0001` and that it contains `grid.t00X.nc` and `state.0000000000.t00X.nc` files.
## Managing MITgcm Output
We will store copies of our output in two places to ensure that we maintain an archive copy but also have a place where we can access the files that we're interested in:
1. A `.tar` file of all the output will be saved in `gfdl:/archive/$USER/exp2/`. The `archive` directory is backed up on tape and a copy of the file will be saved in perpetuity.
1. A copy of all the output will also be saved in `gfdl:/work/$USER/`. Here you will likely have a system of directories to organize the output from your different experiments and runs.
First go to your scratch directory and make a `.tar` file of the output directory:
```
$ cd /lustre/f2/scratch/$USER/exp2/
$ tar -vcf run_test.tar run_test/
```
Open up a new terminal and log into `analysis`, then copy the `.tar` file on `gaea` over to `archive`:
```
$ ssh analysis
$ mkdir /archive/$USER/exp2/
$ cd /archive/$USER/exp2/
$ module load gcp
$ gcp gaea:/lustre/f2/scratch/$USER/exp2/run_test.tar .
```
From the same `analysis` terminal, copy the `.tar` file from `archive` to `work` and un-tar the file:
```
$ mkdir /work/$USER/exp2/
$ cd /work/$USER/exp2/
$ cp /archive/$USER/exp2/run_test.tar .
$ tar -vxf run_test.tar
```
Sign in with Wallet
Connect another wallet