---
# System prepended metadata

title: Run SKRIPS (WRF-MITgcm) on CURC cluster
tags: [WRF-MITgcm]

---

# Run SKRIPS (WRF-MITgcm) on CURC cluster

Notes on running WRF–MITgcm on the CURC cluster.
Author: Yuan-Jen Lin (Feb 4, 2026)

## Preparation
### Enter a compute node 
We can use `sinteractive` (see [CURC page](https://curc.readthedocs.io/en/latest/running-jobs/interactive-jobs.html) for details).

```
sinteractive --partition=amilan --qos=normal --time=01:00:00 --nodes=1 --ntasks=16 --mem-per-cpu=3840M
```

<!--
Or `acompile` (less recommended)
```
acompile --time=04:00:00
```
-->

### Load environmental variables
Feel free to source my environment script (account: yuli3660):
```
source /projects/yuli3660/library-env-vars.sh
```
This script is also available on my GitHub repo: [library-env-vars.sh](https://github.com/yuanjenlin/SKRIPS_configuration/blob/main/library-env-vars.sh)

(Tip: `module list` to see loaded modules)

### Create case folder 
Under /scratch or another location you prefer:
```
mkdir /scratch/alpine/$USER/$CASE
```

## WRF Preprocessing System (WPS)
### Copy WPS directory 
You can copy from `/projects/yuli3660/model/WPS` to your case folder
```
cp -r /projects/yuli3660/model/WPS /scratch/alpine/$USER/$CASE
```

### Edit `namelist.wps`
* Domain center at (`ref_lon`, `ref_lat`)
* For the ‘lambert’, ‘mercator’, and ‘polar’ projections, dx and dy are given in meters, and for the ‘lat-lon’ projection, dx and dy are given in degrees 
* `e_we` and `e_sn` should be **grid points**, so add +1 to the number of intervals
* Domain length in longitude = (`e_we` – 1) × dx
* Domain length in latitude = (`e_sn` – 1) × dy
  
Below is one of my `namelist.wps`, for example.
Check [WRF User Guide](https://www2.mmm.ucar.edu/wrf/users/wrf_users_guide/build/html/wps.html#) for details! 
```
&share
 wrf_core = 'ARW',
 max_dom = 1,
 start_date = '2023-03-01_00:00:00',
 end_date   = '2023-03-05_00:00:00',
 interval_seconds = 21600
/

&geogrid
 parent_id         =   1,
 parent_grid_ratio =   1,
 i_parent_start    =   1,
 j_parent_start    =   1,
 e_we              = 1281,
 e_sn              = 321,
 geog_data_res = 'default',
 dx = 0.03125,
 dy = 0.03125,
 map_proj  = 'lat-lon',
 ref_lat   =    0.0,
 ref_lon   =  180.0,
 pole_lat  =   90.0,
 pole_lon  =    0.0,
 stand_lon =    0.0,
 geog_data_path = '/projects/yuli3660/model_input/wps-geog/'
/

&ungrib
 out_format = 'WPS',
 prefix = 'FILE',
/

&metgrid
 fg_name = 'FILE'
/
```

### Run geogrid
```
./geogrid.exe 2>&1 | tee geogrid.log
```
Sends both stdout(1) and stderr (2) to the screen, and simultaneously writes them to `geogrid.log`.
This will generate `geo_em.d01.nc`.

### Download Input Data
* For example, I downloaded NCEP GDAS/FNL Data from NCAR: https://rda.ucar.edu/datasets/d083003/
  * Only `[].f00.grib2` data are needed.
* See the list of [GRIB Datasets Available from NCAR](https://www2.mmm.ucar.edu/wrf/site/input_data.html)
* Also check [NCEP Products Inventory from NOAA](https://www.nco.ncep.noaa.gov/pmb/products/gfs/#GDAS)
  
### Link the GRIB files to specific filenames
WPS expects filenames such as GRIBFILE.AAA, GRIBFILE.AAB, …, GRIBFILE.ZZZ.
For example, I saved NCEP GDAS/FNL data under /scratch/alpine/yuli3660/gdas1.fnl0p25, so I run
```
./link_grib.csh /scratch/alpine/yuli3660/gdas1.fnl0p25/2023/202303/*.grib2
```

### Vtable
Create a symbolic link named `Vtable` pointing to `ungrib/Variable_Tables/Vtable.GFS`
```
ln -sv ungrib/Variable_Tables/Vtable.GFS Vtable
```

### Run ungrib
```
./ungrib.exe 2>&1 | tee ungrib.log
```
Sends both stdout(1) and stderr (2) to the screen, and simultaneously writes them to `ungrib.log`.
This will generate files such as `FILE:2023-03-01_00`, etc.

If you face segmentation fault
```
ulimit -s unlimited
```

### Run metgrid
```
./metgrid.exe 2>&1 | tee metgrid.log
```
This will generate files such as `met_em.d01.2023-03-01_00:00:00.nc`


## WRF
### Copy WRF directory 
You can copy from `/projects/yuli3660/model/WRF` to your case folder
```
cp -r /projects/yuli3660/model/WRF /scratch/alpine/$USER/$CASE
```

### Modify `namelist.input`

* `restart`: set to `.true.` if it is a restart run
* `restart_interval`: restart output file interval in minutes. 
(e.g. using 5760 for a 4-day run will create restart files at the end of the simulation)
* `time_step`: time step for model integration seconds. 
It should be less than 6xDX in km. For example, if DX~3km, it is good to set `time_step`=15 sec (it is < 6xDX and it is a divisor of 60)
* `num_metgrid_levels` and `num_metgrid_soil_levels`: should match `met_em*.nc` (from WPS)
* `dx`,`dy`: in meters and should match `met_em*.nc` (from WPS) 
Note: for a lat-lon projection, `namelist.wps` uses `dx`,`dy` in degrees, while `namelist.input` uses `dx`,`dy` in meters.
* `e_we`,`e_sn`: should match `namelist.wps` (from WPS)
* `radt`: minutes between radiation physics calls.
Recommend 1 minute per km of dx; set to the same value for all domains. (e.g., if DX~3km, set `radt`=3 min)

References:
https://www2.mmm.ucar.edu/wrf/users/wrf_users_guide/build/html/physics.html
https://www2.mmm.ucar.edu/wrf/users/wrf_users_guide/build/html/namelist_variables.html

### Modify `namelist.WRF_IO`
Below is one of my `namelist.WRF_IO`, for example.
```
## Remove HGT from output
-:h:5:HGT,LANDMASK
## Variables sent to MITgcm
+:h:5:OCNMASK,SST_INPUT,XLAT,XLONG,Q2,T2,U10,V10,QFX,RAINCV,RAINSHV,RAINNCV
+:h:5:SWUPB,SWDNB,LWUPB,LWDNB,HFX,LH
## Variables read from MITgcm
+:i:5:SST,UOCE,VOCE
############################################
## Need these variables at initial condition
+:h:0:OCNMASK,SST_INPUT,UOCE,VOCE,QFX
+:i:0:OCNMASK,SST_INPUT,LH,HFX,SWUPB,SWDNB,LWUPB,LWDNB
## Read reanalysis SST (or climatology) every time step
+:i:4:SST_INPUT
```

### Link `met_em*.nc` (from WPS) to WRF run dir
```
cd <WRF_run_directory>
ln -s ../../WPS/met_em.* .
```

### Run `real.exe`
```
mpirun ./real.exe
```

<!--
```
mpirun -n 16 ./real.exe
```
-->

## MITgcm

### Generate bathymetry
Download [ETOPO2 bathymetry data](https://sos.noaa.gov/catalog/datasets/etopo2-topography-and-bathymetry-natural-colors/)

Create a model_input folder under your case folder, for example:
```
mkdir /scratch/alpine/$USER/$CASE/model_input
```
I wrote a python script `skrips_gen_bathy.py` that generates the bathymetry file for MITgcm from ETOPO2 data and saves it to the above model_input folder. Edit the casename and input/output paths if needed.

See the code from my GitHub repo: [skrips_gen_bathy.py](https://github.com/yuanjenlin/SKRIPS_workflows/blob/main/skrips_gen_bathy.py)

### Generate IC and OBCS
- IC (initial condition)
- OBCS (open boundary conditions)

I wrote a python script `skrips_gen_ic_obcs.py` that generates IC (initial condition) and OBCS (open boundary conditions) of variables U, V, T, S for MITgcm from the GLORYS12V1 product. Edit the casename and input/output paths if needed.

See the code from my GitHub repo: [skrips_gen_ic_obcs.py](https://github.com/yuanjenlin/SKRIPS_workflows/blob/main/skrips_gen_ic_obcs.py
)

### Vertical layer thickness 
The files `delZ_*v.txt` contain vertical layer thicknesses (`dz`) hardcoded in `meshGen.m` by Rui Sun. I have saved `delZ_58v.txt` and `delZ_60v.txt` for future use. `zc` and `zf` can be calculated from (`dz`).
![zc](https://hackmd.io/_uploads/S1wt_DaLlg.png)

## SKRIPS 
Extract the contents of the `SKRIPS.tar` archive, which was previously created from `/projects/rusu1402/runCPL_tropical_pacific`

```
tar -xvf /projects/yuli3660/model/SKRIPS.tar
mv runCPL_tropical_pacific SKRIPS
```

```
cd <SKRIPS_dir>
patch -i /projects/yuli3660/model/diff.patch -p1
```

Check the MITgcm size definition file `mitSettingRS/SIZE.h`. Set `nPx` and `nPy` such that $nPx \times nPy$ equals the number of MPI ranks. $sNx \times nSx \times nPx$ and $sNy \times nSy \times nPy$ equal to the numbers of X and Y points, respectively.


Compile WRF-MITgcm coupled frankenstein model.

```bash
ln -fsv mitgcm_optfile.ifort utils/mitgcm_optfile.intel
./install.sh # Answer Y, then Y
```

Optional: remove *.bin from Rui Sun
```
cd /projects/yuli3660/test_coarse/SKRIPS/runCase
rm *.bin
```

Check the ESMF namelist file `runCase/namelist.rc`. Set `cpuATM` and `cpuOCN` to the number of MPI ranks.

Check
- `data`: modify `endTime` (seconds of the simulation), `xgOrigin`, `ygOrigin`, `delX`, `delY`, `delZ`, `bathyFile`, `uVelInitFile`, `vVelInitFile`, `hydrogThetaFile`, `hydrogSaltFile`
- `data.cal`: modify `startDate_1`
- `data.obcs`: modify `OB_Jnorth`, `OB_Jsouth`, `OB_Iwest`, `OB_Ieast` and obcs file names
- `data.exf`: modify `obcsNstartdate1` and `obcsNperiod`, depending on your OBCS files. Same for S/W/E boundaries.

Check
- `namelist.input`
- `namelist.WRF_IO`

Link Data 
- MITgcm model input
```
ln -s /projects/yuli3660/model_input/tepexc_0.3d_20230301/* .
```

- WRF
```
ln -s /projects/yuli3660/test_coarse/WRF/run/wrf*d01 .
```

Link `esmf_application`
```
ln -sf ../coupledCode/esmf_application .
```

Copy the job script `/projects/yuli3660/model/submit_job.sh` and set the number of MPI ranks in the `mpirun -np ... ./esmf_application` command line.

Submit the job script & good luck.

```bash
sbatch submit_job.sh
```
## Restart
### WPS
- `namelist.wps`: modify `start_date` and `end_date`
- Make sure new NCEP NCEP GDAS/FNL Data is downloaded and linked
- Run ungrid.exe
- Run metgrid.exe

### WRF
- `namelist.input`: 
    - modify `start_[year/month/day/hour]` and `end_[year/month/day/hour]`
    - It does not matter if you set `restart` to `.true.` or not, because what we want are the boundary conditions, by running `real.exe` (initial conditions of the restart run come from the "wrfrst" file).
- Link new `met_em*` data
```
ln -fs ../../WPS/met_em.* .
```
- Run real.exe

### SKRIPS
- Link wrf data again (note that you can delete `wrfinput_d01`, because it is the new WRF initial condition, which you don't need - the initial condition is the "wrfrst" file)
```
ln -fs /projects/yuli3660/test_coarse/WRF/run/wrf*d01 .
```

- `namelist.input`:
    - `start_[year/month/day/hour]`
    - `end_[year/month/day/hour]`
    - restart = .true.
    Add these two lines in &time_control:
    - override_restart_timers = .true.,
    - write_hist_at_0h_rst = .true.,
    - Ref: https://forum.mmm.ucar.edu/threads/wrfout-files-with-a-different-number-of-frames-in-a-restart-run.17064/

- `namelist.rc`:
    - `Start[Year/Month/Day/Hour]`
    - `Stop[Year/Month/Day/Hour]`


- `data`:
    - `startTime`: in sec
    - `endTime`: in sec

- `data.exf`: create and link new OBCS file and modify the corresponding start dates.  

* Note that you do not need to modify `data.cal` file.

## Others

In `data` `PARM05`, should I set ... ?
No.
```
 checkIniTemp    = .FALSE.
 checkIniSalt    = .false.
```