# Run SKRIPS (WRF-MITgcm) on CURC cluster Notes on running WRF–MITgcm on the CURC cluster. Author: Yuan-Jen Lin (Feb 4, 2026) ## Preparation ### Enter a compute node We can use `sinteractive` (see [CURC page](https://curc.readthedocs.io/en/latest/running-jobs/interactive-jobs.html) for details). ``` sinteractive --partition=amilan --qos=normal --time=01:00:00 --nodes=1 --ntasks=16 --mem-per-cpu=3840M ``` <!-- Or `acompile` (less recommended) ``` acompile --time=04:00:00 ``` --> ### Load environmental variables Feel free to source my environment script (account: yuli3660): ``` source /projects/yuli3660/library-env-vars.sh ``` This script is also available on my GitHub repo: [library-env-vars.sh](https://github.com/yuanjenlin/SKRIPS_configuration/blob/main/library-env-vars.sh) (Tip: `module list` to see loaded modules) ### Create case folder Under /scratch or another location you prefer: ``` mkdir /scratch/alpine/$USER/$CASE ``` ## WRF Preprocessing System (WPS) ### Copy WPS directory You can copy from `/projects/yuli3660/model/WPS` to your case folder ``` cp -r /projects/yuli3660/model/WPS /scratch/alpine/$USER/$CASE ``` ### Edit `namelist.wps` * Domain center at (`ref_lon`, `ref_lat`) * For the ‘lambert’, ‘mercator’, and ‘polar’ projections, dx and dy are given in meters, and for the ‘lat-lon’ projection, dx and dy are given in degrees * `e_we` and `e_sn` should be **grid points**, so add +1 to the number of intervals * Domain length in longitude = (`e_we` – 1) × dx * Domain length in latitude = (`e_sn` – 1) × dy Below is one of my `namelist.wps`, for example. Check [WRF User Guide](https://www2.mmm.ucar.edu/wrf/users/wrf_users_guide/build/html/wps.html#) for details! ``` &share wrf_core = 'ARW', max_dom = 1, start_date = '2023-03-01_00:00:00', end_date = '2023-03-05_00:00:00', interval_seconds = 21600 / &geogrid parent_id = 1, parent_grid_ratio = 1, i_parent_start = 1, j_parent_start = 1, e_we = 1281, e_sn = 321, geog_data_res = 'default', dx = 0.03125, dy = 0.03125, map_proj = 'lat-lon', ref_lat = 0.0, ref_lon = 180.0, pole_lat = 90.0, pole_lon = 0.0, stand_lon = 0.0, geog_data_path = '/projects/yuli3660/model_input/wps-geog/' / &ungrib out_format = 'WPS', prefix = 'FILE', / &metgrid fg_name = 'FILE' / ``` ### Run geogrid ``` ./geogrid.exe 2>&1 | tee geogrid.log ``` Sends both stdout(1) and stderr (2) to the screen, and simultaneously writes them to `geogrid.log`. This will generate `geo_em.d01.nc`. ### Download Input Data * For example, I downloaded NCEP GDAS/FNL Data from NCAR: https://rda.ucar.edu/datasets/d083003/ * Only `[].f00.grib2` data are needed. * See the list of [GRIB Datasets Available from NCAR](https://www2.mmm.ucar.edu/wrf/site/input_data.html) * Also check [NCEP Products Inventory from NOAA](https://www.nco.ncep.noaa.gov/pmb/products/gfs/#GDAS) ### Link the GRIB files to specific filenames WPS expects filenames such as GRIBFILE.AAA, GRIBFILE.AAB, …, GRIBFILE.ZZZ. For example, I saved NCEP GDAS/FNL data under /scratch/alpine/yuli3660/gdas1.fnl0p25, so I run ``` ./link_grib.csh /scratch/alpine/yuli3660/gdas1.fnl0p25/2023/202303/*.grib2 ``` ### Vtable Create a symbolic link named `Vtable` pointing to `ungrib/Variable_Tables/Vtable.GFS` ``` ln -sv ungrib/Variable_Tables/Vtable.GFS Vtable ``` ### Run ungrib ``` ./ungrib.exe 2>&1 | tee ungrib.log ``` Sends both stdout(1) and stderr (2) to the screen, and simultaneously writes them to `ungrib.log`. This will generate files such as `FILE:2023-03-01_00`, etc. If you face segmentation fault ``` ulimit -s unlimited ``` ### Run metgrid ``` ./metgrid.exe 2>&1 | tee metgrid.log ``` This will generate files such as `met_em.d01.2023-03-01_00:00:00.nc` ## WRF ### Copy WRF directory You can copy from `/projects/yuli3660/model/WRF` to your case folder ``` cp -r /projects/yuli3660/model/WRF /scratch/alpine/$USER/$CASE ``` ### Modify `namelist.input` * `restart`: set to `.true.` if it is a restart run * `restart_interval`: restart output file interval in minutes. (e.g. using 5760 for a 4-day run will create restart files at the end of the simulation) * `time_step`: time step for model integration seconds. It should be less than 6xDX in km. For example, if DX~3km, it is good to set `time_step`=15 sec (it is < 6xDX and it is a divisor of 60) * `num_metgrid_levels` and `num_metgrid_soil_levels`: should match `met_em*.nc` (from WPS) * `dx`,`dy`: in meters and should match `met_em*.nc` (from WPS) Note: for a lat-lon projection, `namelist.wps` uses `dx`,`dy` in degrees, while `namelist.input` uses `dx`,`dy` in meters. * `e_we`,`e_sn`: should match `namelist.wps` (from WPS) * `radt`: minutes between radiation physics calls. Recommend 1 minute per km of dx; set to the same value for all domains. (e.g., if DX~3km, set `radt`=3 min) References: https://www2.mmm.ucar.edu/wrf/users/wrf_users_guide/build/html/physics.html https://www2.mmm.ucar.edu/wrf/users/wrf_users_guide/build/html/namelist_variables.html ### Modify `namelist.WRF_IO` Below is one of my `namelist.WRF_IO`, for example. ``` ## Remove HGT from output -:h:5:HGT,LANDMASK ## Variables sent to MITgcm +:h:5:OCNMASK,SST_INPUT,XLAT,XLONG,Q2,T2,U10,V10,QFX,RAINCV,RAINSHV,RAINNCV +:h:5:SWUPB,SWDNB,LWUPB,LWDNB,HFX,LH ## Variables read from MITgcm +:i:5:SST,UOCE,VOCE ############################################ ## Need these variables at initial condition +:h:0:OCNMASK,SST_INPUT,UOCE,VOCE,QFX +:i:0:OCNMASK,SST_INPUT,LH,HFX,SWUPB,SWDNB,LWUPB,LWDNB ## Read reanalysis SST (or climatology) every time step +:i:4:SST_INPUT ``` ### Link `met_em*.nc` (from WPS) to WRF run dir ``` cd <WRF_run_directory> ln -s ../../WPS/met_em.* . ``` ### Run `real.exe` ``` mpirun ./real.exe ``` <!-- ``` mpirun -n 16 ./real.exe ``` --> ## MITgcm ### Generate bathymetry Download [ETOPO2 bathymetry data](https://sos.noaa.gov/catalog/datasets/etopo2-topography-and-bathymetry-natural-colors/) Create a model_input folder under your case folder, for example: ``` mkdir /scratch/alpine/$USER/$CASE/model_input ``` I wrote a python script `skrips_gen_bathy.py` that generates the bathymetry file for MITgcm from ETOPO2 data and saves it to the above model_input folder. Edit the casename and input/output paths if needed. See the code from my GitHub repo: [skrips_gen_bathy.py](https://github.com/yuanjenlin/SKRIPS_workflows/blob/main/skrips_gen_bathy.py) ### Generate IC and OBCS - IC (initial condition) - OBCS (open boundary conditions) I wrote a python script `skrips_gen_ic_obcs.py` that generates IC (initial condition) and OBCS (open boundary conditions) of variables U, V, T, S for MITgcm from the GLORYS12V1 product. Edit the casename and input/output paths if needed. See the code from my GitHub repo: [skrips_gen_ic_obcs.py](https://github.com/yuanjenlin/SKRIPS_workflows/blob/main/skrips_gen_ic_obcs.py ) ### Vertical layer thickness The files `delZ_*v.txt` contain vertical layer thicknesses (`dz`) hardcoded in `meshGen.m` by Rui Sun. I have saved `delZ_58v.txt` and `delZ_60v.txt` for future use. `zc` and `zf` can be calculated from (`dz`). ![zc](https://hackmd.io/_uploads/S1wt_DaLlg.png) ## SKRIPS Extract the contents of the `SKRIPS.tar` archive, which was previously created from `/projects/rusu1402/runCPL_tropical_pacific` ``` tar -xvf /projects/yuli3660/model/SKRIPS.tar mv runCPL_tropical_pacific SKRIPS ``` ``` cd <SKRIPS_dir> patch -i /projects/yuli3660/model/diff.patch -p1 ``` Check the MITgcm size definition file `mitSettingRS/SIZE.h`. Set `nPx` and `nPy` such that $nPx \times nPy$ equals the number of MPI ranks. $sNx \times nSx \times nPx$ and $sNy \times nSy \times nPy$ equal to the numbers of X and Y points, respectively. Compile WRF-MITgcm coupled frankenstein model. ```bash ln -fsv mitgcm_optfile.ifort utils/mitgcm_optfile.intel ./install.sh # Answer Y, then Y ``` Optional: remove *.bin from Rui Sun ``` cd /projects/yuli3660/test_coarse/SKRIPS/runCase rm *.bin ``` Check the ESMF namelist file `runCase/namelist.rc`. Set `cpuATM` and `cpuOCN` to the number of MPI ranks. Check - `data`: modify `endTime` (seconds of the simulation), `xgOrigin`, `ygOrigin`, `delX`, `delY`, `delZ`, `bathyFile`, `uVelInitFile`, `vVelInitFile`, `hydrogThetaFile`, `hydrogSaltFile` - `data.cal`: modify `startDate_1` - `data.obcs`: modify `OB_Jnorth`, `OB_Jsouth`, `OB_Iwest`, `OB_Ieast` and obcs file names - `data.exf`: modify `obcsNstartdate1` and `obcsNperiod`, depending on your OBCS files. Same for S/W/E boundaries. Check - `namelist.input` - `namelist.WRF_IO` Link Data - MITgcm model input ``` ln -s /projects/yuli3660/model_input/tepexc_0.3d_20230301/* . ``` - WRF ``` ln -s /projects/yuli3660/test_coarse/WRF/run/wrf*d01 . ``` Link `esmf_application` ``` ln -sf ../coupledCode/esmf_application . ``` Copy the job script `/projects/yuli3660/model/submit_job.sh` and set the number of MPI ranks in the `mpirun -np ... ./esmf_application` command line. Submit the job script & good luck. ```bash sbatch submit_job.sh ``` ## Restart ### WPS - `namelist.wps`: modify `start_date` and `end_date` - Make sure new NCEP NCEP GDAS/FNL Data is downloaded and linked - Run ungrid.exe - Run metgrid.exe ### WRF - `namelist.input`: - modify `start_[year/month/day/hour]` and `end_[year/month/day/hour]` - It does not matter if you set `restart` to `.true.` or not, because what we want are the boundary conditions, by running `real.exe` (initial conditions of the restart run come from the "wrfrst" file). - Link new `met_em*` data ``` ln -fs ../../WPS/met_em.* . ``` - Run real.exe ### SKRIPS - Link wrf data again (note that you can delete `wrfinput_d01`, because it is the new WRF initial condition, which you don't need - the initial condition is the "wrfrst" file) ``` ln -fs /projects/yuli3660/test_coarse/WRF/run/wrf*d01 . ``` - `namelist.input`: - `start_[year/month/day/hour]` - `end_[year/month/day/hour]` - restart = .true. Add these two lines in &time_control: - override_restart_timers = .true., - write_hist_at_0h_rst = .true., - Ref: https://forum.mmm.ucar.edu/threads/wrfout-files-with-a-different-number-of-frames-in-a-restart-run.17064/ - `namelist.rc`: - `Start[Year/Month/Day/Hour]` - `Stop[Year/Month/Day/Hour]` - `data`: - `startTime`: in sec - `endTime`: in sec - `data.exf`: create and link new OBCS file and modify the corresponding start dates. * Note that you do not need to modify `data.cal` file. ## Others In `data` `PARM05`, should I set ... ? No. ``` checkIniTemp = .FALSE. checkIniSalt = .false. ```