---
# System prepended metadata

title: OM4p5 JRA55do1.5

---

# OM4p5 JRA55do1.5

## Setup Gitlab repo

```
git clone https://gitlab.gfdl.noaa.gov/Jan-erik.Tesdal/xml.git om4_jra
cd om4_jra/

git remote add upstream https://gitlab.gfdl.noaa.gov/Brandon.Reichl/xml.git
git fetch upstream
```

```
git branch -a
```

Switch to Brandon's branch
```
git checkout Add_XML_OM4p25_JRA55do1p5_0netfw
```

## Test p25 xml

Open shell and load fre
```
tcsh
module load fre/bronx-19
```

Modify xml
```
vim OM4p25_JRA55do1.5_0netfw.xml
```

Change
> <property name="FRE_VERSION"     value="bronx-19"
> <property name="PROD_SIMTIME"   value="1"/>

```
frelist -x OM4p25_JRA55do1.5_0netfw.xml
fremake -p ncrc4.intel18 -t prod -x OM4p25_JRA55do1.5_0netfw.xml MOM6_SIS2_compile
sbatch /lustre/f2/dev/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223//MOM6_SIS2_compile/ncrc4.intel18-prod/exec/compile_MOM6_SIS2_compile.csh
```
```
frelist --directory=all --platform=ncrc4.intel18 --target=prod -x OM4p25_JRA55do1.5_0netfw.xml MOM6_SIS2_compile
```
* `/lustre/f2/dev/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223//MOM6_SIS2_compile/ncrc4.intel18-prod/exec`

When you are all done, you should have a file in there called `fms_MOM6_SIS2_compile.x`

Run 1 year of OM4p25 JRA1.5
```
frerun -p ncrc4.intel18 -t prod -x OM4p25_JRA55do1.5_0netfw.xml OM4p25_JRA55do1.5_0netfw_cycle1

sbatch -t 20 -q debug /lustre/f2/dev/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p25_JRA55do1.5_0netfw_cycle1/ncrc4.intel18-prod/scripts/run/OM4p25_JRA55do1.5_0netfw_cycle1
sbatch /lustre/f2/dev/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p25_JRA55do1.5_0netfw_cycle1/ncrc4.intel18-prod/scripts/run/OM4p25_JRA55do1.5_0netfw_cycle1
```

```
frelist --directory=all --platform=ncrc4.intel18 --target=prod -x OM4p25_JRA55do1.5_0netfw.xml OM4p25_JRA55do1.5_0netfw_cycle1
```

* `/lustre/f2/scratch/gfdl/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p25_JRA55do1.5_0netfw_cycle1/ncrc4.intel18-prod/archive/ascii/19580101.ascii_out.tar/19580101.fms.out`
> MOM Date   1959/01/01 00:00:00   8760: En 2.255593E-03, MaxCFL  0.13386, Mass 1.381833347635E+21, Salt  34.72030187874, Temp   3.58886748162
>     Total Energy: 43C5A0A53B9F59A6  3.1168542931305953E+18
>     Total Mass:   1.3818333476352224E+21, Change:  -3.9082447858311810E+11 Error:  1.78479E+02 ( 1.3E-19)
>     Total Salt:   4.7977670976004710E+19, Change:  -1.9541223997055433E+09 Error: -5.89756E+00 (-1.2E-19)
>     Total Heat:   1.9797193331257074E+25, Change:   2.3926798001274605E+20 Error:  1.29432E+07 ( 6.5E-19)
>       Total age:   1.3416794243561934E+21 yr kg


* `/archive/Brandon.Reichl/FMS2021.02_mom6_devgfdl_20221223/OM4p25_JRA55do1.5_0netfw_cycle1/gfdl.ncrc4-intel18-prod/ascii/19580101.ascii_out.tar/19580101.fms.out`
> MOM Date   1959/01/01 00:00:00   8760: En 2.255593E-03, MaxCFL  0.13386, Mass 1.381833347635E+21, Salt  34.72030187874, Temp   3.58886748162
>     Total Energy: 43C5A0A53B9F59A6  3.1168542931305953E+18
>     Total Mass:   1.3818333476352224E+21, Change:  -3.9082447858311810E+11 Error:  1.78479E+02 ( 1.3E-19)
>     Total Salt:   4.7977670976004710E+19, Change:  -1.9541223997055433E+09 Error: -5.89756E+00 (-1.2E-19)
>     Total Heat:   1.9797193331257074E+25, Change:   2.3926798001274605E+20 Error:  1.29432E+07 ( 6.5E-19)
>       Total age:   1.3416794243561934E+21 yr kg


## OM4p5 xml
    
```
cp OM4p25_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw.xml
```
Edit OM4p5_JRA55do1.5_0netfw.xml
```
vim OM4p5_JRA55do1.5_0netfw.xml
```
```
frelist -x OM4p5_JRA55do1.5_0netfw.xml
```
```
frelist --directory=all --platform=ncrc4.intel18 --target=prod -x OM4p5_JRA55do1.5_0netfw.xml MOM6_SIS2_compile
frerun -p ncrc4.intel18 -t prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle1
```
```
sbatch -t 20 -q debug /lustre/f2/dev/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/ncrc4.intel18-prod/scripts/run/OM4p5_JRA55do1.5_0netfw_cycle1
frelist --directory=all --platform=ncrc4.intel18 --target=prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle1
```
Standard output in 
* `/lustre/f2/scratch/gfdl/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/ncrc4.intel18-prod/stdout/run`

```
frerun -o -p ncrc4.intel18 -t prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle1
```

JRA river runoff
* `/lustre/f2/dev/Niki.Zadeh/archive/input/xanadu/Runoff_JRAv1.4_OM4p5`


## Generate river runoff for JRA 1.5 p5

Clone [this](https://github.com/raphaeldussin/OM4_025_runoff_JRA/tree/JRAv1.5.0.1) repo:
```
cd /archive/Jan-erik.Tesdal
git clone --recursive https://github.com/raphaeldussin/OM4_025_runoff_JRA.git
cd OM4_025_runoff_JRA
```
    
Branch **1.5.0** is for period 1958-2019 and **1.5.0.1** for period 2020 onwards. Start with 1.5.0.1 branch.
```
git checkout JRAv1.5.0.1
```
    
Simply copy the `ocean_hgrid` and `ocean_mask` for OM4p5. You can find `ocean_hgrid.nc` and `ocean_mask.nc` for OM4p5 in `/archive/gold/datasets/OM4_05/mosaic_ocean.v20180227.unpacked`
```
cp /archive/gold/datasets/OM4_05/mosaic_ocean.v20180227.unpacked/ocean_hgrid.nc . 
cp /archive/gold/datasets/OM4_05/mosaic_ocean.v20180227.unpacked/ocean_mask.nc .
```

Start `tmux` session:
```
csh
conda activate base
tmux
```

Create a conda env with the provided env file, and run make
```bash
conda create --name pad_jra --file conda_env_runoff
conda activate pad_jra
```
    
Before running the script you want to do a `dmget` of the `friver_*` and `licalv_*` files.
> JRA_DIR = /archive/gold/datasets/reanalysis/JRA55-do/v1.5.0.1/padded
> JRA_FILES = $(wildcard $(JRA_DIR)/friver_*.nc $(JRA_DIR)/licalvf_*.nc)
    
```
dmget /archive/gold/datasets/reanalysis/JRA55-do/v1.5.0.1/padded/friver_*.nc
dmget /archive/gold/datasets/reanalysis/JRA55-do/v1.5.0.1/padded/licalvf_*.nc
```  
    
Generate JRA runoff files
```
make
```

Update checksums
```
md5sum *compressed.nc > hash.md5
```
    
Tar up the files

```
mkdir input
cp *.compressed.nc input/
tar -chvzf input.tar.gz input/*
```
    
On GAEA create a new directory and transfer the files
```
cd /lustre/f2/dev/gfdl/Jan-erik.Tesdal/input
mkdir Runoff_JRAv1.5_OM4p5
cd Runoff_JRAv1.5_OM4p5
module load gcp
gcp -v gfdl:/archive/Jan-erik.Tesdal/OM4_025_runoff_JRA/input.tar.gz .
tar -xvzf input.tar.gz
mv input/* .
rm -fr input
rm input.tar.gz
```
    
Fork Raf's repo and set your fork as the origin. On PP/AN:
```
git remote set-url origin https://github.com/jetesdal/OM4_05_runoff_JRA.git
git remote add upstream https://github.com/raphaeldussin/OM4_025_runoff_JRA.git
```
    
Update repo:
```
rm -fr input
rm input.tar.gz
git add hash.md5
git commit -m "Updated hash.md5"
git push origin JRAv1.5.0.1
```
    
Switch to branch 1.5.0
```
git checkout JRAv1.5
```

Edit `Makefile` to change `JRA_DIR` to
> JRA_DIR = /archive/gold/datasets/reanalysis/JRA55-do/v1.5.0_no_null_char/padded

Before running `make` you want to do a `dmget` of the `friver_*` and `licalv_*` files.
```
dmget /archive/gold/datasets/reanalysis/JRA55-do/v1.5.0_no_null_char/padded/friver_*.nc
dmget /archive/gold/datasets/reanalysis/JRA55-do/v1.5.0_no_null_char/padded/licalvf_*.nc
```  

Tar up the files

```
mkdir input
cp *.compressed.nc input/
tar -chvzf input.tar.gz input/*
```
    
On GAEA transfer the files
```
cd /lustre/f2/dev/gfdl/Jan-erik.Tesdal/input/Runoff_JRAv1.5_OM4p5
module load gcp
gcp -v gfdl:/archive/Jan-erik.Tesdal/OM4_025_runoff_JRA/input.tar.gz .
tar -xvzf input.tar.gz
mv input/* .
rm -fr input
rm input.tar.gz
```
    
Update directory in `OM4p5_JRA55do1.5_0netfw.xml`:
>   <property name="JRA_dataset_p5"   value="/lustre/f2/dev/gfdl/Jan-erik.Tesdal/input/Runoff_JRAv1.5_OM4p5"/>

Regenerate runscript:
```
cd /ncrc/home1/Jan-erik.Tesdal/om4_jra
frerun -o -p ncrc4.intel18 -t prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle1
```
    
Check output:
```
frelist --directory=all --platform=ncrc4.intel18 --target=prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle1
```

On GAEA Model output is in 
```
/lustre/f2/scratch/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/ncrc4.intel18-prod/archive
```
    
On PP/AN model output is
```
/archive/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/gfdl.ncrc4-intel18-prod
```

## Update rho2 coords in xml

### (1) MOM override section
> #override NUM_DIAG_COORDS = 3
> #override DIAG_COORDS = "z Z ZSTAR", "rho2 RHO2 RHO", "rho2l35 RHO2L35 RHO"
> #override DIAG_COORD_DEF_Z = "WOA09"
> #override DIAG_COORD_DEF_RHO2 = "FILE:diag_rho2_74layer.nc,interfaces=rho2"
> #override DIAG_COORD_DEF_RHO2L35 = "RFNC1:35,999.5,1028,1028.5,8.,1038.,0.0078125"

### (2) Add file and definition for rho2 coords
     <dataFile label="input" target="INPUT/" chksum="" size="" timestamp="">
       <dataSource platform="$(platform)">/lustre/f2/dev/Raphael.Dussin/input/coords/diag_rho2_74layer.nc</dataSource>
     </dataFile>

### (3) Add diagnostics in diagTable append:
> "ocean_annual_rho2l35",  12, "months", 1, "days", "time"
> "ocean_model_rho2l35", "volcello", "volcello", "ocean_annual_rho2l35", "all", "mean", "none",2
> "ocean_model_rho2l35", "umo", "umo", "ocean_annual_rho2l35", "all", "mean", "none",2
> "ocean_model_rho2l35", "vmo", "vmo", "ocean_annual_rho2l35", "all", "mean", "none",2

### (4) Add postprocessing for `ocean_annual_rho2l35` in `awg_include/xml_building_blocks/om4p25_postprocess.xml`
```xml
      <component type="ocean_annual_rho2l35"  start="$(PP_START_YEAR)" source="ocean_annual_rho2l35" sourceGrid="ocean-tripolar">
        <timeSeries freq="annual"   chunkLength="$(CHUNK_LENGTH_A)" source="ocean_annual_rho2l35"/>
        <timeAverage source="annual" interval="$(CHUNK_LENGTH_A)"/>
      </component>
```

## Switch to C5

To update to c5 use `fre/bronx-20`.

Note, when using `bronx-20`, you have to comment out (or remove) all the the `gnu` based compiler options in the main XML, otherwise you will get an error like
```
*************************** 1 XML validation errors ****************************
awg_include/xml_building_blocks/platforms.xml:
    Line 150 - Element 'compiler', attribute 'type': 'gnu' is not a valid value of the atomic type 'compilerType'.
    Line 150 - Element 'compiler', attribute 'type': [facet 'enumeration'] The value 'gnu' is not an element of the set {'intel', 'intel-classic', 'intel-oneapi', 'pgi', 'gcc', 'cce', 'nvhpc'}.
********************************************************************************
*FATAL*: The XML file '/ncrc/home1/Jan-erik.Tesdal/om4_jra/OM4p25_JRA55do1.5_0netfw.xml' is not valid
*FATAL*: The XML file '/ncrc/home1/Jan-erik.Tesdal/om4_jra/OM4p25_JRA55do1.5_0netfw.xml' can't be parsed
```

Edit `OM4p5_JRA55do1.5_0netfw.xml`, in the <setup> block add:
```xml
<xi:include href="awg_include/xml_building_blocks/platforms.xml" xpointer="xpointer(//freInclude/platform[@name='ncrc5.intel22'])"/>
<xi:include href="awg_include/xml_building_blocks/platforms.xml" xpointer="xpointer(//freInclude/platform[@name='gfdl.ncrc5-intel22'])"/>
```

In `awg_include/xml_building_blocks/platforms.xml`, add:

```xml
<platform name="ncrc5.intel22">
     <freVersion>$(FRE_VERSION)</freVersion>
     <compiler type="intel-classic" version="2022.0.2"/>
     <project>$(GFDL_GROUP)</project>
     <directory stem="$(FRE_STEM)">
       <root>$DEV/$USER/$(FRE_STEM)</root>
       <scripts>$DEV/$USER/$(FRE_STEM)$(DEBUGLEVEL)/$(name)/$(platform)-$(target)/scripts</scripts>
       <state>$DEV/$USER/$(FRE_STEM)$(DEBUGLEVEL)/$(name)/$(platform)-$(target)/state</state>
       <src>$DEV/$USER/$(FRE_STEM)/$(BUILD_DATE)/$(name)/src</src>
       <exec>$DEV/$USER/$(FRE_STEM)/$(BUILD_DATE)/$(name)/$(platform)-$(target)/exec</exec>
       <archive>$SCRATCH/$USER/$(stem)$(DEBUGLEVEL)/$(name)/$(platform)-$(target)/archive</archive>
       <stdout>$SCRATCH/$USER/$(stem)$(DEBUGLEVEL)/$(name)/$(platform)-$(target)/stdout</stdout>
       <work>$SCRATCH/$USER/work/$(stem)/$FRE_JOBID</work>
     </directory>
     <property name="EXEC_ROOT"         value="$DEV/$USER/$(stem)/$(BUILD_DATE)"/>
     <property name="F2003_FLAGS"       value=" -DINTERNAL_FILE_NML -g "/>
     <property name="AWG_INPUT_HOME"    value="$(AWG_INPUT_HOME_GAEA)"/>
     <property name="NB_ROOT"           value="/nbhome/$USER/$(FRE_STEM)$(PLATINFO)/$(name)"/>
     <property name="ARCHIVE_NNZ"        value="/lustre/f2/dev/Niki.Zadeh/archive"/>
     <property name="REFERENCE"         value="/lustre/f2/dev/Niki.Zadeh/archive/answers/$(reference_tag)/$(name)/$(platform)-$(target)"/>
     <property name="FMS_ARCHIVE_ROOT"    value="$PDATA/gfdl/fms"/>
     <property name="CMIP6_ARCHIVE_ROOT"  value="$PDATA/gfdl/cmip6/datasets"/>
     <property name="NCORE_PER_NODE"      value="36"/>
     <property name="NPES_IO_LAND"         value="18"/>
     <csh><![CDATA[
       module load git
       module unload cray-libsci
       module unload darshan
       $(append_to_setup_csh)
     ]]></csh>
    </platform>
</freInclude>
```
    
Also in the same file, add:
```xml
<platform name="gfdl.ncrc5-intel22">
      <xi:include xpointer="xpointer(//freInclude/platform[@name='gfdl.default']/node())" />
</platform>
```

Recompile model with `--plattform/-p ncrc5.intel22` on a C5 login node (`gaea5*`).

```bash
# Run fremake
fremake -p ncrc5.intel22 -t prod -x OM4p5_JRA55do1.5_0netfw.xml MOM6_SIS2_compile

# Run compile script
sbatch /ncrc/home1/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223//MOM6_SIS2_compile/ncrc5.intel22-prod/exec/compile_MOM6_SIS2_compile.csh
```
    
Look at directories
```
frelist --directory=all --platform=ncrc5.intel22 --target=prod -x OM4p5_JRA55do1.5_0netfw.xml MOM6_SIS2_compile
```

If the compiling successfully completed, you should see the file `fms_MOM6_SIS2_compile.x`
```
ls -lah /ncrc/home1/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/MOM6_SIS2_compile/ncrc5.intel22-prod/exec/fms_MOM6_SIS2_compile.x
```
    
Next we generate the runscript
```
frerun -p ncrc5.intel22 -t prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle1
```
    
1-year test run
```
sbatch /lustre/f2/dev/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/ncrc5.intel22-prod/scripts/run/OM4p5_JRA55do1.5_0netfw_cycle1
```

Check output:
```
frelist --directory=all --platform=ncrc5.intel22 --target=prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle1
```

On GAEA Model output is in 
```
/lustre/f2/scratch/gfdl/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/ncrc5.intel22-prod/archive
```
    
On PP/AN model output is
```
/archive/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/gfdl.ncrc5-intel22-prod
```
    
Compare checksums between C4 and C5:
* `gfdl.ncrc5-intel22-prod`
> MOM Date   1958/12/27 00:00:00   
>     Total Energy: 43BCD5DFB40A3BC2
>     Total Mass:   1.3815353609147614E+21           
>     Total Salt:   4.7967439731286311E+19         
>     Total Heat:   1.9798415209849202E+25
    
* `gfdl.ncrc4-intel18-prod`
> MOM Date   1958/12/27 00:00:00
>     Total Energy: 43BCE8227C8E2DAA  
>     Total Mass:   1.3815353603132348E+21
>     Total Salt:   4.7967439728278675E+19
>     Total Heat:   1.9798573745570062E+25

## Run complete simulation
JRA 1.5 goes to 2019 and we have the 1.5.0.1 extension for the most recent years (2020-2022, 2023 partial at this time). Thus, each cycle should be 65 years long (1958-2022) or 62 years long (1958-2019).
>   <property name="PROD_SIMTIME"   value="62"/>

Regenerate the runscript
```
frerun -e -p ncrc5.intel22 -t prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle1
```
    
Submit run
```
sbatch /lustre/f2/dev/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/ncrc5.intel22-prod/scripts/run/OM4p5_JRA55do1.5_0netfw_cycle1
```
    
### Last three years (2020-2022)

Note: You need to rename the files in
```
/lustre/f2/dev/gfdl/Jan-erik.Tesdal/input/Runoff_JRAv1.5_OM4p5
```


```
vas_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0-1_gr_202001010000-202012312100.padded.nc
vas_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0-1_gr_202101010000-202112312100.padded.nc
vas_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0-1_gr_202201010000-202212312100.padded.nc
```
    
Change
 `1-5-0-1` --> `1-5-0`
    
Renamed files
```
vas_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0_gr_202001010000-202012312100.padded.nc
vas_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0_gr_202101010000-202112312100.padded.nc
vas_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0_gr_202201010000-202212312100.padded.nc
```
    
In the xml change
>  <property name="JRA_version"     value="1-5-0"/>

full run
>   <property name="PROD_SIMTIME"   value="65"/>

Regenerate the runscript
```
frerun -e -p ncrc5.intel22 -t prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle1
```
  
## Postprocessing

On PP/AN, model output is located in
```
/archive/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/gfdl.ncrc5-intel22-prod/
```

Global mean metrics and postprocessing 
```
/home/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/gfdl.ncrc5-intel22-prod
```

### Redo postprocessing
    
Remove state files
```
rm -f /home/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/gfdl.ncrc5-intel22-prod/state/postProcess/ocean_*
    
rm -f /home/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/gfdl.ncrc5-intel22-prod/state/postProcess/ice*
```
    
Remove postprocessing script
```
rm -f /home/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/gfdl.ncrc5-intel22-prod/scripts/postProcess/OM4p5_JRA55do1.5_0netfw_cycle1_*
```
    
Generate postprocessing scripts for the first 5-year chunk
```
# Load FRE
module load fre/bronx-20

# Run frepp
frepp -s -c split -P gfdl.ncrc5-intel22 -T prod -t 19620101 -d /archive/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/gfdl.ncrc5-intel22-prod/history -x /home/Jan-erik.Tesdal/ncrc/om4_jra/OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle1
```

<p style='color:red; font-size:20px; font-weight:bold;'>Section transports are missing:</p>

> ERROR: No history files matching $hDate*/*.ocean_Davis_Strait.nc    
> ERROR: No history files matching $hDate*/*.ocean_Indonesian_Throughflow.nc
    
    
# OM4p5_JRA55do1.5_0netfw_cycle2

- Copy restart file

```bash
cd /lustre/f2/dev/gfdl/Jan-erik.Tesdal/ICgen
cp /lustre/f2/scratch/gfdl/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle1/ncrc5.intel22-prod/archive/restart/20230101.tar .
mv 20230101.tar OM4p5_JRA55do1.5_0netfw_cycle1.20230101.tar
```
 
- Update path in the `initCond` statement:
```bash
cd ~/om4_jra
vim OM4p5_JRA55do1.5_0netfw.xml
```
>      <dataFile label="initCond" target="INPUT/" chksum="" size="" timestamp="">
>       <dataSource platform="$(platform)">/lustre/f2/dev/gfdl/Jan-erik.Tesdal/ICgen/OM4p5_JRA55do1.5_0netfw_cycle1.20230101.tar</dataSource>
>      </dataFile>
    
- Make sure that you have `input_filename = 'r'` such that restart is used even for first job:

>      <namelist name="MOM_input_nml" >
>          output_directory = './',
>          input_filename = 'r'
>          restart_input_dir = 'INPUT/',
>          restart_output_dir = 'RESTART/',
>          parameter_filename = 'INPUT/MOM_input','INPUT/MOM_layout','INPUT/MOM_override','INPUT/MOM_saltrestore'
>      </namelist>
>      <namelist name="SIS_input_nml" >
>          output_directory = './',
>          input_filename = 'r'
>          restart_input_dir = 'INPUT/',
>          restart_output_dir = 'RESTART/',
>          parameter_filename = 'INPUT/SIS_input','INPUT/SIS_layout','INPUT/SIS_override'
>      </namelist>
    
- Make sure the xml includes statement to delete `coupler.res`

> if ( $currentSeg == 1 ) then
>    #Take current date from coupler_nml
>    if(-e $work/INPUT/coupler.res ) \rm -rf $work/INPUT/coupler.res
> endif
    
- Change to 1-year run
> <property name="PROD_SIMTIME" value="1"/>

- Generate runscript
```bash
frerun -p ncrc5.intel22 -t prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle2
``` 

- Submit runscript
```bash
sbatch /lustre/f2/dev/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle2/ncrc5.intel22-prod/scripts/run/OM4p5_JRA55do1.5_0netfw_cycle2
```    
First, try with `sbatch -t 20 -q debug`
    
Compare checksums between cycle 1 and 2:
* cycle 1
> MOM Date   1958/12/27 00:00:00 
>     Total Energy: 43BCD5DFB40A3BC2

* cycle 2
> MOM Date   1958/12/27 00:00:00
>     Total Energy: 43BF7C6202ACAC02

* cycle 3
> MOM Date   1958/12/27 00:00:00   
>     Total Energy: 43BEF137E25DFECA
    
## Run complete simulation

>   <property name="PROD_SIMTIME"   value="62"/>

Regenerate the runscript
```
frerun -e -p ncrc5.intel22 -t prod -x OM4p5_JRA55do1.5_0netfw.xml OM4p5_JRA55do1.5_0netfw_cycle2
```
    
Submit run
```
sbatch /lustre/f2/dev/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle2/ncrc5.intel22-prod/scripts/run/OM4p5_JRA55do1.5_0netfw_cycle2
```
    
## Postprocessing

On PP/AN, model output is located in
```
/archive/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle2/gfdl.ncrc5-intel22-prod/
```

Global mean metrics and postprocessing 
```
/home/Jan-erik.Tesdal/FMS2021.02_mom6_devgfdl_20221223/OM4p5_JRA55do1.5_0netfw_cycle2/gfdl.ncrc5-intel22-prod/
```
    
<p style='color:red; font-size:20px; font-weight:bold;'>Diagnotics are missing:</p>

* `zos`, `tos` and `sos` in `ocean_annual`