# FOCI-OpenIFS

## To do:

* Fix post processing in ESM-Tools
* Fix restarts. (OpenIFS is ok, but not NEMO). Dirk needed!
* Implement Tco399L91 and VIKING10
* Tune cloud parameters and ice albedo to stop overall cooling.
* Use HD rather than runoff_mapper (work with Stefan and Eric)
* Use XIOS (waiting on ECMWF)
* Couple OIFS-WAM-NEMO
* Eliminate ice thickness from OASIS coupling. Not used by OIFS anyway.
* Try using "opt" for mapping strategy in OASIS. Can reduce MPI communication times especially in high-res simulations.
* Which is the best conservation post processing to use in OASIS? GLBPOS, GLOBAL etc.
* Global conservation with NEMO and AGRIF?

## Fixes:

* Upgraded to OASIS3-MCT4.
* Reduced OpenIFS output for faster run
* Now coupling oce surf currents for relative wind stress in OpenIFS. Set LECURR=.true.
* Use CMIP6 forcing data
* Can run long runs!
* Upgraded to XIOS 2.5 (r1910). Needed for NEMO v4.

## Works

* Tco95/O96, L91 coupled to NEMO 3.6 / LIM2 via OASIS3-MCT4. XIOS 2.5 (detached) on Mistral.
* 35 min/SY. > 40 SYPD.

## Install

 esm_master install-focioifs-2.0 

### Get input data

Make sure Git supports LFS.

git clone [e-mail]:foci/foci_input2/openifs_cy43_tco95.git
git clone [e-mail]:foci/foci_input2/openifs_cy43_general.git
git clone [e-mail]:foci/foci_input2/nemo_orca05.git
git clone [e-mail]:foci/foci_input2/oasis3_openifs43r3-tco95_orca05.git

### Configure and run

Take runscript in

runscripts/focioifs/

configure as you need (change paths etc) and then run

esm_runscript <runscript_name> -e <experiment_name> -c 

This will run it in "check mode", i.e. set it up but not submit job. If you get this warning:

MISSING FILES:
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/input/oifs//vtables/vtables: 
        --> /work/bb0519/foci_input2/OPENIFS43R3_GENERAL/vtables/vtables
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/restart/oasis3mct//areas.nc: 
        --> /work/bb0519/foci_input2/OASIS3_OPENIFS43R3-TCO95_ORCA05//areas.nc
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/restart/oasis3mct//masks.nc: 
        --> /work/bb0519/foci_input2/OASIS3_OPENIFS43R3-TCO95_ORCA05//masks.nc
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/restart/oasis3mct//grids.nc: 
        --> /work/bb0519/foci_input2/OASIS3_OPENIFS43R3-TCO95_ORCA05//grids.nc
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//cmip6_data/o3_histo: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/forcing/oifs//cmip6_data/o3_histo
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//cmip6_data/o3_scenarios: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/forcing/oifs//cmip6_data/o3_scenarios
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//vtables/vtables: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/input/oifs//vtables/vtables
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//95_4/o3chem_l62: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/input/oifs//95_4/o3chem_l62
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//95_4/o3chem_l60: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/input/oifs//95_4/o3chem_l60
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//95_4/o3chem_l31: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/input/oifs//95_4/o3chem_l31
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//95_4/o3chem_l19: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/input/oifs//95_4/o3chem_l19
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//95_4/o3chem_l137: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/input/oifs//95_4/o3chem_l137
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//areas.nc: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/restart/oasis3mct//areas.nc
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//masks.nc: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/restart/oasis3mct//masks.nc
--  /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/work//grids.nc: 
        --> /work/bb0519/b350090/esm-experiments//focitest17/run_18500101-18501231/restart/oasis3mct//grids.nc

This is normal. Then remove the -c and run the runscript again.

## Problems

### restart files and output files not moved properly

Restart and output files are not being moved properly.
Somehow they sometimes get an extra prefix or suffix.
OpenIFS restart files are ok, but not NEMO. OpenIFS output is moved but gets the wrong name.

### clone command for oasis listed twice

(esm-tools-foci2) b350071@mlogin100% esm_master get-focioifs-1.0 -c
['oifs', 'nemo', 'oasis3mct']
    Executing commands in this order:
         mkdir -p focioifs-1.0
         cd focioifs-1.0
         git clone -b 4.0 https://[e-mail]/modular_esm/oasis3-mct.git oasis
         git clone -b 4.0 https://[e-mail]/modular_esm/oasis3-mct.git oasis
         git clone -b dirk_on_mistral https://[e-mail]/modular_esm/oifs-40r1.git 40r1
         git clone -b esm-tools https://[e-mail]/foci/src/xios.git xios-2.0_r982
         git clone -b esm-tools https://[e-mail]/foci/src/nemo.git nemo-3.6
         git clone -b esm-tools-focioifs https://[e-mail]/ec-earth/runoff-mapper.git rnfmap-focioifs1
         cd ..
(esm-tools-foci2) b350071@mlogin100%

### Crash: divide-by-zero in cloud scheme

On HLRN-Lise I got a crash with Intel 2019 + IMPI 2019 in the cloud scheme. The trace back points to:

src/ifs/phys_ec/cloudsc.F90:1817
1817	      ZICENUCLEI(JL)=1000.0_JPRB*EXP(12.96_JPRB*(ZVPLIQ-ZVPICE)/ZVPLIQ-0.639_JPRB)

Turns out that this was due to bad SST values. New runoff caused SST > 100C.

### Freshwater conservation

Mass fixers are on option 2, which use McGregor mass fixers for q, o3, cloud variables etc.
Diagnostics show that the mass fixers are doing their job. They correct water mass after SL advection.

I suspect that at least part of the problem is excess snow. OpenIFS basically stops counting snow if snow mass > 1000 kg/m2.

Snow mass is PSNS in callpar.F90 and PSNSE1 is snow mass tendency (kg/m2/s).
PSNS is INTENT(IN) for callpar and PSNSE1 is INTENT(INOUT).

callpar first sets PSNSE1 = 0 (section 1.4).
Then PSNSE1 is the output from SURFTSTP (section 12.1)
Then we compute excess snow, ZEXSNOW, remove from PSNSE1.
PSNS is copied into ZSNM1M in callpar then sent to SURFTSTP as INTENT(IN).