# OpenIFS with ESM-Tools release 4
## About
In a nutshell, OpenIFS is the atmospheric model from ECMWF IFS, i.e. without the ocean model, data assimilation, the parallel I/O server etc.
It also does not include the experimental non-hydrostatic code.
OpenIFS 43r3 is based on IFS 43r3.
https://www.ecmwf.int/en/publications/ifs-documentation
OpenIFS is NOT open source. A license is required. GEOMAR and AWI have institute-wide licenses.
Although you can install ESM-Tools in any directory I suggest a directory structure below. If you follow it, it's easier for me (Sebastian Wahl) to find your data or source code directories in case you need support.
# Running OpenIFS 43r3
## Do this first!
* Get an account on DKRZ Mistral, HLRN or Juwels. Sebastian Wahl (GEOMAR) can help with this.
* Join the OpenIFS forum, https://confluence.ecmwf.int/display/OIFSUF/OpenIFS+User+Forums (good place to ask questions etc.)
* Get access to OpenIFS repo on DKRZ gitlab from either Jan Streffing (AWI) or Joakim Kjellsson (GEOMAR).
* Get access to the FOCI repo on GEOMAR GitLab from Sebastian Wahl (GEOMAR)
## Machine specifics
### HLRN4 (Berlin / Goettingen)
OpenIFS's compile scripts didn't work on `blogin` due to missing Perl modules. I went for my own local perl installation in case further problems occur. Now install local perl with (just copy everything below into your shell, takes app 1-2 mins to complete):
```bash
# Create a folder where you want to install different Perls, and cd into it:
# Note that it doesn't need to be your home folder. Put it wherever you want to maintain such software:
version="5.32.0"
export PERL_BASE="$HOME/perl"
mkdir -p $PERL_BASE
cd $PERL_BASE
# Download source tarball into a subfolder named src, and untar:
curl --create-dirs -L -o src/perl-${version}.tar.gz http://www.cpan.org/src/5.0/perl-${version}.tar.gz
cd src
tar -zxf perl-${version}.tar.gz
# Configure and build, making sure that this non-standard location is baked into the perl binary:
cd perl-${version}
./Configure -des -Dprefix=$PERL_BASE/perl-${version} -Dotherlibdirs=$PERL_BASE/perl-${version}/lib/perl5
make
make install
# Install modules needed for OpenIFS / XIOS
curl -sL https://cpanmin.us | $PERL_BASE/perl-${version}/bin/perl - --notest -l $PERL_BASE/perl-${version} App::cpanminus Time::Piece URI
```
Add the following to your `.bash_profile`:
```bash
# Set $PERL5LIB to find these libraries, and set $PATH to use this Perl instead of the system Perl:
export PERL_BASE="$HOME/perl"
export PERL5LIB="$PERL_BASE/perl-5.32.0/lib/perl5:$PERL_BASE/perl-5.32.0/lib/perl5/x86_64-linux:$PERL5LIB"
export PATH="$PERL_BASE/perl-5.32.0/bin:$PATH"
```
## Get the model
First download the ESM-Tools:
```bash
cd ; ln -s $WORK/esm .
mkdir -p $WORK/esm # on HLRN this resolves to /scratch/usr/$USER
cd $WORK/esm
git clone https://github.com/esm-tools/esm_tools
cd esm_tools
./install.sh
```
Install the preprocess plugin needed for OIFS
```bash
mkdir -p ~/esm/plugins ; cd ~/esm/plugins
git clone https://github.com/esm-tools-plugins/preprocess
cd preprocess
module load anaconda3 # machine dependant on JUWELS you need module load Python/3.6.8 at the time of writing
pip install --user -e .
```
Now verify that the plugin is installed correctly with by running `esm_plugins`:
```bash
glogin1:~ $ esm_plugins
The following plugins are installed and available:
preprocess
```
You will also need to switch to the development branch which includes OpenIFS support.
```bash
git checkout geomar_awicm3
```
Create a directory where you want to keep model code, e.g.
```bash
mkdir ~/esm/models
cd ~/esm/models
```
Joakim usually uses `~/model_codes` for his source codes but that doesn't work on on JUWELS where space in HOME is very limited
Then download and compile OpenIFS 43r3.
```bash
esm_master get-oifs-43r3-v1
esm_master comp-oifs-43r3-v1
```
This should download and compile OASIS, XIOS and OpenIFS.
If you encounter problems related to your locale, e.g. LC_ALL, or errors related to Perl, contact Sebastian Wahl.
## Get input data
Input data may already be available on e.g. Mistral (DKRZ) or Berlin (HLRN).
Check with Sebastian Wahl what the current status is.
If data is not yet available, you'll need to retrieve it.
Go to the directory where you want to store input data (this will be several Gb) e.g. on HLRN-Berlin,
```bash
mkdir /scratch/usr/$USER/foci_input2/
cd /scratch/usr/$USER/foci_input2/
```
Before you do any downloading you'll need to install Git LFS.
On most machines this is done by
```bash
module load git
git lfs install
```
Now get the data needed for any OpenIFS run (gas conc, radiation etc)
```bash
git clone https://git.geomar.de/foci/foci_input2/openifs_cy43_general.git
```
This will ask for your password 3-4 times. Sebastian Wahl can help you set up password-less logins.
Then you will need the forcing data for the specific configuration you want.
I recommend starting with Tco95/L91 (global 100 km resolution, 91 levels):
```bash
git clone https://git.geomar.de/foci/foci_input2/openifs_cy43_tco95.git
```
If you want to run full AMIP runs, you should also download the AMIP surface forcing (> 2Gb for T95).
```bash
git clone https://git.geomar.de/foci/foci_input2/openifs_cy43_tco95-amip.git
```
That's it!
## Run OpenIFS (HLRN-Berlin example)
Go to the ESM-Tools runscript directory, copy a runscript and modify it so that it fits you, e.g. start/end date, directory names etc.
```bash
cd esm_tools/runscripts/oifs/
cp oifs-43r3-tco95-amip-blogin-initial-daily_JK.yaml my-oifs-run.yaml
vi my-oifs-run.yaml
```
Then run
```bash
esm_runscripts my-oifs-run.yaml -e oifs_test1
```
where oifs_test1 is the name of your run.
You should find output in your run directory under outdata/oifs.
## Moving forward
Now you can go on to make longer experiments, or increase the resolution, or change parameter settings.
For instance, try setting
```bash
perturb: 1
ensemble_id: 1
```
in your runscript. This is perturb the initial state and allow for ensemble runs.
## FAQ
### What does OpenIFS include?
OpenIFS is an atmosphere model + HTESSEL land-surface model (with FLAKE lake model) + ECWAM wave model.
These are "baked into" the model so that there is only one executable.
ECWAM has a separate namelist.
### How do OpenIFS grids compare to ECHAM?
OpenIFS 43r3 uses cubic truncation (shortest wave is 4 grid points), while ECHAM uses quadratic (3 grid points).
OpenIFS also uses a reduced octahedral Gaussian grid rather than the reduced Gaussian in ECHAM.
So ECHAM Tq127 = N96, but OpenIFS Tco95 = O96.
See also https://confluence.ecmwf.int/display/OIFS/4.+OpenIFS%3A+Grid+and+Resolution
Note also that the vertical grids in OpenIFS are either L91 or L137 compared to ECHAM L95.
### What about ozone?
OpenIFS uses a simplified ozone chemistry parametrization which is much cheaper than a full blown chemistry scheme.
In essence this is a seasonally varying photochemically balanced state.
Perturbations from this state can occur due to dynamics and radiation etc.
Equivalent Chlorine is read from a file each year which gives a long-term trend in O3, i.e. ozone depletion.
However, the radiation code does not see this parametrized O3 concentration but instead read the CMIP6 forcing.
If you want the model to see O3 concentration perturbed by dynamics etc, then you need to turn that on.
Joakim Kjellsson can help out.
See https://acp.copernicus.org/articles/7/2183/2007/
### Supported resolutions at GEOMAR
* Tco95 (100 km), L91
* Tco199 (50 km), L91
* Tco399 (25 km), L91
* Tco399 (25 km), L137
### Supported experiments
* AMIP 1.1.6 (CMIP6). 1 deg monthly SST/ice. 1979-2017.
* HighResMIP (CMIP6). 0.25 deg daily SST/ice. 1948-2050.
* PAMIP (CMIP6). 1 deg monthly SST/ice. 2000 only.
# Development
## Solved
1. ~~Can not read namelist chapters. OpenIFS wants them as uppercase, but everything in ESM-Tools namelists is lowercase. Causes immediate crash.~~
Solved: Paul G made OpenIFS namelists all uppercase.
2. ~~Can only have one "add_export_vars" in oifs.yaml. A bit annoying!~~ Solved: Dirk fixed with with esm_parser 4.0.1
3. ~~I think ESM-Tools does not set NPROC correctly in fort.4.~~ Solved: Dirk fixed this.
4. ~~Can not run AMIP run with old bash script. input_dir, forcing_dir etc not set by runscript, only by default in oifs.yaml. Using a yaml file as runscript is the only thing that works.~~ Not solved, but who cares?
5. ~~Can not set start_date and end_date properly. I've set the run to be 5 days, the restart to be monthly, but the run ends up as 1 year. Why? Also, it is not 1 year, only 364 days.~~ Solved: Dirk changed "ndays" to "nday" in runscript and that did it. If it does not understand the restart frequency it will always assume 1 year.
6. Need to be able to control SPPT. Restart files etc.
## Problems
* ~~We need to make sure that ifs.stat, NODE file, and restart files are moved after a completed run.~~ Solved. All files are copied correctly into place with the latest version.
* ~~I am setting foci_cpl_nemo_lim = false in oifs.yaml. Can I then set foci_cpl_nemo_lim = true in focioifs.yaml and expect that to take precedence?~~ If it does not work this way, let me (Sebastian Wahl) know, I'll work on it and/or forward it to Paul and Dirk at AWI.
## Questions
* ~~How do we do pre-processing with grib-api and CDO to set dates in input data?~~ It needs the installation of the preprocess plugin to be installed. I added it to the install instructions above. The preprocess plugin looks for a shell script called `<model>-<version>-preprocess.sh` (which resolves to `oifs-43r3-preprocess.sh` for this manual). It's on the list to do this automagically upon OpenIFS installation in the future: https://github.com/esm-tools/esm_tools/issues/112
* How do we bring post processing from old ESM-Tools in here?
* With a similar strategy than the preprocessing. We need to discuss what needs doing during postprocessing for OpenIFS.
* I dont understand the directory structure. If my run is called "oifstest1", then the work dir is oifstest1/run_20010101_20011231/work. But what is oifstest1/work used for? Same with scripts dir etc.
* That's something I've always found VERY VERY confusing too. I'm pressing hard for simplifying this in the future.
Sign in with Wallet
Connect another wallet