# MISTRAL
## FOCI coupled (ECHAM6-NEMO3.6)
On mistral:

mkdir ~/models/esm_tools && cd ~/models/esm_tools
git clone -b foci2 [e-mail]:esm-tools/esm_tools.git
cd esm_tools/
module unload netcdf_c/4.3.2-gcc48
module load anaconda3
./install.sh

To upgrade an existing installation in `~/.local/bin` use `pip install --user -e --upgrade .`

esm_master install-foci-default

Compiles all three components.
## Current status Sebastian
...working on runscripts..., testing my setup, trying to find the logfiles for ECHAM and NEMO as the run hangs with a ctl_open which means one of the NEMO input files cannot be found.
#### TODO
1. get `iodef.xml` etc. from `nemo/CONFIG/<config>/` and not from namelists/nemo to be inline with OD setup of CONFIG structure.
2. get `namelist_cfg` etc. from `nemo/CONFIG/<config>`
#### Suggestions / Questions / Comments:
1. if a date entry, e.g. ini_parent_date: is set to `""`, esm_calendar.py crashes in line 305
1. many undocumented variables such as `${esm_master_dir}`
2. The concept behind the `general:` section is unclear. It seems to occur in coupled setups such `configs/awicm/awicm.yaml` but not in the basic setups of the component models such as `configs/echam/echam.yaml`.
3. Better error messages if errors in yaml files, more than `TypeError: 'NoneType' object is not iterable` etc.
4. Keep comments in namelists, the nemo namelists are unreadable without the comments.
5. Documentation and a list of all magic variables such as `${esm_namelist_dir}` would be helpful.
6. Documentation on log files and folder structure (see also below) needed! Some log files are in `scripts/`, some in `log/`. Why is the error log at the top of the log file in `scripts/`?
7. Why do dirs `forcing`, `bin` etc exist in `focitest4/` and `focitest4/run_*/`. The first dirs are all empty. Confusing...
8. Move `comp-*sh` scripts to subdir, they clutter the clean esm-toos directory
9. Can't find any logfiles from the model components e.g. `atmout`, `ocean.output`, ...
10. Allow to call runscript with path (full) path e.g. `esm_runscripts configs/testet_runsfoci-mistral-initial_monthly_swahl.run -e focitest4 -c`
11. What flags are allowed in `configs/machines/mistral.yaml` and `configs/batch_system/slurm.yaml`. All these vars like `partition_flag, additional_flag, ...` are undocumented, it's hard to guess what can be used here. A general `mistral.yaml` for compiling and running all models included in the ESM Tools seems to ambigious. Rather provide a baseline config, and move everything else to e.g. `foci.yaml` or `echam.yaml`.
12. How can I use my own `mistral.yaml`, the default one is cluttered?
13. Why are there some many module load and unload entries in the `.sad` file:

module purge
module load gcc/4.8.2
module unload intel
module load intel
module load cdo nco
module unload netcdf
module load netcdf_c/4.3.2-gcc48
module load python/3.5.2
module load cmake/3.13.3
module unload intel
module load intel/18.0.4 intelmpi/2018.5.288
module load autoconf/2.69
module unload intel
module load intel/18.0.4 intelmpi/2018.5.288
module load autoconf/2.69

## Current status Joakim

I can compile FOCI. Running crashes with

135: oasis_init_comp: Not Calling MPI_Init
181: oasis_init_comp: Not Calling MPI_Init
272: oasis_init_comp: Not Calling MPI_Init
136: oasis_init_comp: Not Calling MPI_Init
186: oasis_init_comp: Not Calling MPI_Init
275: oasis_init_comp: Not Calling MPI_Init
138: oasis_init_comp: Not Calling MPI_Init
187: oasis_init_comp: Not Calling MPI_Init
281: oasis_init_comp: Not Calling MPI_Init
142: oasis_init_comp: Not Calling MPI_Init
  0: forrtl: severe (64): input conversion error, unit -5, file Internal Formatted Read
  0: Image              PC                Routine            Line        Source             
  0: echam6             00000000015856BE  Unknown               Unknown  Unknown
  0: echam6             00000000015B3F74  Unknown               Unknown  Unknown
  0: echam6             00000000015B2332  Unknown               Unknown  Unknown
  0: echam6             00000000013FE545  Unknown               Unknown  Unknown
  0: echam6             000000000141F813  Unknown               Unknown  Unknown
  0: echam6             00000000013D84D3  Unknown               Unknown  Unknown
  0: echam6             0000000000D1BD68  mo_mpi_mp_p_start         455  mo_mpi.f90
  0: echam6             0000000000A1C417  MAIN__                    135  echam6.f90
  0: echam6             000000000041207E  Unknown               Unknown  Unknown
  0: libc-2.12.so       00002B287E390D20  __libc_start_main     Unknown  Unknown
  0: echam6             0000000000411F69  Unknown               Unknown  Unknown
  0: forrtl: error (76): Abort trap signal
  0: Image              PC                Routine            Line        Source             
  0: echam6             000000000159473E  Unknown               Unknown  Unknown
  0: libpthread-2.12.s  00002B287DEDF7E0  Unknown               Unknown  Unknown
  0: libc-2.12.so       00002B287E3A44F5  gsignal               Unknown  Unknown
  0: libc-2.12.so       00002B287E3A5CD5  abort                 Unknown  Unknown
  0: echam6             0000000001586861  Unknown               Unknown  Unknown
  0: echam6             00000000015856BE  Unknown               Unknown  Unknown
  0: echam6             00000000015B3F74  Unknown               Unknown  Unknown
  0: echam6             00000000015B2332  Unknown               Unknown  Unknown
  0: echam6             00000000013FE545  Unknown               Unknown  Unknown
  0: echam6             000000000141F813  Unknown               Unknown  Unknown
  0: echam6             00000000013D84D3  Unknown               Unknown  Unknown
  0: echam6             0000000000D1BD68  mo_mpi_mp_p_start         455  mo_mpi.f90
  0: echam6             0000000000A1C417  MAIN__                    135  echam6.f90
  0: echam6             000000000041207E  Unknown               Unknown  Unknown
  0: libc-2.12.so       00002B287E390D20  __libc_start_main     Unknown  Unknown
  0: echam6             0000000000411F69  Unknown               Unknown  Unknown
srun: error: m10181: task 0: Aborted
srun: Terminating job step 20442419.0
  0: slurmstepd: error: *** STEP 20442419.0 ON m10181 CANCELLED AT 2020-04-14T11:18:55 ***

I can not push my changes to github yet, so its hard for anyone else to reproduce this error. But this is the furthest we can get.

## Questions to Dirk
1. How do install modified source files from the ESM-Tools? With `pip --update`?
2. How to use esm_master etc without installation into `~/.local/...` or an environment (I user python's `venv` at the moment to handle different installations.)
3. Hierachy of yaml files?
4. How do I set "input_dir" for OpenIFS? Setting "input_dir" has no effect. It is always set by the oifs.yaml? Is stuff in yaml files default but runscript overwrites?

## What is FOCI2?

### Can we make something like "FOCI" and allow that "FOCI-1.0" is ECHAM6+NEMO while "FOCI-1.1" is OpenIFS+NEMO? Or can components not change with version?

Dirk informs us that inside the configs/foci/ we currently only have foci.yaml.
But we could have oifs.yaml and echam.yaml there which set the specifics for OpenIFS and ECHAM, and then load these settings into foci.yaml.
So foci.yaml would include NEMO, XIOS and OASIS stuff which is the same oifs and echam.
Then how do we indicate by name what is ECHAM and OpenIFS?
FOCI-2.0 vs 2.1. Or FOCI-2.0-OpenIFS, or what?

## Oddity when compiling ECHAM6 as part of FOCI (Mistral)

Compiling ECHAM6 I get this at the end

[100%] Linking Fortran executable echam6
support/libsupport.a(util_file.c.o): In function `util_tmpnam':
util_file.c:(.text+0xa8): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
/sw/rhel6-x64/gcc/binutils-2.24-gccsys/bin/ld: warning: libhdf5.so.9, needed by /sw/rhel6-x64/netcdf/netcdf_fortran-4.4.2-parallel-impi-intel14/lib/libnetcdff.so, may conflict with libhdf5.so.10
[100%] Built target echam6

Similar warning about HDF5 appear in XIOS. Goes away if we make sure that HDF5_ROOT and HDF5ROOT are the same.