# OpenIFS 43r3 + XIOS ## What works? * OpenIFS 43r3v1 (vendor/v1-from-ecmwf branch) with XIOS disabled in ESM-Tools * OpenIFS 43r3v1 with XIOS enabled as released by ECMWF and using T21 test case * OpenIFS 43r3v1 with XIOS enabled as released by ECMWF in Tco95L91 config in ESM-Tools. ## What does not work? * OpenIFS 43r3 (vendor/v1-from-ecmwf branch) + XIOS when running in ESM-Tools hang. * Using Tco95L91 case, I can not write variables "sro". Message is "netcdf: numeric representation". ## Where does the model crash? Traceback shows that the hang is at Line 239 in mpl_init_mod.F90. ```bash CALL MPI_BCAST(CLENV,ICOUNT,INT(MPI_BYTE),IROOT,MPL_COMM,IERROR) ``` And it's called from DR_HOOK('MASTER') in master.F90. Update: 4/9/2020: Jan might have put XIOS call after main loop in master. That could be what breaks it. ## Where has it been tested On Mistral (DKRZ, Hamburg) and Lise (ZIB, Berlin). Both using Intel chips, Intel Fortran and Intel MPI. "Lise" is pretty new and uses latest versions of Intel compilers. Mistral is older and uses older versions. Same result on both machines. ## Thoughts * The problem only happens with the branch "vendor/v1-from-ecmwf". Clearly something went wrong in the merge. * If I run the T21 test case, it's all fine. But running same config in ESM-Tools gives me netcdf error "numeric representation". Could it be compile problems? Even if I compile OpenIFS same way, error persists. Try to recompile XIOS somehow? ## XML files for OpenIFS XIOS will always expect to find a file iodef.xml in the work directory. The file context_ifs.xml contains settings for the OpenIFS-XIOS coupling. We must always have NFRHIS = NFRPOS. Both numbers tell OpenIFS how often to call XIOS. For example, NFRHIS = 2 means OpenIFS will send data to XIOS every 2nd time step. OpenIFS will transform spectral variables (U,V,T,Z,p) to grid-point space each time XIOS is called, so NFRHIS should be as large as possible to avoid unnecessary calculations. NFRHIS, NFRPOS should be the greatest common divisor of all freq_op values in file_def.xml For T95: time step = 3600s, NFRHIS = 1, NFRPOS = 1, freq_op = 1h. We can write output 3hr, 6hr or 1d, For T199: time step = 1800s, NFRHIS = 2, NFRPOS = 2, freq_op = 1h. For T399: time step = 600s, NFRHIS = 6, NFRPOS = 6, freq_op = 1h.