# ESMValTool tutorial at ITC

# Collaborative Document
# https://tinyurl.com/esmvaltool-itc

Welcome to The Workshop Collaborative Document.

This Document is synchronized as you type, so that everyone viewing this page sees the same text. This allows you to collaborate seamlessly on documents.

This is the Document for today: [link](https://tinyurl.com/esmvaltool-itc)

## 👮Code of Conduct

Participants are expected to follow these guidelines:
* Use welcoming and inclusive language.
* Be respectful of different viewpoints and experiences.
* Gracefully accept constructive criticism.
* Focus on what is best for the community.
* Show courtesy and respect towards other community members.

## ⚖️ License

All content is publicly available under the Creative Commons Attribution License: [creativecommons.org/licenses/by/4.0/](https://creativecommons.org/licenses/by/4.0/).

## 🙋Getting help

To ask a question, just raise your hand.

If you need help from a helper, place a <color> post-it note on your laptop lid. A helper will come to assist you as soon as possible.

## 🖥 Workshop website

🛠 Setup

[Workshop website and Setup](https://www.itc.nl/research/research-facilities/labs-resources/itc-big-geodata/training/introduction-to-esmvaltool/)

## 👩‍🏫👩‍💻🎓 Instructors

Sarah Alidoost, Pranav Chandramouli, Bouwe Andela

## 🧑‍🙋 Helpers

Pranav Chandramouli, Bouwe Andela, Serkan Girgin

## 👩‍💻👩‍💼👨‍🔬🧑‍🔬🧑‍🚀🧙‍♂️🔧 Roll Call
Name/ pronouns (optional) / job, role / social media (twitter, github, ...) / background or interests (optional) / city

* Santiago/Msc Student - MSE/Ecuador
* Sry Handini Puteri / MSc Student - NHR / Indonesia / LinkedIn: Sry Handini Puteri
* Shreya Deep Bansod/ MSc Student - NHR / India / https://www.linkedin.com/in/shreya-deep-bansod-5078b8112/
* Md Saiful Islam/ GEM-MSc/ ITC-NRS/ Bangladesh/ https://www.linkedin.com/in/md-saiful-islam-408088190/
* Aamir Imran /PhD - ITC -Climat Risk Assessment/ Pakistan
* Salsabila Prasetya / MSc - ITC NHR / Indonesia / https://www.linkedin.com/in/salsabila-prasetya/
* Alma Raunak / MSc Student - NRM / Kazakhstan https://www.linkedin.com/in/alma-raunak-655503270/
* Swagatalaxmi Das / Msc / NHR / India /https://www.linkedin.com/in/swagatalaxmi-das-a072a0216/
* Prajwal Khanal / Phd Student @ ITC-WRS/ Nepal/ Land-Atmosphere Interaction, Machine Learning / https://twitter.com/prajzwal08
* Jiangtao CAI/ PhD students / ITC-WRS/ China / Ecohydrology / https://twitter.com/c_giotto
* Nils Tjaden / researcher GeoHealth @ ITC-EOS / https://github.com/nbtjaden
* Mewcha A.Gebremedhin/ITC-/Ethiopia/Hydrogeology
https://www.linkedin.com/in/mewcha-amha-96565371/
* Resul Comert/Turkiye / ITC visiting
* Serkan Girgin / Researcher / @BigGeodata / / Enschede
* Zengjing song / PhD student in ITC-WRS
* Tanuj Pareek / Researcher at ITC-PGM / India /
https://www.linkedin.com/in/tanuj-pareek-61b3a1b2/
* Chakshu Gururani/ MSc @ ITC - NHR/ India
https://www.linkedin.com/in/chakshu-gururani/
* Qianqian Han/ PhD student in WRS/ China
* Seckin Fidan/Turkiye/ITC/visiting PhD
* Hakan Tanyas / Researcher/ AES Enchede
* Ting Duan / PhD student in WRS China / Vegetation characterization, microwave remote sensing, machine learning
* Kiana Karimi / PhD student / WRS
* Enting Tang / PhD student / WRS/ Linkedln: https://www.linkedin.com/in/enting-tang/

## 🗓️ Agenda

|Time| Topic|
| ----------- | ----------- |
|9:30 - 10:00 | Welcome and icebreaker|
|10:00 - 10:30|Introduction to ESMValTool|
|10:30 - 12:00|ESMValTool tutorial (Part 1)|
|12:00 - 13:00|Lunch break|
|13:00 - 14:30|ESMValTool tutorial (Part 2)|
|14:30 - 15:00|Coffee break|
|15:00 - 16:00|Tailor recipes/configuration to use cases|
|16:00|END|

## Notes

### Mamba installation
- [Download and install mamba](https://github.com/conda-forge/miniforge)
### command to install wget
`apt-get update` or `sudo apt-get update`

to install wget:
`apt-get install wget`

to download the file:
`wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh`

to see the downloded file:
`ls`

to install mamba:
`bash Mambaforge-Linux-x86_64.sh or bash <name of the file></name>` (answer yes for all the yes/no)

### WSL communication error!
If WSL isn’t able to communicate with the internet, then DNS server seems out of order. To fix this, execute the following
sudo nano /etc/resolv.conf
and add the following
nameserver 8.8.8.8 (This will enable ubuntu subsystem to communicate with internet)# is used as the comment so above line should be the first line starting without an #. Type Ctrl + S to save the file and Ctrl+X to close the file. This should solve the internet connectivity issue

to confirm installation:

`source ~/.bashrc`
`which mamba`

to update mamba:

`mamba update --name base mamba`

### esmvaltool installation

`mamba create --name esmvaltool esmvaltool 'python=3.10'`

### activate esmvaltool conda environment
mamba activate esmvaltool

### Test esmvaltool installation
esmvaltool --help

### Configure esmvaltool
esmvaltool config get_config_user

### Open folder with config file
explore.exe .

### Check list of recipes
esmvaltool recipes list

### Make folders for esmvaltool
mkdir esmvaltool_workshop

### Setting up config file:
Change output_dir:

~/esmvaltool_workshop/esmvaltool_output

Change download_dir:

~/esmvaltool_workshop/climate_data

Change rootpath: default:

~/esmvaltool_workshop/climate_data

Check search_esgf is 'when_missing'

### Run the recipe:
esmvaltool run examples/recipe_python.yml

### Development installation using source code
- to install git
`sudo apt-get install git`

- to clone repository (to copy esmvaltool source code):
`cd esmvaltool_workshop`
`git clone https://github.com/ESMValGroup/ESMValTool.git`

`cd ESMValTool`
- make sure the `esmvaltool` conda environment is activated:
`mamba activate esmvaltool`
-install esmvaltool in development (editable) mode:
`pip install -e .`
- run recipe
`esmvaltool run <path to recipe_python.yml>`

### recipe Consecutive dry days
documentation of [Consecutive dry days](https://docs.esmvaltool.org/en/latest/recipes/recipe_consecdrydays.html)

- recipe file name `recipe_consecdrydays.yml`
- cmor conventions for cmip6 data [cmor tables](https://github.com/PCMDI/cmip6-cmor-tables/tree/master/Tables)
- look at the python script in `esmvaltool/diag_scripts/droughtindex/diag_cdd.py`

### esmvaltool in jupyter notebook (using python API)
example notebooks can be found [here](https://github.com/ESMValGroup/EGU22-short-course/blob/main/notebooks/Introduction_to_ESMValTool_with_output.ipynb).

### git tips!
`git diff`

## 🔧 Exercises
1) Can you figure out which version of ESMValTool has been installed?
* Prajwal : command: esmvaltool version output: ESMValCore: 2.8.0 ESMValTool: 2.8.0
* Tanuj: `esmvaltool version`
output: ESMValCore: 2.8.0 ESMValTool: 2.8.0
* Qianqian: ESMValCore: 2.8.0, ESMValTool: 2.8.0
* Jiangtao: ESMValCore: 2.8.0,ESMValTool: 2.8.0

* Swagata: esmvaltool version/ output:2.8.0

* Saiful: esmvaltool version
ESMValCore: 2.8.0
ESMValTool: 2.8.0

* Nils: esmvaltool version
ESMValCore: 2.8.0
ESMValTool: 2.8.0

* Mewcha
ESMValCore: 2.8.0
ESMValTool: 2.8.0

* Ting Duan : ESMValCore: 2.8.0 ESMValTool: 2.8.0
### Exercise 4
Saving preprocessed data:

1) Later in this tutorial, we will want to look at the contents of the preproc folder. This folder contains preprocessed data and is removed by default when ESMValTool is run. In the configuration file, which settings can be modified to prevent this from happening?

| Name| Answer|
| ---- | ---- |
| Santiago| remove_preproc_dir: false
| Sry Handini Puteri| remove_preproc_dir: false
| Shreya Deep Bansod |remove_preproc_dir: false
| Md Saiful Islam |remove_preproc_dir: false
| Aamir Imran |
| Salsabila Prasetya | remove_preproc_dir: false
| Alma Raunak |remove_preproc_dir: false
| Swagatalaxmi Das| remove_preproc_dir: false
| Prajwal Khanal |remove_preproc_dir: false
| Jiangtao CAI |remove_preproc_dir: false
| Mewcha A.Gebremedhin| remove_preproc_dir: false
| Resul Comert |remove_preproc_dir: false
| Serkan Girgin |
| Zengjing song |
| Tanuj Pareek |remove_preproc_dir: false
| Chakshu Gururani| remove_preproc_dir:false
| Qianqian Han |remove_preproc_dir: false
| Hakan Tanyas |remove_preproc_dir: false
| Ting Duan |remove_preproc_dir: false
| Kiana Karimi| remove_preproc_dir: false
| Nils Tjaden | remove_preproc_dir: false
| Enting Tang | remove_preproc_dir: false
| Muhammad Badar Munir | remove_preproc_dir: false
| Seckin Fidan |remove_preproc_dir: false

### Exercise 9
Output files and directories:

After the banner and general information, the output starts with some important locations.

1) Did ESMValTool use the right config file?
2) What is the path to the example recipe?
3) What is the main output folder generated by ESMValTool?
4) Can you guess what the different output directories are for?
5) ESMValTool creates two log files. What is the difference?

| Name| Answer|
| ---- | ---- |
| Santiago| 1. yes, 2.examples/recipe_python.yml, 3.esmvaltool_workshop/esmvaltool_output/4.
| Handini Puteri| 1. yes , 2. CRIB: esmvaltool run examples/recipe_python.yml , 3.esmvaltool_output , 4. For not making it overlap for different scenarios? 5. main_log.txt & main_log_debug.txt, handle different diagnostic for different customization
| Shreya Deep Bansod |1. yes, 2. ~/examples/recipe_python.yml, 3. esmvaltoom_output, 4.?, 5. log.txt
| Md Saiful Islam |1. Yes, 2. ~/examples/recipe_python.yml, 3. esmvaltool_output, 4.
| Aamir Imran |1. yes , 2. examples/recipie_python.hml, 3.esmvaltool_output 4.
| Salsabila Prasetya | 1. yes ; 2. examples/recipe_python.yml ; 3. esmvaltool_output
| Alma Raunak |1. Yes;2. /home/alma/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/esmvaltool/recipes/examples/recipe_python.yml ; 3.home/alma/esmvaltool_workshop/esmvaltool_output; 4.
| Swagatalaxmi Das| 1. Yes 2.examples/recipe_python.yml , 3.esmvaltool_workshop/esmvaltool_output 4. RUNDIR, WORKDIR, PREPROCDIR, PLOTDIR
| Prajwal Khanal | 1. Yes 2. /home/prajzwal/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/esmvaltool/recipes/examples/recipe_python.yml 3. ~esmvaltool_output 4. work, plots, work, 5. run/main_log , run/timeseries/log
| Jiangtao CAI |1.Yes;2.examples/recipe_python.yml;3.esmvaltool_output;4.For different simulation configuration; 5. varying levels of detail.
| Mewcha A.Gebremedhin| 1. Yes, 2. examples/recipe_python.yml, 3.esmvaltool_workshop/esmvaltool_output, 4.esmvaltool_workshope/esmvaltool_output/recipe_python_20230417_115919/run/timeseries/script1/log.txt
| Resul Comert |1. yes 2
| Serkan Girgin |
| Zengjing song |1. Yes 2.examples/recipe_python.yml; 3 esmvaltool_workshop; 4.
| Tanuj Pareek |1. Yes, 2. ~/recipes/examples/recipe_python.yml, 3. ~/esmvaltool_workshop/esmvaltool_output/ 4. RECIPE: for recipe itself, RUNDIR: running the main file, WORKDIR, PREPROCDIR, PLOTDIR: saving plots; 5. main_log, main_log_debug
| Muhammad Badar Munir | 1. Yes, 2. /home/badar/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/esmvaltool/recipes/examples/recipe_python.yml, 3. /home/badar/esmvaltool_workshop/esmvaltool_output, 4.RECIPE, RUNDIR, PREPROCDIR, PLOTDIR; 5. home/badar/esmvaltool_workshop/esmvaltool_output/recipe_python_20230417_114856/run/map/script1/log.txt;
| Chakshu Gururani| 1. Yes 2. /home/cgururani/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/esmvaltool/recipes/examples/recipe_python.yml 3. esmvaltool_output 4.
| Qianqian Han |1.yes 2.examples/recipe_python.yml 3.esmvaltool_output 4. 5.
| Hakan Tanyas |1. yes; 2. examples/recipe_python.yml; 3. esmvaltool_workshop
| Ting Duan |1. Yes 2. examples/recipe_python.yml 3. esmvaltool_workshop/esmvaltool_output 4. maybe for different data types 5. one for time series and the other one for map
| Kiana Karimi| 1) Yes, 2)examples/recipe_python.yml, 3)esmvaltool_output, 4)
| Enting Tang | 1. Yes; 2.recipe path: examples/recipe_python.yml; 3. ~/esmvaltool_workshop/esmvaltool_output/recipe_python_20230417_114757; 4.
| Nils Tjaden | 1. Yes; 2. ~/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/esmvaltool/recipes/examples/recipe_python.yml; 3. ~/esmvaltool_workshop/esmvaltool_output/; 4. plots, log files etc, working data; 5. main log, debug log
| Seckin Fidan |1.Yes; 2.recipe_python.yml; 3. esmvaltool_workshop/esmvaltool_output; 4.

## Tips & Tops
### Tips (what could be improved)
* Working with some datasets and modified codes for those datasets would have made workshop easier. Actual usage of the ESMValtool is still ambiguous. Similar to previous workshop like using datasets and comparing results was better in the introduction to raster and vector data using python. Diving a bit deep would have been better.
* Adding more picture in the introduction would make it easier to imagine the output.
* Before the lecture, a short survey can be created to collect what kinds of recipes that students are intersted (e.g., precipitation or dry days), and then show it in the tutorial
* Working on the specific dataset or study case would be easier to understand.

### Tops (what did you like)
* Very good for a brief and light description about the ESMValtool. Clean and short.
* Today's session showed a great potential of climate data analysis using the ESMValTool
* The second tutorial is very powerful to show the details of the output, like the graph, the map and etc.

* Very powerful in handling super complex netCDF dataset and useful description of the tool

* Really nice introduction to esmvaltool and was very intuitive.
* Helpful for beginner.
* The session was easy to follow and the error were sorted quickly with proper explanation (it was nice to understand why the error occured)

* It would have been good to have concise mnaual material for the specific training so that trainee can easily follow
* Nice detailed instruction, and helpful with the installlation of ESMvaltool.
* Very nice tutorial about this useful tool
* nice introduction to understand esmvaltool. Thanks
* very informative and innovative in the context of climate analysis
* Good introduction, established a base for the tutorial
* Helpful instructors, solved all doubts patiently
* Very organized lecture, look forward to more advanced lectures to work with ESMvalTool, CRIB and git. Thanks :D

## 📚 Resources

- [esmvaltool website](https://esmvaltool.org/)
- [esmvaltool documentation](https://docs.esmvaltool.org/en/latest/)
- [esmvaltool community](https://github.com/ESMValGroup/Community/discussions)
- [making your recipe or diagnostics](https://docs.esmvaltool.org/en/latest/develop/index.html)
- [discussion page](https://github.com/ESMValGroup/ESMValTool/discussions)
- [tutorial](https://esmvalgroup.github.io/ESMValTool_Tutorial/index.html)
- [Netherlands eScience Center-calls for proposals](https://www.esciencecenter.nl/calls-for-proposals/)
- [Alternative Installations](https://docs.esmvaltool.org/en/latest/quickstart/installation.html)
- [Example Recipe: recipe_python.yml](https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/recipes/examples/recipe_python.yml)
- [Diagnostic script: diagnostics.py](https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/diag_scripts/examples/diagnostic.py)