Tech lead meeting notes

Names legend: - Bouwe Andela [@bouweandela](https://github.com/bouweandela) - (former) Breixo Solino [@bsolino](https://github.com/bsolino) - Emma Hogan [@ehogan](https://github.com/ehogan) - Felicity Chun [@flicj191](https:github.com/flicj191) - (former) Klaus Zimmermann [@zklaus](https://github.com/zklaus) - Manuel Schlund [@schlunma](https://github.com/schlunma) - (former) Mike Tetley [@headmetal](https://github.com/headmetal) - (former) Remi Kazeroni [@remi-kazeroni](https://github.com/remi-kazeroni) - Romain Beucher [@rbeucher](https://github.com/rbeucher) - Saskia Loosveldt [@sloosvel](https://github.com/sloosvel) - Valeriu Predoi [@valeriupredoi](https://github.com/valeriupredoi) - Bettina Gier [@bettina-gier](https://github.com/bettina-gier) - Charles Turner [@charles-turner-1](https://github.com/charles-turner-1) - Julien Lenhardt [@jlenh](https://github.com/jlenh) --- Here is a template that can be copied and pasted for each meeting :) # <day> <month> <year> ## Participants - Attendees: - Absentees: - Chair: - Minutes taker: ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - Review of milestones for the next release - ... ## Agenda - ... --- # 6 February 2026 ## Participants - Attendees: @bouweandela @ehogan @schlunma @jlenh @valeriupredriupredoi @flicj191 - Absentees: @bettina-gier @charles-turner-1 @rbeucher - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - Review of milestones for the next release ## Agenda - release 2.14 update: @schlunma (with @jlenh and @flicj191 as release buddies): - ESMValCore: couple of minor PRs still to be merged (today), one big one #2918 that is still to be decided if merged for 2.14 - ESMValCore RC ideally to be rolled out start next week (9-13 February) - @ehogan asking about the new config structure if is there is still some backwards functionality; @bouweandela giving an overview about that, and what it'll be included in release 2.14 - @schlunma: reading non-CMOR standard data? @bouweandela: yes so long as it is CF-compliant, possible skip all checks and simply ingest the data into preprocessing; Bouwe also discussing the need for CMOR standrds for ESMValTool (need to have strict standards for inter-comparison) - @bouweandela suggesting to use the Release Slack channel! - ESMValTool Milestones: few bookeeping (renaming files etc), couple cmorizers, one MO diagnostic, ESA-related cmorizers that @schlunma will ask Axel - **we should do tech review** - @jlenh, @flicj191 "volunteered" - thank you, @bouweandela looking at distributing them - email response: tech support for training sessions organized by Ranjini - who will be there, and if we can provide support: @bouweandela will be at least at the one after the Workshop, @valeriupredoi will be too - @bouweandela: showing us traffic metrics from visitors to ESMValCore/Tool documentation and GitHub pages (newly implemented via Matomo) --- # 9 January 2026 ## Participants - Attendees: @bouweandela @ehogan @bettina-gier @charles-turner-1 @rbeucher @schlunma @jlenh @valeriupredriupredoi - Absentees: @flicj191 - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - Review of milestones for the next release ## Agenda - @ehogan: Happy New Year! :tada: Any thoughts on whether the TLT like to use the Thursday after the ESMValTool workshop in May (Thu, May 14) at the MO to meet with AVD? Please let me know! :blush: - ESA-CCI training on Thursday, need to determine whether there's a good time to meet that doesn't clash with this. Ask Ranjini / Alistair about the training. Could meet during a coding session during one of the workshop days instead. - Upcoming 2.14 release - Manuel is release manager, Tina will shadow, ACCESS-NRI (Felicity with help from someone else in the team) will do the next (v2.15.0) release, usually previous release manager helps the next one, but people could shadow the one before, if they wanted? - Bouwe parallel load of Dataset via Dask: could be an issue in iris, Bouwe will check that, if it is, then it'll be dropped from 2.14 - Intake-esm support --- # 5 December 2025 ## Participants - Attendees: @bouweandela @ehogan @bettina-gier @sloosvel @jlenh @valeriupredriupredoi - Absentees: everyone else - Chair: - Minutes taker: ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) (few topics proposed by @ehogan from 7 November 2025 call) - @ehogan: V offered to give an overview of AIVAL in my absence (thank you V!). A question from me: Would it be possible to add a standing agenda item to discuss technical work related to AIVAL? Or should we do this via a separate meeting? - @ehogan: Would the TLT like to use the Thursday after the ESMValTool workshop in May (Thu, May 14) at the MO to meet with AVD? Please let me know! :blush: - @bouweandela: Look at the milestones for the next release - ESMValCore: - stuff by @schlunma that needs review, but we need Manu around - fix in iris that follows CF for parametric vertical coords: doesn't really work ATM, but needs addressing - support for Python 3.14: stuck on fiona ATM - loading Datasets in parallel in Dask (one DS per worker) -> probably to be moved to next Milestone - documentation issue by @ehogan -> probably to be moved to next Milestone - @jlenh planning to clean up release docs (with updates) - thank you! - @jlenh asking about decadal support: @sloosvel mentions issues with the data - needs be complete for decadal analysis - EC-Earth project - @sloosvel: question about the new configuration: do we need to edit config-developer.yml? @bouweandela: legacy config kept there for bkwds compat, will be removed; so far the CMOR config and preproc output filename, but Bouwe will move them to new config file format too - @valeriupredoi: Zarr loading via front-end (recipe), all depends on the source and node (catalog, esgf index, Pangeo-style etc) For ESMValTool: - sealevel rise patterns - RTW -> next Milestone - add ENES-RI to Zenodo communities - EN4 cmorizer (the one with many HDF5 files, each opened/closed multiple times), being investigate by @jlenh - suggested by @bouweandela to play around with Dask configuration - @all-of-us to have a look see if we can find stuff open and needing review --- # 7 November 2025 ## Participants - Attendees: @sloosvel @bouweandela @rbeucher @valeriupredoi @bettina-gier @jlenh - Absentees: @flicj191 @schulnma (Papa Leave) @ehigal (Leave) @charles-turner-1 - Chair: @valeriupredoi - Minutes taker: @valeriupredoi (severely lacking coffee) ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - items proposed by @ehogan to be discussed: discussion to be done when @ehogan present: - @ehogan: V offered to give an overview of AIVAL in my absence (thank you V!). A question from me: Would it be possible to add a standing agenda item to discuss technical work related to AIVAL? Or should we do this via a separate meeting? - @ehogan: Would the TLT like to use the Thursday after the ESMValTool workshop in May (Thu, May 14) at the MO to meet with AVD? Please let me know! :blush: - @sloosvel announcing her leaving BSC - @bouweandela @valeriupredoi @rbeucher commenting on the [new ESGF2 support](https://github.com/ESMValGroup/ESMValCore/pull/2765) by @bouweandela ## Agenda - ... --- # 3 October 2025 ## Participants - Attendees: @valeriupredoi @bouweandela @charles-turner-1 @flicj91 @sloosvel - Absentees: @rbeucher @bettina-gier @schulnma @jlenh - Chair: @bouweandela and @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items ## Agenda - @ehogan: As a way to prioritise issues, would it be worth using a project board, or at least reviewing issues at TLT meetings? -> @bouweandela: we do have Milestones (M) (and are used during release cycles, but there a lot of outstanding issues/PRs that do get pushed from M to M often) - @ehogan: I discovered recently that variable names will be changing for CMIP7: https://github.com/WCRP-CMIP/CMIP7-CVs/issues/172. There's also some example test tables in the CMOR repo https://github.com/PCMDI/cmor/blob/main/TestTables/CMIP7_atmos2d.json#L40 with the branded variable suffix. Will this cause issues for ESMValTool? :thinking_face: -> few changed elements like underscores; @sloosvel: there is already a CMIP5-CMIP6 list and we can maybe add to that - @sloosvel: Just a reminder that the new release schedule is now available in the documentation: https://docs.esmvaltool.org/en/latest/community/release_strategy/release_strategy.html#upcoming-releases - we are going through the milestones listed - @ehogan will be looking at the split docs PR #2732 which is nearly ready - @bouweandela: REF launch event next week: https://staging.climate-ref.org/explorer/themes?theme=atmosphere and https://github.com/Climate-REF/climate-ref/ --- # 2 September 2025 ## Participants - Attendees: @jlenh @rbeucher @sloosvel - Absentees: @valeriupredoi @bouweandela @ehogan @schlunma @bettina-gier @flicj191 @charles-turner-1 - Chair: - Minutes taker: @sloosvel ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - @ehogan: As a way to prioritise issues, would it be worth using a project board, or at least reviewing issues at TLT meetings? ## Agenda - ESMValTool fetaure freeze to be done tomorrow to include the following ENSO recipe: https://github.com/ESMValGroup/ESMValTool/pull/3905 - ESMValCore second release candidate to be done either today or tomorrow - @bouweandela will be unavailable on Tuesdays, can the TLT meet on a different day? - @sloosvel to create a poll to find another date for the TLT meeting --- # 1 July 2025 ## Participants - Attendees: @sloosvel @bettina-gier @ehogan @bouweandela @schlunma @flicj191 - Absentees: @charles-turner-1 @rbeucher @jlenh @valeriupredoi - Chair: @bouweandela - Minutes taker: @schlunma ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) ## Agenda - Review of Pending Action Items - none - @bouweandela: Announce release? Documentation already updated - no updates - @ehogan: RTW - V, apologies for not getting in touch with you to discuss this before now! Can we discuss it during this meeting? :blush: We are using the GitHub REST API for the RTW [status report](https://esmvaltool.dkrz.de/shared/esmvaltool/rtw/status_report.html) and the documentation states that "you can authorize a GitHub App, which can then make API requests on your behalf.". How easy would it be to authorize the [esmvalbot](https://github.com/apps/esmvalbot) to do this for us? - @bouweandela: esmvalbot is a GitHub probot app and can already comment on issues, so it will probably be possible to use it; can also create new bot (see https://github.com/probot/probot, https://docs.github.com/en/apps/creating-github-apps/about-creating-github-apps/about-creating-github-apps#github-apps-that-act-on-their-own-behalf) - @bouweandela: Alternative: setup GitHub action that reads output from DKRZ and reports possible problems (open issues, see https://github.com/marketplace/actions/open-issue-action) - @ehogan: On DKRZ, usually only 5 jobs can be run in parallel (would take very long to run all recipes), but we have been given special settings to be able to run 50 jobs in parallel; needs to be implemented though - @schlunma: this will probably be last TLT meeting before parental leave, will be back around December --- # 3 June 2025 ## Participants - Attendees: @sloosvel @bettina-gier @charles-turner-1 @rbeucher @ehogan @jlenh - Absentees: @bouweandela @schlunma @flicj191 - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - none ## Agenda - Note: our Zoom link has changed! - @charles-turner-1: work on IntakeESM integration, since it's cloud-based, integratio with Xarray (trace is: URI -> S3 bucket -> esmvalcore `Dataset` object -> get `ncdata` perhaps, -> few extra complications); work at https://github.com/intake/intake-esm/pull/717 - @valeriupredoi and @bettina-gier: Julia removal (and its friends) + Python 3.13 support - now all done! - @ehogan: CONFIG environment variable - working with @schlunma on fixing it (since it broke RTW) - @jlenh: should have docs on how install ESMValTool centrally on an HPC --- # 6 May 2025 Meeting cancelled in favour of the workshop :blush: --- # 1 April 2025 ## Participants - Attendees: @bouweandela @sloosvel @charles-turner-1 @jlenh @valeriupredoi @ehogan @rbeucher - Absentees: @flicj191 @schlunma @bettina-gier - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| | 2025-xx-xx | xx | xx | xx | ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - all caught up with - @charles-turner-1 Intake cataloging/find files/datasets - @bouweandela: maintainance - R issues: upper limit pin on R is keeping old dependencies and environment not solving well, we need help fixing the broken R recipes (that needed the R pin); also help fully replacing prospector with ruff; @sloosvel: is happy to ask BSC folks about the R recipes (thank you!) https://github.com/esMValGroup/esMValTool/issues/3674 - @jlenh: fire diagnostics for REF: integration of diagnostics is done; some checks still needed (e.g. units for burnt area); draft pull requestnot yet, branch https://github.com/ESMValGroup/ESMValTool/tree/ref_fire_diagnostic - @jlenh: REF - if there is a place where output (plots, data) can go to; @bouwenadela: yes and no, we have a board https://github.com/orgs/Climate-REF/projects/2/views/1 (and of course, the documentation pages for each recipe) --- # 4 March 2025 ## Participants - Attendees: @bouweandela @sloosvel @bettina-gier @charles-turner-1 @jlenh @valeriupredoi @ehogan - Absentees: @rbeucher @flicj191 @schlunma - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| | 2024-12-03 | @rbeucher | Arrange meeting on Intake | happened | | 2024-12-03 | ~~@ehogan~~ @bouweandela | Split docs meeting | happened, @flicj191 will start working on this, see https://github.com/esmValGroup/esMValTool/issues/3860 for further information - item done | | 2025-02-04 | @valeriupredoi | set up TLT-SLT call for broken recipes | item done | ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - Status of release - Core has been released! - Tool stable release out today: one test not picking up the Core 2.12, and remember to pin iPython - @ehogan: Sharing from the User Engagement Team (UET): Two tutorial-related items requiring feedback from the TLT: - Proposal to divide the tutorial into two repositories. This has the benefit of enabling adding other advanced tutorials, e.g. from ACCESS-NRI & MO, domain-specific, etc., without requiring coordination / merging with the main tutorial(s) below: - a basic tutorial that gets updated regularly (and tagged to match the ESMValTool version) - an advanced tutorial that is more community driven and may not be updated as regularly - @boueandela: this has always been the aim - so it's all cool - Rather than using the new Software Carpentry template (I am struggling to see the benefits of continuing to use the Software Carpentry template), can we consider using what we are currently using for the documentation? Benefits include: - there is no need to maintain difficult-to-build R-based environments - the tutorial(s) could be hosted via Read the Docs - making changes to the tutorial(s) should be easier, due to familiarity of the structure / syntax (it would be the same as the documentation) - the new Software Carpentry theme (example: https://swcarpentry.github.io/git-novice/) does not look dissimilar to the (Sphinx + PyData) theme - discussion on this: we like, but TLT can't really spare time to work, apart from helping with configuration and setup - We could provide a template repository within ESMValGroup that people could use as a starting point for their own tutorial(s). - @bouweandela and @charles-turner-1: happy with Intake? It's a lot of technical work with packaging and internal `$PATH` issues, conda env issues etc - @jlenh: ESMValTool `main` branch used for the REF, adding dependencies: @bouwenadela careful with license, and then be maintained - @bettina-gier: tests for CMORizers? That a good idea - pls open a GH issue or Discussion item - we also need to figure out where to continue the call: GMeets, new Zoom link or Teams; @valeriupredoi will do some due dilligence asap ## Agenda - ... --- # 4 February 2025 ## Participants - Attendees: @bouweandela @schlunma @rbeucher @sloosvel @bettina-gier @charles-turner-1 @valeriupredoi - Absentees: @ehogan @flicj191 - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| | 2024-12-03 | @rbeucher | Arrange meeting on Intake | happened | | 2024-12-03 | ~~@ehogan~~ @bouweandela | Split docs meeting | happened, @flicj191 will start working on this, see https://github.com/esmValGroup/esMValTool/issues/3860 for further information | | 2025-02-04 | @valeriupredoi | set up TLT-SLT call for broken recipes | Pending | ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - @schlunma: - Add Julien from SMHI to the tech lead team? ## Agenda - Pending actions: both have been addressed: docs split will be started by @flicj91 (cheers!) and meeting about the Intake lib has happened - @schlunma: broken recipes discussed in detail: SLT discussed [broken recipe policy](https://docs.esmvaltool.org/en/latest/community/broken_recipe_policy.html) - Current policy: - Broken recipe and diagnostic code are removed from the ESMValTool main branch -> @bouweandela raises the issue of complete removal (NOT for an example like Julia recipe!), there are ways of "archiving" old code but not removing, but those are difficult on the technical side - Scientific documentation of the recipe (and diagnostics) will be kept in the user and developer guide with an additional note and link to the last release in which the recipe was still fully functional -> @bouweandela: code will not actually work since deps are too old, unlikely to have an out-the-box solution - an idea to have them on Zenodo, @valeriupredoi: since both deps and data will become unavailable (could bundle conda-lock files) it'll be difficult - Policy does not cover the case of removing unused/outdated recipes/diagnostics from the testing and maintenance framework of the tool - Idea: - Introducing an "archive" folder in the ESMValTool repository (with subfolders "recipes" and "diagnostic_scripts") - Possibility to move broken/unused/outdated recipes and/or diagnostic scripts to this new folder (instead of deleting the code) - Reasons: - Make sure diagnostics and recipes that have been advertised in presentations and papers remain part of the ESMValTool distribution (as "inactive" components) - Ensure long-term and easiest possible access to old code for scientists - Make maintenance and testing of the tool as well as releases easier by retiring unused/outdated recipes (by the recipe maintainer) - Examples include recipes relying on CMIP5 data that are not available any more, the NCL-based "perfmetrics" package or @schlunma's paper from 2020 - Special care has to be taken when retiring diagnostic scripts to avoid breaking of other diagnostics / recipes - The science team is in favor of this idea - @schlunma will report this idea at the next meeting of the Technical Lead Development Team (TLT) - Way forward is to open a pull request to adapt/extend the broken recipe policy - Need to have a final decision very soon: SLT have more or less decided on "archiving", TLT is discussing now, but as @bouweandela mentions - it's a joint TLT-SLT matter, so @valeriupredoi will set up a TLT-SLT call - Upcoming release: RTW will not be used for this release (not all the recipes are there, there are a few teething issues that need ironing out - @sloosvel is opening issues, thank you!); @bouweandela mentions we could start setting it up (at least for "short" recipes); @sloosvel - MO have a GUI that could be ported - somewhere (@bouweandela: where though?) - that could be a good start; @sloosvel is running the RTW on about 100 "short" recipes on Levante, those can be used for the release, but there is little time to add the beefier ones before the release; @sloosvel will announce the release schedule soon. - add Julien (SMHI) to the TLT - will ask him! --- # 7 January 2025 ## Participants - Attendees: @schlunma @bouweandela @valeriupredoi @charles-turner-1 @flicj191 @ehogan @bettina-gier - Absentees: @sloosvel - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| | 2024-12-03 | @rbeucher | Arrange meeting on Intake | Pending | | 2024-12-03 | ~~@ehogan~~ @bouweandela | Split docs meeting | Pending | ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - @ehogan: RTW / postponing the release? ([RTW project board](https://github.com/orgs/ESMValGroup/projects/27/views/1)) ## Agenda - The only thing we talked about other than wishing each other happy new year, and complaining about the weather, was the potential, andpossible delay of v2.12 release by a few (roughly three) weeks, as prompted by @ehogan; this is due to the necessity to includeas many recipes as possible in the Recipe Test Workflow (RTW); @ehogan will have a chat with @sloosvel about this, and @valeriupredoi offered to organize an extra TLT just before the release process starts, if need be. AOB: @bettina-gier has kindly volunteered to organize the main ESMValTool call (many thanks, Tina!!) --- # 3 December 2024 ## Participants - Attendees: @schlunma @bouweandela @rbeucher @valeriupredoi @charles-turner-1 @flicj191 @ehogan @sloosvel @bettina-gier - Absentees: - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| | 2024-12-03 | @rbeucher | Arrange meeting on Intake | Pending | |2024-12-03| @ehogan | Split docs meeting | Pending | |2024-08-13|all|Review [RTW pull request](https://github.com/ESMValGroup/ESMValTool/pull/3210)|Done, thank you everyone! :tada: | ## Agenda - Review of Pending Action Items - @ehogan: From last time: A new Geoscience RSE community has recently been formed (https://society-rse.org/weather-and-climate-rsecon24-birds-of-a-feather-session-retrospective-blog-post/) and we are interested in organising a townhall meeting for RSEs at EGU25 next year; will there be an ESMValTool presence there and should we get involved? - @ehogan: I have created a RTW project board (https://github.com/orgs/ESMValGroup/projects/27/views/1) to help @sloosvel and I (others are welcome!) work together to get RTW working for v2.12.0 release :tada: @bettina-gier: where to put smaller test recipes, Emma says tag her - still to be decided; @bouweandela: test ones on CircleCI - but data must be small, since they get downloaded/run every night, see [CI test configuration](https://github.com/ESMValGroup/ESMValTool/blob/ba9cfe426f9f2a97dba672784ccba8462eea7d6e/.circleci/config.yml#L177-L231) -> these test Core mostly; - @ehogan: Chris is back working on ESMValTool until he finishes with his Apprenticeship (~May 2025). Is there anything, perhaps something non-critical-path-Dask-related, that he could get involved in? - @bouweandela: could run all recipes with the `dask.Distributed` scheduler, not all work, it's a case-by-case challenging work, @ehogan proposes using the simple scheduler first; @bouweandela also proposes `pyproject.toml`-ing ESMValTool; @schunma is very keen on Dask optimization with `Distributed`, @bouweandela proposes Manu work mostly on it, with Chris testing; - @ehogan: Using the new Software Carpentry template [ESMValTool_Tutorial: Issue #335](https://github.com/ESMValGroup/ESMValTool_Tutorial/issues/335) - @ehogan: Should we arrange a meeting to come up with a documentation design following this [comment on "Agenda for ESMValTool community workshop: 19-21 November"](https://github.com/ESMValGroup/Community/discussions/192#discussioncomment-11258741) - @rbeucher is keen to get Felicity onboard; Emma will organize the first meeting on this via a Doodle, this will be a **to-do item** here; - @ehogan: Somehow at the MO we have ended up with multiple date ranges for HadISST: ``` % ls /data/users/esmval/ESMValTool/temporary/obs/Tier2/HadISST/ OBS_HadISST_reanaly_1_Amon_ts_187001-201712.nc OBS_HadISST_reanaly_1_Amon_ts_187001-202112.nc OBS_HadISST_reanaly_1_OImon_sic_187001-201712.nc OBS_HadISST_reanaly_1_OImon_sic_187001-202112.nc OBS_HadISST_reanaly_1_Omon_tos_187001-201712.nc OBS_HadISST_reanaly_1_Omon_tos_187001-202112.nc ``` I found [ESMValTool: PR #3072: Update HadISST cmorizer with recent years](https://github.com/ESMValGroup/ESMValTool/pull/3027), so I understand why the date ranges changed, but is it expected that ESMValTool will error with this obs setup? Temporary solution: delete older datasets, keep latest dataset with latest version; - @ehogan: Slack for more general use Ask the UET to join, possibly wider community to join as well; - @rbeucher: intake discussion - @charles-turner-1 started looking at adding datasets; a meeting would be useful with Bouwe, Tina, the Met Office contractor might have switched focus, if at all. @rbeucher will organize a meeting, **this is a to-do item** --- # 12 November 2024 ## Participants - Attendees: @schlunma @bouweandela @rbeucher @valeriupredoi @charles-turner-1 - Absentees: @flicj191 @ehogan @sloosvel - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2024-10-01| @ehogan | Open an Issue or Discussion item about geospatial benchmark suite | Done: [ESMValTool: Discussion #3764: Would we like to help create a geospatial benchmark suite?](https://github.com/ESMValGroup/ESMValTool/discussions/3764) |2024-08-13|all|Review [RTW pull request](https://github.com/ESMValGroup/ESMValTool/pull/3210)|pending| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - @ehogan: I know I'm not here :stuck_out_tongue_closed_eyes: but a new Geoscience RSE community has recently been formed (https://society-rse.org/weather-and-climate-rsecon24-birds-of-a-feather-session-retrospective-blog-post/) and we are interested in organising a townhall meeting for RSEs at EGU25 next year; will there be an ESMValTool presence there and should we get involved? ## Agenda - @valeriupredoi [Python 3.13 support PR](https://github.com/ESMValGroup/ESMValCore/pull/2566) - @bouweandela offered to look, cheers - @rbeucher @charlesturner we briefly discussed Intake-ESM contributions, already started by NRI (Charles is now a maintainer of Intake-ESM) - more ANON at the workshop - @bouweandela Having ESMValCore as a subproject of the ESMValTool documentation is confusing to users and various things about the setup are broken [ESMValGroup/ESMValTool#3490](https://github.com/ESMValGroup/ESMValTool/issues/3490). Would it be an option to split the documentation of the two projects? - Bouwe mentions a clear separation makes things easier and clearer, but there may be administrative/political headaches (umbrella project ESMValGroup etc); @schlunma: references changed to URLs may be quite a bit of work; @valeriupredoi: we need to find time/person; @romain: Felicity could do it (pending on Steering Committee approval) - @schlunma: Brief update on paper: paper needs internal clearance (formality!), also reruns with iris 3.11 - the more hefty upgrade needed, and a ESMValCore=2.11.1 release, hoping end of November submission --- # 1 October 2024 ## Participants - Attendees: @sloosvel @schlunma @bouweandela @ehogan @rbeucher @valeriupredoi - Absentees: @headmetal (not TLT anymore) @flicj191 - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2024-08-13|@bouweandela @schlunma|[Variable derivation for ERA5 in the fix](https://github.com/ESMValGroup/ESMValCore/issues/1806) -> [draft PR](https://github.com/ESMValGroup/ESMValCore/pull/2551) open|done| |2024-08-13|all|Review [RTW pull request](https://github.com/ESMValGroup/ESMValTool/pull/3210)|pending| |2024-08-13|all|Find a new release manager|done| |2024-09-03|@schlunma|Re-ping UET with e-mail and ask for review of new [configuration PR](https://github.com/ESMValGroup/ESMValCore/pull/2448)|done| |2024-09-03|all|Brainstorm possible topics for new ESiWACE computational performance improvement proposal (call will be published [here](https://www.esiwace.eu/services/software-support), likely towards the end of the year)|for spring, not now| |2024-09-03|@schlunma|Circulate paper about ESMValTool Dask improvements|done| ## Agenda - Review of Pending Action Items: one new set to DONE: next release manager @sloosvel (many thanks, Saskia!) - @schlunma: Vote on potential new TLT members: @bettina-gier Tina: will be working on IntakeESM, Healpix (technical stuff), PR reviews - there was a vote, and all in favour! @valeriupredoi and @schlunma: add her to the mailinglist, Slack, Google Cal event for the TLT; also Mike Tetley @headmetal to be removed, and add Charles Taylor (email off Slack); - @bouweandela: Can we disable coverage collection by default when running tests locally [ESMValGroup/ESMValCore#2456](https://github.com/ESMValGroup/ESMValCore/pull/2456#discussion_r1634728779) -> takes about 2x slower to run tests: we like it, so long as there is documentation on how to turn it back on; - @bouweandela: Switch from yapf to ruff and fix our pre-commit set up? See e.g. [ESMValGroup/ESMValTool#2161](https://github.com/ESMValGroup/ESMValTool/discussions/2161) and [ESMValGroup/ESMValCore#2490](https://github.com/ESMValGroup/ESMValCore/pull/2490#issuecomment-2342899385); @schlunma any plans for esmvaltool? @bouweandela: we'll try see how it goes; - @bouweandela: Consider adopting spec0 [ESMValGroup/ESMValCore#2533](https://github.com/ESMValGroup/ESMValCore/issues/2533) -> all happy about it, @bouweandela points us to a checker, also we should start preparing docs for it, and adopt the new standard; - @ehogan: Would we like to help create a geospatial benchmark suite, as described in this blog post from Coiled (Matt Rocklin's dask-as-a-service company): [Large Scale Geospatial Benchmarks](https://docs.coiled.io/blog/geospatial-benchmarks.html)? Martin and Bill in AVD suggested that ESMValTool might well be able to help out with one of the asks, which is code that does large-scale geospatial data processing; @ehogan promised to open an Issue or Discussion item about it; - @schlunma: What do we about NCL? There are (again) problems with our environment, see [ESMValGroup/ESMValCore#2535](https://github.com/ESMValGroup/ESMValCore/issues/2535): separate module maybe; @bouweandela: leave it to user to install it; @schlunma: diagnostics are old, most devs switching to Python for new stuff -> **topic to be brought at the workshop**; @rbeucher: a few recipes (eg Perfmetrics) that are good, to be ported to Python: possible solutions include separate conda envs and docker containers; this item has a **Workshop TODO**; - @schlunma: Very brief update on ESMValTool+Dask paper: to be re-circulated today(ish) and some of us still need to send comments; @bouweandela give feedback on current performance status: few benchmark discrepancies to look at ie perfomance is *even* better (eg used dev branch of `iris`); - @valeriupredoi: work on modernizing and re-deploying our ESMValBot https://github.com/ESMValGroup/ESMValBot/pull/37 - @valeriupredoi: `codacy-ruff` plugin to Codacy: https://github.com/ESMValGroup/ESMValTool/issues/3739 - @rbeucher: `intake-esm` catalog support: more and more interest in adding datasets to the ESM catalog; @bouwenadela: looking at XCube also, a proposal in the pipes, so there may be time to implement `intake` support; coordinate with Tina, so there is no duplication -> a meeting to be set up; - @schlunma wants to merge PR that introduces new (dask-like) configuration system today ([ESMValGroup/ESMValCore#2448](https://github.com/ESMValGroup/ESMValCore/pull/2448)): this needs ne merged **ASAP**; - @ehogan: It's [International RSE Day](https://society-rse.org/international-rse-day-october-10th-2024/) on October 10! :blush: yay! - AOB: good news! We have our next Release Manager (for v2.12.0): @sloosvel - thank you, Saskia! ETA for release January 2025. --- # 3 September 2024 ## Participants - Attendees: @sloosvel @schlunma @bouweandela - Absentees: @valeriupredoi @ehogan @headmetal @rbeucher @flicj191 - Chair: @bouweandela - Minutes taker: @schlunma ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2024-08-13|@bouweandela @schlunma|[Variable derivation for ERA5 in the fix](https://github.com/ESMValGroup/ESMValCore/issues/1806)|pending| |2024-08-13|all|Review [RTW pull request](https://github.com/ESMValGroup/ESMValTool/pull/3210)|pending| |2024-08-13|all|Find a new release manager|pending| |2024-09-03|@schlunma|Re-ping UET with e-mail and ask for review of new [configuration PR](https://github.com/ESMValGroup/ESMValCore/pull/2448)|pending| |2024-09-03|all|Brainstorm possible topics for new ESiWACE computational performance improvement proposal (call will be published [here](https://www.esiwace.eu/services/software-support), likely towards the end of the year)|pending| |2024-09-03|@schlunma|Circulate paper about ESMValTool Dask improvements|pending| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items ## Agenda - Review of Pending Action Items - Add new action items - Support for intake: @bouweandela and Tina might be able to work on that --- # 13 August 2024 ## Participants - Attendees: @valeriupredoi @schlunma @bouweandela - Absentees: @sloosvel @ehogan @headmetal @rbeucher @flicj191 - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - @schlunma: how can we support variable derivation for ERA5 in the fix? https://github.com/ESMValGroup/ESMValCore/issues/1806 - @valeriupredoi: development dependencies test https://github.com/ESMValGroup/ESMValCore/issues/2492 - @valeriupredoi: PRs that need ESMValCore a long time before release cycle https://github.com/ESMValGroup/ESMValTool/pull/3697 - @valeriupredoi: ESMValTool ready for Python 3.12 bar `prospector` https://github.com/ESMValGroup/ESMValTool/pull/3501 - @schlunma: pin iris? https://github.com/ESMValGroup/ESMValCore/pull/2500 - @bouweandela: ESMValBot maintainance ## Agenda - let's come back to how can we support variable derivation for ERA5 in the fix? https://github.com/ESMValGroup/ESMValCore/issues/1806 - @valeriupredoi: development dependencies test https://github.com/ESMValGroup/ESMValCore/issues/2492 - should run only on `main` nightly; @valeriupredoi will open a PR fixing this in CircleCI configuration - @valeriupredoi: PRs that need ESMValCore a long time before release cycle https://github.com/ESMValGroup/ESMValTool/pull/3697 - this is a fluffy topic that involves bugfix releases scheduled reguarily that, in turn, depends on the RTW (Recipe Test Workflow) that is still in prototype/pre-production, hence... - ...we should have another look at the RTW PR in `main` https://github.com/ESMValGroup/ESMValTool/pull/3210 - @valeriupredoi: ESMValTool ready for Python 3.12 bar `prospector` https://github.com/ESMValGroup/ESMValTool/pull/3501 - the only showstopper is, as we mentioned, `prospector` which, according to @bouweandela, and everyone else, is a bit dead; @valeriupredoi will start the integration process of Ruff into Codacy (since we really want Codacy umbrella) after holidays in September; - @schlunma: pin iris? https://github.com/ESMValGroup/ESMValCore/pull/2500 - that PR is now Ready-for_Review, it is great to have two supported Iris versions (latest, one before latest) but if API changes are large, we can go with just the latest on occasion (like now) - @bouweandela: ESMValBot maintainance: the Bot needs updating (really old deps), and needs migration to the new Levante tenancy; @valeriupredoi will help with this September onwards -> --- # 2 July 2024 ## Participants - Attendees: @bouweandela @ehogan @flicj191 @schlunma @sloosvel - Absentees: @headmetal @rbeucher @valeriupredoi - Chair: @schlunma - Minutes taker: @bouweandela ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items ## Agenda - @ehogan: Update on the release :blush: - @ehogan: Where would be the best place to store KGOs for the RTW on DKRZ? - @ehogan: Would it be possible to get [gh (cli for GitHub)](https://github.com/cli/cli) installed on DKRZ, so the RTW can open issues when a recipe fails? - @schlunma started a document for the "Dask-ESMValTool" paper (only bullet points for now). Who wants to contribute? Would be great to brainstorm ideas for content together - @schlunma: If we find a suitable topic, it would be possible to submit another ESiWACE3 proposal (starting end of 2024), but we would have bigger chances if this is not submitted by DLR (because we already got the first one funded) -> relevant/desired for us? - @schlunma: it looks like ESGF will retire CMIP5 and CMIP3 and CORDEX-CMIP5 datasets very soon - what do we do abuot this? SLT thought about "archive repository" - @ehogan: Do we test against Iris release candidates? Should we? ## Topics discussed - @ehogan Release testing underway, would like some help with the final reviews. Hopefully the release will happen today. @schlunma and @bouweandela offered to help out. - Recipe test workflow: the recommended place for somewhat longer term storage is the bd0854 [group workspace](https://docs.dkrz.de/doc/levante/file-systems.html), but note that space is limited. Longer term we would like to store the known good outputs on Zenodo. - Add `gh` to ESMValTool development dependencies in environment(_osx).yml, it is not installed on Levante as a module - @schlunma Please contact me if you would like to be involved in writing the paper on improving ESMValCore computational performance. - The current ESiWACE3 service project runs until September 2024. There is an option to submit a new ESiWACE3 proposal for any issues we cannot solve before the project is over. Please report remaining computational performance issues (e.g. slow data load, high memory use, long runtime, etc) so we can use them for the next proposal. A GitHub issue is usually a good place, but this discussion [ESMValGroup/ESMValCore#1763](https://github.com/ESMValGroup/ESMValCore/discussions/1763) can also be used. - CMIP5/CMIP3 data availability: If CMIP5 and CMIP3 data would be no longer available from ESGF, this would affect 125 out of 160 recipes currently available in ESMValTool. We discussed several options: - Try to convince the ESGF developers that they should keep the data available because it is necessary for reproducible science. Papers like GMD demand robust archives for publications and it looks like ESGF is not capable of develivering this: https://www.geoscientific-model-development.net/policies/code_and_data_policy.html - Use the (fraction of?) the data that is archived at local centers, though it is not yet clear how this will be made available. - Re-publish relevant data on a more reliable repository, e.g. Zenodo, and update ESMValCore so it can download from there - Update recipes to CMIP6 where it makes sense to do so (e.g. general recipes, example recipes) - Retire recipes reproducing specific papers - Testing Iris release candidates could be done with the Recipe Test Workflow @ehogan Will take this into consideration for the long term development of the recipe test workflow. @bouweandela will make a nightly ESMValCore test run with the Iris development version to catch issues early. --- # 7 May 2024 ## Participants - Attendees: @bouweandela @ehogan @sloosvel @flicj191 @headmetal @rbeucher - Absentees: @schlunma - Chair: @valeriupredoi - Minutes taker: @valeriupredoi @bouwenadela ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@bouweandela|Shared gmail account to handle to Zenodo credentials| done - check email | ## Agenda - @ehogan: Testing NumPy v2? - @ehogan: Do not use Iris v3.7.0 or v3.8.0; instead use 3.7.1, 3.8.1 or >= 3.9 to avoid a regridding memory leak - @ehogan: CircleCI and GitHub workflow; can they be consolidated? - @valeriupredoi Prospector woes, can anyone review [this PR](https://github.com/landscapeio/prospector/pull/658) - @valeriupredoi Issue with ESMPy (`esmpy`) not being on PyPI - @ehogan: Release stuff ## Topics discussed - Testing with NumPy v2: there is a [draft pull request](https://github.com/ESMValGroup/ESMValCore/pull/2395) by @valeriupredoi open to do tests - Pinning Iris: @ehogan will open pull requests to pin Iris to newer versions (will restrict to `>3.8.0`) - The reason we have both CircleCI and GitHub Actions is that is has historically grown like this. CircleCI is better because it is a dedicated CI provider, but GitHub Actions is nice because it provides many predefined workflows. It seems a bit of a waste of time to change this ie there are a lot more pressing issues to follow up on (e.g. @bouweandela suggest updating to `pyproject.yml`) - Release stuff: discussion about dedicated release branch vs mergeback into `main` branch: @bouweandela explains a linear history, target is different otherwise. Two open pull requests in milestone that can probably be merged today, one that has seen no activity in the past 2 years, so it seems unlikely that anything will happen before the deadline. That means that @ehogan will probably be able to do the ESMValCore feature freeze today. - Issue with ESMPy (`esmpy`) not being on PyPI. @bouweandela: this package has never been on PyPI, so why does it cause problems now? An issue with the condamamba env that makes `pip` actually look for the pinned package on PyPI - I'll check all is fine and we should get the PR merged today (with keeping `esmpy` in the `setup.py` list) - @valeriupredoi Prospector woes, can anyone review [this PR](https://github.com/landscapeio/prospector/pull/658) - please please :)) # 2 April 2024 ## Participants - Attendees: @bouweandela @ehogan @schlunma @sloosvel - Absentees: @flicj191 @headmetal @rbeucher @valeriupredoi - Chair: @bouweandela - Minutes taker: @bouweandela ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@bouweandela|Shared gmail account to handle to Zenodo credentials|pending| ## Agenda - Review of Pending Action Items - @schlunma Attending Iris Peleton meetings - @ehogan Update on release planning ## Topics discussed - Review of Pending Action Items: we dropped the action items waiting on @ehogan until she has time to work on them. - [Scitools Peleton meetings](https://github.com/orgs/SciTools/projects/13): these meetings are a great opportunity to get involved with Iris development. @bouweandela will attend the meetings regularly at least until the end of the [ESiWACE3 ESMValTool service project](https://github.com/orgs/ESMValGroup/projects/20) in October. The time is not very practical for @schlunma, so he will attend occasionally when he is working on something iris related. The meetings are at 11 AM CEST, so it should be possible to join them from Australia if there is an interest. - @ehogan Release date of ESMValCore and ESMValTool v2.11 is likely end of April/beginning of May --- ## Topics discussed # 5 March 2024 ## Participants - Attendees: @bouweandela @ehogan @valeriupredoi @schlunma @sloosvel - Absentees: @headmetal @rbeucher @flicj191 - Chair: @valeriupredoi - Minutes taker: @valeriupredoi / @bouweandela ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@ehogan|KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news| |2023.07|@bouweandela|Asking the community to check the output once for all| Waiting for KGO recording solution to be ready. @ehogan: see [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for some additional thoughts on this| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| | 2023.11 | @ehogan | Create a security policy PR | See [ESMValTool: issue: #3455](https://github.com/ESMValGroup/ESMValTool/issues/3455) and [ESMValTool: PR: #3456](https://github.com/ESMValGroup/ESMValTool/pull/3456) | | 2023.11 | @ehogan | Present summary of formatting / linting tools at an upcoming monthly community meeting | Not yet started, but will continue to document on [ESMValTool: discussion: #3206](https://github.com/ESMValGroup/ESMValTool/discussions/3206) with the aim of presenting at the March monthly community meeting | ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) ## Agenda - @ehogan: I have some questions about the release :blush: - @schlunma: We at DLR consider writing a paper about computational improvements in ESMValTool (dask and co.) - @schlunma: Allows specifying DRS per rootpath in configuration file (https://github.com/ESMValGroup/ESMValCore/pull/1894) - @schlunma: Do we really need CMOR warnings? (https://github.com/ESMValGroup/ESMValCore/issues/2351)@schlunma: squash CMOR warnings - @ehogan: The presentation I gave at RSECon23 is now available on [YouTube](https://www.youtube.com/watch?v=UVgvQXgWSRQ) :tada: ## Topics discussed - @ehogan: release questions: one issue with the steps taken during the release - @bouweandela proposes a meeting to discuss the procedures (Bouwe is the replacement past release manager, so he's kind enough to help); Thursday 14th March 10am UK - @flicj191 Felicity and Rhaegar Zeng volunteered to look around; - @schlunma: paper on performance improvelemts with Dask, @bouweandela and @valeriupredoi are go for it - @schlunma: specify DRS per rootpath: previous agreement at the June workshop was to include this in a larger-scale refactoring of the configuration - is this realistically feasable/possible, and when, and if not, we should merge https://github.com/ESMValGroup/ESMValCore/pull/1894; @bouweandela will try write up a synposis/set of specs for configuration overhaul - implementation looks distant, so we should merge the PR - @schlunma: squash CMOR warnings - they produce huge warn messages output: @bouweandela mentions that they can be useful for data providers to check their data quality - piping it through to a dedicated file for CMOR warnings, @schlunma was nice to volunteer to implement the solution; - @ehogan: The presentation I gave at RSECon23 is now available on [YouTube](https://www.youtube.com/watch?v=UVgvQXgWSRQ) :tada: - Emma will open a PR to the Community GH repo - @bouweandela re-raising the topic about shared email address for Zenodo: [Gmail with delegated access looks like an option](https://support.google.com/mail/answer/138350?hl=en#zippy=%2Cadd-a-delegate) - @bouweandela is an absolute legend and he's taking over this from despicable V - @valeriupredoi will contact Codacy to ask for support for Ruff --- ## Topics discussed # 13 February 2024 ## Participants - Attendees: @headmetal @rbeucher @flicj191 @bouweandela @ehogan @valeriupredoi - Absentees: @schlunma - Chair: @valeriupredoi - Minutes taker: @valeriupredoi / @bouweandela ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| | 2024.01 | @bouweandela | Inviting new members to the tech lead team (ACCESS-NRI and SMHI) | ACCESS-NRI now here! :) SMHI does not have time at the moment unfortunately. |2023.07|@ehogan|KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news| |2023.07|@bouweandela|Asking the community to check the output once for all| Waiting for KGO recording solution to be ready. @ehogan: see [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for some additional thoughts on this| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| | 2023.11 | @ehogan | Create a security policy PR | See [ESMValTool: issue: #3455](https://github.com/ESMValGroup/ESMValTool/issues/3455) and [ESMValTool: PR: #3456](https://github.com/ESMValGroup/ESMValTool/pull/3456) | | 2023.11 | @ehogan | Present summary of formatting / linting tools at an upcoming monthly community meeting | Not yet started, but will continue to document on [ESMValTool: discussion: #3206](https://github.com/ESMValGroup/ESMValTool/discussions/3206) with the aim of presenting at the March monthly community meeting | ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) ## Agenda - Review of Pending Action Items - TLT meeting scheduling - @valeriupredoi - Next release - OBS data versioning [ESMValGroup/ESMValTool#3452](https://github.com/ESMValGroup/ESMValTool/issues/3452) @bouweandela - @ehogan: brief CMEW update - Managing connections and dealing with knowledge gaps when personnel move @bouweandela ## Topics discussed - @valeriupredoi: TLT meeting scheduling - Next release: likely in April/May by @ehogan and team - @rbeucher is asking what we do for a release: @bouweandela giving a brief overview of the workload needed - TLT members have left/moved, so we are short on people - @ehogan - MetO still working on the RTW (recipe test workflow) - it works on a small scale (prototype) - recipes break since dependencies change but the maintainers don't run them periodically - @rbeucher - could run different recipes on different machines to speed the process up: comparison vs previous run could still be made? - OBS data versioning [ESMValGroup/ESMValTool#3452](https://github.com/ESMValGroup/ESMValTool/issues/3452) @bouweandela - to be looked at by @valeriupredoi as well - CMEW update (V will write a short paragraph offline) - Managing connections and dealing with knowledge gaps when personnel move @bouweandela: no single point of failure - we need redundencies in place ie people that know that "special stuff" too: - @rbeucher is asking about the Consortium ie at least 1 FTE per institute: @bouweandela responds: variablity in FTE/institute depending on available funding; MetO about the same (@ehogan and Alistair, mainly) - @bouweandela recommends that we work together on things whereever this is possible within the constraints posed by funding sources - @bouweandela recommends ensuring that we write down critical information - @ehogan proposes we create a table of expertises among ourselves, so we discover gaps, and identify the overlaps; @bouweandela also proposes teams' description - useful for identifying knowledge bases -> @bouweandela proposes contact at the Steering Committee as well. - @rbeucher is asking about PR reviewes - can they help? Yes, please, cheers! Contributions most welcome, and documentation in the PR review process (@bouweandela is giving a brief overview of the PR merging process). - @rbeucher bunch of Australia-specific (ocean mainly) recipes (not ESMValTool-ed, but lots interested in porting them to ESMValTool) -> great to have them, @bouweandela poroposes bringing own specialist reviewers, faster and more accurate review. --- # 16 January 2024 ## Participants - Attendees: @bouweandela @ehogan @zklaus @schlunma @valeriupredoi - Absentees: @sloosvel - Chair: - Minutes taker: ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| | 2024.01 | @bouweandela | Inviting new members to the tech lead team (ACCESS-NRI and SMHI) | |2023.07|@ehogan|KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news| |2023.07|@bouweandela|Asking the community to check the output once for all| Waiting for KGO recording solution to be ready. @ehogan: see [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for some additional thoughts on this| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| | 2023.11 | @ehogan | Create a security policy PR | See [ESMValTool: issue: #3455](https://github.com/ESMValGroup/ESMValTool/issues/3455) and [ESMValTool: PR: #3456](https://github.com/ESMValGroup/ESMValTool/pull/3456) | | 2023.11 | @ehogan | Present summary of formatting / linting tools at an upcoming monthly community meeting | Not yet started, but will continue to document on [ESMValTool: discussion: #3206](https://github.com/ESMValGroup/ESMValTool/discussions/3206) with the aim of presenting at the January or February monthly community meeting | ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - Update about CMEW @ehogan - OBS data versioning [ESMValGroup/ESMValTool#3452](https://github.com/ESMValGroup/ESMValTool/issues/3452) @bouweandela - Managing connections and dealing with knowledge gaps when personnel move @bouweandela ## Agenda - ... ## Topics discussed - OBS data at SPIRIT (ESM2025) @schlunma - @zklaus provided path and location at `/modfs/modtools/esmvaltool/obs` with `/home/kzimmer` - @bouweandela: @zklaus' replacement (tough term!) in the Tech Lead Team: Klaus would nominate Joakim Loew; for Steering Committee/Group to be discussed (possibly Ulrika) - Inviting new members to the tech lead team (ACCESS-NRI and SMHI) @bouweandela - Release manager for and time of v2.11 release @bouweandela - @ehogan may be able to do it in March, but may be more certain towards the end of January; maybe worth asking at the community meeting; - Planning next releases @bouweandela - best to start with asking the ACCESS-NRI folks, as suggested by the Steering Group, to join the Tech Lead Team (as mentioned above), then we have a large enough TLT where we find release managers from medium to (longer term); - New meeting to catch up with the iris developers @bouweandela - related to the optimization work done by @bouweandela and his colleague: it appears that iris is starting to be the performance bottleneck; Bouwe will start the motion with an email soon(ish); - To be looked at within the next few days/week - Review of some PRs (descending priority) @schlunma 1. https://github.com/ESMValGroup/ESMValCore/pull/2231 @bouweandela will have a look 2. https://github.com/ESMValGroup/ESMValCore/pull/2183 @valeriupredoi will have a look 3. https://github.com/ESMValGroup/ESMValCore/pull/2279 @valeriupredoi will have a look 4. https://github.com/ESMValGroup/ESMValCore/pull/2189 @valeriupredoi will have a look 5. https://github.com/ESMValGroup/ESMValCore/pull/2270 @ehogan will have a look --- # 5 December 2023 (usual scheduled meeting) ## Participants - Attendees: @bouweandela @ehogan @schlunma @valeriupredoi - Absentees: @zklaus @sloosvel - Chair: @valeriupredoi - Minutes taker: @ehogan and @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@ehogan|KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news| |2023.07|@bouweandela|Asking the community to check the output once for all| Waiting for KGO recording solution to be ready. @ehogan: see [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for some additional thoughts on this| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| | 2023.11 | @ehogan | Create a security policy PR | See [ESMValTool: issue: #3455](https://github.com/ESMValGroup/ESMValTool/issues/3455) and [ESMValTool: PR: #3456](https://github.com/ESMValGroup/ESMValTool/pull/3456) | | 2023.11 | @ehogan | Present summary of formatting / linting tools at an upcoming monthly community meeting | Not yet started, but will continue to document on [ESMValTool: discussion: #3206](https://github.com/ESMValGroup/ESMValTool/discussions/3206) with the aim of presenting at the January or February monthly community meeting | | 2023.11 | @zklaus | v2.10.0 release | | ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - @ehogan: Release manager for February - @ehogan: Is anyone interested in attending the Climate Model Evaluation Workflow (CMEW) demos in January? ## Agenda - none (short meeting) ## Topics discussed - Bouwe and V are interested in attending the CMEW demos :) - The next TLT meeting will be on Tuesday, Jaunary 16, 2024 :) - @schlunma drew attention to https://github.com/ESMValGroup/ESMValTool/issues/3452 --- # 7 November 2023 (usual schedule meeting) ## Participants - Attendees: @bouweandela @ehogan @zklaus @valeriupredoi - Absentees: @schlunma @sloosvel - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@ehogan|KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news| |2023.07|@bouweandela, @remi-kazeroni|Asking the community to check the output once for all| Waiting for KGO recording solution to be ready. @ehogan: see [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for some additional thoughts on this| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| ## Proposed topics for discussion [UFMM] == undiscussed for many (previous) meetings - Agenda section at the start of the meeting) - [UFMM - discussed below] @ehogan: would it be possible to make a decision on whether we should use `black`, please? See [ESMValTool: discussion: #2161](https://github.com/ESMValGroup/ESMValTool/discussions/2161) for more details. I would be very happy to help with this, given the benefits! :blush: - [UFMM - discussed below] @ehogan: I would also really like to discuss the use of pre-commit.ci, and I realise I am not here for this meeting, but I wanted to add it to the list so I don't forget about it / maybe we can discuss it in the October meeting? See [ESMValTool: discussion: #3206](https://github.com/ESMValGroup/ESMValTool/discussions/3206) for more details. - [UFMM - discussed below] @ehogan: Should we add a "Security policy" to ESMValTool? See https://github.com/ESMValGroup/ESMValTool/community for more details. - [UFMM - discussed below] @ehogan: I have added this to the September agenda as I wanted to bring it to the attention of the TLT before October (happy to discuss further on Slack when I'm back from RSECon23) :blush: Would it be possible to discuss supporting those at the MO looking to develop ESMValTool to support more CORDEX data? I am happy to help, but I'm not that familiar with CORDEX / regional data. Here is some information from Katy Richardson, a Senior Scientist at the MO: - Met Office has a small amount of funding to support capability development for loading and processing CORDEX datasets in ESMValTool, with an initial focus on the CORDEX Africa model simulations - We are aware of numerous issues encountered with loading RCM data, mainly linked with specific coordinate systems, projections and metadata, and associated issues with e.g. area averaging calculations. There are varying issues across the different RCM models used in the CORDEX project - some more complex than others. - We are looking for support in prioritising a solution to these issues and incorporating a solution into an ESMValTool release to support future RCM and CORDEX analysis with ESMValTool. - The funding is from September 2023 - March 2024. Current plans are for this strand of the funded work to run between October 2023 and January 2024. - [still ongoing - discussed below] @all @zklaus: how are the releases going? - [still ongoing] @bouweandela: Who will maintain the ESMValBot and ESMValTool VM now that Remi is gone? - we should talk about having a talk - [still ongoing] @bouweandela: Plan future releases - we decided to talk about possible RMs at the wider ESMValTool call, just a reminder - [still ongoing, updated] @valeriupredoi: non-contiguous bounds when crossing 360deg https://github.com/ESMValGroup/ESMValCore/issues/2229 - it'd be good to have some tests done with Rebecca's PR, as suggested by @schlunma - Versions of observational data (``obs4MIPS``, discussed below) ## Meeting notes - @ehogan: use of ``black``, ``ruff``, ``precommit.ci``: - @zklaus: ``ruff`` has a standalone library of linters, but ``prospector`` uses existing standards; K prefers ``black`` to ``yapf``; there could be risks due to different standards implementations - @ehogan: better to finish up a draft proposal, that includes pros/cons - @bouweandela: ``precommit.ci`` has a cmd-line interface/outlook; Codacy is more user-friendly; the issues with different versions are the different pins in different envs (.pre-commit-config.yaml, environment.yml, setup.py, Codacy); pre-commit only allows the analysis of complete files (instead of just the changed lines), resulting in many issues getting raised that were present before, this may cause people to feel unjustly blamed for those issues if they are raised on their pull requests. - @bouweandela: we may need to move to ruff at some point, e.g. pycodestyle, one of the linters we use through prospector, is already deprecated because all of its functionality is available in ruff. I would prefer to wait until Codacy supports ruff though. - **fair consensus**: change from ``yapf`` to ``black`` yes but should be discussed with community, but not so much Codacy to ``precommit.ci``; bring it up to monthly meetings; - @ehogan will work on a white paper-like distilled analysis of these tools; - @ehogan: security policy: we should add at least a line, @bouweandela brought up other packages SP: if you find anything send an email to Tech Lead Team etc etc - @ehogan: CORDEX@MetOffice work going on, but it appears the issues are mostly ``iris``-related; @zklaus: don't look at it mostly as an in-house issue, share away with people that are interested; funding there is! - @zklaus: 2.10 releases: Core RC is out (yay!); Tool is next, outstanding issue - ``obs4MIPS`` changed standards, we have not picked up their latest version; Romain did update the version, but recipes don't work: - @bouweandela: code need version information from the directory structure - **question at hand**: shall we update the ``obs4MIPs`` version for the release or after? - there are small and big issues; @bouweandela prefers Romain finishes the PR fully; **consesnus**: we wait for after the release! - another issue is the coexistence of ``frequency`` and ``time_frequency`` facets for the obs4MIPs project on ESGF, one is wrong, but hard to get to fix it upstream (at ESGF!); also the ``institute`` for CCCma [ESMValGroup/ESMValCore#1937](https://github.com/ESMValGroup/ESMValCore/pull/1937) and various others; - **TODO for 2.10**: missing bits to be readded: https://github.com/ESMValGroup/ESMValTool/issues/3405 - @valeriupredoi: Corectness conference at NCAR, presentation to be written by V with much support from @bouwenadela (many thanks!) - @valeriupredoi: Python 3.12: almost there, we need to raise the flake8 issue at https://github.com/PyCQA/flake8/issues - @ehogan volunteered to open that issue (many thanks!) --- # 24 October 2023 (decided by participants, not scheduled by default) ## Participants - Attendees: @bouweandela @ehogan @zklaus @valeriupredoi - Absentees: @schlunma @sloosvel - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@ehogan|KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news| |2023.07|@bouweandela, @remi-kazeroni|Asking the community to check the output once for all| @ehogan: see [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for some additional thoughts on this| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| ## Proposed topics for discussion - Agenda section at the start of the meeting) - @ehogan: would it be possible to make a decision on whether we should use `black`, please? See [ESMValTool: discussion: #2161](https://github.com/ESMValGroup/ESMValTool/discussions/2161) for more details. I would be very happy to help with this, given the benefits! :blush: - @ehogan: I would also really like to discuss the use of pre-commit.ci, and I realise I am not here for this meeting, but I wanted to add it to the list so I don't forget about it / maybe we can discuss it in the October meeting? See [ESMValTool: discussion: #3206](https://github.com/ESMValGroup/ESMValTool/discussions/3206) for more details. - @ehogan: Should we add a "Security policy" to ESMValTool? See https://github.com/ESMValGroup/ESMValTool/community for more details. - @ehogan: I have added this to the September agenda as I wanted to bring it to the attention of the TLT before October (happy to discuss further on Slack when I'm back from RSECon23) :blush: Would it be possible to discuss supporting those at the MO looking to develop ESMValTool to support more CORDEX data? I am happy to help, but I'm not that familiar with CORDEX / regional data. Here is some information from Katy Richardson, a Senior Scientist at the MO: - Met Office has a small amount of funding to support capability development for loading and processing CORDEX datasets in ESMValTool, with an initial focus on the CORDEX Africa model simulations - We are aware of numerous issues encountered with loading RCM data, mainly linked with specific coordinate systems, projections and metadata, and associated issues with e.g. area averaging calculations. There are varying issues across the different RCM models used in the CORDEX project - some more complex than others. - We are looking for support in prioritising a solution to these issues and incorporating a solution into an ESMValTool release to support future RCM and CORDEX analysis with ESMValTool. - The funding is from September 2023 - March 2024. Current plans are for this strand of the funded work to run between October 2023 and January 2024. - @all @zklaus: how are the releases going? - @bouweandela: Who will maintain the ESMValBot and ESMValTool VM now that Remi is gone? - @valeriupredoi: [Mambaforge/Miniforge/Crapiforge](https://github.com/ESMValGroup/ESMValCore/issues/2138#issuecomment-1755414763) - @bouweandela: Plan future releases - @valeriupredoi: non-contiguous bounds when crossing 360deg https://github.com/ESMValGroup/ESMValCore/issues/2229 ## Agenda - @zklaus: release situation for 2.10 - things are a bit more difficult this time round (new names for obs4MIPS, changes to recipe handling) - @bouweandela fixed a couple issues; further help is needed - Some files are not found in the ``download`` directory if the ESGF node where the data is from is down - @bouweandela (and us) agrees we should get the ESMValCore relase candidate out anyway - we'll test all recipes when CEDA comes bac online - PRs to change obs4MIPS name changes (by Romain) - @bouweandela mentions we should merge them to get the data finding to be using modern names - a couple bottom lines: we'll test everything that don't need CEDA, and we'll get Core RC out - @bouweandela: ESMValBot and the VM (former: Remi): we need one maintainer for these: V @valeriupredoi volunteers, Bouwe will meet V at some point talk about it; - Mambafore: we'll stick with it for now, until major problems appaear (if any) - future releases: February and June 2024, but we need to find release managers - let's talk at the wider ESMValTool call! - next call on November 7th (first Tuesday of the month), as scheduled (auto-invite should be on) --- # 10 October 2023 (TLT extraordinaire - ESMValCore=2.10 release) ## Participants - Attendees: @bouweandela @ehogan @zklaus @schlunma @valeriupredoi - Absentees: @sloosvel - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@ehogan|KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news| |2023.07|@bouweandela, @remi-kazeroni|Asking the community to check the output once for all| @ehogan: see [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for some additional thoughts on this| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| |2023.09|@schlunma|Talk about ERA5|pending| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - @zklaus: Do we need an `orphan` label for PRs/Issues that have been abandoned by their author? - @bouweandela: Who is looking after CMORizer pull requests now that Remi is gone? - @bouweandela: Who will maintain the ESMValBot and ESMValTool VM now that Remi is gone? - @valeriupredoi: [Mambaforge/Miniforge/Crapiforge](https://github.com/ESMValGroup/ESMValCore/issues/2138#issuecomment-1755414763) - @bouweandela: Plan future releases We didn't get around to discussing @ehogan's items during this meeting, so they have been moved to the next meeting :blush: ## Topics Discussed - @zklaus: release is on track, just one last PR to be merged (#2191) that is ready too to be merged (reviewed and ready) - @zklaus: `orphan` label is indeed needed -> @vpredoi will put it in the labels list right now - @schlunma - ERA5/Grib - PBs of ERA5 data provided by DKRZ, in GRIB file format; - use via`iris-grib` is the package needed to read GRIB files; finally the environment now works with `iris-grib`, only CircleCI tests are failing (local fine) - @bouweandela mentions the Circle Test is done in a pre-built container that may not have the dependency installed; - grid - ERA5 is on a reduced Gaussian grid, @schlunma wrote a bespoke regridder (lazy and all) that would be good to be in Scitools-Iris (at a later stage, after they have overhauled their regridders) - for now include it in a private function in ESMValCore (before inclusion in Iris)? Opinions below: - @zklaus: this is a useful functionality for other datasets, so probably not private - @schlunma: probably not a good idea to put in the ESMValCore mechanism (hard to get it out when time comes) - debating ensues...will add the conclusion - Decided: we have it as a public/tested regrid scheme - @zklaus Q: online download from CDS is the same as data from DKRZ? @schlunma A: yes, but there are more data available on DKRZ; - @ehogan KGO solution: KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news -> please have a read and see if you can contribute! -> in terms of storage for KGOs, Zenodo may be a good solution (possible decided solution). Emma will test the few recipes in RTW with Zenodo first, and then we'll see. --- # 3 October 2023 ## Participants - Attendees: @bouweandela @zklaus @ehogan - Absentees: @valeriupredoi @sloosvel @schlunma - Chair: @zklaus - Minutes taker: @zklaus ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@ehogan|KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news| |2023.07|@bouweandela, @remi-kazeroni|Asking the community to check the output once for all| @ehogan: see [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for some additional thoughts on this| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| |2023.09|@bouweandela|Application for DockerHub Open Source account (accepted)|done| |2023.09|@schlunma|Talk about ERA5|pending| ## Agenda - Review of Pending Action Items - Upcoming release - libwebp vulnerability We didn't get around to discussing @ehogan's items during this meeting, so they have been moved to the next meeting :blush: ## Topics discussed ### Review of Action Items #### KGO Recording Emma reports on her research. More details in the linked discussion. Commercial solutions like AWS S3 or Git LFS at Github.com seem prohibitively expensive. Possible candidate tools: * https://www.fatiando.org/pooch/latest/ Storage provider candidates: * Zenodo * b2share #### Community review Will be done when storage solution is in place. #### Shared email account no update #### Docker community account Approved by Docker #### Pypi organization No response yet #### Talk about GRIB and ERA5 Will be moved to proposed agenda for next meeting. ### Upcoming release The feature freeze for the Core should have happened yesterday. Will be announced now. Bouwe as previous release manager and Klaus as release manager will schedule a virtual meeting to kickstart the process. We will push the feature freeze forward by one week. Template for the discussion at https://github.com/ESMValGroup/ESMValCore/discussions/2071 ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) --- # 19 September 2023 ## Participants - Attendees: @bouweandela (a bit late) @bsolino @zklaus @sloosvel @valeriupredoi - Absentees: @schlunma @ehogan - Chair: @valeriupredoi - Minutes taker: @valeriupredoi ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@ehogan|KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news| |2023.07|@bouweandela, @remi-kazeroni|Asking the community to check the output once for all| @ehogan: see [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for some additional thoughts on this| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| |2023.09|@bouweandela|Application for PyPI premium account (made)|pending| |2023.09|@schlunma|Talk about our Lord Savior GRIB|pending| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - @zklaus: authentication on PyPI; PyPI now supports organizations - (Question) @valeriupredoi: situation on Docker (wrt pay plan/open organization status renewal) - @bouweandela: Upcoming release? - (Announcement) @valeriupredoi: I can't make the October 3rd TLT (UKESM Retreat) ## Agenda - @ehogan is not here so we'll re-add her points above to the next call Proposed topics (and delete them from here); - @zklaus: authentication on PyPI; PyPI now supports organizations - (Question) @valeriupredoi: situation on Docker (wrt pay plan/open organization status renewal) - @bouweandela: Upcoming release? - (Announcement) @valeriupredoi: I can't make the October 3rd TLT (UKESM Retreat) - Upcoming release - GRIB support ## Topics discussed - @zklaus: authentication on PyPI; two separate items authntication and ownership (individuals and, now, organizations): for organizations we can create an "ESMValGroup" organization that has ownership; action item: go for organization and who wants to be added? @sloosvel: how individual credentials are stored? @zklaus: have an account in PyPI that gets added to the organization - we are all in favour to create an organization! @zklaus is already creating one as we speak - organization creation submitted for review on 19 September 2023! Authentication now: open an issue and discuss there https://github.com/ESMValGroup/ESMValCore/issues/2195 - Docker situation: we got kicked out of the premium plan (this is TBD at next meeting) - @bouweandela reapplied for the free plan, he recommends we wait and see. - Upcoming release: we should review @schlunma's contributions plus @bsolino's new HTML contributions; - grib support: ERA5 weirdness: DKRZ downloaded data is wonky - the cause could be the netCDF-conversion server by ECMWF (which is experimental, as @bouweandela mentions) - this brings in the question if/how much we need to support ``grib`` and its special regridding need, which @schlunma has implemented IO/regridding for in https://github.com/ESMValGroup/ESMValCore/pull/2178 - we should talk to Manu! --- # 5 September 2023 # This call has been postponed for 19 September 2023 ## Participants - Attendees: @bouweandela @bsolino @zklaus @schlunma @remi-kazeroni @sloosvel @valeriupredoi - Absentees: @ehogan (I will be at RSECon23 :tada:) - Chair: - Minutes taker: ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@ehogan|KGO recording solutions|See [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for the latest news| |2023.07|@bouweandela, @remi-kazeroni|Asking the community to check the output once for all| @ehogan: see [ESMValTool: discussion: #3247](https://github.com/ESMValGroup/ESMValTool/discussions/3247) for some additional thoughts on this| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| ## Proposed topics for discussion (to be ordered then added to the Agenda section at the start of the meeting) - Review of Pending Action Items - ... ## Agenda - ... ## Topics discussed - ... --- # 1 August 2023 ## Participants - Attendees: @valeriupredoi @bouweandela @ehogan @schlunma @bsolino - Absentees: @remi-kazeroni @zklaus - Chairs and minutes takers: ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.07|@ehogan|KGO recording solutions|pending| |2023.07|@bouweandela, @remi-kazeroni|Asking the community to check the output once for all|pending| |2023.07|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|pending| ## Agenda - Review of pending Action Items (see updated table above) - @ehogan: RSECon23 presentation :) - @schlunma: Allow fix_file/load to return/read CubeList objects [#2160](https://github.com/ESMValGroup/ESMValCore/pull/2160) - Manuel @schlunma Use smarter (units-aware) weights [#2139](https://github.com/ESMValGroup/ESMValCore/pull/2139) - @schlunma: Run CMOR checks once per datasets [#2157](https://github.com/ESMValGroup/ESMValCore/pull/2157) - Synchronization of main branches between public and private repositories ``main`` ## Proposed topics for discussion - Review of pending Action Items - Synchronization of main branches between public and private repositories ``main`` - Manuel @schlunma Use smarter (units-aware) weights [#2139](https://github.com/ESMValGroup/ESMValCore/pull/2139) - @schlunma: Run CMOR checks once per datasets [#2157](https://github.com/ESMValGroup/ESMValCore/pull/2157) - @schlunma: Allow fix_file/load to return/read CubeList objects [#2160](https://github.com/ESMValGroup/ESMValCore/pull/2160) - @ehogan: RSECon23 presentation :) ## Topics discussed ### @ehogan: RSECon23 presentation :) Emma will share the presentation with the User Engagement Team, but she is wondering if that'd be a good idea to have us look at it too; good luck! ### Allow fix_file/load to return/read CubeList objects Issue [#2160](https://github.com/ESMValGroup/ESMValCore/pull/2160) brought forth by @schlunma: - files that can not be read by ``iris.load_grib``, we need to process ``xarray`` arrays; question is - why can't we use Iris (Bouwe) - it's messy, and there are issues with Iris' GRIB loader; - Emma @ehogan mentions that Iris people are keen on the xarray=iris interoperability, so positive responses may be expected - ``fix_file`` may not be the func to be used, add a secondary/intermediary function? - question is we can pass ``load`` a ``CubeList`` `ie` an abstract object rather than a POSIX path --- # 4 July 2023 ## Pending Action Items | Meeting | Assignees | Item | Status | |-|-|-|-| |2023.06|@zklaus|Changelog generation tools|Migration to Towncrier approved| |2023.06|@ehogan|KGO recording solutions|-| |2023.06|@bouweandela, @remi-kazeroni|Asking the community to check the output once for all|-| |2023.06|@valeriupredoi|Shared gmail account to handle to Zenodo credentials|-| ## Participants - Attendees: @valeriupredoi @sloosvel @bouweandela @remi-kazeroni @bsolino @zklaus - Absentees: @schlunma @ehogan - Chairs and minutes takers: @valeriupredoi and @remi-kazeroni ## Proposed topics for discussion - Review of pending Action Items - Status of the release v2.9 (@bouweandela, @remi-kazeroni, see https://github.com/ESMValGroup/ESMValTool/discussions/3198 and https://github.com/ESMValGroup/ESMValCore/discussions/2071) - Changing the Tier of an OBS dataset: do we first need to make tier key optional in recipes? (@remi-kazeroni, see https://github.com/ESMValGroup/ESMValCore/issues/2112) - Improving the concatenation of cubes (@sloosvel, see https://github.com/ESMValGroup/ESMValCore/issues/2073) ## Agenda 1. Status of the release v2.9 2. Changelog generation tools 3. [Improving the concatenation of cubes](https://github.com/ESMValGroup/ESMValCore/issues/2073) 4. [Changing the Tier of an OBS dataset](https://github.com/ESMValGroup/ESMValCore/issues/2112) 5. AOB ## Topics discussed ### Status of 2.9.0 release (@bouweandela) ESMValCore=2.9.0 is now released (on PyPI and conda-forge), currently runs on Levante going with new ESMValCore. Plan is to have ESMValTool released on Thursday 6 July 2023, including release on conda-forge. Caveat emptor: we still have to understand the performance issues with the new Dask/iris combo. (see https://github.com/ESMValGroup/ESMValTool/discussions/3198 and https://github.com/ESMValGroup/ESMValCore/discussions/2071). ### Changelog generation tools Numpy, Pytest, Pip all use Towncrier. Dask (and Distributed) uses a git oneliner that produces something similar to what we are trying to replace, see [here](https://github.com/dask/dask/blob/main/docs/release-procedure.md). Scipy seems to be doing it mostly manually, with a PR based list as a starting point for the maintainer. I (@zklaus) suggest to adopt Towncrier. We can adapt a config from Numpy or Pytest, see [here](https://github.com/pytest-dev/pytest/blob/6995257cf470d2143ad1683824962de4071c0eb7/pyproject.toml#L67). @zklaus giving us a more detailed set of instructions: one can put a bunch of rst files in a directory, then the Changelog file gets generated from these. The rst files need a naming convention e.g. `<PULL REQUEST>.<TYPE>.rst` that gets piped to Towncrier. @bsolino asks: is this an extra layer on top of the automated bit? @zklaus says: it's an addition to the oldschool Changelog, some packages do this, as well as maintaining the oldschool Changelog. If one wants to have a look only at the commits, then commit log it. @bouweandela says: looks clean and user-orineted (very helpful for new users wanting to read what's actually been done in the past). @remi-kazeroni asks: when should we start adopting it? @zklaus says: let's first decide to adopt it; if so, let's first do it for ESMValCore, as a testbed. @remi-kazeroni proposes an ad-hoc vote to adopt Towncrier: vote result: unanimity for adoption! Therefore, we are aiming for first use next release (with Klaus as RM; he'll have done us a nice configuration for it). **Thanks a lot, Klaus!** - [ ] **Action Item** @zklaus will investigate further and open a PR. On an unrelated note, the search made me stumble over [Pooch](https://www.fatiando.org/pooch/latest/); maybe worth checking out - for auto-downloads of data (maybe auxilary data as @bouweandela mentions). ### Concatenation process improvements (@sloosvel) There is room for improving the concatenation mechanism, both at ESMValCore, and iris ends. Proposal involving iris: - iris ignores a number of netCDF attributes/coordinates, so we can use this method: check once for a certain attribute/coordinate validity, then ignore once the first check has passed; - @bouweandela responds: a bit clunky to configure this from the recipe layer - could be tied to the ``cmor_level__check``, we probably don't want to add extra user confuguration; - @zklaus mentions using the netCDF layer (even before iris), but adds that the proper long term solution is to improve iris; - @sloosvel: proposing an interim solution to tie to the ``cmor_check_level`` option <-> @zklaus is a bit worried about the users' abuse of this option; - @bouweandela proposes talking to the data providers to output fewer files: - @sloosvel says it could be a flat no, due to the data production setup that's optimized for their (data providers') needs; - @zklaus mentions that there is new data being produced, so the old infrastructure will probably not change; - bottom line: raise this with iris and tell them to haul rears! ### Changing the Tier of an OBS dataset Topic brought in by @remi-kazeroni, discussed in https://github.com/ESMValGroup/ESMValCore/issues/2112; - @zklaus says: it's in the ``Dataset`` facet, manipulation can happen there but that needs to be done in the Core; - @bouweandela: there are risks due to concatenation/merging of data from different Tiers if people store the same data in two different TierX folders. We could think of a new drs with no Tiers. @remi-kazeroni: that would be helpful for users not sharing data (i.e. working on personal computers). Tiers are mostly needed to share data on large clusters (e.g. DKRZ); - @remi-kazeroni says: we should think more carefully about this. Note so easy to make it backward compatible (recipe-wise and OBS data pool-wise). ### AOB - @bouweandela: Workshop announcement: *"Correctness and Reproducibility for Climate and Weather Software"* at NCAR, 9-10 November 2023. Deadline: August 1. @valeriupredoi is willing to help writing an abstract. --- # 29 June 2023, ESMValTool workshop @ SMHI, hybrid ## Participants - Attendees: @bouweandela @zklaus @remi-kazeroni @sloosvel @valeriupredoi @schlunma @ehogan @bsolino - Absentees: None - Chairs and minutes takers: @valeriupredoi and @remi-kazeroni ## Topics discussed 1. [New policy to generate Core Changelog](https://github.com/ESMValGroup/Community/discussions/86#discussioncomment-5381404)? - See workshop minutes. Consensus on starting a new policy, similar to what is done by Iris. - [x] **Action Item** @zklaus will look into tools to generate the changelog and report back next time. 2. [Recording KGO](https://github.com/ESMValGroup/Community/discussions/86#discussioncomment-5474852) - @ehogan: do we need to do that per site? - @zklaus: suggests to start with one set of KGOs - @bouweandela: which platform to use? The VM is not good enough (no backup). @zklaus: git-lfs but we may need to pay - @bouweandela, @ehogan: which types of files to store? Consensus to store both plots and netcdf files - @bouweandela, we could add an option to remove large amounts of netcdf files produced by a handful of recipes. @remi-kazeroni: see issue https://github.com/ESMValGroup/ESMValTool/issues/3183 - @bouweandela: requirements: version controlled, accessible to TLT, backed-up - @sloosvel: shall we consider a thredds server if not too complicated to set-up - Decided at the workshop: make a new issue to record a list of recipes with KGOs and proper explanations what that will be used for. - [ ] **Action Item** @ehogan will look into possible solutions and report at a next TLT. - [ ] **Action Item** @bouweandela, @remi-kazeroni will open an issue to ask people to check the recipe output before these enter the list of KGOs 3. [Gmail-based solution](https://github.com/ESMValGroup/Community/discussions/86#discussioncomment-5381460) - @valeriupredoi checked a possibility that could be used for our Zenodo. @bouweandela: requirement is to be able to send emails from this account - [ ] **Action Item** @valeriupredoi will set up a google group for that. 4. Next meetings - Decision to use HackMD to record the minutes, see [link](https://hackmd.io/@CS80aeLkSQOjqsG5kzUKqg/ByUP4nYQq) - As soon as the meeting is over, the convener will setup a HAckMD document for next time with Action Items and section to suggest discussion topics. - Meetings will be on t**he first Tuesday of the month, 16:00 (European time)**. - [x] **Action item** @valeriupredoi will set up a recurring meeting with a Zoom link.

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.