owned this note
owned this note
Published
Linked with GitHub
# New Xarray meeting notes
https://us02web.zoom.us/j/87503265754?pwd=cEFJMzFqdTFaS3BMdkx4UkNZRk1QZz09
Archive: https://hackmd.io/Vv6g2ABzTPKbe2MWBQqS1w
## Aug 28, 2024
### Attendees
- Justus Magin / @keewis
- Matt Savoie / @flamingbear
- Tom Nicholas /
- Deepak Cherian /
- Daniel Kaufman / @danielfromearth
### 60 second updates
- Justus: sprint on xarray + duckarrays testing framework with Tom (https://github.com/xarray-contrib/xarray-array-testing/)
- Tom:
- duckarrays testing
- using `conventions.decode_cf_variables` without decoding actual values
- https://github.com/zarr-developers/VirtualiZarr/pull/224
- going to NUMFocus summit next week
- Deepak:
- groupby multiple arrays
- providing input to dask things; shuffling, blockwise reshape, auto rechunking, reshaping;
- https://github.com/dask/dask/pull/11350/files
- Matt: Nada
### Agenda
- Anyone else want to come and represent Xarray at the NUMFocus summit next week in Boston?
- Eni?
- decode_cf
- PRs needing review:
- speed up docs build: https://github.com/pydata/xarray/pull/9395
- Shuffling API:
- https://github.com/pydata/xarray/pull/9320
- `Dataset.shuffle_by() -> Dataset`
- `DatasetGroupBy.shuffle() -> DatasetGroupBy`.
## Aug 14, 2024
### Attendees
- Justus Magin / @keewis
- Owen Littlejohns / @owenlittlejohns
- Eni Awowale/ @eni-awowale
- Tom Nicholas / @TomNicholas
- Deepak Cherian / @dcherian
### 60 second updates
- Moving old datatree issues
- Tom
- Working on chunkmanager PR
- Deepak
- groupby shuffle
- https://github.com/pydata/xarray/pull/9320
- engaging with dask
- https://github.com/dask/dask/issues/11314
- https://github.com/dask/dask/pull/11303
- https://github.com/dask/dask/pull/11273
- Justus
- merged the python 3.9 dropping PR
### Agenda
- VirtualZarr ideas?
- Eni's open_groups PR
- https://github.com/pydata/xarray/pull/9243
## July 31, 2024
### Attendees
- Matt Savoie / @flamingbear
- Tom Nicholas / @TomNicholas
- Justus Magin / @keewis
- Owen Littlejohns / @owenlittlejohns
### 60 second updates
- Matt
- is trying to wrap head about copying trees. [#9285](https://github.com/pydata/xarray/issues/9285) should not be as hard as I'm making it.
- Tom
- Trying to coordinate to push datatree over the finish (well first release) line
- Fixes for a bunch of small datatree bugs
- PR for allowing chunked arrays that aren't dask/cubed through xarray
- https://github.com/pydata/xarray/pull/9286
- rename chunkmanagers vs "ComputeManagers"
- Justus:
- released 2024.07.0 yesterday (new script to extract contributors from git commits)
- pint-xarray: accessor entrypoints / PintIndex
### Agenda
- DataTree should avoid any in-case modification
- Auto-copy on setting parent?
- Remove the ability to assign .parent entirely?
- Need to keep .parent accessible in order to walk up through tree
- Who is submitting to AGU today?
- (Tom is, on VirtualiZarr)
- (Ryan is)
- Owen is
- Stephan maybe
- ChunkManager vs ComputeManager https://github.com/pydata/xarray/pull/9286
- Justus tell us about the PintIndex (postponed to next time)
## July 17, 2024
Cancelled -- only Stephan Hoyer and Justus Magin showed up.
## July 3, 2024
### Attendees
- Justus Magin / @keewis
- Tom Nicholas / @TomNicholas
- Matt Savoie / @flamingbear
- Joe Hamman / @jhamman
### 60 second updates
- Tom
- Reviewed datatree coordinate inheritance PR properly (https://github.com/pydata/xarray/pull/9063)
- Now unblocked for releasing datatree in `main`
- Mostly actually wrote code for virtualizarr
- Justus
- lots of fixes for numpy2 (for the dependencies we couldn't test before)
- https://github.com/pydata/xarray/pull/9136 should be ready for merging
- other bug fixes (hypothesis test for datetime ExtensionArrays, arrays as attributes)
- nested duck arrays: finding `cupy` underneath arbitrary layers (especially dask)
- Matt
- Watching [#9063 ](https://github.com/pydata/xarray/pull/9063#) get merged.
- Joe
- Just working on zarr-python
-
### Agenda
- Codecs separate from xarray?
- Keeps coming up in virtualizarr
- https://github.com/zarr-developers/VirtualiZarr/issues/68#issuecomment-2197682388
- Can we get zarr-python to open a netCDF file by using chunk manifests + defining enough new codecs?
- One difference is that zarr codecs take arrays -> arrays but xarray decoding takes Variables -> Variables
- action: open a new issue to consolidate the discussion
- Interesting question of subclassing xarray.Dataset in virtualizarr
- https://github.com/zarr-developers/VirtualiZarr/issues/171
- Cupy + dask
- https://github.com/pydata/xarray/issues/7721 (discussion of the issue)
- https://github.com/keewis/nested-duck-arrays
## June 19, 2024
### Attendees
- Matt Savoie / @flamingbear
- Justus Magin / @keewis
- Tom Nicholas / @TomNicholas
- Stephan Hoyer / @shoyer
- Owen Littlejohns / @owenlittlejohns
- Eni Awowale/ @eni-awowale
- David Auty / @autydp
- From NASA EED-3, knows Matt and Owen
### 60 second updates
- Matt
- Hope to continue datatree inheritance discussion.
- Justus: numpy2-compatible release last week
### Agenda
- DataTree coordinate inheritance question
- Release timeline
- Can we release by the time of Eni and Tom's SciPy talk about DataTree? (~July 10th)
- How much feedback from community do we need?
- Stephan: Got plenty already
- David: Has "quirky" data at NASA
- Would probably prefer more lenient data model
- Stephan: Prefer not to have "fallback mechanisms"
- David: Wants to use datatree for analysis, ideally changing the structure as little as possible
- Tom: What do we think about this `open_as_dict_of_datasets` idea? Would that help?
- Tom: Solves problem of interrogating data / displaying groups
- Stephan: Makes some sense - analogous to how `open_mfdataset` works for 90% of cases
- As if we had made a `open_mf_as_grid_of_datasets` function to create an interrogatable intermediate structure
- Stephan: Function to write a messy dataset too? (lower priority)
- Matt: In favour
- David: Can you open just a subtree of a file? Tom: Yes if we add a group kwarg to `open_datatree`
- Eni: Useful if `open_datatree` failed on alignment it gave very clear report of what should be fixed
- Justus: `preprocess` arg could be useful for "massaging"
- Tom: Could use python's new Exception Groups feature for showing all errors at once
- Stephan: Should also think about saving out a "crooked datatree"
- Consensus?!
- Plan going forward
- Everyone who is interested look in detail at Stephan's PR (https://github.com/pydata/xarray/pull/9063)
- Likely to spawn smaller issues / PRs about reprs and so on
- Need separate PR for `open_as_dict_of_datasets` (or `open_datatree_as_dict`?) (Tom can raise issue for this)
- Orthogonal to Stephan's PR
- Tutorial for tidying up a messy nested netCDF file into a nice sane aligned DataTree (similar to the "Tidy Xarray" idea)
## June 5, 2024
### Attendees
- Justus Magin / @keewis
- Tom Nicholas / @TomNicholas
- Kai Mühlbauer / @kmuehlbauer
- Stephan Hoyer / @shoyer
- Joe Hamman / @jhamman
- Deepak Cherian / @dcherian
- Matt Savoie / @flamingbear
- Mathias Hauser / @mathause
### 60 second updates
- Justus:
- numpy 2: more progress, still not done (lots of edge cases): https://github.com/pydata/xarray/pull/8946
- Joe
- still heads down on zarr 3, alpha release coming this week
- I will be at the CZI annual meeting next week, kicking off our latest grant
- We're hiring (https://jobs.gusto.com/postings/earthmover-xarray-community-developer-498dca94-335e-4d5c-a6c7-83ca19772512)
- Tom:
- Owen, Matt, and Eni accepted our invitation to join the core team!
- Mostly just this discussion about datatree coordinate inheritance behaviour (https://github.com/pydata/xarray/issues/9056)
- Kai:
- a bit of issue clearance
- jumped now on the open_datatree-stuff
- Deepak:
- job posting: https://jobs.gusto.com/postings/earthmover-xarray-community-developer-498dca94-335e-4d5c-a6c7-83ca19772512
- @deepak (https://discourse.pangeo.io/t/potential-for-adapting-pythia-foundations-for-different-disciplines-e-g-neuro/4239/3?u=tomnicholas)
- user survey: https://docs.google.com/forms/u/2/d/1x9bOIelnUsDMyI1tF4bN7TWK0v4nBDiwhpxh9mi6PaI/edit
- last call for comments.
- Stephan:
- DataTree inheritance model: https://github.com/pydata/xarray/pull/9063
### Agenda
- Numpy 2: dtype casting in where
- separate meeting to discuss in detail
- Miscellaneous PRs
- https://github.com/pydata/xarray/pull/5704
## May 22nd, 2024
### Attendees
- Matt / @flamingbear
- Justus / @keewis
- Tom Nicholas / @TomNicholas
- Mathias Hauser / @mathause
### 60 second updates
- Matt
- no update
- Justus
- numpy 2: array api fixes (ready for a final review!) https://github.com/pydata/xarray/pull/8854
- numpy 2: where dtype casting. Stephan helped me figure out a clean way to implement this, but didn't have time to do this, yet.
- Tom
- No real updates on xarray itself
- Q about ChunkManager https://github.com/pydata/xarray/issues/8733#issuecomment-2111146588
- Joe
- Still cranking on zarr v3
- Deepak is heads down this week
- Mathias
- no update
### Agenda
## May 8th, 2024
### Attendees
- Deepak
- Justus
- Matt
- Tom
- Mathias
- Ryan
### 60 second updates
- Deepak
- optimizing zarr region writes / appends
- iterating on Xarray User Survey
- Justus
- more numpy 2 compat... we're now down to failures with just the array api and the casting changes due to NEP 50
- Tom
- Trying to start some deprecation cycles
- PR to concat without creating indexes
- Ryan
- Trying to nerd-swipe someone into making some useful indexes
- https://github.com/pydata/xarray/discussions/8955#discussioncomment-9226372
- https://discourse.pangeo.io/t/example-which-highlights-the-limitations-of-netcdf-style-coordinates-for-large-geospatial-rasters/4140/26
- Tom: even simpler case - no indexes!
- PR for concat is all that is needed
- Immediate next case is pandas index that is disconnected from variable data
- https://github.com/TomNicholas/VirtualiZarr/issues/18#issuecomment-2025423042
- Mathias
- no news
- Matt
- Also no update
### Agenda
- NamedArray update
- Stalled, Anderson has run out of time
- 80% of the way there for decoupling the backends from lazy indexing.
- Action item: make a todo list for whats left and needed.
- PRs to review/merge
- Tom: Please someone merge this indexes PR https://github.com/pydata/xarray/pull/8872
- I can't release v0.1 of VirtualiZarr until it's in xarray main...
- Deepak : https://github.com/pydata/xarray/pull/8998
- Numpy 2 compat:
- should we switch casting behavior to NEP 50? https://github.com/pydata/xarray/pull/8946
- https://numpy.org/neps/nep-0050-scalar-promotion.html
- array api is close: https://github.com/pydata/xarray/pull/8854
- Release plan:
- release before that doesn't fully support numpy 2 yet
- release one version with numpy 2 and py3.9
- then drop python 3.9
## April 24th, 2024
### Attendees
- Justus Magin / @keewis
- Matt Savoie / @flamingbear
- Kai Mühlbauer / @kmuehlbauer
- Tom Nicholas / @TomNicholas
- Joe Hamman / @jhamman
- Owen Littlejohns / @owenlittlejohns
- Deepak Cherian / @dcherian
- Stephan
### 60 second updates
- Justus: upstream-dev CI / numpy 2 compat
- Tom:
- On a train, probably can't call in
- Looking at changing backends.NetCDFDataStore to only open file once when reading many groups
- Kai are you or others planning to take this on?
- I want to change internal invariants to stop checking for default pandas indexes
- https://github.com/pydata/xarray/pull/8960#discussion_r1573306634
- @deepak what option are you referring to? I don't see a kwarg to `assert_equal`... https://github.com/pydata/xarray/blob/b0036749542145794244dee4c4869f3750ff2dee/xarray/testing/assertions.py#L88-L120
- https://github.com/pydata/xarray/blob/b0036749542145794244dee4c4869f3750ff2dee/xarray/testing/assertions.py#L385-L387
- +We should plumb it through.+ Wrong: look here: https://github.com/pydata/xarray/blob/b0036749542145794244dee4c4869f3750ff2dee/xarray/tests/__init__.py#L286
- (we don't use the public API directly in tests)
- COol.
- This might be a good time to write an "Assertions" section into the [docs page on testing](https://docs.xarray.dev/en/stable/user-guide/testing.html#testing-your-code)
- Kai: not much, considering helping with datatree backend stuff together with @aladinor and @mgrover1, need to check which way to go (from the xarray side, or from the external backend side)
- Matt: also not much recently. Always datatree. listening.
- Owen: Continued datatree migration, current PR open: https://github.com/pydata/xarray/pull/8967
- Joe:
- Foo proposal was funded! Deepak and I are hoping to hire a near-full time dev with bio experience to come work with us
- Going to try zarr-python-3 in xarray next week.
- Deepak - not much, pushed on public grouper api
### Agenda
- Break behaviour of dataset constructor?
- https://github.com/pydata/xarray/issues/8959
- `ds = xr.Dataset(data_vars={'x': ('x', [0])})`
- promotes to coordinate
- Start with `PendingDeprecationWarning`
- Add a separate more explicit construction method/kwarg? Or use the new behavior in case a `Coordinates` object is passed
- numpy 2:
- array api: https://github.com/pydata/xarray/pull/8854
- main concerns: dispatching between numpy issubdtype and arrayapi isdtype <- this is kinda hairy
- stephan will take a look
- copy parameter to `__array__` (typing, mostly): https://github.com/pydata/xarray/pull/8939/files
- dtype casting rules: https://github.com/pydata/xarray/pull/8946#issuecomment-2068949796
- general rule: determine dtype without python scalars (which are "weak dtypes" in jax), then cast python scalars to array using that dtype. If that doesn't work, either raise or determine a fallback
- implementation `as_shared_dtype`
- `concatenate` and `stack` shouldn't really allow python scalars (?)
- may be specific to `where`, in which case that code could also go there
## April 10, 2024
### Attendees
- Matt Savoie / @flamingbear
- Kai Mühlbauer / @kmuehlbauer
- Tom Nicholas / @TomNicholas
- Justus Magin / @keewis
- Deepak Cherian / @dcherian
- Owen Littlejohns / @owenlittlejohns
### 60 second updates
- Matt : Good meeting for Datatree yesterday. [PR](https://github.com/flamingbear/xarray/pull/11) to [existing PR](https://github.com/pydata/xarray/pull/8879) for simplifying iterators is ready. Owen will ping Tom later today when he merges.
- Justus:
- "source" encoding from `fsspec` objects
- h5netcdf + character sets
- Tom
- Mostly thinking about the virtualizarr stuff (i.e. not propagating xarray indexes and dealing with encoding)
- Chance of me being able to think about datatree inheritance has gone up since NCAR machines are all down...
- Kai: not much xarray related (beside some h5netcdf char encoding ;-)
- Owen: [Open PR for iterators.py](https://github.com/pydata/xarray/pull/8879) - Will update based on recent feedback.
- Deepak :
- merged in stateful tests (https://github.com/pydata/xarray/pull/8658)
- Explanation of hypothesis testing strategies https://docs.xarray.dev/en/stable/user-guide/testing.html#hypothesis-testing
### Agenda
- upstream tests:
- https://github.com/pydata/xarray/issues/8844
- string dtypes (needs volunteer)
- array API tests
- https://github.com/pydata/xarray/pull/8854
```
xarray/tests/test_duck_array_ops.py::TestOps::test_where_type_promotion: AssertionError: assert dtype('float64') == <class 'numpy.float32'>
+ where dtype('float64') = array([ 1., nan]).dtype
+ and <class 'numpy.float32'> = np.float32
xarray/tests/test_duck_array_ops.py::TestDaskOps::test_where_type_promotion: AssertionError: assert dtype('float64') == <class 'numpy.float32'>
+ where dtype('float64') = array([ 1., nan]).dtype
+ and <class 'numpy.float32'> = np.float32
```
- encoding and virtualizarr
- https://github.com/TomNicholas/VirtualiZarr/issues/68
- https://github.com/fsspec/kerchunk/blob/a0c4f3b828d37f6d07995925b324595af68c4a19/docs/source/tutorial.rst
## March 27, 2024
### Attendees
- Deepak Cherian
- Alex Ford / @asford
- Tom Nicholas / @TomNicholas
- Matt Savoie / @flamingbear
- Stephan Hoyer
### 60 second updates
- Deepak : upstream-dev fixes
- Tom:
- Datatree meetings
- Xarray without indexes
- With a few un-merged PRs I can actually xr.concat Datasets without indexes at all
- See [example in VirtualiZarr](https://github.com/TomNicholas/VirtualiZarr/blob/main/docs/usage.md#manual-concatenation-ordering)
- Alex F
- First time attending.
- Question on possible wrapping of torch-tensors in xarray
- We have working internal fork, interested in upstreaming
- Matt
- good meeting yesterday with agreement to move forward faster not smarter. Basically move most code without improvements and identify places we want to work later.
- Stephan
- Benoit might be working on indexes again in a couple of months, funding from NASA grant at UW.
### Agenda
- torch inside xarray
- Relevant issues
- https://github.com/pydata/xarray/issues/3232
- https://github.com/data-apis/array-api-compat
- as a comparison point: [JAX-Xarray](https://github.com/google-deepmind/graphcast/blob/main/graphcast/xarray_jax.py)
- https://github.com/pytorch/pytorch/issues/58743
- Why xarray?
- "problem of dimension tracking", sequence information, hypercubes, align in to canonical coordinate frame, write gradient aware calculations inside that coordinate system,
- like named dims, tried NamedTensor, switched to Xarray, have many coordinate variables,
- Pain points?
- pytorch isn't compliant with array API standard
- can be mostly solved using the array-api-compat shim library
- non-numpy dtypes
- Not really covered in the array API standard
- Might need special-casing within xarray