NumPy2 lighning talk: Refactor np.MaskedArray to use __array_function__ instead of inheriting from np.ndarray

Even with all the checks added for MaskedArrays inside NumPy code, there are still issues around properly propogating masks. In PR 22914, @greglucas proposed adding MaskedArray.__array_ufunc__ building on work done in other PRs that were subsequently reverted.

On the mailing list, Allan Haldane announced a third-party implementation of MaskedArray using __array_function__ https://github.com/ahaldane/ndarray_ducktypes. The post mentions two open problems:

  • what to do about the mask of arr.imag = MaskedArray([1,X,1])
  • how to ducktype scalars when using __array_function__ ? Do we always need an associated "MaskedScalar" type?

In the PR, @greglucas says:

My desires from a new class:

  • No auto-masking of "bad" computation locations, let the user do that apriori if they want I haven't seen anyone argue for this yet, and this looks like it goes way back in git blame.
  • No warnings when doing a computation of a bad value under a mask. It would be annoying to get divide by zero warnings on your computations when you've already pre-masked the data (see my example at the top of the PR for the motivation here!)
  • np.func gets forwarded to np.ma.func whenever possible (implement __array_ufunc__ and __array_function__). It is quite surprising right now to do np.stack(ma, ma) and get a Masked instance back, but with an incorrect mask! (it would be better to just get a plain ndarray IMO)
  • Proper subclass wrapping/heirarchy preservation. Should fall out of (3) if implemented properly with NotImplemented being returned.

@mhvk adds

There is only one item I'd add: think carefully how to deal with underlying data other than ndarray. For instance, part of the reason I made a separate implementation in astropy was that the current MaskedArray simply cannot be used to create MaskedQuantity – with MaskedArray holding a Quantity attributes like the unit become hidden

There is also PR 22913 to implement this idea on the NumPy MaskedArray, without removing the inheritance from ndarray.

Implications for Array API compatibility

Masked arrays are currently out of scope for the Array API.

Who is in favor?

There are many open issues around MaskedArray. A refactor would enable cleaning up athe API space and smoothing out many of the rough edges, making maintenance easier.

What will break and how could we work around the breakage?

Why not make the change?

Is it worth it?

<Discussion></Discussion>

Decision

<Discussion></Discussion>

Last changed by 
 
Matti Picus
0
166

Read more

NEP 53 — Deciding on adoption Google Highway to underpin NumPy's SIMD functionality

switching the C++ wrappers from universal intrinsics to Highway, but in any case the work to move to C++ is not completemeson compiler switches or attributes (with a subtopic of MSVC). As far as I could tell there were concerns about CPU feature detection as well but I am not sure that is an issue since we need some kind of meson feature detection anywayspecifying features for runtime dispatch or more to the point to prevent runtime dispatch on certain featuresrewriting existing kernels using Highway intrinsics, but in any case we need to refactor to use C++dispatching inner loops on selected features: a desire to setup dispatching only once and not on every kernel callrewriting the _simd testing module to use C++ and to Highway

Jul 3, 2024
NumPy2 lighning talk: Change `NPY_MAX_DIMS` to 64 (from 32)

Originally proposed in issue 5744, and on the mailing list. The motivation seem a bit unclear. There is a clear use case for this in simulating quantum systems. While 64 is better than 32, it is not clear that will be the end of this discussion. In the issue there is a suggestion to make the limit dynamic. In PyArrayObject the value is not used, but it is used heavily in the structs behind PyArrayIterObject. PyArrayMultiIterObject, PyArrayMapIterObject, PyArrayNeighborhoodIterObject. It is also used as a shortcut in code like npy_intp dimensions[NPY_MAXDIMS]; for (i=0; i<PyArray_NDIMS(arr); i++) { dimensions[i] = PyArrayDIMS(arr, i); } /* Some code that broadcasts or otherwise manipulates "dimensions" */ ret = new_array(..., dimensions, ...); SciPy has about 30 such uses.

Apr 3, 2023
NumPy2 lighning talk: Change `np.int`_ to `np.int64` on all platforms

Today np.int_ reflects the C long type, which is 32-bits for MSVC (windows) and 64-bits on all other 64-bit platforms, and 32-bits on 32-bit platforms. The relevant issues discussing this: 9464, 12332, 17531 (which points out that the canonical name for np.int32 on win64 is np.intc), 17640 18540 Implications for Array API compatibility Under Dtypes, the Array API spec says: The default integer data type should be the same across platforms, but the default may vary depending on whether Python is 32-bit or 64-bit. So this change would bring NumPy into compliance with the Array API Who is in favor?

Apr 3, 2023
Work package 1 "Faster emulation of the RISC-V Sail model: Build Tooling, Documentation, CI Integration"

Using the Sail RISC-V model for emulation and testing currently has significant downsides compared to other existing emulators for RISC-V, such as Spike or QEMU. Specifically, the Sail-based emulators are 1000x-5000x slower, which makes them unworkable for many applications. This is one of the main pain-points of using the Sail model in RISC-V development. Pydrofoil (PyPy&apos;s Sail backend) [1] significantly increases the speed of the RISC-V Sail model. However, Pydrofoil is not currently easily usable by the RISC-V community: it is not documented and hard to use by anyone except the authors. In this work package we propose to provide build tooling,

Nov 11, 2022
Read more from Matti Picus
Published on HackMD