Dispatching discussion - 12/15/21

Intros - Ralf

• Background: discussion here:
  • https://labs.quansight.org/blog/2021/11/pydata-extensibility-vision
  • https://discuss.scientific-python.org/t/a-proposed-design-for-supporting-multiple-array-types-across-scipy-scikit-learn-scikit-image-and-beyond/131
• Coordinated approach, break from the new-package-per-hardware paradigm
  • Decision-making, timeframes - coordinated across projects

Intros - Stefan

• Focus the discussion: support for other array objects vs. selective dispatching

Discussion

• Error conversion between hardware devices? What if there's a CPU step in the middle of a GPU processing chain?

  • Ralf: this should error: no automagic conversion between array types
  • Strictly-typed dispatch, no implicit conversions
  • Implicit conversions are difficult, both for users and developers

• Can't pass arbitrary array objects into a foreign array library

  • How does this work with NEP18/NEP35 & type-based dispatching
    • The __array_function__ and other array protocols are numpy-specific for the purposes of this discussion; not part of the array API standard
    • Sebastian: could have a duck-array fallback for this

• High-level question: do we want to do this? Is this useful?

  • Having talked to HW-folks, e.g. AMD,

  • What are the options when there is a more performant algorithm that relies on new hardware. Where do they plug in?:

    • Copy API and have a new library
    • Monkeypatch functionality in existing library

  • These options are not particularly robust

  • Monkeypatching makes the system opaque - if you get strange results, it's difficult to get the failures back to the users.

  • Explicit import switching: difficult to support in packages that are built e.g. on top of scipy

• What's the scope for this proposal? Multi-GPU libraries?

  • The assumptions change (?) when working with clusters, colocated devices
  • A: It seems like this should be okay as long as the distributed nature is entirely encapsulated in the backend
  • Take dask as an example - currently adheres closely to numpy API, but when you start really using distributed-specific features (e.g. blocksize) then things may not be as performant/have new challenges
  • What about lazy systems?
    • Can subset features that don't work in a lazy-implementation e.g. and mark them as such

• Is there any CI/CD infrastructure in place for testing on specialized hardware

  • Public CI systems for HW is (should be?) a requirement
  • This is a bigger problem than just this project (see apple M1)
  • Every feature should be tested somewhere, but who's responsible? Hardware-specific code will not live in the consumer libraries (e.g. scikits)
    • Functions, including dispatch, should be tested by the backend implementer
      • Dask is kind of a special case, more of a meta-array than a backend implementer.
    • What about uarray?
      • in scipy.fft, cupy has the test for dispatching on their end
  • It'd be nice if backend implementers could find a way to make a consumer library's test suite runnable w/ their backend

• Is there a way of defining the interface/design patterns to look at the design in a more abstract way

  • Maybe abstract design principles should be better-documented

• Numerical precision questions in testing: should be a mechanism for users to specify what precision they need from backends for testing

  • One opinion: this should be up to the backend. Backend implementer documents what their precision is, upstream authors need to be aware
    • Gets more tricky when you run the original test suite on a new backend

• Clarifying the distinction between what's been done in NumPy (e.g. NEPS) and what's being proposed for scipy

  • A SPEC (NEP-like document, cross-project) for documenting/decision-making
  • Re: numpy vs. scipy. The dispatching machinery in NumPy was not made public and wasn't designed to handle backend dispatching
  • NumPy dispatching only does type-dispatching, not backend-selection dispatching
    • __array_function__ and __array_ufunc__ seem to work well in RAPIDS ecosystem, can wait-and-see about adoptying any new dispatching mechanism in numpy

• Backend registration, discovery, versioning etc. How will this work?

  • Register on import?
  • plugin-style where consumer looks for backends
  • Backend versioning?
    • Keeping backends and interfaces in sync
  • Re: compatibility - have the library dictate to backend implementers that they should follow the libraries interface? Backends accept any signature and only use what it understands, ignoring the rest?
  • Related problem: rollout. Not everyone is publishing their libraries (or backends) sychronously.
    • +1, having an abstract API might help because the API can be versioned
  • Re: chaning APIs
    • the majority of current API changes are adding new kwargs, not backward incompatible. This is handleable
  • Extremely important - needs to be layed out more concretely

• Have some way for backend-implementers to indicate back to users how much of a given API they support. This usually isn't immediately obvious to users. Whose job is it to inform users how much of any given API is supported by a backend. Can be addressed via tooling

  • numba has a similar problem (currently manually updated), but some tooling for this
  • Similar problem in dask via __array_function__
  • Libraries may be able to reject backend registration if it's known to be uncompatible - depends on how much information libraries have about what backends support
  • Nuances in compatibility tables: e.g. in cupy, there may be instances where the signature looks the same as numpy, but only a subset of options for a particular kwarg are supported.

• Is it possible for library/API authors to make their tests importable/usable by backends?

  • Important for the previous point: how do implementers/libraries stay in sync

• Who are the arbiters for deciding what's supported?

  • No arbiters - hope that people are honest/correct about it, but no formal role/procedure for verifying support
  • Library authors can't reasonably put requirements on backend implementers in terms of what fraction of the API must be supported
  • Opinion: it's best for backend implementers to have freedom to decide what they provide or not

• In the future, libraries may be written where the user interfaces and backend interfaces are decoupled from the start by design

• Dispatching between libraries, e.g. scipy.ndimage and scikit-image

  • Another example: scipy.optimize.minimize w/in scikit-learn

• Two distinct concerns: duck-typing vs. type-based dispatching and backend selection

  • Supporting various array types would be more work for library maintainers
  • Minimize duplication of pure-Python code by backend implementers
  • Combine array_api with a function-level dispatcher to support most use-cases

• Having scikit-learn be the API is viewed as a positive, +1 for dispatching

• What about a backend system for computational kernels shared by multiple GPU libs? (E.g. CUDA solvers)

  • Seems not high priority right now.
  • There could be a "glue" layer for this internally

Next steps

• Good ideas from the discussion above:

  • Sphinx(?) tooling for support tables
  • Making test suites reusable

• Things to document/describe better

• Collaborative long-term plans between libraries and potential backend-implementers

• Get a SPEC started from a distillation of the discussion on discuss.scientific-python.org

• Come up with a minimal set of principles that will guide the effort

  • a, b, c need to be implemented before approaching technology decisions
    • Docs w/ tables
    • Allow function overrides, etc.
  • Should have this in place before discussion about specific technological approachs (e.g. uarray)

• A single SPEC has a lot less info than the blog posts + discussion + this meeting

  • Dynamic summarization of ongoing discussions would be useful

• Where to start with the concrete implementation? A formal SPEC

• Explicitly layout the target audience, potential (hoped-for?) impact?

• Organize another call - open to the public

Afternotes

• single-dispatch (stlib) vs multi-dispatch