Video call link: https://meet.google.com/jrp-qgcf-nxh
If not working: https://colgate.zoom.us/j/276854695

Monday April 28, 2025

Attendees: Dan, CJ, Stéfan

PRs Ready for Review:

• PR 22873: nD with unary functions
  • sum/mean/min/max/argmin/argmax
  • refactor nD->2D conversion to a helper func
  • refactor dot/tensordot/matmul to use helper
  • Other changes:
    • switched int axis to tuple in validateaxis
    • Added more tests for min/max(explicit=True)cuz different codepath
    • When axis=None, use sputils._todata().sum() instead of self @ ones
    • assign to out instead of res.sum(axis=(), out=out). (Feels closer to intent of out…??)
• PR 21743 test_base.py split.
  • Removed list of tests that dont use get_index_dtype and tests that the list was correct
• PR 22897 nD binopt users:
  • eq/ne/gt/ge/lt/le
  • multiply/_divide
  • maximum/minimum
  • refactor _convert_to_2d tools
  • refactor dot/tensordot/matmul
  • other changes
    • Pass around op in comparison/bool funcs. Use op.name when calling _binopt. Use op(0, other) for whether to warn.
    • switched long names to argminmax, arminmax_axis

Think about:

• has_canonical_format: in test suite _binopt is called ### times with ### attribute and check both True amd ### both False.

  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →

• Right now eq and ne return True/False if shapes do not match!?!? Shouldn't we raise? Can we deprecate/remove this for sparray?

• Many astype w/o copy kwarg could be copy=False. I changed a couple. More in a later PR.

• 21925 broadcasting of _binop at C++ level

  • row broadcasting is easy. col requires a few differences
  • do we put col broadcasting into the existing functions, or as a separate func?
  • Idea: create small C++ structure to hold shape and/or stride info. Perhaps like:

int ndim; 
int nnz;
int shape[];   // length ndim 
int strides[]; // length ndim     (or compute these)
I indptr[];    // length shape[0] + 1
I indices[];   // length nnz
T data[];      // length nnz

• 19984 binop general/canonical discussion (Nothing implemented since last mtg)
  • Does this capture last mtg discussion?
    • Would be nice to capture the egregious cases and boost doc warnings. Egregious cases (based on code) when the vector created in General is long and sparse.

Monday April 14, 2025

Attendees: Dan and CJ

PRs Ready for Review:

• Ready for review/merge (approved already):

  • 22647 validateaxis
    Image Not Showing Possible Reasons

    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported

    Learn More →
    MERGED
  • 22527 add returns sections to construct.py function
    Image Not Showing Possible Reasons

    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported

    Learn More →
    MERGED
  • 22169 refactor CSC to use CSR sparsetools
    Image Not Showing Possible Reasons

    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported

    Learn More →
    MERGED

• Think about:

  • Next nD Steps:
    • PR has sum/mean, min/max, binopt, multiply, add, sub, minimum/maximum
    • ?split out sum/mean/min/max from bigger PR?
    • add other nD features:
      • CSR support (or new format name csx to do csr in nD)
      • indexing?
      • construction functions?
      • array_api nD functions like expanddims
  • 21925 broadcasting of _binop at C++ level
    • row broadcasting is easy. col requires a few differences
    • do we put col broadcasting into the existing functions, or as a separate func?
    • if we split col broadcasting, do we support both general and canonical for this case?
  • 19984 binop general/canonical discussion
    - simple heuristic to pick general or sort+canonical
    - or docs to push users to choose
    - Let's aim for this: Would be nice to capture the egregious cases and boost doc warnings. Egregious cases (based on code) when the vector created in General is long and sparse.
  • 21743 test_base.py split.  2 issues:
    • right now Test64bit sparray tests are run when "slow" tests are run. spmatrix tests always.
    • Test64bit is hard to separate from the rest of test_base.py when testing
    • Q1: should we turn on sparray beyond "slow" tests? (Ans: Yes) Should we change some spmatrix to be slow? (Ans: No – we will remove them with spmatrix so won't be long. And we'll see if people complain.)
    • Q2: is it worth splitting out Test64Bit  to a separate module to allow partial testing? (Ans: Yes)

Monday March 17, 2025

Attendees: CJ, Stefan

PRs Ready for Review:

• Ready for review/merge (*approved already):

  • 22647 validateaxis
  • 22535 * add returns sections to construct.py function
  • 22169 refactor CSC to use CSR sparsetools

• Read through (easy?)

  • 22535 update roadmap sparse

• Think about:

  • 21925 broadcasting of _binop at C++ level
    • row broadcasting is easy. col requires a few differences
    • do we put col broadcasting into the existing functions, or as a separate func?
    • if we split col broadcasting, do we support both general and canonical for this case?
  • 21743 test_base.py split.  2 issues:
    • right now Test64bit sparray tests are run when "slow" tests are run. spmatrix tests always.
    • Test64bit is hard to separate from the rest of test_base.py when testing
    • Q1: should we turn on sparray beyond "slow" tests? should we change spmatrix at all?
    • Q2: is it worth splitting out Test64Bit  to a separate module to allow partial testing?
  • 19984 binop general/canonical discussion
    - simple heuristic to pick general or sort+canonical
    - or docs to push users to choose

Monday March 3, 2025

Attendees: Dan, CJ

Active PRs and Issues

• PR 22535: Update sparse part of roadmap

• PR 22527: Add Returns sections to _construct.py

• nD features:

  • PR 22540: nD binop split from broadcasting
  • After that PR, COO nD supports most features. Still need:
    • triu,tril,vstack,hstack,block_array
    • diagonal
    • _matmul_sparse
  • nD invites other features (in array-api)
    • expand_dims, permute_dims, flip, move_axis, squeeze, etc in manipulation funcs
    • keepdims kwarg wherever used in array-api?
    • indexing
    • nD CSR with (cached?) conversion to 2D for most ops
    • nD DOK

• binop general/canonical discussion

  • Do we come up with a simple heuristic or add docs to push users to choose?
  • This will interact with changes to implement broadcasting. So good to resolve now(?).

Monday February 17, 2025

Attendees: CJ, Dan

News:

• A number of PRs/backports for SciPy v1.15 have been merged.
  • selecting wrong dtype for coo coords #22353
  • revert NotImplemented return values in dot and matmul #22373 Need to raise b/c numpy handles sparse as object dtype.
  • handle A[1,:,None] index for spmatrix #22472
• Lower priority doc & err msg changes not backported.
• We've helped a number of libraries switch to sparray.
  • networkx, dipy (summer 2024)
  • scikit-image
  • pyamg
  • cvxpy
  • sklearn tests for sparray (I think this is pass 1 of migration doc, but haven't looked hard)
• To do:
  • scikit-learn pass 2
  • mne-python
  • librsb & pyRSB
  • pytorch and pytorch.vision
  • supyx.sparse
  • sfepy (finite element)
  • fipy (pde solver – finite element)
  • scikit-umfpack
  • trilinos
  • scvers

PR discussions, Topics/Questions:

• nD features:
  • PR 22540: nD binop split from broadcasting
    • coo copies python code for binopt & minmax from _compressed.py and _data.py
    • minmax calls mixin methods without subclassing. works, but why not subclass?
  • PR 21613: broadcasting with copies
  • Currently nD-add/sub creates duplicates without calling sum_duplicates
  • Q: should CSR support nD with an axis arg? Or call it something other than CSR? (1D currently uses the name CSR)
• Q: How to reliably time binop sparsetools?
  • Issue 19984: predictably slow "add"
    • Considering general vs sort & canonical, general is predictably slower for some parameter regions. (most of the time it is faster)
    • Posts from commit 618c4ea and #1385 (comment) suggest that nbell put general back in due to an incorrect timing reported on the issue.
    • Need to check whether sorting followed by canonical is faster than general. My timing comment shows that the general method is usually faster for unsorted indices. But some conditions make it slower than sort & canonical. The conditions seem to be large M with small nnz. But large M suggests should switch to CSC format.
  • PR 21925: broadcasting without copy will need timing for this
    • Currently general & canonical
    • broadcast indptr can be easily included
    • broadcast indices could be separate func or separated in an if clause.
    • C++ changes: PR 21925 csr.h
  • PR 22169: CSC binop via CSR and _swap Timing? (no difference? but how to check that)

Follow-up items for 1.16

• dont copy when broadcasting
• support int32 index arrays for nD
• improve coo_todense_nd performance?
• avoid assigning to self? Maybe assign self/other -> A/B
• choose priority between CSR nD and broadcasting w/o copy

Monday December 9, 2024

Attendees: Dan, CJ, Stéfan

News:

• SciPy v1.15 maintenance branch PRs
  • completed Dec 7
    Image Not Showing Possible Reasons

    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported

    Learn More →
  • PR 21435: nD matmul, dot, tensordot
  • PR 21858: safely_cast_index_dtype
    • Migration Guide refers to this, but it is private (inside _sputils). Is that a problem?
  • PR 21944: add broadcast_shapes
  • PR 21753: migration guide updates
  • PR 21905: migrate scipy.io to sparray
  • PR 21888: random_state -> rng
  • PR 21824: set idx_dtype at creation where possible
  • Image Not Showing Possible Reasons

    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported

    Learn More →

Follow-up items for 1.16

• dont copy when broadcasting
• support int32 index arrays for nD
• improve csr_todense_nd performance??
• avoid assigning to self? Maybe assign self/other -> A/B

PR Discussion:

• Try Broadcasting in sparsetools.
  • PR 21925 Feedback on approach?
    • C++ changes: PR 21925 csr.h
    • Python changes: PR 21925 changes in _binopt
• nD elementwise mixed with broadcasting
  • PR 21613
  • lots here. should we split? or try to fix/merge/follow-up
  • broadcasting works, but copies data
  • coo copies binopt & minmax from _compressed
  • minmax calls mixin methods without subclass (which might be ok?)

sparray migration PR Tracking:

Conversion of SciPy subpackages to sparray

• sparray migration: csgraph docs: PR 21873 merged
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
• sparray migration: csgraph code: PR 21779 merged
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
• sparray migration: linalg: PR 21710 merged
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
• sparray migration: optimize: 1st pass PR 21785 merged
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
• sparray migration: io: PR 21905
• scikit-image
• sparray migration optimize pass 2 – in progress.
• pyamg
• scikit-sparse
• cvxpy
• sklearn

Monday November 25, 2024

Attendees: Dan, CJ

News:

• SciPy release timing for 1.15 is Dec 6 for the maintenance branch.

Topics Discussion:

• For v1.15
  • Finish idx_dtype handling
    • first 2 PRs in PR discussion
  • switch scipy.io and optimize to sparray.
  • looks like this might get merged gh-21435 Add nd COO support for matmul, dot and tensordot
  • Migration Guide
  • Improve csr_todense_nd performance??
• Broadcasting in sparsetools.
  • two PRs. not trying to do views. Just changing loop control with B/C in mind.
    • csr.h to get idea of approach. _binopt for what changes in python code.
    • need new broadcast_shapes function. numpy checks memory available so doesn't work for 32-bit systems.
• Other nD topics:
  • 32-bit indexing for COO nd

PR Review Discussions:

• idx_dtype:
  • PR 21824: set idx_dtype inside sparse. approved 2 wks ago, but some new additions, and better handling of previous changes by moving inside constructors like arange, np.concatenate, etc. Also fixes Issue 20389: maintain int64 idx_dtype in vstack
  • PR 21858: safely cast index arrays helper function for idx_dtype.
    • two reviews so far. no approvals yet.
    • remove msg kwarg? allows e.g. "for SuperLU" in errmsg "indptr values too large for SuperLU". See @stefanv and @perimosocordiae comments.
• Migration to sparray
  • PR 21905: scipy.io migration to sparray Requires new keyword arg to indicate sparray/spmatrix. Can we go over the deprecation plan to see if it looks OK?
    • maybe need note in sparray migration guide about mmread(file, sparray=True)
  • PR 21573: update sparray migration guide Should be ready.
• PR 21743: unmark sparray 64bit slow move sparray test64bit tests out of slow. Separate test_base.py from test_64bit.py
• Broadcasting:
  • PR 21925: broadcast binopt Change to csr_binop_csr to do broadcasting inside C++. Look at csr.h changes
  • PR 21944: broadcast_shapes()new function broadcast_shapes needed to avoid numpy restrictions on dense array size.
• nD PRs:
  • Issue 21923: interest in nD functions From CVXPY. Want np funcs like extend_dims, nd-kron, swap_axes.
  • gh-21435 Add nd COO support for matmul, dot and tensordot
  • gh-21613: 2D broadcasting and nD COO binops and reductions(min/max/sum)
  • Final PR from nD internship: vstack/hstack/etc construction functions. Written but not submitted yet.
• randoms_state -> rng. Merged
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
  PR 21888: SPEC 7 Keyword arg name switched to rng. After deprecation period, only Generators will be used. Implemented for rand, random and random_array.

sparray migration PR Tracking:

Conversion of SciPy subpackages to sparray

• sparray migration: csgraph docs: PR 21873 merged
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
• sparray migration: csgraph code: PR 21779 merged
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
• sparray migration: linalg: PR 21710 merged
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
• sparray migration: optimize: 1st pass PR 21785 merged
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
• sparray migration: io: PR Requires new keyword arg to indicate sparray/spmatrix. Can we go over the deprecation plan to see if it looks OK?
• sparray migration optimize pass 2 – in progress. waiting for HIGHS library to be setup. That's in. linprog still not removed.
• PR 21799: is_sptriangular function merged
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →

More minutes in the 2024 hackmd file

More minutes in the 2023 hackmd file