Sparse summit follow up meeting

--- tags: scipy, sparse, scientific-python-summit --- Video call link: https://meet.google.com/jrp-qgcf-nxh If not working: https://colgate.zoom.us/j/276854695 -------------------------- # Monday December 22, 2025 ### Attendees: - [sparse.linalg.norm dtype PR](https://github.com/scipy/scipy/pull/24121) All looks good after addition in doc_string of `int` in the list of possible return types - Handle `!=` when shapes don't agree but broadcasting can work. - Broadcasting breaks `==` and `!=` backward compatibility: "returns False" to "does broadcasting" - We need to deprecate the "returns False" behavior and replace it with broadcasting. - Discussion from last time: - let's raise a ValueError saying the user should fix it[use A.shape != B.shape]. Provide an equals[and not equals] method that can do broadcasting. - As Dan understands that discussion, we are expecting people who want broadcasting to use these new methods for now and then switch to != and == after the deprecation period. - proposed naming of new methods: `A.broadcast_ne` and `broadcast_eq`. Added to [PR 21925](https://github.com/scipy/scipy/pull/21925) in this [recent commit](https://github.com/scipy/scipy/pull/21925/commits/11af609ce5a30c95dea0f79c9fced862461ebf3c). Should this be separated from the broadcasting PR? How? - A plan for timing of deprecation of spmatrix: - 1.15 had fully functional sparray. - 1.18 could deprecate spmatrix (Let's do it!) - 1.19 move docs about spmatrix to separate page from main sparse page - 1.20 remove spmatrix - This would remove spmatrix in Jun 2027. - We can say "no earlier than 1.20" in the deprecation warning. - We should update the roadmap along with the deprecation PR, in January after the 1.17 release lands. - [Broadcasting binops in C++ 21925](https://github.com/scipy/scipy/pull/21925) - Timing shows that the new broadcastable binop takes similar time to the old version when broadcasting is not needed. "similar" means within the noise of asv, and within the noise of %timeit (ad hoc) experiments. - Errors when expanding broadcasting PR to all binop operators reveals: - `divide` doesnt create `nan` and `inf` correctly for implicit zeros. - `multiply` doesn't handle `inf` times implicit 0 by returning `nan` (result is implicit 0). - Both can be fixed with methods that check implicit zero locations so should be warned `SparseEfficiencyWarning`. This extra effprt could be hidden behind a check of `np.isinf(other)`. - sksparse related PRs (mostly `sparse.linalg.linsolver.py`) - [scikit-sparse broesler PRs](https://github.com/scipy/scipy/pulls/broesler) -------------------------- # Monday December 8, 2025 ### Attendees: CJ, Dan, Stéfan - Handle `!=` when shapes don't agree but broadcasting can work. - Broadcasting breaks `==` and `!=` backward compatibility: returns False to does broadcasting - [PR 23091](https://github.com/scipy/scipy/issues/23091) already discussed that (for sparse array) we should switch `==` and `!=` to match numpy results. (Raise when not broadcastable). Currently returns `False` when shapes don't match. - What kind of change/deprecation do we need to handle this? - Proposal 1: Immediate change to broadcast when possible, else deprecation warning and return False for 2 cycles. Then broadcast when possible, raise when not. - Proposal 2: Immediate change to sparray: broadcast when possible. Raise otherwise. Deprecate spmatrix 2-cycles. Then switch to broadcast/raise behavior. - Proposal 3: Deprecate all broadcasting for `eq` and `ne` for 2 release cycles. Then broadcast when possible and raise when not. - let's raise a ValueError saying they should fix it. Provide an equals method that can do broadcasting. - Let's form a plan for timing of deprecation of spmatrix: - 1.15 had fully functional sparray. - 1.18 could deprecate spmatrix (Let's do it!) - 1.19 move docs about spmatrix to separate page from main sparse page - 1.20 remove spmatrix - This would remove in Jun 2027. Thoughts? - We can say "no earlier than 1.20" in the deprecation warning. - We should update the roadmap along with the deprecation PR, in January after the 1.17 release lands. - Release notes in [sparse for SciPy 1.17](https://github.com/scipy/scipy/pull/24100) (Merged -- but changes can still be made) - Headsup up about [PR #24101 updating use of sksparse](https://github.com/scipy/scipy/pull/24101/files) (to use newer scikit-sparse solvers). Mostly in `sparse/linalg/lin_solver.py` but also in other modules and in modules over in `scipy/optimize`. - [Broadcasting binops in C++ 21925](https://github.com/scipy/scipy/pull/21925) - How do we pick level of splitting C++ functions into smaller more specific functions? - Time end-to-end code. - Compile time: good ways to check? - Filesize change: - for file `build/scipy/sparse/sparsetools/_sparsetools.<...>.so`? - Other .so files to watch? `_csparsetools.<...>.so`? - I played a little with `struct`s. - Only 1 unsized array is allowed in C++ struct. - OK to use pointers in the struct instead? - Could use struct of struct with 1 unsized array inside each? E.g. `struct csr_stuff {I n_row, I n_col, I* indptr, I* indices, T* data}` - Q: is `typedef struct` preferred? Or just the `struct`? - We'll also want to store nnz (which gives the size of the `indices`). - Storing raw pointers + size is fine. - We can build the struct inside `csr_binop_csr` and avoid touching the API surface. - [PR 23797 add `copy=False` to `astype`](https://github.com/scipy/scipy/pull/23797) :tada: Merged! - [sparse.linalg.norm issue 23651](https://github.com/scipy/scipy/issues/23651) convert to float before `sum(abs(data))`? (I didn't work on this since last meeting) -------------------------- # Monday November 10, 2025 ### Attendees: - [PR 23923 fix spurious conversions from csc to csr in binops](https://github.com/scipy/scipy/pull/23923) I think this is pretty well discussed. :tada: merged -------------------------- # Monday October 13, 2025 ### Attendees: ### Quick Reviews? - [expand_dims/permute_dims/swapaxes PR 23534](https://github.com/scipy/scipy/pull/23534) Approved - [nD kron 23677](https://github.com/scipy/scipy/pull/23677) Approved ### Topics: - Broadcasting binopt [PR 21925](https://github.com/scipy/scipy/pull/21925) - new commit splitting column broadcasting into separate functions. See [recent comment](https://github.com/scipy/scipy/pull/21925#issuecomment-3395758371) for description. - Reversing `op` args? Now use `reverse ? op(b, a) : op(a, b)` Better way in C++? - C++ lambda function?, - separate function reverse_op(op) -> new_op? - Define many new reverse op funcs for `lt, le, gt, ge, minus, divide`? - Will this choice be impacted by templating costs? How to think about that? - How do we pick level of splitting into specialty functions? - Time end-to-end code. - Compile time: good ways to check? - Filesize change: - for file `build/scipy/sparse/sparsetools/_sparsetools.<...>.so`? - Other .so files to watch? `_csparsetools.<...>.so`? - I played a little with `struct`s. - Only 1 unsized array is allowed in C++ struct. - OK to use pointers in the struct instead? - Could use struct of struct with 1 unsized array inside each? E.g. `struct csr_stuff {I n_row, I n_col, I* indptr, I* indices, T* data}` - Q: is `typedef struct` preferred? Or just the `struct`? - Broadcasting breaks `==` and `!=` backward compatibility: - PR 23091 already discussed that we should switch `==` and `!=` to match numpy results. (Raise when not broadcastable). Currently returns False/True when shapes don't match. - Broadcasting will change return type of the broadcastable case from bool to sparse. - What kind of change/deprecation do we need to handle this? - Proposal 1: Immediate change to broadcast when possible, else deprecation warning and return True/False for 2 cycles. Then broadcast when possible, raise when not. - Proposal 2: Immediate change to sparray: broadcast when possible. Raise otherwise. Deprecate spmatrix 2-cycles. Then switch to broadcast/raise behavior. - Proposal 3: Deprecate all broadcasting for `eq` and `ne` for 2 release cycles. Then broadcast when possible and raise when not. Others? Which? - `has_canonical_format` and `setdiag`: [PR 23676](https://github.com/scipy/scipy/pull/23676) made requested fixes. Re-review. - Further ideas about how to simplify `has_canonical_format`? - [sparse.linalg.norm issue 23651](https://github.com/scipy/scipy/issues/23651) ?convert to float before `sum(abs(data))`? (I didn't work on this since last meeting) -------------------------- # Monday September 29, 2025 ### Attendees: CJ, Dan, Stéfan ### Topics: - PR that is already approved but needs merging - [loadmat docs and spmatrix 23620](https://github.com/scipy/scipy/pull/23620) (merged :tada:) - Broadcasting binopt [PR 21925](https://github.com/scipy/scipy/pull/21925) - The PR looks like: ``` if column broadcasting: handle row broadcasting do binopt with columns repeated else: handle row broadcasting do usual binopt ``` - We could weave the column broadcasting into the rest of binopt using `if` clauses. But separating that logic should be faster. - We could make a separate function for column broadcasting, but I'm not sure how that affects library size. Certainly results in bigger api. - We talked about using a struct to hold the shape of the input arrays. What is advantage over passing in many ints? Does the struct already exist in numpy and we would just be using it? - We could just pass in the shape of A and B and compute which needs broadcasting in the C++ code. - TODO: Dan will update the current PR by splitting broadcasting case into its own function. Move as much as possible into C++. Try struct for internal C++ calls. - `set_diag` handling of has_canoncial_format - The (fairly) recent change to CSR setdiag fails to maintain the flags `has_canonical_format`/`has_sorted_indices` in the case when we convert to COO, use the COO set_diag and then convert back to CSR. - [Issue-comment 23644](https://github.com/scipy/scipy/issues/23644#issuecomment-3313055054) and [PR 23676](https://github.com/scipy/scipy/pull/23676) propose a fix to mimic what we do with insert_many (when flag is not set at the start, don't `sum_duplicates`. But when flag is set at the start, keep it in that state by calling `sum_duplicates`. Is this the right approach? - [Related discussion](https://discuss.scientific-python.org/t/sparse-array-has-canonical-format-and-has-sorted-indices-as-read-only-properties/2119) - **Big Picture**: What is the history of `has_canonical_format`? - Why a property in CSR? (not in COO) - In what ways do COO canonical flag handling differ from CS[RC]? - Which methods don't maintain canonical? - What is good long-term API for set/get canonical format? - TODO: review 23676 and probably merge -- and think about other ways to simplify canonical for the long term. Perhaps have a separate format for people who want to use noncanonical representations. Only relevant for some formats. - Next nD functionality [Issue 21923](https://github.com/scipy/scipy/issues/21923) (requested by CVXPY): - [expand_dims/permute_dims/swapaxes PR 23534](https://github.com/scipy/scipy/pull/23534) - [nD kron 23677](https://github.com/scipy/scipy/pull/23677) - [An issue 23651](https://github.com/scipy/scipy/issues/23651) arises for integer dtype with `sparse.linalg.norm` when `ord` is 1, -1, inf, -inf. When data is int, norm returns np.int64. Looks like it should return np.float64 to avoid overflow issues. Is Warren right that we should convert to a float before computing (to avoid overflow)? - It'd be nice to allow folks to get norm without having to convert the whole matrix to float dtype. But how could we avoid overflows? Docs could be changed to just say, the dtype is whatever it gives you. - How does numpy handle the cases where overflow can occur? - Merged :tada: Recent post on a very old issue: DOK.update with a check that inputs are valid keys? [Issue 8338](https://github.com/scipy/scipy/issues/8338) Dan did this in [DOK update(): ](https://github.com/scipy/scipy/pull/23625) -------------------------- # Monday August 18, 2025 ### Attendees: (none) ### PRs to Review: - COO indexing in nD [PR 23218](https://github.com/scipy/scipy/pull/23218) - merged! :tada: - Broadcasting binopt [PR 21925](https://github.com/scipy/scipy/pull/21925) - We could weave the column broadcasting into the rest of binopt using `if` clauses. But separating that logic should be faster: - if column broadcasting: - handle row broadcasting - do binopt with columns repeated - else: - handle row broadcasting - do binopt - We could make a separate function for column broadcasting, but I'm not sure how that affects library size. Certainly results in bigger api. - We talked about using a struct to hold the shape of the input arrays. What is advantage over passing in many ints? Could we pass in a separate nparray of shape? Or does the struct already exist in numpy and we would just be using it? - Next nD functionality [Issue 21923](https://github.com/scipy/scipy/issues/21923) (requested by CVXPY): - nD construct functions. (especially nD-kron) - `permute_dims()`/`swapdims()`. Not sure if same as transpose, but permute_dims is array-api, swapdims is numpy and transpose is ??. - `expand_dims()` - other nD functions? -------------------------- # Monday August 4, 2025 ### Attendees: ### PRs to Review: - COO indexing in nD [PR 23218](https://github.com/scipy/scipy/pull/23218) *approved* -------------------------- # Monday July 21, 2025 ### Attendees: ### PRs to Review: - benchmarks for sparray [PR 23291](https://github.com/scipy/scipy/pull/23291) *Merged* Ralf asked for sparray benchmarks to be added as parametrizations of the current `sparse.py` benchmarks. This way the results are all in the same table. Nothing added for nD or sparray specifically. And this doesn't handle csgraph or sparse.linalg benchmarks which still use spmatrix. This just runs the current sparse benchmarks for both spmatrix and sparray. - change default dtype of 'diags_array' and 'diags' [PR #23340](https://github.com/scipy/scipy/pull/23340) *Merged* - currently uses `np.common_type` on diagonals input dtype. `common_type` converts to integers to floating point (inexact dtypes). - This PR switches to `np.result_type` which selects dtype like other arithmetic -- leaving integers as integers. - COO indexing in nD [PR 23218](https://github.com/scipy/scipy/pull/23218) - changed comments to better explain approach. For `getitem` we: - build a mask of coords that satisfy slice and integer index constraints. - match array indices with coords (track whether each position in the array index matches each coords at all axes). - reorder coords to put array indices at beginning if needed by numpy rules. - add newaxis coords if requested. - For `setitem` with notation `A[key]=x` we: - broadcast `x` to the shape for `key` - form `x_coords/x_data` for x.nonzero - match `key` to existing `coords` (like getitem) and remove from `A` (set to zero) - map axes of `x_coords` to axes of `coords` (array indices can reorder axes) - build `coords` and `data` for new entries - concatenate new and existing `coords` and `data` - mark `has_canonical_format` as `False` - splitting `get` and `set` into two PRs is messy because `_TestGetset` and `TestGetSet1D` test both so we have to muck around with the `sparse_test_class` function in test_base. So I hope we can review `getitem` separately from `setitem` while keeping them in this PR. -------------------------- # Monday July 7, 2025 ### Attendees: ### PRs to Review: - make `_validate_indices` and `_asindices` functions [PR 23267](https://github.com/scipy/scipy/pull/23267) instead of methods - benchmarks for sparray [PR 23291](https://github.com/scipy/scipy/pull/23291) - My naive timings suggest that spmatrix and sparray perform about the same. But others would know better how to read the benchmarks. - Should we switch to sparray as if it is a code-change (instead of a test change) to ease plots across that change? Or is it better to have tracking of both performances over time? - COO indexing in nD [PR 23218](https://github.com/scipy/scipy/pull/23218) - changed comments to better explain approach. For `getitem` we: - build a mask of coords that satisfy slice and integer index constraints. - match array indices with coords (track whether each position in the array index matches each coords at all axes). - reorder coords to put array indices at beginning if needed by numpy rules. - add newaxis coords if requested. - For `setitem` with notation `A[key]=x` we: - broadcast `x` to the shape for `key` - form `x_coords/x_data` for x.nonzero - match `key` to existing `coords` (like getitem) and remove from `A` (set to zero) - map axes of `x_coords` to axes of `coords` (array indices can reorder axes) - build `coords` and `data` for new entries - concatenate new and existing `coords` and `data` - mark `has_canonical_format` as `False` - We can refactor `getitem` and `setitem` code to `_index.py` if desired. Current code is only for COO. So its hard to predict which parts will be common to other formats that might support nD later. - tests: for `get` and `set` are not separated in `test_base.py` (they are in `TestGetSet`) and splitting that is possible but sometimes subtle. So easier to keep them both in one PR - benchmarks: - sparray and spmatrix times seem within variation - we need to change csgraph and linalg benchmarks to use sparray too at some point. - separate PR to copy `sparse.py` to `sparse_array.py`. Get feedback from benchmark users. ### List of Todos - deprecate `eq` and `ne` for sparrays returning `True/False` when shapes differ. - Many `astype` w/o `copy` kwarg could be `copy=False`. I changed a couple. More in a later PR. -------------------------- # Monday June 23, 2025 ### Attendees: CJ and Dan ### PRs to Review: - DIA tocsr() efficiency [PR 23009](https://github.com/scipy/scipy/pull/23009) - remove Python 2 relics from `_spbase` [PR 23138](https://github.com/scipy/scipy/pull/23138) - COO indexing in nD, [PR 23218](https://github.com/scipy/scipy/pull/23218) - currently `__getitem__/__setitem__` are in `_coo.py` to avoid circular import using `_ravel_coords`. Would be better long term in `_index.py` but the code is pretty COO specific right now. Maybe refactor when a second nD format has a more complete vision. - has_canonical_format set to False after a COO `set` operation. Note this `has_canonical_format` is **not** a property unlike CSR/CSC where it is. ### Think about: - has_canonical_format: - in test suites for SciPy, networkx, scikit-image, scikit-learn, and cvxpy every call to `_binopt` has matching canonical structure and `has_canonical_format` property. We could probably use the property in place of the check. :rocket: - Indexing of e.g. 2D CSR when the index key would produce 3D array... Do we convert to COO? e.g. `A[None, [[1, 2], [0, 1]]]` or `A[index3d]` - Initial thought: raise, and tell user to first convert to COO - Are we missing an optimization here? Say you select only four elements out, converting to COO first is expensive vs constructing output directly. - Do as much of slicing as you can in CSR, then go to 3D? - It's hard to introspect to know what ### List of Todos - deprecate `eq` and `ne` for sparrays returning `True/False` when shapes differ. - Many `astype` w/o `copy` kwarg could be `copy=False`. I changed a couple. More in a later PR. #### Scratch space ``` DIA -> CSR 400 × 400: main: 225.0±2.00μs this: 71.5±0.09μs no nnz: 51.9±2.0μs +copy: 54.1±0.2μs cntnz: 50.4±0.2μs 10000 × 10000: main: 1740±0.05μs this: 175±1μs no nnz: 157±0.5μs +copy: 198±1.0μs cntnz: 195±1.0μs 62500 × 62500: main: 15.50±0.60ms this: 1.06±0.02ms no nnz: 1.07±0.04ms +copy: 4.06±0.20ms cntnz: 1.38±0.04ms ``` -------------------------- # Monday June 9, 2025 ### Attendees: CJ, Stefan, Dan ### PRs Ready for Review: - [PR 21743 test_base.py split](https://github.com/scipy/scipy/pull/21743) free-threading fails now fixed. - [PR 22936 new tests for has_canonical_format](https://github.com/scipy/scipy/pull/22936) supplements testing of the `has_canonical_format` property. ### Think about - Right now `eq` and `ne` return True/False (not an array at all!) if shapes do not match!? Shouldn't we raise? Can we deprecate/remove this for sparray? - NumPy gives ValueError: operands could not be broadcast together — we should match that behavior - At this point, will need deprecation cycle - has_canonical_format: - in test suites for SciPy, networkx, scikit-image, scikit-learn, and cvxpy every call to `_binopt` has matching canonical structure and `has_canonical_format` property. We could probably use the property in place of the check. :rocket: - The tests added in the above PR show how it might occur that they not agree. Basically, someone has to set the property incorrectly, or mess with the inner arrays after 1st accessing the property. Once the cache of the flag is made, further changes to inner arrays are not protected. Most people dont access the property in their code, so the caching doesnt cause a problem. - Many `astype` w/o `copy` kwarg could be `copy=False`. I changed a couple. More in a later PR. - Dan focusing lately on indexing for COO including nD. `_validate_index` will need update for `None` and array-index with more than 2D result. Dan's got it mostly working for `__getitem__`. Also a start on `__setitem__`. - new methods in an nD subclass of `IndexMixin`? or just as methods in `_coo.py`? (can be refactored later either way) - I'm thinking indexing of e.g. 2D CSR converting to COO if the index key would produce 3D array. Like `A[None, [[1, 2], [0, 1]]]` or `A[index3d]` - Initial thought: raise, and tell user to first convert to COO - Are we missing an optimization here? Say you select only four elements out, converting to COO first is expensive vs constructing output directly. - Do as much of slicing as you can in CSR, then go to 3D? - It's hard to introspect to know what -------------------------- # Monday May 12, 2025 ### Attendees: No meeting ### Release schedule for SciPy 1.16: - [Timetable](https://discuss.scientific-python.org/t/proposed-release-schedule-for-scipy-1-16-0/1883): maint branch May 17. - Should we add milestone labels for the nD PRs? Others? (Dan'll add them -- can remove them if needed) ### PRs Ready for Review: - **Merged** :rocket: [PR 22873 nD unary functions](https://github.com/scipy/scipy/pull/22873): nD with unary functions - all suggestions/concerns addressed and approvals from CJ and from Joren regarding typing. - **Merged** :rocket: [PR 22897 nD binop functions](https://github.com/scipy/scipy/pull/22897) nD binopt functions: - updated to move general code to `_base.py` with subclass private methods to `_csr.py` and `coo.py`. - [PR 21743 test_base.py split](https://github.com/scipy/scipy/pull/21743) free-threading fails now fixed. - [PR 22936 new tests for has_canonical_format](https://github.com/scipy/scipy/pull/22936) supplements testing of the `has_canonical_format` property. ### Think about - Right now `eq` and `ne` return True/False (not an array at all!) if shapes do not match!? Shouldn't we raise? Can we deprecate/remove this for sparray? - has_canonical_format: - in test suites for SciPy, networkx, scikit-image, scikit-learn, and cvxpy every call to `_binopt` has matching canonical structure and `has_canonical_format` property. We could probably use the property in place of the check. :rocket: - The tests added in the above PR show how it might occur that they not agree. Basically, someone has to set the property incorrectly, or mess with the inner arrays after 1st accessing the property. Once the cache of the flag is made, further changes to inner arrays are not protected. Most people dont access the property in their code, so the caching doesnt cause a problem. - Many `astype` w/o `copy` kwarg could be `copy=False`. I changed a couple. More in a later PR. -------------------------- # Monday April 28, 2025 ### Attendees: Dan, CJ, Stéfan ### PRs Ready for Review: - [PR 22873](https://github.com/scipy/scipy/pull/22873): nD with unary functions - sum/mean/min/max/argmin/argmax - refactor nD->2D conversion to a helper func - refactor dot/tensordot/matmul to use helper - Other changes: - switched int `axis` to tuple in `validateaxis` - Added more tests for `min/max(explicit=True)`cuz different codepath - When `axis=None`, use `sputils._todata().sum()` instead of `self @ ones` - assign to `out` instead of `res.sum(axis=(), out=out)`. (Feels closer to intent of out...??) - [PR 21743](https://github.com/scipy/scipy/pull/21743) test_base.py split. - Removed list of tests that dont use `get_index_dtype` and tests that the list was correct - [PR 22897](https://github.com/scipy/scipy/pull/22897) nD binopt users: - `eq/ne/gt/ge/lt/le` - `multiply/_divide` - `maximum/minimum` - refactor `_convert_to_2d` tools - refactor dot/tensordot/matmul - other changes - Pass around `op` in comparison/bool funcs. Use op.name when calling `_binopt`. Use `op(0, other)` for whether to warn. - switched long names to `argminmax`, `arminmax_axis` ### Think about: - has_canonical_format: in test suite `_binopt` is called ### times with ### attribute and check both True amd ### both False. :rocket: - Right now `eq` and `ne` return True/False if shapes do not match!?!? Shouldn't we raise? Can we deprecate/remove this for sparray? - Many `astype` w/o `copy` kwarg could be `copy=False`. I changed a couple. More in a later PR. - [21925](https://github.com/scipy/scipy/pull/21925) broadcasting of _binop at C++ level - row broadcasting is easy. col requires a few differences - do we put col broadcasting into the existing functions, or as a separate func? - Idea: create small C++ structure to hold shape and/or stride info. Perhaps like: ``` int ndim; int nnz; int shape[]; // length ndim int strides[]; // length ndim (or compute these) I indptr[]; // length shape[0] + 1 I indices[]; // length nnz T data[]; // length nnz ``` - [19984](https://github.com/scipy/scipy/pull/19984) binop general/canonical [discussion](https://github.com/scipy/scipy/issues/19984#issuecomment-2613783999) (Nothing implemented since last mtg) - Does this capture last mtg discussion? - Would be nice to capture the egregious cases and boost doc warnings. Egregious cases (based on code) when the vector created in General is long and sparse. -------------------------- # Monday April 14, 2025 ## Attendees: Dan and CJ ### PRs Ready for Review: - Ready for review/merge (*approved already*): - [22647](https://github.com/scipy/scipy/pull/22647) validateaxis :tada: MERGED - [22527](https://github.com/scipy/scipy/pull/22527) add returns sections to `construct.py` function :tada: MERGED - [22169](https://github.com/scipy/scipy/pull/22169) refactor CSC to use CSR sparsetools :tada: MERGED - Think about: - Next nD Steps: - PR has sum/mean, min/max, binopt, multiply, add, sub, minimum/maximum - ?split out sum/mean/min/max from bigger PR? - add other nD features: - CSR support (or new format name csx to do csr in nD) - indexing? - construction functions? - array_api nD functions like `expanddims` - [21925](https://github.com/scipy/scipy/pull/21925) broadcasting of _binop at C++ level - row broadcasting is easy. col requires a few differences - do we put col broadcasting into the existing functions, or as a separate func? - if we split col broadcasting, do we support both general and canonical for this case? - [19984](https://github.com/scipy/scipy/pull/19984) binop general/canonical [discussion](https://github.com/scipy/scipy/issues/19984#issuecomment-2613783999) - simple heuristic to pick general or sort+canonical - or docs to push users to choose - Let's aim for this: Would be nice to capture the egregious cases and boost doc warnings. Egregious cases (based on code) when the vector created in General is long and sparse. - [21743](https://github.com/scipy/scipy/pull/21743) test_base.py split. 2 issues: - right now Test64bit sparray tests are run when "slow" tests are run. spmatrix tests always. - Test64bit is hard to separate from the rest of test_base.py when testing - Q1: should we turn on sparray beyond "slow" tests? (Ans: Yes) Should we change some spmatrix to be slow? (Ans: No -- we will remove them with spmatrix so won't be long. And we'll see if people complain.) - Q2: is it worth splitting out Test64Bit to a separate module to allow partial testing? (Ans: Yes) -------------------------- # Monday March 17, 2025 ## Attendees: CJ, Stefan ### PRs Ready for Review: - Ready for review/merge (*approved already): - 22647 validateaxis - 22535 * add returns sections to `construct.py` function - 22169 refactor CSC to use CSR sparsetools - Read through (easy?) - 22535 update roadmap sparse - Think about: - 21925 broadcasting of _binop at C++ level - row broadcasting is easy. col requires a few differences - do we put col broadcasting into the existing functions, or as a separate func? - if we split col broadcasting, do we support both general and canonical for this case? - 21743 test_base.py split. 2 issues: - right now Test64bit sparray tests are run when "slow" tests are run. spmatrix tests always. - Test64bit is hard to separate from the rest of test_base.py when testing - Q1: should we turn on sparray beyond "slow" tests? should we change spmatrix at all? - Q2: is it worth splitting out Test64Bit to a separate module to allow partial testing? - 19984 binop general/canonical [discussion](https://github.com/scipy/scipy/issues/19984#issuecomment-2613783999) - simple heuristic to pick general or sort+canonical - or docs to push users to choose -------------------------- # Monday March 3, 2025 ## Attendees: Dan, CJ ### Active PRs and Issues - [PR 22535: Update sparse part of roadmap](https://github.com/scipy/scipy/pull/22535) - [PR 22527: Add Returns sections to `_construct.py`](https://github.com/scipy/scipy/pull/22527) - nD features: - [PR 22540: nD binop split from broadcasting](https://github.com/scipy/scipy/pull/22540) - After that PR, [COO nD supports most features](https://github.com/scipy/scipy/issues/21169#issuecomment-2363676145). Still need: - triu,tril,vstack,hstack,block_array - diagonal - `_matmul_sparse` - nD invites other features (in [array-api](https://data-apis.org/array-api/latest/API_specification/index.html)) - expand_dims, permute_dims, flip, move_axis, squeeze, etc in [manipulation funcs](https://data-apis.org/array-api/latest/API_specification/index.html) - keepdims kwarg wherever used in array-api? - indexing - nD CSR with (cached?) conversion to 2D for most ops - nD DOK - binop general/canonical [discussion](https://github.com/scipy/scipy/issues/19984#issuecomment-2613783999) - Do we come up with a simple heuristic or add docs to push users to choose? - This will interact with changes to implement broadcasting. So good to resolve now(?). -------------------------- # Monday February 17, 2025 ## Attendees: CJ, Dan ### News: - A number of PRs/backports for SciPy v1.15 have been merged. - [selecting wrong dtype for coo coords #22353](https://github.com/scipy/scipy/pull/22353) - [revert NotImplemented return values in dot and matmul #22373](https://github.com/scipy/scipy/pull/22373) Need to raise b/c numpy handles sparse as object dtype. - [handle `A[1,:,None]` index for spmatrix #22472](https://github.com/scipy/scipy/pull/22472) - Lower priority doc & err msg changes not backported. - We've helped a number of libraries switch to sparray. - networkx, dipy (summer 2024) - scikit-image - pyamg - cvxpy - sklearn tests for sparray (I think this is pass 1 of migration doc, but haven't looked hard) - To do: - scikit-learn pass 2 - mne-python - librsb & pyRSB - pytorch and pytorch.vision - supyx.sparse - sfepy (finite element) - fipy (pde solver -- finite element) - scikit-umfpack - trilinos - scvers ### PR discussions, Topics/Questions: - nD features: - [PR 22540: nD binop split from broadcasting](https://github.com/scipy/scipy/pull/22540) - `coo` copies python code for `binopt` & `minmax` from `_compressed.py` and `_data.py` - minmax calls mixin methods without subclassing. works, but why not subclass? - [PR 21613: broadcasting with copies](https://github.com/scipy/scipy/pull/21613) - Currently nD-add/sub creates duplicates without calling sum_duplicates - Q: should CSR support nD with an axis arg? Or call it something other than CSR? (1D currently uses the name CSR) - Q: How to reliably time `binop` sparsetools? - [Issue 19984: predictably slow "add"](https://github.com/scipy/scipy/issues/19984#issuecomment-2613783999) - Considering `general` vs `sort & canonical`, `general` is predictably slower for some parameter regions. (most of the time it is faster) - Posts from [commit 618c4ea](https://github.com/scipy/scipy/commit/618c4ea9fe4fd2c4642f02de61c123b1844cc8d4) and [#1385 (comment)](https://github.com/scipy/scipy/issues/1385#issuecomment-17023024) suggest that nbell put `general` back in due to an incorrect timing reported on the issue. - Need to check whether sorting followed by canonical is faster than general. [My timing comment](https://github.com/scipy/scipy/issues/19984#issuecomment-2613783999) shows that the general method is usually faster for unsorted indices. But some conditions make it slower than `sort & canonical`. The conditions seem to be large `M` with small `nnz`. But large M suggests should switch to CSC format. - [PR 21925: broadcasting without copy](https://github.com/scipy/scipy/pull/21925) will need timing for this - Currently `general` & `canonical` - broadcast `indptr` can be easily included - broadcast `indices` could be separate func or separated in an if clause. - C++ changes: [PR 21925 csr.h](https://github.com/scipy/scipy/pull/21925/files#diff-cce26e9307eb0faebfb555c944539bd5ae2b4050055ab1ec97ca837df53e8229) - [PR 22169: CSC binop via CSR and `_swap`](https://github.com/scipy/scipy/pull/22169) Timing? (no difference? but how to check that) ### Follow-up items for 1.16 - [ ] dont copy when broadcasting - [ ] support int32 index arrays for nD - [ ] improve `coo_todense_nd` performance? - [ ] avoid assigning to self? Maybe assign self/other -> A/B - choose priority between CSR nD and broadcasting w/o copy -------------------------- # Monday December 9, 2024 ## Attendees: Dan, CJ, Stéfan ### News: - SciPy v1.15 maintenance branch PRs - completed Dec 7 :tada: - [PR 21435: nD matmul, dot, tensordot](https://github.com/scipy/scipy/pull/21435) - [PR 21858: safely_cast_index_dtype](https://github.com/scipy/scipy/pull/21858) - Migration Guide refers to this, but it is private (inside `_sputils`). Is that a problem? - [PR 21944: add broadcast_shapes](https://github.com/scipy/scipy/pull/21944) - [PR 21753: migration guide updates](https://github.com/scipy/scipy/pull/21753) - [PR 21905: migrate scipy.io to sparray](https://github.com/scipy/scipy/pull/21905) - [PR 21888: random_state -> rng](https://github.com/scipy/scipy/pull/21888) - [PR 21824: set idx_dtype at creation where possible](https://github.com/scipy/scipy/pull/21824) - :tada: ### Follow-up items for 1.16 - [ ] dont copy when broadcasting - [ ] support int32 index arrays for nD - [ ] improve csr_todense_nd performance?? - [ ] avoid assigning to self? Maybe assign self/other -> A/B ### PR Discussion: - Try Broadcasting in sparsetools. - [PR 21925](https://github.com/scipy/scipy/pull/21925) Feedback on approach? - C++ changes: [PR 21925 csr.h](https://github.com/scipy/scipy/pull/21925/files#diff-cce26e9307eb0faebfb555c944539bd5ae2b4050055ab1ec97ca837df53e8229) - Python changes: [PR 21925 changes in _binopt](https://github.com/scipy/scipy/pull/21925/files#diff-e420b0a47f194c165015c67e94adb2da2bff46ae7fc1aa4b1a652a1dfd3d60b0) - nD elementwise mixed with broadcasting - [PR 21613](https://github.com/scipy/scipy/pull/21613) - lots here. should we split? or try to fix/merge/follow-up - broadcasting works, but copies data - `coo` copies `binopt` & `minmax` from _compressed - minmax calls mixin methods without subclass (which might be ok?) ### sparray migration PR Tracking: Conversion of SciPy subpackages to sparray - [x] sparray migration: [csgraph docs: PR 21873](https://github.com/scipy/scipy/pull/21873) merged :tada: - [x] sparray migration: [csgraph code: PR 21779](https://github.com/scipy/scipy/pull/21779) merged :tada: - [x] sparray migration: [linalg: PR 21710](https://github.com/scipy/scipy/pull/21710) merged :tada: - [x] sparray migration: [optimize: 1st pass PR 21785](https://github.com/scipy/scipy/pull/21785) merged :tada: - [x] sparray migration: [io: PR 21905](https://github.com/scipy/scipy/pull/21905) - [x] scikit-image - [ ] sparray migration optimize pass 2 -- in progress. - [ ] pyamg - [ ] scikit-sparse - [ ] cvxpy - [ ] sklearn -------------------------- # Monday November 25, 2024 ## Attendees: Dan, CJ ### News: - [SciPy release timing for 1.15](https://discuss.scientific-python.org/t/proposed-release-schedule-for-scipy-1-15-0/1502) is Dec 6 for the maintenance branch. ### Topics Discussion: - For v1.15 - Finish `idx_dtype` handling - first 2 PRs in PR discussion - switch `scipy.io` and `optimize` to sparray. - looks like this might get merged [gh-21435 Add nd COO support for matmul, dot and tensordot](https://github.com/scipy/scipy/pull/21435#issuecomment-2498261129) - Migration Guide - Improve csr_todense_nd performance?? - Broadcasting in sparsetools. - two PRs. not trying to do views. Just changing loop control with B/C in mind. - [csr.h](https://github.com/scipy/scipy/pull/21925/files#diff-cce26e9307eb0faebfb555c944539bd5ae2b4050055ab1ec97ca837df53e8229) to get idea of approach. [_binopt](https://github.com/scipy/scipy/pull/21925/files#diff-e420b0a47f194c165015c67e94adb2da2bff46ae7fc1aa4b1a652a1dfd3d60b0) for what changes in python code. - need [new broadcast_shapes function](https://github.com/scipy/scipy/pull/21944). numpy checks memory available so doesn't work for 32-bit systems. - Other nD topics: - 32-bit indexing for COO nd ### PR Review Discussions: - idx_dtype: - [ ] [PR 21824: set idx_dtype](https://github.com/scipy/scipy/pull/21824) inside sparse. approved 2 wks ago, but [some new additions](https://github.com/scipy/scipy/pull/21824#issuecomment-2476964166), and better handling of previous changes by moving inside constructors like `arange`, `np.concatenate`, etc. Also fixes [Issue 20389: maintain int64 idx_dtype in vstack](https://github.com/scipy/scipy/issues/20389) - [ ] [PR 21858: safely cast index arrays](https://github.com/scipy/scipy/pull/21858) helper function for idx_dtype. - two reviews so far. no approvals yet. - remove `msg` kwarg? allows e.g. "for SuperLU" in errmsg "indptr values too large for SuperLU". See @stefanv and @perimosocordiae comments. - Migration to sparray - [ ] [PR 21905: `scipy.io` migration to sparray](https://github.com/scipy/scipy/pull/21905) Requires new keyword arg to indicate sparray/spmatrix. Can we go over the deprecation plan to see if it looks OK? - maybe need note in sparray migration guide about `mmread(file, sparray=True)` - [ ] [PR 21573: update sparray migration guide](https://github.com/scipy/scipy/pull/21753) Should be ready. - [ ] [PR 21743: unmark sparray 64bit `slow`](https://github.com/scipy/scipy/pull/21743) move sparray test64bit tests out of `slow`. Separate `test_base.py` from `test_64bit.py` - Broadcasting: - [ ] [PR 21925: broadcast binopt](https://github.com/scipy/scipy/pull/21925) Change to csr_binop_csr to do broadcasting inside C++. [Look at `csr.h` changes](https://github.com/scipy/scipy/pull/21925/files#scipy/sparse/spasretools/csr.h) - [ ] [PR 21944: `broadcast_shapes()`](https://github.com/scipy/scipy/pull/21944)new function `broadcast_shapes` needed to avoid numpy restrictions on dense array size. - nD PRs: - [Issue 21923: interest in nD functions](https://github.com/scipy/scipy/issues/21923) From CVXPY. Want np funcs like `extend_dims`, `nd-kron`, `swap_axes`. - [ ] [gh-21435 Add nd COO support for matmul, dot and tensordot](https://github.com/scipy/scipy/pull/21435) - [ ] [gh-21613: 2D broadcasting and nD COO binops and reductions(min/max/sum)](https://github.com/scipy/scipy/pull/21613) - Final PR from nD internship: vstack/hstack/etc construction functions. Written but not submitted yet. - randoms_state -> rng. Merged :tada: [PR 21888: SPEC 7](https://github.com/scipy/scipy/pull/21613) Keyword arg name switched to `rng`. After deprecation period, only Generators will be used. Implemented for `rand`, `random` and `random_array`. ### sparray migration PR Tracking: Conversion of SciPy subpackages to sparray - [x] sparray migration: [csgraph docs: PR 21873](https://github.com/scipy/scipy/pull/21873) merged :tada: - [x] sparray migration: [csgraph code: PR 21779](https://github.com/scipy/scipy/pull/21779) merged :tada: - [x] sparray migration: [linalg: PR 21710](https://github.com/scipy/scipy/pull/21710) merged :tada: - [x] sparray migration: [optimize: 1st pass PR 21785](https://github.com/scipy/scipy/pull/21785) merged :tada: - [ ] sparray migration: [io: PR ]() Requires new keyword arg to indicate sparray/spmatrix. Can we go over the deprecation plan to see if it looks OK? - [ ] sparray migration optimize pass 2 -- in progress. waiting for HIGHS library to be setup. That's in. `linprog` still not removed. - [x] [PR 21799: is_sptriangular function](https://github.com/scipy/scipy/pull/21799) merged :tada: ----------------------------------- ## More minutes in [the 2024 hackmd file](https://hackmd.io/R5HaLsWnQ6KNq11bqoxf9Q) ## More minutes in [the 2023 hackmd file](https://hackmd.io/N8hOBnLlTUSiwPa9zAn_vg)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.