# nx-parallel meetings

meeting link: https://colgate.zoom.us/j/281534728
notes: https://hackmd.io/vlGWcfRsQF64OTIm7bCtpw?both

### Date - 25th Oct, 2024

**Attendees** : Aditi, Derek

### Topics

- https://github.com/networkx/nx-parallel/pull/86
- https://github.com/networkx/nx-parallel/pull/72
- https://scikit-learn-general.narkive.com/LR2bD7eS/grid-search-joblib#:~:text=Post%20by%20Satrajit%20Ghosh%0Adoes,scope%20of%20joblib%2C%20I%20believe

### Date - 11th Oct, 2024

**Attendees** : Aditi, Rick, Dan

### Topics

- temporary fixing https://github.com/networkx/networkx/issues/7671 : https://github.com/networkx/nx-parallel/pull/87
- test this at networkx's level
- discussions on `spatch`
- Tests for backends - to check if the entry_points and other things are working fine for each backend
- these can be centralised tests that backends can use

### Topics for next meeting

- handling cases like in https://github.com/networkx/nx-parallel/pull/72
- i.e. `Parallel(n_jobs=(no_of_primary - 1) ** 2, require="sharedmem")`

### Date - 4th Oct, 2024

**Attendees** : Aditi, Derek

### Topics

- Demo of a really nice (private) networkx backend by Derek!
- discussions on distributed support in nx-parallel
- https://ml.dask.org/joblib.html
- https://joblib.readthedocs.io/en/latest/auto_examples/parallel/distributed_backend_simple.html
- https://docs.dask.org/en/stable/deploying.html#distributed-computing

- Discussion on https://github.com/networkx/nx-parallel/pull/86
- Derek will address the review comments and then get more perspectives on it
- Should we add the graph conversion `if` condition in the same decorator that we use for setting configs or a separate decorator or just leave it how it is right now i.e. not in a decorator?
- more perspectives needed on this

### Date - 27th Sept, 2024

**Attendees** : Aditi, Derek

### Topics

- https://github.com/networkx/nx-parallel/pull/80 got merged!! :tada::tada:
- `execute_parallel` function's concept
- move `create_iterables` inside it
- should we have `"nodes"`, `"edges"`, `"isolates"`, `"pairs"` in `create_iterables`?
- [Aditi] This seems restricting in some ways.

- derek will create a PR about it!

### Date - 20th Sept, 2024

**Attendees** : Aditi, Derek

### Topics

- making PR https://github.com/networkx/nx-parallel/pull/80 ready to be merged!
- update the pre-commit hooks for them to act more uniformly across machines, editors etc.
- chunks of size `n` or `n` number of chunks?
- memory management or time
- we do pass `n_jobs` as `n` in the `chunks` function
- but smaller chunks might be better but also bad in terms of memory as more the number of chunks more will be the memory usage. and what about time?
- https://joblib.readthedocs.io/en/latest/auto_examples/memory_basic_usage.html , https://github.com/scikit-learn/scikit-learn/blob/6fd2f8372227ac394cd3dd5cd96d110fa49355b8/sklearn/pipeline.py
- Adding caching in nx-parallel : https://joblib.readthedocs.io/en/latest/auto_examples/nested_parallel_memory.html#sphx-glr-auto-examples-nested-parallel-memory-py
- shared memory not possible in nx-parallel
- [Derek] more flexibility in nx-parallel is possible beyond allowing the modification of parallel parameters from `joblib.parallel_config`

### Date - 13th Sept, 2024

**Attendees** : Aditi, Derek, Erik

### Topics

- Type annotation - removing it from PR#80 for now and make an issue on it instead
- Are we going to test them if we have them?
- nx_cugraphs - https://github.com/scientific-python/docstub
- Type annotation : sklearn doesn’t use it

- Too early to tell if we should include Map-Reduce in nx-parallel but we could have a general infrastructure to simplify things

- Heatmaps makes sense
- Avg to reduce noise?
- plotting and timing in pytest-benchmarks?
- Interactive plot (dash) - https://dash.plotly.com/interactive-graphing
- Algo drop-down
- Live plot - function of vertices, edge probability is a config?
- https://docs.rapids.ai/api/cugraph/stable/nx_cugraph/nx_cugraph/
- https://github.com/python-graphblas/graphblas-algorithms

- Config - should we sync the two configs i.e. networkx's and joblib's?
- Speeding up generator functions - removing nested for loop might help!
- adding graph conversion and chunking logic in a decorator; include in the config decorator only?

## Notes related to nx-parallel (from the last NetworkX comm. meeting)

### Date - 4th Sept, 2024

### Topics:

- TODO : Add a parallel implementations for finding all cycles in a graph in nx-parallel([@Nikoleta-v3](https://github.com/Nikoleta-v3) requires this)
- TODO : Better testing of configs and chunking in nx-parallel
- Having a map-reduce arch. in nx-parallel? My(Aditi) main concerns
- might not be very first-time contributor friendly;
- and might not be able to fit future non-embarrassingly parallel graph algo’s implementations into this arch.
- for reference - [PR#7](https://github.com/networkx/nx-parallel/pull/7), [Issue#30](https://github.com/networkx/nx-parallel/issues/30)
- [Derek] Adding another types of parallelism in nx-parallel —> running same algorithm on multiple graphs or subgraphs within a graph in parallel
- Why would someone not use joblib for this kind of thing but instead would want to use nx-parallel?
- If those graphs have a common property so the parallelism would be done in a particular way.
- Other topics:
- benchmarking - https://github.com/networkx/nx-parallel/issues/12
- improving configs in nx-parallel? - https://github.com/networkx/nx-parallel/issues/76
- handling already fast algorithms - https://github.com/networkx/nx-parallel/issues/79, https://github.com/networkx/nx-parallel/issues/77
- showcasing speedups in a better way - https://github.com/networkx/nx-parallel/issues/51 (also memory usage and other CPU usage, processes related, threads related statistics, etc.)

- TODO : Creating a website for nx-parallel (Aditi)
- related to [PR#80](https://github.com/networkx/nx-parallel/pull/80)
- Thoughts on having type annotations in nx-parallel?
- Why is `GraphIteratorType` a class and not just a list? and should we have all types in a separate `types.py`?
- Introducing lower-level parallelism in nx-parallel?
-