---
# System prepended metadata

title: nx-parallel meetings

---

# nx-parallel meetings

meeting link: https://colgate.zoom.us/j/281534728
notes: https://hackmd.io/vlGWcfRsQF64OTIm7bCtpw?both

### Date - 25th Oct, 2024

**Attendees** : Aditi, Derek

### Topics

- https://github.com/networkx/nx-parallel/pull/86
- https://github.com/networkx/nx-parallel/pull/72
- https://scikit-learn-general.narkive.com/LR2bD7eS/grid-search-joblib#:~:text=Post%20by%20Satrajit%20Ghosh%0Adoes,scope%20of%20joblib%2C%20I%20believe


### Date - 11th Oct, 2024

**Attendees** : Aditi, Rick, Dan

### Topics

- temporary fixing https://github.com/networkx/networkx/issues/7671 : https://github.com/networkx/nx-parallel/pull/87
    - test this at networkx's level
- discussions on `spatch`
- Tests for backends - to check if the entry_points and other things are working fine for each backend
    - these can be centralised tests that backends can use

### Topics for next meeting

- handling cases like in https://github.com/networkx/nx-parallel/pull/72
    - i.e. `Parallel(n_jobs=(no_of_primary - 1) ** 2, require="sharedmem")`

### Date - 4th Oct, 2024

**Attendees** : Aditi, Derek

### Topics

- Demo of a really nice (private) networkx backend by Derek!
- discussions on distributed support in nx-parallel
    - https://ml.dask.org/joblib.html
    - https://joblib.readthedocs.io/en/latest/auto_examples/parallel/distributed_backend_simple.html
    - https://docs.dask.org/en/stable/deploying.html#distributed-computing

- Discussion on https://github.com/networkx/nx-parallel/pull/86
    - Derek will address the review comments and then get more perspectives on it
- Should we add the graph conversion `if` condition in the same decorator that we use for setting configs or a separate decorator or just leave it how it is right now i.e. not in a decorator?
    - more perspectives needed on this

### Date - 27th Sept, 2024

**Attendees** : Aditi, Derek

### Topics

- https://github.com/networkx/nx-parallel/pull/80 got merged!! :tada::tada:
- `execute_parallel` function's concept
    - move `create_iterables` inside it
        - should we have `"nodes"`, `"edges"`, `"isolates"`, `"pairs"` in `create_iterables`? 
            - [Aditi] This seems restricting in some ways.

- derek will create a PR about it!

### Date - 20th Sept, 2024

**Attendees** : Aditi, Derek

### Topics

- making PR https://github.com/networkx/nx-parallel/pull/80 ready to be merged!
- update the pre-commit hooks for them to act more uniformly across machines, editors etc.
- chunks of size `n` or `n` number of chunks?
    - memory management or time
    - we do pass `n_jobs` as `n` in the `chunks` function
    - but smaller chunks might be better but also bad in terms of memory as more the number of chunks more will be the memory usage. and what about time?
- https://joblib.readthedocs.io/en/latest/auto_examples/memory_basic_usage.html , https://github.com/scikit-learn/scikit-learn/blob/6fd2f8372227ac394cd3dd5cd96d110fa49355b8/sklearn/pipeline.py
- Adding caching in nx-parallel : https://joblib.readthedocs.io/en/latest/auto_examples/nested_parallel_memory.html#sphx-glr-auto-examples-nested-parallel-memory-py
    - shared memory not possible in nx-parallel
- [Derek] more flexibility in nx-parallel is possible beyond allowing the modification of parallel parameters from `joblib.parallel_config`

### Date - 13th Sept, 2024

**Attendees** : Aditi, Derek, Erik

### Topics

- Type annotation - removing it from PR#80 for now and make an issue on it instead
    - Are we going to test them if we have them?
    - nx_cugraphs - https://github.com/scientific-python/docstub
    - Type annotation :  sklearn doesn’t use it

- Too early to tell if we should include Map-Reduce in nx-parallel but we could have a general infrastructure to simplify things

- Heatmaps makes sense 
    - Avg to reduce noise?
    - plotting and timing in pytest-benchmarks?
    - Interactive plot (dash) - https://dash.plotly.com/interactive-graphing 
        - Algo drop-down
        - Live plot - function of vertices, edge probability is a config?
    - https://docs.rapids.ai/api/cugraph/stable/nx_cugraph/nx_cugraph/
    - https://github.com/python-graphblas/graphblas-algorithms

- Config - should we sync the two configs i.e. networkx's and joblib's?
- Speeding up generator functions - removing nested for loop might help!
- adding graph conversion and chunking logic in a decorator; include in the config decorator only?


## Notes related to nx-parallel (from the last NetworkX comm. meeting)

### Date - 4th Sept, 2024

### Topics:

- TODO : Add a parallel implementations for finding all cycles in a graph in nx-parallel([@Nikoleta-v3](https://github.com/Nikoleta-v3) requires this)
- TODO : Better testing of configs and chunking in nx-parallel
- Having a map-reduce arch. in nx-parallel? My(Aditi) main concerns 
    - might not be very first-time contributor friendly; 
    - and might not be able to fit future non-embarrassingly parallel graph algo’s implementations into this arch.
    - for reference - [PR#7](https://github.com/networkx/nx-parallel/pull/7), [Issue#30](https://github.com/networkx/nx-parallel/issues/30)
- [Derek] Adding another types of parallelism in nx-parallel —> running same algorithm on multiple graphs or subgraphs within a graph in parallel
    - Why would someone not use joblib for this kind of thing but instead would want to use nx-parallel?
        - If those graphs have a common property so the parallelism would be done in a particular way.
- Other topics:
    - benchmarking - https://github.com/networkx/nx-parallel/issues/12
    - improving configs in nx-parallel? - https://github.com/networkx/nx-parallel/issues/76
    - handling already fast algorithms - https://github.com/networkx/nx-parallel/issues/79, https://github.com/networkx/nx-parallel/issues/77
    - showcasing speedups in a better way - https://github.com/networkx/nx-parallel/issues/51 (also memory usage and other CPU usage, processes related, threads related statistics, etc.)

- TODO : Creating a website for nx-parallel (Aditi)
- related to [PR#80](https://github.com/networkx/nx-parallel/pull/80)
    - Thoughts on having type annotations in nx-parallel?
    - Why is `GraphIteratorType` a class and not just a list? and should we have all types in a separate `types.py`?
- Introducing lower-level parallelism in nx-parallel?
- 