# Python Packages by Domain Stack
Scientific Python libraries for analyzing celestial bodies.
- Astropy: Common core package for Astronomy in Python.
- Poliastro: Interactive Astrodynamics and Orbital Mechanics, with a focus on ease of use, speed, and quick visualization.
- SunPy: A community-developed, free and open-source solar data analysis environment.
- SkyPy: Package for simulating the astrophysical sky.
Scientific Python tools for packaging.
- conda-forge: A community-led collection of recipes, build infrastructure and distributions for the conda package manager.
- conda: Package, dependency and environment management for any language.
- scikit-build: The official adaptor for CMake in Python packaging to build binary extensions, along with Python packages for CMake, Ninja, and other tooling.
- cibuildwheel: The PyPA tool for building binary wheels for all supported platforms.
Scientific Python packages for improving performance when coding in Python
- Dask: A library that allows parallelism for analytics and performance at scale.
- Cython: An optimizing static compiler. It makes writing C extensions as easy as Python itself.
- Numba: JIT compiler that translates a subset of Python and NumPy code into fast
- CuPy: Array Library compatible with NumPy and SciPy for GPU-accelerated Computing with Python
- pybind11: A C++ API for CPython/PyPy.
Scientific Python libraries for visualization
- matplotlib: Library for creating static, animated, and interactive visualizations.
- bokeh: A library for creating interactive visualizations for modern web browsers.
- scikit-image: A collection of algorithms for image processing.
- napari: A multi-dimensional image viewer for Python.
- holoviz: A set of high-level tools to simplify visualization in Python.
Scientific Python organizations that provide support to open source Python projects
- NumFocus: Non-profit organization that promotes open practices in research, data, and scientific computing.
- SciPy: Annual Scientific Computing with Python conference.
- OpenTeams: A Network of Enterprise Solution Architects using open Source talent to build and maintain your softwaresolutions.
- Scientific Python: A community effort to better coordinate the Scientific Python ecosystem and grow the community.
Scientific Python packages that contain developer tools.
- IPython: A command-line interface to Python, for interactive computing with an interactive shell and a kernel for Jupyter.
- Jupyter: A set of web services for interactive computing across all programming languages.
- Spyder: A Scientific Python Developement Environment.
Scientific Python libraries with math tools.
- sympy: Library for symbolic mathematics.
- scipy: Collection of fundamental algorithms for scientific computing in Python
### Building blocks
Scientific Python packages that make it easier to build and work in blocks.
- numpy: Fundamental package for numerical computation. It defines the n-dimensional array data structure, the most common way of exchanging data within packages in the ecosystem.
- zarr: A library and a format for chunked, compressed and N-dimensional arrays storage.
- xarray: A package that makes working with labeled multi-dimensional arrays simple, efficient, and fun.
- NetworkX: A package for the creation, manipulation, and study of complex networks.
- Awkward Array: Manipulate JSON-like data with NumPy-like idioms.
### Machine Learning
Scientific Python packages for machine learning.
- scikit-learn: A set of simple and efficient tools for predictive data analysis.
- PyTorch Ignite: High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.
Scientific Python libraries for working with DataFrames.
- Pandas: A package that provides fast, powerful, flexible, and easy to use data analysis and manipulation.
- GeoPandas: An open source project to make working with geospatial data in Python easier.
- Pandera: A Statistical Data Testing Toolkit that provides a flexible API for performing data validation on dataframes.
Scientific Python tools for writing tests.
- Pytest: A mature full-featured Python testing tool that helps you write better programs
- Hypothesis: A library for creating unit tests which are simpler to write and more powerful
- Ghostwriter: A command-line interface which can write property-based tests for you.
Communities around specific disiplines.
- scikit-hep: A community project from High Energy Physics building tools useful in HEP and beyond.
# MEETING 1
Stéfan van der Walt (Scientific Python)
Juanita Gomez (Scientific Python)
Pamphile Roy (Scientific Python)
Paige Martin (Pangeo)
Ryan Abernathey (Pangeo)
Jim Pivarski (Scikit-HEP)
Henry Schreiner (Scikit-HEP)
Levi Wolf (Geography)
Martin Fleischmann (Geography)
Pey Lian Lim (Astropy)
Isaac Virshup (scverse)
## Meeting notes
- Initial idea: Place for new users of the Scientific Python ecosystem where they find information about tools that they can use for a specific domain.
Challenge: Linear structure is hard since there are packages that are used for a lot of domains.
Possible solution: Using tags
- Using Stacks
- Two different ideas: The landing page plus infrastructure (cookie-cuttter)
- Assigning responsabilities in order to maintain this. People from each of the stacks. -> How do we decide on this people? Voting?
Is this made for new commers or for people already involved in the ecosystem?
- It is useful to have information about who is maintaining which package to know who to reach out depending on the needs.
- It is useful for new comers to get to know packages from the different domains.
For example Astropy has https://www.astropy.org/affiliated/
It makes sense to have a person in charge of keeping the information updated but also allowing people to collaborate and having a review process in order to add new packages.
Counterpoint: The ecosystem is amazing because of decentralization, a lot of people creating.
There are a lot of resources that have the information about Python packages but we have the support of people from the community maintaining packages so we should have a high-bar standard for the packages that we include. -> On the topic of reviewing, it is essential to coordinate with PyOpenSci. https://www.pyopensci.org
Two possible concerns:
- People don't really look up for packages in lists, they usually learn from them in tutorials.
- There is a security concern when including all the packages
We can make a distinction between "core packages" but having a list can be useful to teach people what is used by most of the people.
There is not a lot of documentation on how to use several packages together. -> This is the idea of the Pangeo inicitative: https://projectpythia.org
- Where should this information leave? -> Probably each domain should host information on their own domain but we could share tooling within the community in order to make it easier. Example: https://discourse.pangeo.io/t/statement-of-need-integrating-jupyterbook-and-jupyterhubs-via-ci/2705
List of tools developed and used by several packages that we can help coordinate:
- Memory flags, DevOps https://github.com/astropy/astropy-project/issues/118 (Astropy)
- [Scikit-HEP cookie](https://github.com/scikit-hep/cookie), [developer pages](https://scikit-hep.org/developer), and [repo review](https://scikit-hep.org/developer/reporeview)
A need from the Scientific Python community: Link to legacy code-> So it would be useful to have a centralize place to point to this information.
- Organize a sprint to build the pages (we can work on some infrastructure before)
## Chat transcript
**08:06:34** From Henry Schreiner : Just figured out that I had no sound, sorry! Fixed.
**08:07:49** From pllim : Pey Lian
**08:10:08** From Juanita Gomez : I’ll be taking notes here under "Meeting 1”: https://hackmd.io/1wioifmCTY2UVmGT-jcZsg. Feel free to add anything there!
**08:10:32** From Isaac Virshup : Do we need to come back to jim?
**08:10:48** From Pamphile Roy : https://hackmd.io/1wioifmCTY2UVmGT-jcZsg
**08:10:55** From Pamphile Roy : Remove the dot :)
**08:11:06** From Juanita Gomez : oops
**08:12:54** From Henry Schreiner : It worked for me with the dot too, it was ignored when I clicked the link ¯\_(ツ)_/¯
**08:15:21** From Levi John Wolf (he/him) : my apologies for arriving in late! Levi Wolf w/ PySAL & Geopandas.
**08:16:52** From Jim Pivarski : "Stack" vs "framework"? (There's the VanderPlas layers diagram for illustrating stacks.)
**08:17:01** From pllim : https://www.astropy.org/affiliated/
**08:19:14** From Ryan’s iPhone : On the topic of reviewing, it is essential to coordinate with PyOpenSci.
**08:19:29** From Stefan van der Walt : Definitely; we will be talking to Leah.
**08:19:34** From Ryan’s iPhone : https://twitter.com/leahawasser/status/1565339596132560896?s=21&t=Ayow8xbhW4W-hvp2UB8-5Q
**08:29:55** From Pamphile Roy : Sorry I will have to jump out, something came out now. I might be able to jump back again, but in any case I will watch the recording. Thanks all, great to see everyone 😃
**08:36:29** From Ryan’s iPhone : Agree security is important. But also probably beyond the scope of what we can accomplish here. Companies like anaconda have big businesses about providing security guarantees for open source package. Hard to match that from a volunteer effort.
**08:39:58** From Ryan’s iPhone : https://projectpythia.org/
**08:43:15** From Ryan’s iPhone : On the topic of notebook build processes https://discourse.pangeo.io/t/statement-of-need-integrating-jupyterbook-and-jupyterhubs-via-ci/2705
**08:45:44** From pllim : How widely can this meeting's recording and notes be shared? Am I allowed to share this with Astropy Coordination Committee members (5 people)?
**08:46:45** From Stefan van der Walt : Sure, we'll post on YouTube so others can watch and comment.
**08:48:36** From Jim Pivarski : And much of that is broader than our HEP domain.
**08:51:18** From pllim : The diagram here gives you an idea on the Astropy side though it is a little outdated by now (e.g., we no longer use Travis CI) -- https://github.com/astropy/astropy-project/issues/118
**08:52:00** From Ryan’s iPhone : 👍 that cookie cutter sounds super interesting.
**08:52:06** From Isaac Virshup : Minor tangent: does anyone have a process for what happens when you update your cookie-cutter? E.g. how do users update to the newer standards?
**08:53:19** From Henry Schreiner : https://scikit-hep.org/developer & https://github.com/scikit-hep/cookie