# Python Packaging
[TOC]
## 87. Section Intro - Python Packaging
- Why packaging?
- 1.Distributing your code
- 2.Non-painful import statements
- 3.Reproducibility
- it work on my machine meme
## 88. PYTHONPATH and imports in Python
```
/home/eric/repos/packaging/
├── my_folder/
│ ├── __init__.py
│ └── my_nested_file.py
├── my_file.py
└── my_other_file.py
```
```python
$PYTHONPATH:/home/eric/repos/packaging python my_folder/my_nested_file.py
```
臨時設定Python模組的搜尋路徑,在不打包的情況下,import寫在某個路徑下的模組
```my_nested_file.py```
```python
from my_otherfile import CONSTANT as CONSTANT2
```
## 89. PYTHONPATH hacking (continued)
- 加入 sys.path
```python=
sys.path.inert(0,"/home/eric/repos")
```
## 90. Our first setup.py file
### 1.Module（模組）
- 單一 .py 檔案
```python
my_module.py
```
- 匯入
```python
import my_module
```
### 2.Package（套件）
- 一個有 __init__.py 的資料夾。一個可被匯入模組集合。
```python
my_package/
├── __init__.py
├── module1.py
└── module2.py
```
- 匯入
```python
from my_package import module1
```
### 3.Sub-package（子套件）
- 套件裡面的另一個套件(有 __init__.py 的資料夾)。
```python
my_package/
├── __init__.py
├── module1.py
└── sub_package/
├── __init__.py
└── module2.py
```
- 匯入
```python
from my_package.sub_package import module2
```
### 4.Distribution Package（發佈套件）
把你的模組/套件"打包成一個可安裝的壓縮包"的版本，可以上傳到PyPI或透過pip安裝。
包含：
setup.py 或 pyproject.toml
原始碼（模組/套件）
README.md、LICENSE 等
|Name | Description | Example |
| -------- | -------- | -------- |
Module | 單一 .py 檔案 | XX_module.py
Package | 有 __init__.py 的資料夾 | XX_package/
Sub-package | Package 裡的子資料夾 | XX_package/sub_package/
Distribution Package | 可被 pip 安裝的壓縮包 | XX_project-0.1.0.tar.gz
- 環境預設有package
- pip
- setuptools
```python
from setuptools import setup, find_packages
setup(
name='my_project', # pip install XXX
version='0.1.0', # 版本
packages=find_packages(), # 自動找所有 __init__.py folder
install_requires=[ # 相依套件
'numpy>=1.20.0',
'pandas',
],
author='XXX',
description='A simple example Python package',
license='MIT',
)
```
- document
```python=
python setup.py --help
```
```python=
python setup.py build sdist
# creating a "source" distribution of a package
# 建立 setup.py 的設定來建立一個 原始碼壓縮包,方便別人安裝或上傳到 PyPI
```
https://docs.python.org/3.10/distutils/sourcedist.html
output:
- *.egg-info/（紀錄metadata）
- build folder
- dist folder -> .tar.gz
- 打包安裝後，你就不需要再手動設 PYTHONPATH 來執行模組
## 91. setuptools docs; find_packages
- 解壓縮
```python
pip install ./dist/packaging-0.0.0.tar.gz
```
- find_packages
- 自動尋找Python套件中所有包含 __init__.py 的資料夾
- https://setuptools.pypa.io/en/latest/userguide/quickstart.html#dependency-management
## 92. pip install --editable
- 傳統安裝pip install . 安裝後改程式碼要重新安裝,用來打包發佈
- pip install -e .(pip install --editable)
## 93. install_requires and reproducibility
- install_requires 相依套件
## 94. Shortcomings of sdist format
- 產生Python套件的分發格式（distribution formats）
| 類型 | sdist (.tar.gz) | wheel (.whl)|
| -------- | -------- | -------- |
格式全名| Source Distribution |Binary Wheel（PEP 427 定義）
內容物| 原始碼 + setup.py |已編譯好的檔案 + metadata
用途 | 提供源碼給 pip 來編譯安裝 | 提供預編譯檔案直接安裝
是否需要編譯器（EX:gcc） | 可能需要 | 不需要
pip 安裝速度 | 慢 | 快
適用對象 | 開發者 | 使用者
## 95. Wheels
- install https://pypi.org/project/wheel/
```python
pip install wheel
```
- 用setup.py建立.whl格式
```python
python setup.py bdist_wheel
```
## 96. pyproject.toml origin story; Build dependencies explained
```python=
setup_requires=[
"wheel"
]
```
```toml=
# pyproject.toml
[build-system]
requires = ["wheel"]
```
## 97/98/99. Escaping config file hell; setup.cfg/Removing setup.cfg thanks to PEP 621/Removing setup.py; PEP 517; Build backends
|setup.py |pyproject.toml |setup.cfg |
| -------- | -------- | -------- |
python code | toml格式設定檔 | ini格式設定檔
傳統，定義專案的安裝、依賴等 | 用來統一所有的配置項 | setuptools 用來配置和定義版本、依賴等的文件
- 1. ```setup.py``` => ```pyproject.toml```
```python=
from setuptools import setup
setup(
name='my_project',
version='0.1',
packages=['my_package'],
install_requires=[
'numpy',
'flask'
],
extras_require={
'dev': ['pytest', 'sphinx'],
'docs': ['sphinx']
}
)
```
- 2.```setup.cfg``` => ```pyproject.toml```
```ini=
[metadata]
name = my_project
version = 0.1
[options]
packages = find:
install_requires =
numpy
flask
[options.extras_require]
dev =
pytest
sphinx
docs =
sphinx
```
- output:
```TOML=
[project]
name = "my_project"
version = "0.1"
dependencies = [
"numpy",
"flask"
]
[project.optional-dependencies]
dev = ["pytest", "sphinx"]
docs = ["sphinx"]
[build-system]
requires = ["setuptools>=42"]
build-backend = "setuptools.build_meta"
```
## 100/101. Data files - part 1; example use case for including data/ Part 2; MANIFEST.in and build-backend-specific alternatives
- MANIFEST.in 配置文件
- 讓 setuptools、distutils 或其他打包工具，哪些文件需要被包含/排除進 package。
```
recursive-include data *.json #recursive-include
```
## 102. Dependency management; Dependency graphs and locking dependencies
## 103. Dependency graphs
- 列出目前環境已安裝的套件，顯示依賴關係的工具(安裝[pipdeptree](https://pypi.org/project/pipdeptree/))
```
pip install pipdeptree
```
- 查看
```
pipdeptree
```
```
xgboost==2.1.0
├── numpy [required: Any, installed: 2.0.0]
└── scipy [required: Any, installed: 1.14.0]
└── numpy [required: >=1.23.5,<2.3, installed: 2.0.0]
```
- 反向查詢(--reverse)，哪些package需要這個package作為依賴。
```
pipdeptree --reverse --packages numpy
```
```
numpy==2.0.0
├── contourpy==1.3.1 [requires: numpy>=1.23]
│ └── matplotlib==3.10.1 [requires: contourpy>=1.0.1]
├── matplotlib==3.10.1 [requires: numpy>=1.23]
├── pandas==2.2.2 [requires: numpy>=1.26.0]
│ └── streamlit==1.43.2 [requires: pandas>=1.4.0,<3]
├── pydeck==0.9.1 [requires: numpy>=1.16.4]
│ └── streamlit==1.43.2 [requires: pydeck>=0.8.0b4,<1]
├── scikit-learn==1.5.1 [requires: numpy>=1.19.5]
├── scipy==1.14.0 [requires: numpy>=1.23.5,<2.3]
│ ├── scikit-learn==1.5.1 [requires: scipy>=1.6.0]
│ └── xgboost==2.1.0 [requires: scipy]
├── streamlit==1.43.2 [requires: numpy>=1.23,<3]
└── xgboost==2.1.0 [requires: numpy]
```
- graphviz 畫圖
- Constraint tree(約束樹):
安裝每個套件時，會帶來它需要的其他套件(dependencies)和對這些套件版本的要求(constraints)
```
A 需要 B >=1.0, <2.0
C 需要 B >=1.5, <1.6
```
## 104. Constraint trees and "dependency hell"
- Dependency Hell(依賴地獄):
當不同套件對同一依賴要求版本互相衝突，導致根本無法同時安裝/或者要調整才能安裝
## 105. Constraint trees are unstable across package versions
- 避免
- 1.虛擬環境（Virtual Environment）
- 2.鎖定套件版本（用 requirements.txt 或 pyproject.toml + poetry.lock）
## 106. Optional dependencies AKA "extras"
- 選擇性的額外依賴（optional dependencies）
- 某些功能，才需要的額外套件, PEP 621, 用 [project.optional-dependencies] 在 pyproject.toml 裡
```
[project.optional-dependencies]
dev = [
"pytest>=6.0",
"black>=22.3",
"mypy>=1.0",
]
viz = [
"matplotlib>=3.5",
"seaborn>=0.12",
]
aws = [
"boto3>=1.26"
]
```
```
pip install .[dev]
```
## 107. Shopping for dependencies with the Snyk Python package index
- 套件健檢 [Snyk](https://snyk.io/advisor/)
- package health
```
pip install snyk
```
登入
[snyk document](https://docs.snyk.io/)
```
snyk auth
```
掃描你的專案中所有的```requirements.txt```、```pyproject.toml```等檔案
```
snyk test --all-projects
```
Sign in with Wallet
Connect another wallet