## Section 12: Intro to Continuous Delivery — Publishing to PyPI
- [108. Section introduction](#108.-Section-introduction)
- [109. DevOps and Waterfall vs. Agile](#109.-DevOps-and-Waterfall-vs.-Agile)
- [110. Publishing to PyPI; Test vs. Prod PyPI](#110.-Publishing-to-PyPI;-Test-vs.-Prod-PyPI)
- [111. Generating API tokens for PyPI](#111.-Generating-API-tokens-for-PyPI)
- [112. Running twine upload](#112.-Running-twine-upload)
- [113. Task runners - part 1 - CMake and Makefile](#113.-Task-runners---part-1---CMake-and-Makefile)
- [114. Task runners - part 2 - Alternatives to CMake (Just, PyInvoke, Bash)](#114.-Task-runners---part-2---Alternatives-to-CMake-(Just,-PyInvoke,-Bash))
- [115. Task runners - part 3 - Bash as a task runner](#115.-Task-runners---part-3---Bash-as-a-task-runner)
- [116. Publishing to TestPyPI](#116.-Publishing-to-TestPyPI)
- [117. Publishing to Prod PyPI; Why we should move deployment to GitHub Actions](#117.-Publishing-to-Prod-PyPI;-Why-we-should-move-deployment-to-GitHub-Actions)
## 108. Section introduction
- **DevOps** = Software Development + IT Operations
- **Goal**: Build and deliver software efficiently
- **Key DevOps practices**:
- Trunk-based development (frequent, small commits)
- Small PRs (if needed)
- CI/CD pipelines
- **Course Goal:**
- You should be equipped as a **DevOps engineer**
- You’ll understand not only software development, but also **publishing and delivery workflows**
- Example implementation: publish a Python package to PyPI

## 109. DevOps and Waterfall vs. Agile
- **Waterfall Project Management**
- Traditional method used in early software development.
- **Problem**: Doesn't handle feedback or uncertainty well.
- Often results in **building the wrong product** due to inaccurate user assumptions.
- **Agile Project Management**
- **Modern alternative** to Waterfall.
- Converts the linear waterfall steps into a **repeating cycle**.
- **Validated learning**: Ship fast, get feedback, adjust direction.
- **Outcome**: Avoids wasting time on unused or irrelevant features.
- **Agile in Practice**
- Common Agile frameworks:
- **Scrum**: 4-week development cycles (sprints) with planning, daily standups, retrospectives.
- **Kanban**: Visual task tracking with "To Do", "In Progress", "Testing", "Done" columns.
- Tools:
- **Jira** for task management (controversial).
- **GitHub Actions**, **Jenkins**, **Bamboo** for automation.
- **DevOps Infinity Loop**
- DevOps is about **fast, reliable, and repeatable software delivery**.
- Loop includes:
- **Plan → Code → Build → Test → Release → Deploy → Operate → Monitor**
- Loop repeats continuously.
- **Goal**: Deliver small changes often, efficiently, and with quality.
- **DevOps vs Agile**
- **Agile** = methodology for building the right thing through iteration.
- **DevOps** = practice for building and shipping software efficiently.
- **CI + CD** are part of DevOps.
- GitHub Actions help automate both development (left side) and delivery (right side).

## 110. Publishing to PyPI; Test vs. Prod PyPI
- **What is PyPI?**
- **PyPI** stands for *Python Package Index* – the central repository where Python packages are published and shared.
- Developers and users can browse, download, and install packages via `pip`.
- **How PyPI Versions Work**
- Each release on PyPI is **immutable**:
- Once a version (e.g., `1.0.0`) is published, it **cannot be replaced**.
- If there's an issue, the version must be **yanked** (marked as unusable), not overwritten.
- **Yanked versions**:
- **A yanked version on PyPI is a released version of a Python package that is marked as not recommended for use, but not completely deleted.**
- Used when a release has major issues (e.g., bugs, security vulnerabilities).
- Still listed in release history, but **no longer installable via pip**.

- **Your Deployment Strategy**
1. **First deploy to Test PyPI** to ensure the package uploads and installs correctly.
2. **Only after success**, publish to the real(produciton) PyPI.
3. Avoid releasing broken packages and filling your release history with yanked versions.

## 111. Generating API tokens for PyPI
- **Step 1: Register Accounts**
- **Register separately** on:
- [https://test.pypi.org](https://test.pypi.org/) (TestPyPI)
- [https://pypi.org](https://pypi.org/) (Production PyPI)

- **Step 2: This Is Important**
- You **cannot proceed** with package publishing steps unless you:
- Have valid credentials.
- Have generated **API tokens** for each environment.
- These tokens are required for **automated publishing pipelines**.

- **Step 3: Generate API Token on TestPyPI**
- Log in to your [TestPyPI account](https://test.pypi.org/manage/account/).
- Go to **Account Settings → API Tokens → Add API Token**.
- Be aware:
- **Broad scope tokens** are powerful – if leaked, they compromise *all* your packages.
- For sensitive environments (e.g., companies), use private PyPI or SSH installs.

- **Step 4: Secure Your Token**
- Save the token in a **secure, private location**:
- Examples: Password manager (e.g., LastPass secure note), `.env` file (excluded from Git), or secret vault.
- **Do NOT store the token in your Git repository.**

- **Step 5: Using the Token**
- Tool used: **twine** (officially recommended for package publishing).
- Format for authentication:
- **Username**: `__token__`
- **Password**: Your generated token

## 112. Running twine upload
- **Tools Used**
- [**twine**](https://twine.readthedocs.io/) – recommended CLI for uploading Python packages to PyPI/TestPyPI.
- [**build**](https://pypa-build.readthedocs.io/) – used to build source distributions (`.tar.gz`) and wheels (`.whl`).
- ### **Steps to Publish with twine**
1. **Install twine**

        pip install twine

2. **Clean `dist/` folder** (optional but recommended)

        rm -rf dist/

3. **Build the package**

        pip install build
        python -m build

- This creates `.tar.gz` and `.whl` files in the `dist/` folder.
4. **Upload to TestPyPI**

        twine upload --repository testpypi dist/*

- When prompted:
- **Username**: `__token__`
- **Password**: Your **Test PyPI API token**

- **Common Issues**
- **Authentication prompt**: Manual input of token is required here, but **not suitable for automation**.
- **Package name conflict**: If a package with your name already exists on PyPI/TestPyPI, you'll get an error (e.g., “not allowed to upload to project”).
- **Solution**: Use a **globally unique** package name.
- **Tip**: Check availability on both [PyPI](https://pypi.org/) and [TestPyPI](https://test.pypi.org/).

- **Recommended Practices**
- For official guidance, consult: [https://packaging.python.org](https://packaging.python.org/)
- This site reflects the latest **PEPs** and canonical Python packaging practices.

## 113. Task runners - part 1 - CMake and Makefile
- **Why We Need a Task Runner**
- **Definition**: **Task runners are tools that automate repetitive development tasks, such as testing, building, linting, or packaging your code**.
- Projects often rely on many CLI tools (e.g., `pre-commit`, `flake8`, `pytest`, `mypy`, `twine`, `build`, `coverage`, `sphinx`, `docker`, etc.).
- Each tool has **many arguments and config options**, making onboarding difficult.
- Task runners help:
- **Document commonly used commands**
- **Reduce cognitive load** for contributors
- **Standardize development workflows**
- Simplify **open source contributions** and **internal team adoption**

- **What is a Makefile?**
- A `Makefile` is a widely used **task runner** that allows developers to define CLI tasks.
- Syntax:

        Makefile
        install:
            echo hello

Run with:

        make install

- **Advantages of Makefile**
- **Portable**: Preinstalled or easily installable on most Linux/macOS systems.
- Can be run with a simple `make <task>` command.
- Useful for simple, one-liner development tasks.
- **Reproducibility**: Helps set up consistent dev environments with commands like `make install`.

- **Drawbacks of Makefile**
- **Tab-sensitive syntax**: Must use tabs (not spaces) to indent task commands.
- Causes confusing errors like `missing separator`.
- **Difficult variable handling**:
- Variables can behave unpredictably.
- Variables declared inside task bodies don’t persist across lines.
- **Poor Bash compatibility**:
- Not actually Bash—defaults to `sh` which lacks common features.
- Multi-line commands, conditionals, and pipelines are fragile.
- **Limited support for logic and expressions**:
- Complex logic becomes hard to maintain or debug.
- **Weak error messages** and steep learning curve.
- Bash already has enough gotchas—Makefile adds more on top.
- **Summary**
- Keep Bash scripts **simple** and avoid clever tricks.
- Use `Makefile` for **simple, portable, reproducible** tasks.
- **For anything more complex, consider using other task runners or Bash scripts**.

## 114. Task runners - part 2 - Alternatives to CMake (Just, PyInvoke, Bash)
- **Problems with Makefiles**
- **Hard to learn** and full of idiosyncrasies.
- **Can't accept arguments** for targets (e.g., `make test myfile.py` isn't supported).
- Complex logic (variables, conditionals, multiline commands) is **fragile** and error-prone.
- Still, it’s:
- **Extremely portable**
- Comes pre-installed on most UNIX-like systems
- Supports **dependency graphs** (e.g., `build: install` runs `install` before `build`)

- **Alternatives to Makefiles**
- [**Just**](https://github.com/casey/just)
- [**Invoke**](pyinvoke.org)
- **Poetry / Tox / Earthly**

- **Drawbacks of All These Alternatives**
- Most require installation steps.
- Varying learning curves.
- None match the **universality** of `make`.

- **A Minimalist Approach: The "Task File"**
- Proposed by **Adrian Cooney** as a simple, Bash-based task runner.
- Just a single `.sh` script (`run.sh`) with named functions like:

        build() {
          python -m build
        }

        test() {
          pytest
        }

- To run a task: `./run.sh build` or `source run.sh && build`
- **Benefits**:
- **Highly portable**
- **No dependencies**
- Easy to understand, modify, and extend
- Works well with **GitHub Copilot** or other AI auto-completion

- **Conclusion**
- While tools like `make`, `just`, and `invoke` have strengths, **plain Bash may be best for simplicity and portability**.
- For this course, the instructor will use a **Bash-based `run.sh` task file** to manage project commands.

## 115. Task runners - part 3 - Bash as a task runner
- **Purpose of Task Runner (run.sh)**
- A **task runner** helps manage common development commands in one place.
- Prevents developers from memorizing or mistyping long CLI commands.
- Offers a portable, familiar way to manage tasks like `install`, `build`, `lint`, and `publish`.

- **Basic Setup of `run.sh`**
- Make the script executable: `chmod +x run.sh`.
- Use `"$@"` to **dynamically** pass command-line arguments to functions.
- Default function is set to `help`, showing available tasks if no input is given.

- **Script Improvements**
- Added `set -e`: causes the script to exit immediately if any command fails.
- Optionally add `set -x` to print every command before execution (useful for debugging).
- Replaced hardcoded `./` with `${DIR}` to ensure relative paths work even if script is run from elsewhere.

- **Tasks Implemented**
1. **`install`**
- Upgrades `pip`
- Installs project in editable mode with dev dependencies
- Uses `${DIR}` for consistent project root path
2. **`build`**
- Runs `python -m build --sdist --wheel` using the build CLI
3. **`publish:test`**
- Publishes to TestPyPI via `twine upload --repository testpypi dist/*`
- Still requires manual username/token entry
4. **`lint`**
- Runs `pre-commit run --all-files`
- Adds `pre-commit` to the `static-code` extras group in `pyproject.toml`

- **pyproject.toml Modifications**
- Introduced extra groups like `[project.optional-dependencies]` for:
- `test`
- `release` (includes `build`, `twine`)
- `static-code` (includes `pre-commit`)
- Combined all extras under `[dev]` for comprehensive local installs
- **Makefile Integration**
- Optional: create a minimal `Makefile` that wraps `run.sh` commands.

        install:
            bash run.sh install

- Provides familiar interface (`make install`) while delegating logic to `run.sh`.

- **Why This Works Well**
- Avoids Makefile limitations (e.g., lack of argument passing, tab sensitivity).
- Keeps commands organized, editable, and extensible.
- Users only need Bash — no new tooling or installs.

## 116. Publishing to TestPyPI
- **Objective**
- Automate publishing a Python package to **TestPyPI** without human input using a `run.sh` task runner.

- **twine Automation Setup**
- `twine upload` supports:
- `-username __token__`
- `-password YOUR_TOKEN`
- `-non-interactive` to disable CLI prompts.
- Use `dist/*` to upload both source and wheel distributions.
- Use backslashes for multi-line Bash commands (no trailing spaces!).

- **Avoid Hardcoding Secrets**
- Never hardcode tokens in scripts—**severe security risk**.
- Use environment variables to securely manage sensitive info.

- **Use `.env` File (Not Committed to Git)**
- Add sensitive tokens like:

        TEST_PYPI_TOKEN=your-token-here

- Ensure `.env` is ignored via `.gitignore`:

        *.env
        *env

- **Load `.env` in Bash Script**
- Use a Bash function:

        function load_dot_env {
            while read -r line; do export "$line"; done < <(grep -v '^#' "$DIR/.env")
        }

- Call `load_dot_env` before running `twine upload`.

- **Update `publish:test` Task**
- Add environment variable usage:

        --username __token__ \
        --password "$TEST_PYPI_TOKEN"

- Call `load_dot_env` before the command.

- **Add a `clean` Task**
- Removes build artifacts (`dist/`, `.egg-info`, `build/`) with:

        rm -rf dist/ build/ *.egg-info

- **Fix Version Conflicts**
- If PyPI returns a "file already exists" error:
- Update version in `pyproject.toml` (e.g., `0.0.0` → `0.0.1`)
- PyPI is **immutable** — you can’t overwrite published versions.

- **Add a `release:test` Pipeline**

- New function `release:test`:

        function release:test {
            run clean
            run build
            run publish:test
        }

- Combines cleanup, build, and upload into one command.

- **TestPyPI Success**
- Upon successful upload, a URL is printed to view the package on [Test PyPI](https://test.pypi.org/).

## 117. Publishing to Prod PyPI; Why we should move deployment to GitHub Actions
- **Add `publish:prod` Task**
- Copy `publish:test` and modify:
- Change repository to `pypi`
- Use `PROD_PYPI_TOKEN` from `.env`
- Chose not to generalize with arguments—favored **readability** over abstraction.

- **Add `release:prod` Pipeline**
- New `release:prod` task:
- Runs `release:test` (which includes: `lint`, `clean`, `build`, `publish:test`)
- Then runs `publish:prod`
- Version bumped to `0.0.3` to avoid PyPI immutability error.
- Successfully published to both **Test PyPI** and **Prod PyPI**.

- **Why This Matters**
- **Scripting releases** reduces reliance on manual, error-prone steps.
- Avoids situations where only one dev knows the deployment “magic.”
- Increases transparency and reliability.

- **Secret Sharing Best Practices**
- Do **not** commit `.env` to Git.
- Use a **shared password manager** (e.g., LastPass, 1Password, Vault).
- GitHub secrets are useful but **not ideal** as your only source of truth (you can’t view them once set).

- **Pitfalls of Local Deployment**
- Local publishing relies on:
- Having correct `.env`
- Being present (on vacation, sick, quit?)
- Breaks **auditing**, **team sharing**, and **security controls**.

- **Why Move to GitHub Actions**
- Ensures:
- **CI/CD consistency**
- **Logs and audit trails**
- **No lost secrets** when devs leave
- **Controlled rollouts** (with optional approvals)