# `Podcast.__init__` Notes

Episode topic: About their work to upgrade the dependency resolution capabilities of pip and improve its user experience

### Interview

- Q: Introductions
- Paul: pip maintainer, CPython Core Developer, BDFL-Delegate (and what that means). Contributor to various things in the past (Vim, Boost).
- Pradyun: dumb college kid. pip maintainer. PyPI moderator. TOML Core dev.
- Tzu-Ping: MacDown. Freelancer based on Taiwan. PyCon Taiwan organiser. pipenv maintainer.

- Q: How did you get introduced to Python?
- Paul:
- Porting stuff to the ACORN
- Ported perl once. :)
- Horrible port, but I like this thing I built.
- Didn't look at Python until got a PC, stuff just worked.
- Frustrated w/ Perl's symbols,
- Pradyun: Dad gave me a book, had too much free time as a kid, learnt Python from the book. Then GSoC happened.
- Tzu-Ping: Started programming in college, and got a iBook G4 -- no Mac programs at that time, so I started writing some. Joined as an iOS dev -- the web-dev person left. Boss went -- hey, you know Django? -- I can try. And... now 70% Python, 0% iOS.

- Q: Can you start by describing the focus of the work that you are doing?

- Replacing the existing resolver, which is... broken in some ways.
- Cover the UX work here.
- What is the scope of the work, and what is the established criteria for when it is considered complete?
- resolver: writing the replacement, getting it feature complete, and rolling it out
- UX: collecting data and analysis which will be used to improve the CLI and messaging & error reporting

- Q: What is your history with working on the pip source code and what interests you most about this project?

- Speak in turn: Paul, Pradyun, TP

- Q: What are the main sources or manifestations of technical debt that exist in pip as of today?

- Pradyun will take this and run with it!
- Originally designed to install everything from source. Lot's of code is for building from source.
- Shifting to wheels + static metadata and more. There is still a transition within the code, even though user facing changes have occured.
- NOT "we're fighting a dragon in there"
- Unifying the "code flow" for various user workflows.
- `RequirementSet`, `InstallRequirement`
- Q: How does it currently handle dependency resolution?
- Pradyun wants to take this and run with it!
- Poorly. :)
- Optimistically.
- "first come first serve" -- assuming that the first choice "first found, wins"

- Q: What are some of the workarounds that developers have had to resort to in the absence of a robust dependency resolver in pip?
- no-deps, pip-tools, poetry
- explicitly add version ranges (or pin version) to requirements so pip does the right thing
- pinning in a library (which is bad).
- adding top level requirements

- Q: How is the new dependency resolver implemented?
- Paul + Tzu-Ping to take this and run with it!
- Paul describing dependency resolution in general, and how Python packaging makes it awkward in certain ways
- how pip is unique: not fully-visible metadata, SAT in pip's use case
- TP describes how ResolveLib came about

- Q: How has your initial design evolved or shifted as you have gotten further along in its implementation?
- Starts with TP, Paul and Pradyun fill in is missing something
- Reporting progress during dependency resolution, presenting error messages / handling errors during resolution.
- Adding pip's features (extras, constraints, etc) wasn't a major problem - tricky, yes - but resolvelib was flexible enough to cover it.

- Q: What are the pieces of information that the resolver will rely on for determining which packages and versions to install? (e.g. will it install setuptools > 45.x in a Python 2 virtualenv?)
- Paul to cover this.
- Distinguish metadata, data in the index (data-requires-python), and wheel tags.
- Name, version, dependency data. Generated from source code.
- Python version restriction (Requires-Python vs wheel tags)
- New resolver fixes the fact that old resolver would work out it was for Python 2 too late.
- Backtracks, rather than just ploughing on and failing

- Q: What are the new capabilities in pip that will be enabled by this upgrade to the dependency resolver?
- Fixing things that were broken. Enables better "environment management" in other tools?
- Library implementation is more shareable, people have other options.
- Simplifying pip-tools -- .
- Easier for Python projects to have shared implementations for dependency resolvers.

- Q: What projects or features in the encompassing ecosystem will be unblocked with the introduction of this upgrade?
- Adding an upgrade-all command.
- Example of funded work helping: encouraging other projects in the ecosystem, to seek funding for long-standing high-impact issues, to get things over the final few hurdles?
- https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Finish_dependency_resolver_for_pip

- Q: What are some of the changes that users will need to make to adopt the updated pip?
- Right now: --unstable-feature
- Eventually: have it by default, "we aim to roll it out later this year."
- Be more careful about dependencies -- ensure current environments are ready with `pip check`.
- pinning in a library is bad.
- But still put in sensible ranges
- Part of the project is to sort out the rollout.

- Q: How do you anticipate these changes in pip, impacting the viability or adoption of Python and its ecosystem within different communities or industries?
- Paul!
- The other way around: Python dependency resolution can be a problem (known in certain circles)
- Possible a better resolver can encourage people to choose Python
- Having good dependency resolution in pip emphasise the value of good, consistent metadata
- Projects would be encouraged to provide such if they can benefit from them

- Q: What are some of the additional changes or improvements that you would like to see in pip or other core elements of the Python landscape?
- Moar static metadata! Other languages don't suffer from the level of dynamic data Python does...
- (good chance to talk about how we handle interoperability standards, and why)
- Standardizing a way for pip to get dependency information from a package index.
- Pradyun: Create reusable libraries for package management workflows (building logic, install logic etc)
- Like ResolveLib

- Q: What are some of the most interesting, unexpected, or challenging lessons that you have learned while working on these updates to pip?
- **take turns!**
- first ever funded work in pip!?
- pip has no resources at all.
- Compare to package managers in other popular langauges -- npm (JS), cargo (Rust), pub (dart), bundler (Ruby).
- PSF's PackagingWG is doing great work in this front!
- It’s a pain to continue supporting both Python 2 and 3 at the same time. We're dropping support for Python 2 from pip in Jan 2021, but we're implementing the resolver with Python 2 compatibility.
- Really hard to know all the ways people use pip :-(

- Paul+Pradyun: It feels way different working on pip as paid-for-time vs volunteered time (should I watch TV or play a game or work on pip?)
- Paul
- more focused, less interruptions
- tend to get less distracted since being paid bring a certain amount of work ethic
- contrasting with PEP 517 implementation
- hey, this is cool, I'm getting paid to do my hobby -- I'm actually doing work, and not my hobby makes tackling difficult problems easier.
- first time getting paid for Python stuff.
- Pradyun:
- first time getting paid. :)
- much faster feedback loops compared to when we only had volunteers.
- face to face time
- Tzu-Ping: Joining pip dev as a paid-for-time.
- Actually my first full-remote project
- pip as a project is managed even more differently
- Much slower feedback loops
- Need to be more indenpendence and self-concious, push things much harder