[Back to PulpCon2023 Schedule](https://hackmd.io/@pulp/pulpcon_2023)

# Pulp Sub-repository discussion session

Slot: Thursday, November 9 (day #4) 9:00am EST, 15:00 CET

Speaker: Quirin Pamp
GitHub/Matrix: quba42
Company: [ATIX](https://atix.de/en/)
Role: pulp_deb maintainer

## Problem Statement

* Some content types may have meaningful subsets within a given repository.
* The Pulp plugin must record not only what repository a particular content unit is stored in (handled by pulpcore using `RepositoryContent`), but also what subset or grouping within that repository.
* Exception: Perhaps there is some natural way to querry for the subset?
* e.g., by label, by some subfield, etc
* e.g.: APT repo component or package index; RPM modulestream; RPM kickstart-tree repos

### Additional Considerations

* Various actions might target the subset rather than the repository as a whole whereas pulpcore functionality almost always operates on the repository version level.
* There is (is there?) a hierarchical organization/grouping implied

## How pulp_deb handles this to date

* Today pulp_deb uses `PackageReleaseComponent` (PRC) content to record which `Package` is present in what `ReleaseComponent` (within a particular repository version).
* Essentially just a table with two foreign keys.
* Also inherits from `Content`
* No associated artifacts

### Problems and limitations

* Doubles the amount of content (per repository version)
* For each Package + PRC combination we record the following:
* pulpcore `RepositoryContent`: This package is in this repository version
* pulpcore `RepositoryContent`: This PRC is in this repository version
* pulp_deb `PackageReleaseComponent`: This package is in this `ReleaseComponent`
* Handling repo version consistency, e.g.: What happens with a PRC when a package is removed from the repository?
* This exposes what should be pulp_deb plumbing to users (who should not have to care about understanding what a `PackageReleaseComponent` is).

=> This has proven to be the main obstacle for improving pulp_deb efficiency.

## How a sub-repository might work

* Each sub-repository represents one `ReleaseComponent`.
* `Packages` are contained in the sub-repositories.
* The parent repository does not contain any `Packages` directly.
* Instead, it contains a `ReleaseComponent`, that points at a particular sub-repository version.
* Both parent and sub-repository are independently versioned.

### How would this solve the problem?

* We would be back down to one pulpcore `RepositoryContent` record per `Package` in the APT repo.

### Open questions and challenges

* Can multiple parent repositories reference the same sub-repository? (I think not)
* concurrency/deadlocks have entered the room...
* How do users interact with the sub-repositories?
* For example: Does the user "add a package" to the subrepository, and then ask the parent repository to update its sub-repository version reference, or is there an API to "add a package" directly to the parent repository which then performs everything that is needed under the hood? Both?
* user needs to know distribution and component-in-that-distro, at create time
* prob should be user-required, no defaults
* How to handle orphaned sub-repositories? On delete cascade?
* Handling repo uniqueness constraints would get more complicated.
* What happens when a sync fails for one sub-repository, but the other sub-repositories have already saved a new version?
* Into the APT repo weeds: Fot optimize sync it would be better to have one sub-repository per package index, rather than one per release-component. However, this strikes me as not very intuitive and then there is the case of architecture=all indices...

## Discussion

* How feasible would it be to introduce a "Sub-repository" concept into Pulp?
* Is there demand for something like this outside of pulp_deb?
* Pitfalls, concerns, alternatives?

### Alternative: Just have the reference on the Package model

* The `Package` model could have a component reference on it.
* This would duplicate packages in the DB when the same package exists in different components (but not the artifacts).
* Con: Feels like we are not "cutting nature at the joints"
* Pro: Might be simpler than the sub-repository approach and also gets rid of PackageReleaseComponents

### Notes
* is "repository" the right word to use for this grouping concept?
* it's internal-only, so does it matter?
* "human beings" need to understand this architecture, and "repository" comes with semantics that we may/do not want/need here
* what is the/a "natural key" of such a grouping?
* what about immutability of repo-version, when a contained-object might change?
* if the sub-repo-version is identified as Content, that belongs-to a Real Repository, that should maintain "version must be immutable"