# pulp_rpm, distributiontrees, subrepos, and concurrency

## Problem
* Concurrent syncs of repos with overlapping remotes containing treeinfos with sub-repos (addons/variants), fails

## Immediate Problem
* sub-repos are named as (addon-id)-(treeinfo-hash). With overlapping content, multiple workers can try to create/sync the same sub-repo at the same time, causing never-happen collisions in naming/version-creation

## Attempted fixes

### subrepo-name-uniqeness
* fixes immediate problem
* works for not-autopublish
* breaks autopublish workflow
* leaves N-1 "orphaned" sub-repositories that can only be cleaned up "by hand"
### ignoring failures
* solves immediate problem
* breaks mirror
### subrepo-naming, publishing only from 'final' DistTree
* combines the failures of the previous two

## Architectural Collision
* DistributionTree is shared content
* Addons/Variants included in repos via their associated DistributionTree
* Sub-repo content included via Addon/Variant
* Addon/Variant only know one repository
* mirror/autopublish attempt to "do things" **inside** the worker that is doing the sync
* mirror/autopublish code lives in a number of places, with subtle, interrelated, implications
* mirror/autopublish **DO NOT** like concurrent operations on the "same" entity

## Possible solution #1
* subrepo-unique-naming
* autopublish becomes a dispatched Task, to be handled by the normal task-scheduling (ie, not from 'inside' the sync-worker)
* orphaned-subrepo-cleanup also becomes a dispatched Task, dispatched per-repo-with-subrepos
* Pros:
* order-dependency and concurrency protected by the tasking system
* Cons:
* WAY more complicated workflow
* Changes current "how to know your publish is done" semantics

## Possible solution #2
* make DistributionTree unique PER REPO
* e.g., "digest" = hash+repo-id
* make subrepos unique PER REPO
* see subrepo-unique-naming
* Pros:
* DistTree and its subrepos now cannot collide across repositories
* concurrent-sync of same-repo handled by tasking
* preserves current sync-task semantics
* preserves current workflow
* Cons:
* Duplicated DistTrees and Subrepos will sync, where current situation would skip them as "already done"

## Conclusion
* Solution #2 works and is minimally invasive
* The expense of syncing subrepos "more often than necessary" is a small price to pay for "works"
* in Pulp4, subrepos need to be first-class objects
* in Pulp4, subsystems need to let the tasking system enforce resource-management and ordering, instead of trying to force complicated orchestration "inside" already-dispatched tasks

### What if you have "old way" DistributionTrees? Don't we need a migration?

No. Current repo-versions continue to work just fine. Next time you sync that repo, it'll create a new version when it otherwise might not have, with the new-naming-scheme DistTree and subrepos. From the outside, it's Business As Usual.

### Does Anything Get Left Behind?

No. Unlike "subrepo-naming-uniqueness", above, this approach keeps everything linked at a repo-version. This means that the now-unique DistTree/Addon/Image/Variant/Subrepos will all be available to orphan-cleanup if/when any containing-repo-version has been deleted.

### Wait - what about copies?
If/when you copy a DistTree from repo-A to repo-B, you'll end up with both repo-A-version and repo-B-version pointing to the same DistTree object/family - i.e., you'll put yourself into the same situation we are in now.

If you do this into multiple repos concurrently, and attempt to publish concurrently, you will end up In Trouble, in the same way you do now.

If you do *not* attempt to publish concurrently, it'll work just as it does today.

Once you've somehow Gotten Away With It - an external recipient syncing such repositories will be creating its own copies, and won't care.

Conclusion: concurrent-copy of DistTrees isn't made worse by this change.

## Links
* [2103246](https://bugzilla.redhat.com/show_bug.cgi?id=2103246)
* [2077363](https://bugzilla.redhat.com/show_bug.cgi?id=2077363)
* [2278](https://github.com/pulp/pulp_rpm/issues/2278) fail on concurrent-ks-repo-syncs
* [2275](https://github.com/pulp/pulp_rpm/issues/2775) Same as 2278, diff symptom (CLOSED as dup of 2278)
* [2304](https://github.com/pulp/pulp_rpm/issues/2304) subrepos need better keys (CLOSED as dup of 2278)