relative_path problem

# relative_path problem ### Agenda * Existing design and the Problem * Potentially valid proposals and their pros/cons * Discussion and/or brainstorm more ideas ### Use cases * As a user, I can publish the same content in different repos under different relative paths. * As a user, I can mirror repository metadata ### Problem Currently, a "relative_path" is tied to ContentArtifact in Pulp. This means that either a content unit can't exist in two places within a repository or across repositories with different relative paths, or it has to be stored as two separate content units. This creates redundant data and potential confusion for users. As a specific example, we need to support mirroring content in pulp_rpm. Currently, for each location at which a single package is stored, we’ll need to create a content unit. We could end up with several records representing a single package. Users may be confused about why they see multiple records for a package and they may have trouble for example deciding which content unit to copy. ### Existing Design *Open in new tab to see a larger image.* ![current models](https://i.imgur.com/gyQ5i90.png) ### Potentially valid proposals #### pulp_deb / pulp_file way Have a relative_path as a part of a content unit uniqueness constraint. Solves the problem of having a content unit in two places within a repository - but not the problem of having many duplicate content units. Metadata mirroring is handled by having special content units for metadata storage. Pros: * no changes in pulpcore * plugin is in control if they need this feature or not * easy to implement Cons: * doesn't solve the content duplication concern * multi-artifact content can't leverage that * it's confusing for users to see a list of same content and choose which one to copy. #### Additive changes only Our existing data model stays the same, we just add the following model. RepositoryContentContentArtifact repo_content (models.ForeignKey): The associated RepositoryContent content_artifact (models.ForeignKey): The associated ContentArtifact relative_path (str): The relative path for this relationship unique_together = (repo_content, content_artifact, relative_path) *Open in new tab to see a larger image.* ![proposal3](https://i.imgur.com/sInWfpk.png) Metadata mirroring is handled by having special content units for metadata storage. Pros: * no change to existing relations * multi-artifact content is handled * avoids the duplicate content problem * general solution * simple data migration Cons: * one more table to deal with for every piece of content * performance impact? * complicated db schema * many places need adjustments (stages, content app, etc) * relative_path info is present in multiple places and somewhat duplicated * might need REST API changes * copy API is difficult, need to provide relative_path for each content or find a convention for defaults #### relative_path in PublishedArtifact only There is no work that we can usefully do at publish time for a mirrored RPM repo because the metadata and locations must be an ***exact*** replica of the original repository. Since all of this information is available to us at sync-time, the proposal in this case is to simply create a Publication (save PublishedArtifacts and PublishedMetadata) at sync time, directly. This allows to have the same content published in many locations without needing to temporarily store it somewhere else to use later during publish, avoiding any model changes. ##### Simple version Pros: * no model changes * no data migration * (mostly) avoids the duplicate content problem * avoids needing to have special models for mirroring metadata Cons: * Doesn't solve the problem in a generalized way, would still be on the plugin writer to implement * Doesn't solve the duplicate content problem for productID (we have a workaround for this currently) * RPM sync gets a complicated special case * Possibly some minor workflow differences due to no separate publish  ##### More complicated version (generalization) We could take this idea further and make it generalized, but it would have a lot of the same drawbacks as Proposal 2. Applying it outside of the mirrored metadata case means that you need to be concerned with copy, and the need to provide a location along with the content unit, or reading locations from existing PublishedArtifacts. It would also mean, probably, adding this feature to DeclarativeVersion somehow. Pros: * all of the above from previous * generalized, also applies to multi-artifact content Cons: * might need REST API changes * copy API is difficult, need to provide relative_path for each content or find a convention for defaults * copy might need to involve PublishedArtifacts (not universally, just for the plugins that utilize this) * big changes needed to stages & add/remove content ### Discussion Observation: There are really 2 different types of publishable repositories. 1. Repositories where there the layout matters, and there is one and *only* one appropriate layout for artifacts. * e.g. file repositories, "mirrored" RPM repositories * "relative_path" is absolute, it's the full path. * in a mirrored RPM repo, you can't move `packages/d/dog.rpm` to `packages/dog.rpm`, nor can you rearrange the structure of a file repo. * very little needs to happen at publish time because there's not much that you can change 2. Repositories where republishing in different layouts is possible, "layout doesn't matter". * e.g. standard RPM repositories and python repositories * "relative_path" is floating, it's really just the filename. The location of this file within the repository can vary depending on the publish-time options and it won't meaningfully affect the consumer. * a non-mirrored RPM repo can be published in different ways, you *can* move `packages/d/dog.rpm` to `packages/dog.rpm * same with Python repositories --- ### Considered Proposals #### Proposal I (skip, no solution for multi-artifact content) Create a RepositoryContentArtifact which maps ContentArtifact and its relative path. RepositoryContent has foreign key to a RepositoryContentArtifact instead of Content. RepositoryContentArtifact content_artifact (models.ForeignKey): The associated content artifact. relative_path (str): the relative path of an artifact in a repository RepositoryContent repo_content_artifact (models.ForeignKey): The associated RepositoryContentArtifact repository (models.ForeignKey): The associated repository. version_added (models.ForeignKey): The RepositoryVersion which added the referenced Content. version_removed (models.ForeignKey): The RepositoryVersion which removed the referenced Content. ##### Code Changes * Stages API: `Content` should no longer be associated with a `Repository`. `RepositoryContentArtifact` should be associated with a repository. Plugin writers need to create `DeclarativeRepositoryContentArtifacts`. Associating `Content` with a `Repository`, need to create a `RepositoryContentArtifact` for every `ContentArtifact` and associate the `RepositoryContentArtifact` with the `Repository`. * Content App: Need to update query to inspect `RepositoryContent.artifact.relative_path` directly and serve the associated `RepositoryContent.artifact.content_artifact` * Complex migration is needed ##### Questions? * How does this work with multi-artifact content?Q: how to list multi-artifact content? Seems like it has a lot of downsides and few if any upsides compared to other proposals. #### Proposal II (skip, no solution for multi-artifact content) Inspired by proposal I. Use `ContentArtifact.relative_path`. RepositoryContent has foreign key to ContentArtifact instead of Content. RepositoryContent content_artifact (models.ForeignKey): The associated ContentArtifact ### Q: how to list multi-artifact content - get all content through content_artifact and then show unique items? repository (models.ForeignKey): The associated repository. version_added (models.ForeignKey): The RepositoryVersion which added the referenced Content. version_removed (models.ForeignKey): The RepositoryVersion which removed the referenced Content. ##### Code Changes * Stages API * Content App * Complex migration is needed ##### Questions? * How does this work with multi-artifact content? * Performance concern to search content by a repo version #### Proposal III Everything stays the same, we just add the following model. RepositoryContentContentArtifact repo_content (models.ForeignKey): The associated RepositoryContent content_artifact (models.ForeignKey): The associated ContentArtifact relative_path (str): The relative path for this relationship unique_together = (repo_content, content_artifact, relative_path) *Open in new tab to see a larger image.* ![proposal3](https://i.imgur.com/sInWfpk.png) ##### Code Changes * Stages API * Content app * Copy workflow * Migration: no changes in existing models/tables ##### Questions * Does it create a circular reference? * Performance concerns? #### Proposal IV Observation: There are really 2 different types of publishable repositories. 1. Repositories where there the layout matters, and there is one and *only* one appropriate layout for artifacts. * e.g. file repositories, "mirrored" RPM repositories * "relative_path" is absolute, it's the full path. * in a mirrored RPM repo, you can't move `packages/d/dog.rpm` to `packages/dog.rpm`, nor can you rearrange the structure of a file repo. * very little needs to happen at publish time because there's not much that you can change 2. Repositories where republishing in different layouts is possible, "layout doesn't matter". * e.g. standard RPM repositories and python repositories * "relative_path" is floating, it's really just the filename. The location of this file within the repository can vary depending on the publish-time options and it won't meaningfully affect the consumer. * a non-mirrored RPM repo can be published in different ways, you *can* move `packages/d/dog.rpm` to `packages/dog.rpm * same with Python repositories Our problems are rooted in trying to serve both needs with the same `relative_path` mechanism, *especially* in the RPM plugin where repositories could behave either way and content synced or uploaded into one must be cross-compatible. We're currently trying to push this problem outside of the domain of "content uniqueness" where it is causing us the most immediate issues, but it might be better to rethink what we are trying to do here and avoid mixing the two entirely. Observation: We already have a mechanism for handling absolute, locked in paths. The `RepositoryContentArtifact` model from Proposal 1 is actually completely identical to `PublishedArtifact` except for the latter's extra PK to `Publication`. There will be a perfect 1-to-1 duplication between `RepositoryContentArtifact` and `PublishedArtifact` - is this actually productive? The entire purpose of storing the content_artifact-relative_path relationship to begin with is to have somewhere to hold it until publication time, when we copy it into the `PublishedArtifact` table. If we know all of that information up-front, why bother with a middleman? `RepositoryContentContentArtifact` from Proposal 3 is also *mostly* identical to `PublishedArtifact`. Proposal: * Rename "relative_path" to "filename" and rewrite existing ones to be just the filename * Add a nullable `Publication` column to the `RepositoryVersion` model and have the serializer only show it if it is "complete" * No new models added, and `ContentArtifact` and `RepositoryContent` remain exactly the same * Migration would be simple * Hardest part would be stripping the path prefix from the "filename" * Repository modification needs a "special case" for "Type 1" repositories - need to add/remove `PublishedArtifact` at the same time as `RepositoryContent`. * This possibly means that `Repository` / `RepositoryVersion` code needs to know what "type" of repository it is to know how it should behave? * Mirrored RPM repositories need to disallow modification, anyway, so we need to keep track of this somewhere. Within repositories that require "absolute" paths, instead of storing the full path somewhere within the "artifact-content-relationship-complex", we just create an incomplete `Publication` for the repository version and save the `PublishedArtifact`s directly. In the general case (e.g. the file plugin), at publish time, the only thing which must happen is to write the metadata, utilizing a query on `PublishedArtifact`. In the case of mirrored RPM repositories, we can go further, and simply finish the entire `Publication` at sync time. We save the metadata directly as `PublishedMetadata`, save our `PublishedArtifacts`, and we're done. Future attempts to publish the repo version become a no-op (or maybe an error) and we probably need some mechanism to block manual modifications. Q: Where does saving `PublishedArtifact`s need to happen? A: It could be done in either the plugin or DeclarativeVersion or RepositoryVersion.add_content() itself... I think it makes the most sense in RepositoryVersion.add_content(), however in that case it must have knowledge about the repository and the content type to decide whether doing so is appropriate Q: How does copy work? A: I'm not sure, but there's some fundamental problems here aren't really tied to any specific proposal. Using the file plugin as an example, if we extract the location of a content unit within the repository from the content unit itself, then that means that when copying that unit into another repository we need to provide that location somehow, which our repo modification API has no provision for. We'll have this problem regardless of which proposal we use. Something which is more specific to this proposal is the fact that copy would now involve `PublishedArtifact`s as well in some circumstances. But if we're adding a new model, we'd have the same sort of concerns as well. Q: How does it work for plugins which do not need a Publication? (is it safe to strip the relative_path to the filename in such case) A: ? need feedback from these plugins on the subject of 'relative_path'. I wouldn't expect any other part of the proposal would be problematic though. * it's ok fo the container plugin * it's ok for the rpm plugin #### Proposal V No changes in pulpcore for the relative path. Have a separate content type for mirrored metadata. All plugins/content units carry relative path themselves. #### Proposal VI (pulp_deb solution) No changes in pulpcore. Have relative_path added to all content types natural_key. Bite the bullet that you can have the same content unit multiple times with different relative_path if they are found in different places (Artifacts stay deduplicated). Downside: This only works with content_units that have at most one file associated. Q: How is it different from the Proposal 5? ### Questions Q1: How does copy work with these proposals? (Any of them, really). Once "relative_path" is removed as a direct property of a content unit and content (say a file) can exist at different locations in different repos, and/or multiple different paths in the same repo, it becomes impossible to say "add this content unit to a repository" without answering the question "where" for content types where the location matters. A: If the current `relative_path` from ContentArtifact is renamed to a `filename`. It can be used as a default relative_path. Or during copy the configured relative path for that repo is used. How does that sound? ###### tags: `PulpCon 2020`