# EPIC https://pulp.plan.io/issues/7670 --- ## Deprecation Requirements - UI - Set/unset the deprecation status on a collection - Query collections in a specific repository by deprecation status - return all deprecated collections - return all non deprecated collections - return all collections - Efficiently get deprecation status in a specific repository for a list of up to 120 collections - Efficiently get deprecation status in a specific repository for a single collection - server operations - synclists - efficiently sync the metadata for individual collections from repo A to repo B when any changes happen with collections that are tracked by the synclist in repo A - copy / move - duplicate an individual collections metadata when it is copied or moved from repo A to repo B - deletion - clean up metadata for collections that have been deleted from the system - CLI - none right now - sync - Efficiently determine if anything has changed since the last sync - Efficiently pull all the metadata for a repo and update it in the database --- ## As a Galaxy V3 api user, I have a /metadata/ endpoint ```/metadata/``` - 200 OK response with: ```json { "deprecated_collections": [ { "namespace": "cisco", "name": "management", }, { "namespace": "another_ns", "name": "another_collection_name", }, { "namespace": "foo", "name": "bar", } ] } ``` This gets added to the Galxy V3 API here: https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/urls.py#L40-L74 ```/metadata/?deprecated_collections="cisco"``` - 200 OK repsonse with: ```json { "deprecated_collections": ["cisco", "another_collection_name", "foo"] } ``` ## Sync Repository Metadata at sync time and parse and save it ## Copy https://pulp.plan.io/issues/7621 When you copy a Collection from repo A to repo B using the ```/modify``` endpoint, you have to copy the 'deprecated' repository with it. Have pulp_ansible's /modify/ endpoint use a similar-but-customized implementation of what pulpcore provides [here](https://github.com/pulp/pulpcore/blob/612611162e0382bbcd5df52b44471e33e5725e7e/pulpcore/plugin/actions.py#L23-L63) ## Remove 'deprecated' from model and existing Galaxy API views The 'deprecated' field should be removed --- ## Proposed solutions ### OneToOneField, Global metadata #### Data model Use a `OneToOneField` to link collection versions with their metadata. This makes it easy to perform joins between metadata and collections. ```python class MutableCollectionMetadata(models.Model): collection_version = models.OneToOneField(CollectionVersion) deprecated = models.Boolean() # other metadata fields can be added as they are needed here last_updated = models.Datetime() ``` #### Collection list Operation (UI) We can use the existing collection api and simply append the one to one field to the queryset with a simple join ```python CollectionVersion.objects.select_related( 'mutable_collection_metadata').filter(deprecated=false) ``` #### Server operations Metadata is global, so there's no need to keep it in sync between repos. #### Sync Provide a `metadata/` endpoint that allows for mutable metadata to be queried by last_updated (which allows the client to only get changes that have happened since the last sync) and allows metadata to be filtered by a `repository` param (this could either be positional in the URL or a query param). This query can be done efficiently with joins: ```python MutableCollectionMetadata.objects.select_related( 'collection_version').filter( last_updated_gt=last_sync_data, colllection_version__ansible_repository__name='selected_repo') ``` The new data can then either be inserted into the database one at a time using update or create, or possibly using a bulk update using posgres upset: https://stackoverflow.com/questions/27047630/django-batching-bulk-update-or-create ##### Sample response `GET content/<distro_path>/api/v3/metadata` ```json [ { "namespace":"foo", "name":"bar", "version": "1.2.3", "deprecated": false, "last_updated": "<timestamp>" } ] ``` --- # Notes * provide a metadata endpoint (per RepositoryVersion) * should have timestamp * bulk in one response body * sync * check metadata at first stage, and check the timestamp * models * RepositoryVersion <=> Collection * serializing * metadata response (prefetch) * deprecated in v3 collection serializers * https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/urls.py#L42 * https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/urls.py#L50-L52 * copy * have a modify implementation --- # Prototype * https://github.com/pulp/pulp_ansible/pull/389