---
# System prepended metadata

title: EPIC

---

# EPIC
https://pulp.plan.io/issues/7670

---

## Deprecation Requirements

- UI
  - Set/unset the deprecation status on a collection
  - Query collections in a specific repository by deprecation status
    - return all deprecated collections
    - return all non deprecated collections
    - return all collections
  - Efficiently get deprecation status in a specific repository for a list of up to 120 collections
  - Efficiently get deprecation status in a specific repository for a single collection
- server operations
  - synclists
    - efficiently sync the metadata for individual collections from repo A to repo B when any changes happen with collections that are tracked by the synclist in repo A
  - copy / move
    - duplicate an individual collections metadata when it is copied or moved from repo A to repo B
  - deletion
    - clean up metadata for collections that have been deleted from the system
- CLI
  - none right now
- sync
  - Efficiently determine if anything has changed since the last sync
  - Efficiently pull all the metadata for a repo and update it in the database

---

## As a Galaxy V3 api user, I have a /metadata/ endpoint

```/metadata/``` - 200 OK response with:

```json
{
	"deprecated_collections": [
        {
			"namespace": "cisco",
			"name": "management",
		},
		{
			"namespace": "another_ns",
			"name": "another_collection_name",
		},
		{
			"namespace": "foo",
			"name": "bar",
		}
	]
}
```

This gets added to the Galxy V3 API here: https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/urls.py#L40-L74

```/metadata/?deprecated_collections="cisco"``` - 200 OK repsonse with:

```json
{
	"deprecated_collections": ["cisco", "another_collection_name", "foo"]
}
```

## Sync Repository Metadata at sync time and parse and save it


## Copy
https://pulp.plan.io/issues/7621

When you copy a Collection from repo A to repo B using the ```/modify``` endpoint, you have to copy the 'deprecated' repository with it.

Have pulp_ansible's /modify/ endpoint use a similar-but-customized implementation of what pulpcore provides [here](https://github.com/pulp/pulpcore/blob/612611162e0382bbcd5df52b44471e33e5725e7e/pulpcore/plugin/actions.py#L23-L63)


## Remove 'deprecated' from model and existing Galaxy API views

The 'deprecated' field should be removed

---

## Proposed solutions

### OneToOneField, Global metadata

#### Data model

Use a `OneToOneField` to link collection versions with their metadata. This makes it easy to perform joins between metadata and collections.

```python
class MutableCollectionMetadata(models.Model):
  collection_version = models.OneToOneField(CollectionVersion)
  deprecated = models.Boolean()
  # other metadata fields can be added as they are needed here
  last_updated = models.Datetime()
```

#### Collection list Operation (UI)

We can use the existing collection api and simply append the one to one field to the queryset with a simple join

```python
CollectionVersion.objects.select_related(
    'mutable_collection_metadata').filter(deprecated=false)
```

#### Server operations

Metadata is global, so there's no need to keep it in sync between repos.

#### Sync

Provide a `metadata/` endpoint that allows for mutable metadata to be queried by last_updated (which allows the client to only get changes that have happened since the last sync) and allows metadata to be filtered by a `repository` param (this could either be positional in the URL or a query param). This query can be done efficiently with joins:

```python
MutableCollectionMetadata.objects.select_related(
    'collection_version').filter(
        last_updated_gt=last_sync_data,
        colllection_version__ansible_repository__name='selected_repo')
```

The new data can then either be inserted into the database one at a time using update or create, or possibly using a bulk update using posgres upset: https://stackoverflow.com/questions/27047630/django-batching-bulk-update-or-create

##### Sample response

`GET content/<distro_path>/api/v3/metadata`
```json
[
    {
        "namespace":"foo",
        "name":"bar",
        "version": "1.2.3",
        "deprecated": false,
        "last_updated": "<timestamp>"
    }
]
```
---

# Notes
* provide a metadata endpoint (per RepositoryVersion)
    * should have timestamp
    * bulk in one response body
* sync
    * check metadata at first stage, and check the timestamp
* models
    * RepositoryVersion <=> Collection
* serializing
    * metadata response (prefetch)
    * deprecated in v3 collection serializers
        * https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/urls.py#L42
        * https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/urls.py#L50-L52
* copy 
    * have a modify implementation


---
# Prototype
* https://github.com/pulp/pulp_ansible/pull/389