---
tags: advisories
---
# Merging Advisories

## Overview of the problem
### Scenario

An advisory with a given name exists in Pulp. A sync-operation finds an advisory with the same name. One of four things must happen:

* Keep the existing advisory
* Delete the existing advisory and take the new one
* Merge the new one with the existing one and keep the result
* Determine that there is a fatal data error and give up

### Places that need to deal with advisory-merges
* sync (**two** advisories collide)
* upstream-repo-updates
* adding a remote to a repository that's already been sync'd
* copy (**two** advisories collide (?))
* repo-to-repo
* upload (**arbitrary number** of advisories with same id)
* **if** repository is specified - otherwise, no merge for errata with same id but different digests
* can copy() result in 'finding' more than 2 advisories with the same id?
* repository-modify (**arbitrary number** of advisories with same id)
* incoming erratum has same id as one already in repo (same as sync case)
* incoming add_contents contains **multiple** errata with same id but different hrefs

### Application-reality Complication

The final complicating determinant is, "don't write to persistent storage unless you have to". This adds one more rule to set:
* "don't write to storage unless you have to"

This problem led to the addition of the ``digest`` field in stored advisories. If an incoming advisory has its digest computed, and it's identical to one that has been stored, then we have nothing to do.

The problem with the existing ``digest`` is it is driven by the output of "whatever createrepo_c spits out this week", and is applied to the entire advisory including reference- and package-lists. This makes it not as reliable as we would like, for the purposes of this discussion.

### Who Should Win?

I'm pulling in a new advisory, and find that something with its ID already exists. Who should I keep, incoming or current?

* Incoming that has only metadata changes (e.g., descriptive text)
* Incoming is adding packages
* Incoming is removing packages
* Incoming has a completely different package-set
* Incoming is older-by-version
* Incoming is older-by-updated_date

What to do in the face of the possible combinations of these difference is the question conflict-resolution is trying to answer.

### Related Issues

* [8249](https://pulp.plan.io/issues/8249) - migrate from pulp2 and sync causes fatal conflict-failures
* [PR 1945](https://github.com/pulp/pulp_rpm/pull/1945) merged
* [8250](https://pulp.plan.io/issues/8250) - ALLOW_UNSAFE_ADVISORY_CONFLICT_RESOLUTION feature
* [PR 1950](https://github.com/pulp/pulp_rpm/pull/1950) submitted
* [7282](https://pulp.plan.io/issues/7282) - Updating an advisory/updaterecord deletes old and doesn't produce new
* ggainey in-progress
* [Pain](https://hackmd.io/csLR-MkgTz2TG6H--mnn7w?both)

## Current Thinking/code
### Determinants
We consider merging advisories only when we have an existing advisory and a new one, which have the same name/id and different checksums(?).

We can do one of four things in the conflict case:
* KEEP: Keep existing and ignore new
* REPLACE: Remove existing and replace with new
* MERGE: Merge new into existing, merge pkglists
* ABORT: fatal error

The attributes that are important for deciding how to resolve the conflict include
* version - not every advisory has a version
* updated_date (or issued_date if updated_date not set)
* pkglist

Advisories are considered "the same" if version AND updated_date are the same.

Advisory A is "older" than Advisory B if updated_date(A) < updated_date(B).

Similarly, Advisory A is newer if updated_date(A) > updated_date(B).

If there is no updated_date, rely on version.

pkglist comparisons can be in four states:
* pkglist(A) == pkglist(B)
* one pkglist is a proper-subset of the other
* pkglist-intersection is non-empty and not a proper subset of the other
* pkglists are disjoint

### Decision-Making Process

('Current' in the sense that I'm in the middle of working on 8249 by adding proper-subset-recognition-logic to the code)

Existing Advisory A, incoming Advisory B, same ID.

* version AND updated_date SAME:
* pkglists-same: REPLACE (metadata-only change)
* pkglist(A).subset(pkglist(B)): REPLACE (adding-pkgs)
* pkglist(B).subset(pkglist(A)): KEEP? (removing-pkgs)
* pklist-intersection EMPTY: MERGE (e.g., combining debuginfo- into base-repo)
* else: ABORT (8249)
* dates-same OR (dates-differ, same-version), intersection EMPTY: [ttereshc thinks this condition potentially incorrectly reflects the current state]
* pkglist(A) EMPTY: REPLACE
* else: ABORT
* dates-differ AND pkg-intersection:
* A.updated < B.updated: REPLACE (8229)
* else: KEEP
* dates-same AND versions-differ AND pkg-intersection:
* A.is_previous_to(B): REPLACE
* else: KEEP

### Fatal Failures
First case:
* IF version and updated_date are the same, AND
* pkglists are not equal, AND
* neither pkglist is a proper subset of the other, AND
* pkglists are not disjoint, THEN
* FAILURE.

Specific use-case: an advisory released for two incompatible repositories at the same time.

Second case:
* IF updated_dates differ OR (updated_dates are the same AND versions differ), AND
* pkglists intersect, AND
* neither pkglist is a proper subset of the other, THEN
* FAILURE.

Specific use-case: merging base and later-version-of debuginfo into the same place.

### ttereshc version of existing logic
Existing Advisory A, incoming Advisory B, same ID.

* dates-same and versions-same:
* pkglists-same: REPLACE (metadata-only change)
* pkglist(A).subset(pkglist(B)): REPLACE (adding-pkgs)
* pkglist(B).subset(pkglist(A)): KEEP? (removing-pkgs)
* pklist-intersection EMPTY: MERGE (e.g., combining debuginfo- into base-repo)
* else: ABORT (8249)
* dates-same and versions-differ:
* pklist-intersection EMPTY: ABORT
* A.is_previous_to(B): REPLACE
* else: KEEP
* dates-differ and any versions (same or different):
* pklist-intersection EMPTY: ABORT
* A.updated < B.updated: REPLACE (8229)
* else: KEEP

The fatal case conditions are the same for this version as well.

## Simplifying (?) the workflow
Some of what follows is driven by discoveries made while investigating the advisory-upload issue [7282](https://pulp.plan.io/issues/7282). See [Advisory Merging Woes](https://hackmd.io/@pulp/advisory_upload_fun) for gory details.

What if Pulp3 stopped trying to guess what the 'right thing' to do is, and instead always went with 'newest is best, **always**'?

'Newest' drives/can-be-driven off of four fields:
* **version**. You'd think this would be It, but it's not. Some repos (SUSE) don't use version. Sometimes an advisory is updated and its version doesn't change. And version is just a string, it doesn't have to be a number - "newer than" is a collation-based guess.
* **last_update**. Assuming no version - clearly we can use "last time this advisory was updated" to define 'newer'. Alas, some repository-types don't fill this field in.
* **issue_date**. If there's no version and no last_updated, an advisory does have an issue_date. But in the repository-modify case, it's possible to have three advisories that have identical version/update/issue dates, but different metadata.

The only thing pulp has control over are the ``core_content`` fields **pulp_created** and **pulp_last_updated**. Of those, **pulp_created** is always there.

### Proposal, No Fatalities!
The current fatal-error heuristics in the merge process are quite possibly saving users who've made an error from themselves. But, they are just heuristics - we don't **know** that we're saving the user (as opposed to just getting in their way).

One simplifying assumption can be summed up as "The User Knows Best," and we drop the fatal-error cases completely.

### Proposal, dates
Advisories have two newer-than functions:
* **advisory_newer**(n,m): prefers version > update_date > issue_date. Returns -1 n newer, 0 equal, 1 n older
* **pulp_newer**(n,m): prefers pulp_last_updated > pulp_created. Same returns.
* **advisory_newest**([n,m,...]): Returns most-recent of advisories in list, can be a tuple
* **pulp_newest**([n,m,...]): Returns singleton newest, period.

When faced with two or more advisories with the same id (e.g., 'RHSA-202:12345'), Pulp3 will
* choose the 'most recent', metadata and related collections
* if **advisory_newer** is equal, choose **pulp_newer** for metadata and merge related collections.

Consider the following 3-advisory scenarios.
* adv(n,m) is advisory_newest, p(n,m) is pulp_newest():
* adv([A,B,C]) => [A]
* A metadata and relations
* adv([A,B,C]) => [B,C]
* union B and C relations, in-storage metadata
* adv([A,B,C]) => [A,B,C]
* union relations, metadata from p([A,B,C])

### Proposal, digests
The digest field has some problems:
* Hash based on output of createrepo_c
* No ordering-control
* Applies to the entire advisory, no way to separate metadata from collection-changes

To address these we could:
* Break createrepo output into multiple sections
* metadata
* package-list
* reference-list
* Enforce field-ordering on metadata
* Enforce ordering on package- and reference-list
* prepare digests for each separate section
* leave current digest 'as is'

This would a) separate us from createrepo_c()'s output-ordering, and b) let us know which parts of an advisory have changed without having to read in package- and reference-lists.

The migration to fill these in 'correctly' if we go this route will, of course, be...exciting.

###### tags: `advisories`, `RPM`