ORAS migrating to OCI Reference types

Status: Work-in-progress

Version: draft

Overview

The two PRs of working group proposal for reference types were merged recently and will be released in Image-spec v1.1 and distribution-spec v1.1.0. See below two PRs for reference types:

Working Group Proposal for Reference Types by michaelb990 · Pull Request #335 · opencontainers/distribution-spec
Working Group Proposal for Reference Types by lachie83 · Pull Request #934 · opencontainers/image-spec

NOTES:
distribution-spec v1.1.0-rc.1 was released, see v1.1.0-rc1
image-spec v1.1.0-rc.2 was released, see v1.1.0-rc.2

This document describes the changes, manifest, and fallback use cases for ORAS-go library and ORAS CLI when migrating to OCI reference types.

NOTES:
EDIT (SteveLasker): Per the Distribution Spec: Enabling the Referrers API
3. After the referrers API is enabled, Registries MUST include all newly pushed image and artifact manifests with a valid refers field in the referrers API response.
This means a registry that supports the _referrers api MUST support the OCI Artifact manifest
Registries that support the referrer API also support OCI artifact manifest. Registries that support OCI artifact manifest may not support referrer API.

Solution and Plan

ORAS will support only OCI artifact in the near future and fall back to use OCI Image manifest with a subject property if a registry doesn't support OCI artifact. There is no plan to support both ORAS artifact and OCI artifact in the same ORAS release. The last version of ORAS-go supporting ORAS artifact is v2.0.0-rc.3, and the last version of ORAS CLI supporting ORAS artifact is 0.15.x.

Check below table for the timeline of releases supporting OCI artifact:

Release	ETA Date	Related issue
ORAS-go v2.0.0-rc.4	End of Oct, 2022	ORAS-go #271
ORAS CLI 0.16.0	Nov 7, 2022	ORAS #406

ORAS-go supports reference types

Changes are required for the following use cases:

Pushing Manifests with Subject
Discovering Referrers
Deleting Manifests

ORAS CLI supports reference types

ORAS push command

A new flag is introduced with the name --compatibility={none,min,max}, so that users can change the default pushing behavior.

With value set to min (default value), pushing OCI artifact manifests to a registry, if the response is 400 Bad Request, fallback to pushing OCI image manifest.
With value set to none, no fallback procedure is triggered for pushing OCI artifact manifests. Push fails if the response is 400 Bad Request.
With value set to max, OCI image manifests are pushed instead of OCI artifact manifests.

NOTES: The flag names are tentative. Any suggestions are welcome.

Should follow below steps:

PUT artifact manifest
If 200, we are good.
Else
- If can fallback (we don't know that the referrers API is enabled, and the error is 400 and MANIFEST_INVALID), PUT OCI image manifest.
- Else abort.

ORAS pull command

No changes

ORAS attach command

A new flag is introduced with the named --compatibility={none,min,max}, so that users can change the default pushing behavior.

With value set to min (default value), OCI artifact manifest with subject field is attached, if the response is 400 Bad Request, fallback to attaching OCI image manifest with subject field.
With value set to none, no fallback procedure is triggered for attaching OCI artifact manifests. Attaching fails if the response is 400 Bad Request.
With value set to max, OCI image manifests are attached instead of OCI artifact manifests.

If the referrers API returns a 404, use the same logic as ORAS-go Pushing Manifests with Subject.

NOTES: The flag names are tentative. Any suggestions are welcome.

Should follow below steps:

oras: Resolve the subject manifest –> 200
oras: PUT artifact manifest
- If it succeeds,
  - oras-go: Check the referrers API is enabled or not.
    - If it is enabled, done / return
- If it fails,
  - oras: set referrers capability to false.
  - oras: PUT image manifest
oras-go: Update the index

ORAS manifest fetch

No changes

ORAS manifest Push

Pushing manifests with subject

If the referrers API returns a 404, use the same logic as ORAS-go Pushing Manifests with Subject.

When pushing an OCI artifact manifests with subject field，if the response is 400 Bad Request, by default, fallback to pushing an OCI image manifest with subject field. Fallback procedure can be disabled using a new flag named --strict.

When user pushes an OCI Image manifest with subject field，by default, image index is updated, user can disable updating image index using the new flag --strict.

NOTES: The flag names are tentative. Any suggestions are welcome

Pushing manifests without subject

No changes when pushing an OCI image manifest without subject field.

When pushing an OCI artifact manifests without a subject field，if the response is 400 Bad Request, by default, fallback to pushing an OCI image manifest without a subject field. Fallback procedure can be disabled using a new flag named --strict (flag name is tentative).

NOTES: The flag names are tentative. Any suggestions are welcome

ORAS manifest Delete

By default, only supplied manifest is deleted.

Use a new flag named -r, --recursive to delete referrer manifests recursively. See below example, by using the flag -r, --recursive, when deleting subject manifest, manifest A, B, C are deleted, and image index are updated.

Use a new flag --strict to disable updating image index.

graph LR

A -- refers --> Subject
B -- refers --> Subject
C -- refers --> Subject
Index --> A
Index --> B
Index --> C

Tag --> Index

Tag -.- Subject

NOTES: The flag names are tentative. Any suggestions are welcome

ORAS blob

No changes

ORAS tag

No changes

ORAS repository

referrers tag schema may mess the results of oras repository list, so a new flag could be introduced to filter out referrers tag schema from the results.

ORAS discover

The same logic as ORAS-go discover referrers.

ORAS copy from target A to target B (include extended copy)

Notes: Registry is one type of target, there are others types of target, for example, OCI-layout.

ORAS doesn't perform manifest conversion, so copy succeeds only if both target A and target B support the type of manifest being copied.

Target	B: Support OCI Artifact	B: Support OCI Image
A: Copy OCI Artifact	Success	Failure
A: Copy OCI Image	Success	Success

Steve Lasker

2022/09/13 16:22:22

This is a little tricky: See: https://github.com/opencontainers/distribution-spec/blob/main/spec.md#enabling-the-referrers-api 3. After the referrers API is enabled, Registries MUST include all newly pushed image and artifact manifests with a valid refers field in the referrers API response. (Edited)

Michael Brown

2022/09/15 20:52:57

I'm inclined to agree with @shizh here. Either oras-go should remove support for ORAS artifacts entirely or it should simply expose the primitives for pushing / pulling either type of manifest and let consumers decide for themselves what to support. If I was creating a new OCI tool today, I would not want my client generating manifests that won't be supported going forward. (Edited)

Sajay Antony

2022/10/11 17:01:47

- Wi

Min compatibility might deter users from considering this as the best practice. I think calling out types would be better if we really wanted this. I would also like to consider delaying introducing this flag until we have an external ask for this from the community if possible. (Edited)

Lixia (Sylvia) Lei

2022/10/12 02:57:15

if the response is `400 Bad Request`, by default, fallback to pushing an OCI image manifest with `subject` field. Fallback procedure can be disabled using a new flag named `--strict`.

`oras manifest push` will only push what users provided, and will not fallback to push OCI image manifest. If the response is a 400, it will just fail. (Edited)

2022/10/12 03:02:10

Same here. (Edited)

Billy Zha

2022/10/17 15:15:47

--compatibility={none,min,max}

Here are some thoughts for the flag naming 1) --enforce-media-type={artifact(none), fallback(min), image(max)} and default to fallback 2) --fallback={never(none), error-only(min), always(max)} (Edited)

2022/10/31 05:11:29

fests with Subject](https://hackmd.io/@yizha1/ORAS_to_OCI_reference_type#Pushing-Manifests-with-Subject). When pushing an OCI artifact manifests with `subject` field，if the response is `400 Bad Request`, by default, fallback to pushing an OCI image manifest with `subject` field. Fallback procedure can be disabled using a new flag named `--strict`. When user pushes an OCI Image manifest with `subject` field，by default, image index is updated, user can disable updating image index using the new flag `--strict`. >NOTES: The flag names are tentative. Any suggestions are welcome #### Pushing manifests without subject No changes when pushing an OCI image manifest without `subject` field. When pushing an OCI artifact manifests without a `subject` field，if the response is `400 Bad Request`, by default, fallback to pushing an OCI image manifest without a `subject` field. Fallback procedure can be disabled using a new flag named `--strict` (flag name is tentative). >NOTES: The flag names are tentative. Any suggestions are welcome ### ORAS manifest Delete By default, only supplied manifest is deleted. Use a new flag named `-r, --recursive

Should be 'No changes' here (Edited)

Nima Talebi

2022/09/13 23:19:22

nit: ...doesn't *necessarily* mean... (Edited)

2022/09/13 23:24:10

Is the disclosure of which API was used by the cli available to the user in some debug mode? (Edited)

Shiwei Zhang

2022/09/14 00:09:45

Even if OCI Referrers API does not return 404, we should query the ORAS artifact referrers API as they are in parallel. (Edited)

2022/09/14 00:45:07

Since `_oras/referrers`is deprecating, should we stop pushing ORAS Artifact manifests? (Edited)

Feynman Zhou

2022/09/14 03:44:24

Is it possible to check if the registry supports referrers API before the ORAS client interacts with (push/pull) it? (Edited)

2022/09/15 20:50:29

ECR does not need this. (Edited)

2022/09/15 20:55:29

Yes, it does. (Looks like @stevelasker said the same below) see: https://github.com/opencontainers/distribution-spec/blob/main/spec.md#enabling-the-referrers-api (Edited)

2022/10/11 15:21:00

With value set

Is "--backward-compatibility" more intuitive here? (Edited)

2022/10/12 00:56:47

--strict

I would recommend not introducing this flag and focus on ensuring that attach would behave the way described. (Edited)

2022/10/12 00:57:35

--recursive

The expectation is that we delete all levels of referrers. Is that correct? (Edited)

2022/10/12 03:03:56

I think so.

2022/10/12 01:02:19

oras repository list

+1 something like `--exclude-digest-tags` (Edited)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

ORAS migrating to OCI Reference types

Overview

Solution and Plan

ORAS-go supports reference types

Pushing Manifests with Subject

Discovering Referrers

Deleting Manifests

ORAS CLI supports reference types

ORAS push command

ORAS pull command

ORAS attach command

ORAS manifest fetch

ORAS manifest Push

Pushing manifests with subject

Pushing manifests without subject

ORAS manifest Delete

ORAS blob

ORAS tag

ORAS repository

ORAS discover

ORAS copy from target A to target B (include extended copy)