owned this note   changed 3 years ago  
Published Linked with GitHub

[OCI WG] Reference Types

tags: oci

Status

This WG has met the mission statement below and has now been archived – see https://github.com/opencontainers/wg-reference-types/blob/main/README.md

Meeting Details:

Template

Enter Date Here

Attendees

  • attendees list

Notetaker

  • notetaker 🥇
  • backup notetaker 🥈

Agenda

  • enter agenda before meeting

Notes

August 23, 2022 – CANCELLED

We're iterating on the PRs, and don't anticipate a need to meet. Go outside! 🏡

August 16, 2022

Attendees

  • Brandon Mitchell
  • Sajay Antony
  • Lachlan Evenson
  • Michael Brown (IBM)
  • Josh Dlistsky
  • Jason
  • Tianon Gravi
  • Ramkumar Chinchani

Notetaker

  • notetaker 🥇
  • backup notetaker 🥈

Agenda

  • enter agenda before meeting
  • Reviewing PR feedback
  • PR 16

Notes

Aug 9, 2022

Attendees

  • attendees list

Notetaker

  • notetaker 🥇
  • backup notetaker 🥈

Agenda

Notes

Aug 2, 2022

Attendees

  • Brandon Mitchell
  • Sajay Antony
  • Yi Chen
  • Lachlan Evenson
  • Nisha Kumar
  • Michael Brown
  • attendees list

Notetaker

  • Brandon Mitchell 🥇
  • backup notetaker 🥈

Agenda

Notes

  • artifactType filtering:
    • Deciding if it makes sense to add this since clients will have filtering.
    • The negative is servers may need to cache multiple responses, one per media type
    • Michael will get a Vote opened up for this, to be closed by end of week.
  • How to handle a "refers" pushed before referred manifest exists?
    • Group seems to like responding to API even before manifest exists
    • Jason will add a PR for this
    • Avoid using 404 for cases other than "API not available" to avoid interfering with the "tag fallback" - Brandon
    • Involve the OCI Maintainers in the decision of the API route since the manifest that is referenced in the URI path and technically the manifest may not exists.
  • Is ordering important:
    • Within the descriptor itself (mediaType, digest, size)
    • How does this impact reproducibility?
    • This hasn't been handled by JSON previously.
    • Would be good to bring up reproducibility to image-spec maintainers, too big of a lift for this WG. (Sajay)
  • Should a list of artifact types be included?
    • We've gotten rid of the short artifact types from the spec
    • The IANA media types are pushed off to IANA
  • Adding more examples?
    • Pushing to have a more generic example instead of different example per type for SBOM/Sig/etc.
  • Live Coding PRs
    • Where should the tag schema fall back be defined?
      • Group is leaning towards distribution-spec
    • Started a PR for defining the push process with the tag schema in oci-playground/distribution-spec

July 26, 2022

Attendees

  • Josh Dlistsky
  • Sajay Antony
  • Brandon Mitchell
  • Jason
  • Michael Brown (AWS)
  • Yi Chen
  • Tianon Gravi
  • Mounic
  • Chris Short
  • attendees list

Notetaker

  • notetaker 🥇
  • backup notetaker 🥈

Agenda

Notes

July 19, 2022

Attendees

  • Brandon Mitchell
  • Sajay Antony
  • Tianon
  • Michael Brown
  • Ramkumar Chinchani
  • Jason Hall
  • attendees list

Notetaker

  • Brandon Mitchell 🥇
  • backup notetaker 🥈

Agenda

Notes

  • Issue 64 - artifactType
    • Concern that depending on the config descriptor media type will not work with Docker Hub (Brandon)
    • Consider specific use cases for Docker Hub (Michael)
    • Ask Docker Hub if they would consider changing their filtering on the Config media type
    • Brandon will update the PR based on comments and take out of draft state
  • Issue 65 - predictable digest tags
    • Majority is in favor of predictable tags
    • We need to pick whether it is a tag per artifact type or one tag for all artifacts
    • One tag per type requires a query to recursively get all artifacts, but removes the search space for tools working with specific types of artifacts
    • Opening a follow on vote in Issue 72
    • There's still an option to roll back these changes before suggesting to OCI
  • Reviewing PRs from Jamstah
    • Some may go away with the decision on 65
  • Change sets to image-spec/dist-spec
    • distribution-spec has a sample PR by Josh
    • Nisha will start with the image-spec changes
    • To handle a migration with Go, it may require renames that would be handled with an alias. That may need a second repository rather than a rename. (Brandon)

July 12, 2022

Attendees

  • Brandon Mitchel
  • Ankit Agarwal
  • Marina Moore
  • Jason Hall
  • Michael Brown (AWS)
  • Tianon Gravi
  • Josh Dolitsky
  • Brandon Mitchell
  • Jesse Butler
  • Ramkumar Chinchani (Cisco)
  • attendees list

Notetaker

  • notetaker 🥇
  • backup notetaker 🥈

Agenda

Notes

July 5, 2022

Attendees

  • Brandon Mitchell
  • Tianon
  • attendees list

Notetaker

  • notetaker 🥇
  • backup notetaker 🥈

Agenda

Notes

June 28, 2022

Attendees

  • Lachlan Evenson
  • Tianon
  • Nisha Kumar
  • Alexander Mazuruk
  • Sajay Antony
  • Michael Brown
  • Jesse Butler
  • Brandon Mitchell

Notetaker

  • Nisha 🥇
  • Lachlan Evenson 🥈

Agenda

  • Next steps for deciding on a proposal (Brandon)

Notes

  • Looks like we've settled on prop E and F. (Brandon)
  • Are we looking at a cherry pick of a cherry pick (Nisha)
    • I think they represent two different paths they can go down (Josh)
    • Should we call a vote with how to proceed?
  • From Jon: We need to query the new fields and references via an API
  • Jesse: Must drive quorum here - "consensus doesn't mean unanimous" use Roberts rules(?)
  • Brandon: will file an issue to collect poll results and we will leave it up for a week.
  • Jason: propose that once we vote we present a united front.
  • Brandon: maybe we should go over E to flush out any issues
  • Any time you have to run the tag list instead of give me the tag - that's a downside to prop e - Brandon
  • Can we be clear about the upgrade path is for prop e? - Nisha
    • I would divide the prop into all the new additions then the fallback method (Nisha)
  • Do we convert Proposal E into a spec or how are we going to communicate this?
    • Josh: would like the entirety of Proposal E be agreed upon
    • Michael: concerned about load bearing annotations. We should make them into fields.
    • Brandon: what about signing keys?
    • General concern: do we want to support server side filtering or do we want to embed annotations to do client side filtering?
  • Quorum vote administrivia
  • Context on upgrade path from Brandon - Nisha will submit PR for formatting
  • Added questions to https://github.com/opencontainers/wg-reference-types/issues/42 for follow up

June 21, 2022

Attendees

  • CANCELED

Notetaker

  • notetaker 🥇
  • backup notetaker 🥈

Agenda

  • Meeting canceled as per wg-reference-types Slack channel

Notes

June 14, 2022

Attendees

  • Josh Dolitsky
  • Tianon
  • Brandon Mitchell

Notetaker

  • notetaker 🥇
  • backup notetaker 🥈

Agenda

  • enter agenda before meeting

Notes

June 7, 2022

Recording: https://youtu.be/9kfiO0C0T7o

Attendees

  • Jason Hall
  • Nisha Kumar
  • Sajay Antony
  • Tonis Tiigi
  • Michael Brown
  • Silvin Lubecki
  • Chris Crone
  • Tianon
  • Lachlan Evenson
  • Josh Dolitsky
  • Brandon Mitchell

Notetaker

  • Nisha 🥇
  • Brandon 🥈

Agenda

  • Enumerate concerns and thoughts on Proposal F (Josh)
    • Document them in issue #53

Notes

  • Silvin: Proposal F
    • Add annotations to describe references
    • Add "well known" annotations for "small" data
    • Append data to original image by uploading a new OCI index manifest adding a "digest tag".
      • Jason: we want to get rid of the digest tag. But is this the best we can come up with?
      • Silvin: this would be only one that would include everyone.
      • Tonis: latest tag will point to the latest image and latest index
      • Jason: when downstream consumer ingested alpine:latest then they will add a signature but this changes the digest of the original image. How can I find that this image is the same as the original image?
      • Tonis: in this case you would use the digest tag
      • Brandon: that would mean we would have to parse each of the index manifest.
      • Brandon: what if the arm64 image exists in other indices?
        • Chris: is that in scope?
        • Chris: you would push to a different repo with all the attached artifacts
        • Brandon: now we have to query a special index along with the top level index
        • Tonis: you can only know the list of signatures which means the image has been rebuilt the image that many times
    • Nested indices
      • Tonis: Pull manifest list, check each manifest if it has the right platform. Descriptors are then sorted. Things that don't have a platform are lower priority. If a index is found it will pull the manifest down and go through the same process.
      • Jason: this algorithm needs to be well defined. Platformless manifests need to be handled.
      • Michael: issues with tag race conditions and customers not wanting tags to be overwritten, the tag based approach may not work with existing registries. Tag needs to be immutable and built filters on tags. Scaling this for big registries require additional changes. How do we want to address this?
      • Chris: should have hard limits on index size
    • Josh: thoughts around the reference type field? There is a mediaType which we can rely on.
      • Silvin: not sure if defining mediaTypes was in scope. Maybe use config?
    • Josh: is the primary goal of adding "small data" to support clients?
      • Silvin: it's to reduce roundtripping of manifests - just pull one manifest and get the signature
    • Tonis: unknown platform - using a keyword "unknown"
      • Brandon: feels like another hack
    • Sajay: ordering seems to be important
      • Brandon: defined in OCI today, but not tested enough
    • Nisha: Do nested indexes need to be covered in other proposals too?
      • Jason: we should do this in general from OCI outside of the working group
    • Josh: Scalability if vulnerability scan is added each day to an image
      • Jason: May result in lots of entries in one list, or a deeply nested index if tags cannot be mutated.
    • Brandon: some runtimes limit length of individual annotations to 4KB, not limited by OCI yet, but maybe should be
    • Nisha: is this creating more tags than other proposals?
      • Want to avoid running tag list API
      • This solution doesn't have unique hash, so tags can be pulled directly
    • Brandon: Race conditions when updating existing image
      • Jason: push OCI to support ETags
      • How should clients respond if ETags are not supported by the registry?
      • How can clients detect ETag support on push?
      • Sajay: could all SBOMs be added to a single index?

May 31, 2022

Recording: https://youtu.be/jfLkC67K4Kk

Attendees

  • Brandon Mitchell
  • Josh Dolitsky
  • Silvin Lubecki
  • Chris Crone
  • Tianon
  • Tonis Tiigi
  • Sajay Antony
  • Patrick Flynn
  • Nisha Kumar
  • Jason Hall
  • attendees list

Notetaker

  • Brandon 🥇
  • Nisha 🥈

Agenda

  • Chris: Proposal F

Notes

  • Proposal F:
    • Chris: Docker will write up a Proposal F based on the discussion here
    • How to make mutating an index atomic
    • ETag concern from before was S3 backend was eventually consistent
    • This is no longer a concern with the AWS implementation of S3
    • Josh: size of attached artifacts
      • Manifest are size limited
      • Artifact can be pushed as a blob
      • Manifest should be small, think that signing falls in this bucket
      • Manifest can also grow when there are lots of small things (many attestations attached to an image)
      • We can't necessarily control how people will add new ref types to an image, may not just be a signature and sbom
      • Some new types may group lots of content into a single manifest instead of lots of ref types to an image
    • Sajay: what does this group do if we propose no change to image/distribution specs
      • OCI artifact definition is already a "no change" description
      • We can still standardize how this is done without proposing new types or API endpoints
    • Sajay: we will have limits on the number of objects that can be attached based on the size of manifests
    • Nisha: concerns with F including missing filtering support, creating tags, difficult to track history
    • Jason: don't treat signatures as special, but all small things should have a different option to inline
    • Chris: don't specify each use case, let tool builders define how they use the OCI specs
      • Brandon: interoperability concerns, want to be sure tooling can be designed to copy any ref type between registries
    • Nisha: creating micro-SBOMs and linking them together, how would this be done with F
      • The linked structure would create a tree
      • Tree should be represented by an OCI index (with the option of nesting indexes)
      • Concern with frequently changing content where existing data remains and gets extended with new information
      • Brandon: GitBom may handle tree design in a way that fits into annotation syntax
    • Josh: how to pick between E or F?
      • Concerned that registries won't update quick, resulting in bad UX for proposal E
    • Josh: do we need to see some new proposal G based on Justin's bigger goals
      • Need to find a solution that handles our immediate goals
      • Can also look towards bigger goals later
      • Brandon: A concern from Justin was E resulted in a data model that didn't fit his longer term vision
      • Nisha: we don't have a place to iterate on bigger visions for OCI
    • Working group shouldn't expand too far out of WG goals, leave that to main OCI group and TOB
    • Jason: Various ways to innovate
      1. Can do something in a vacuum and later propose to OCI
      2. Work within constraints of OCI (like cosign)
      3. Dictating from above often fails (or won't get approved by OCI)

May 24, 2022

Recording: https://youtu.be/m4rQBcDb3N4

Attendees

  • Brandon Mitchell
  • Nisha Kumar
  • Tonis Tiigi
  • Chris Crone
  • Jason
  • Sajay Antony
  • Josh Dolitsky
  • Tianon Gravi
  • Matt Bendix
  • Patrick Flynn
  • Lachlan Evenson
  • Ramkumar Chinchani
  • Michael Brown
  • attendees list

Notetaker

  • Nisha 🥇
  • Chris 🥈

Agenda

Notes

  • Demo
  • Feedback on proposals
    • https://github.com/opencontainers/wg-reference-types/issues/48
    • Anything requiring changes to structs and API will have impact on existing implementations
    • Changing how registries are restructured is problematic for large registries
    • Reg: using index - should digest change when digests are added?
      • Brandon: then it's hard to figure out what you are signing
      • Chris: you could sign a single digest (rather a digest + size) or any struct (list of digests)
      • Tonis: the docker use case is more aligned to proposal D i.e. no changes (?). Logically, an image should only have one signature.
      • Jason: conformance tests for nested indexes. How would you discover an image in a deeply nested index?
      • Brandon: if you are an image originator this makes sense, but downstream, workflows that depend on the digest being static will be affected.
      • Tonis: but there could be different components - if container bits change, workflows can detect it.
      • Jason: the same system can query for attestations and signatures.
      • Tonis: you can't understand what attestations were part of the container.
      • Nisha: this is addressed in one of the use cases, we want to know what was the latest thing appended to an image. We don't want a list of attestations, just the latest one. Proposal C was supposed to handle this. We left this because it had problems with GC.
      • Tonis: You don't always want all the signatures, sometimes you want a subset.
      • Nisha: specific use case is chain of custody. In this case, you want to maintain the chain of signatures for the artifact so that you can track where it came from.
      • Jason: this is a valid use case. Can be tracked by adding metadata to document handovers between parties.
      • Tonis: in the index model (Proposal D-ish), when you hand it over, you append the object with a new signature. The digest changes with the signature addition.
      • Brandon: Folks still need write access to your repositories. Suppose you want to link digests to a different repos.
    • Brandon suggests Tonis submit their proposals and how it addresses use cases
      • TBD things are OK in the proposal
    • Jason: what if there is a vuln against digest abc, we want to get that information linked. If we link in the index then we will have to parse the index. If the digest doesn't change it's easy to match.
      • Chris: you have to crawl the registry anyway.
      • Chris- Proposal C: it does look like an index but versioning breaks backwards compat.
      • Chris: propose start with Proposal D (no changes) but work on v3. Jason: fallback mechanism becomes "the mechanism". This is the nice things in Proposal E. Brandon: registries would want to upgrade.
      • Chris: registries will probably rate limit and give you a crappy experience rather than move to the new API.
    • Action Item: Chris and Tonis to submit a proposal
  • Are OCI layout changes needed?
  • Referres API?
    • Query or an OCI index?
  • The name of the "reference" 🌹
  • Upgrading
  • Docker community concerns
    • Chris: what are the data structures that get stored in registries. Should they be attached to the root object or detached.
    • Tonis: Not sure it fits the requirements for backwards compatible use cases. Performance is an issue. Needs to be snappy with UX.
    • Jason: This is feedback that we should consider performance.
    • Jason: docker pull shouldn't have to verify all the signatures by default. Various UX ways to handle this.
    • Brandon: Have done some thinking about this on the notary side about how to define policies for this.
    • Jason: We should support the distroless use case well even though it is a bit exceptional.
    • Brandon: backwards compatibility is a big concern, particularly for runtimes that don't know anything about references and need to run images.

May 17, 2022

Recording: https://youtu.be/LFoHo9zPxzM

Attendees

  • Brandon Mitchell
  • attendees list

Notetaker

  • No one, it was chaos 🥇

Agenda

  • Live code proposal E (Josh)
  • If time permits - Demo distribution with extension support for the new manifest. [sajay]

Notes

May 10, 2022

Recording: https://youtu.be/Vs5HVCo4Tv0

Attendees

  • Josh Dolitsky
  • Jason Hall
  • Sajay Antony
  • Lachie Evenson
  • Michael Brown
  • Jesse Butler
  • Brandon Mitchell (joined late)
  • Ramkumar Chinchani
  • attendees list

Notetaker

  • Jason Hall 🥇
  • backup notetaker 🥈

Agenda

  • https://github.com/opencontainers/wg-reference-types/pull/44
    • prototyping in OCI extension, upgrade it to a "real" path later
    • OCI extension discovery is an OCI extension
    • let's only make the API move once
    • (Nisha) How do we demonstrate the popularity of an extension?
    • PoC as an extension to distribution at https://github.com/oci-playground/distribution/pull/1
  • Get input on the following - @sajay
    • Discuss if we need artifactType in the new manifest and how does this impact the image spec changes since the updates decided to drop filtering all together?
      • Basically is there a middle ground for minimum viable experience?
      • Is this acknowledging that other filters will never happen, or will there be two tiers/paths for filters? (Brandon)
    • Get concensus on if we should return referrers or leave it upto cloud providers to decide the impact of introducing refer field into the image-spec? Inputs from Jon might be really valuable here. Or maybe bring it upto the OCI call.
    • Is there a need for implementers to vouch for spec changes? (Nisha)
  • enter agenda before meeting
  • Follow up: should ArtifactType be a field or an annotation?

Notes

May 3, 2022

Recording: https://youtu.be/pGEbdLKnirA

Attendees

  • Jason Hall
  • Lachlan Evenson
  • Nisha
  • Tianon
  • Patrick Flynn
  • Josh Dolitsky
  • Michael Brown
  • Marina Moore
  • Dan Lorenc
  • Sajay Antony
  • attendees list

Notetaker

  • Nisha 🥇
  • backup notetaker 🥈

Agenda

  • Discuss proposal E
    • modifying existing image spec. [Sajay]
    • new manifest updates [Lachie]
    • wrap and next steps to OCI TOB [Lachie]
  • Partial demo of proposal E w/ crane and zot [Josh]
  • enter agenda before meeting

Notes

  • Proposal E
    • Continuation of last week's convo: what is the expectation of modifying the existing spec to add a reference field.
    • There is some work on the operator's side: image spec does not say that new mediaTypes need to be parsed.
    • Jason: reg: what should the registry operators do. A registry should ignore it. Registry operators should signal that anything pushed before x date will not be indexed, you will have to repush it.
    • Jason: spec says "ignore field" but doesn't say what it should do if that field becomes meaningful (similar to data field discussion). Suggest adding to the spec to watch out for changes to the fields even though it can be ignored for now. This class of problem is probably common to all the specs.
    • Sajay: if reference field may or may not be honored. Should we just go with the new manifest and let registry operators deal with the change - communicate to registry operators and prepare for changes?
    • Jason: This is hard to motivate rather than implement.
    • Sajay: afraid to touch the image spec because it may break something that will cause backlash.
    • Jason: yes, definitely must have something that supports this
    • Jason: will take the action to bring up these two concerns to image spec maintainers, if there is group consensus.
    • Michael: legitimate concerns about accepting input you don't understand with regards to security. Confuse a manifest list for a manifest. When accepting data with references that mean things, then there should be some response from the registry.
    • Jason: the spec is general about what it supports.
    • Jason will take the action to clarify verbiage about specific and general concerns.
  • Proposal E vs Proposal A
    • Why not take the manifest from Proposal A?
    • ArtifactType, Subject, Referrers
    • Jason: personally doesn't have a preference to names. Would it make sense to add to E why the names changed? May also be cognizant of the image spec maintainers' feelings about the words.
    • (Late follow up from Brandon): Names weren't changed from Proposal A, they were just taken from Proposal B, this was a cherry pick from multiple proposals
    • Lachie: looks like proposal E is not as strong as the other proposals. Eg: Proposal A has examples and well thought out explanations about filtering. Maybe the action item is adding the example from proposal A.
    • Sajay: Agree, but Mike Brown did say to use the extensions API. No strong opinion about names. Gets complicated with filtering. Probably need some clarity with that. The filtering with extensions would help to land the filtering without touching the manifest endpoints.
    • Michael: agree dealing with the filtering part in increments.
    • Josh: Proposal E has strengths but the filtering is getting in the way. Maybe focus on the upgrade path and break the proposals
    • Nisha: seems to be consensus that Proposal E is good, but should be broken down into sub-proposals and staged (not Lachie's interpretation)
    • Lachie's interpretation: strip back Prop E to minimal viable proposal, for TOB to understand, strip out filtering, better examples, (Josh: describe upgrade path, Sajay: use extensions a la Prop A?)
      • E-prime: how do we coalesce toward the concrete proposal
    • Josh: unless we're creating a new spec, we can bypass TOB
      • we might be creating a new spec
      • Lachie: we could drop distribution-spec changes if we do an extension :thinking_face:
    • Nisha: Prop E is good, we have to figure out what to propose first; start with E's new artifact, then filtering and image-spec backcompat changes
      1. Drop filtering 2. Registry endpoints out to extensions 3. Submit artifact manifest and image reference field upgrade path. Need explanation for all the proposals.
    • Michael: adding an endpoint is important though
  • Demo
    • Josh to follow up on questions that came up while implementing the demo. Eg the links annotiation which is a comma separated list.
Select a repo