# OCI Artifacts Support in Flux

[](https://hackmd.io/HS9ZqSCWQNqSefdaWFylJQ)

## Problem Statement

Registries are evolving into generic artifact stores. Many of our users would like to take advantage of this fact in order to store their kubernetes declaration in registries and apply them using flux. Helm already supports oci registries to store Charts.

## Requirements

- Users are using registry providers that implement the [distribution-spec](https://github.com/opencontainers/distribution-spec/) and support OCI Artifacts.
- Users have the tooling to create oci artifacts of their contents and push them to OCI Registries

## Uses cases

- Being able to declare an `OCI Registry` as a source to be reconciled by the source controller
- Being able to use flux `Kustomization` crd to reconcile declaration fetched from OCI registries
- Being able to use flux `HelmRelease` crd to reconcile Helm Charts fetched from OCI registries
- Future controllers (tf-controller) shall be able to use the OCI Registry source

## Analysis of the existing

### OCI Artifacts

[OCI Artifacts](https://github.com/opencontainers/artifacts/blob/main/artifact-authors.md#defining-a-unique-artifact-type) provides a reference for artifact authors and registry implementors to support new artifact types with pre 1.0 implementations of the `distribution-spec`. More particularly, this repository serve 3 primary goals:
1. Artifact Authors - guidance for authoring new artifact types, including a clearing house for [well known](https://github.com/opencontainers/artifacts/blob/main/definitions-terms.md#well-known-type) artifact types.
2. Registry Operators and Vendors - guidance for how operators and vendors can support new artifact types, including how they can opt-in or out of well known artifact types. Registry operators that already implement mediaType filtering will not have to change. The artifact repo will provide context on how new mediaTypes can be used, and how mediaTypes can be associated with a type of artifact.
3. Clearing House for [Well-known Artifacts](https://github.com/opencontainers/artifacts/blob/main/definitions-terms.md#well-known-type) - artifact authors can submit their artifact definitions, providing registry operators a list by which they can easily support.

OCI Artifacts enables these use cases for `OCI distribution-spec` based registries by specifying a manifest.config.mediaType on the content pushed to a registry.
Artifacts are defined by setting the `manifest.config.mediaType` to a globally unique value. Each new artifact type MUST be uniquely defined. Helm types are registered to IANA.

**How does a registry implementing distribution-spec check for artifact types (hardcoded types or is there any lookup)?**
We tried answering that question by looking at the distribution-spec [reference implementation](https://github.com/distribution/distribution). The filtering happens at the manifest mediaTypes only. The request header content-type is set to the manifest mediatype. The supported types by the reference implementation are:

const MediaTypeManifest untyped string = "application/vnd.docker.distribution.manifest.v2+json"
const MediaTypeManifestList untyped string = "application/vnd.docker.distribution.manifest.list.v2+json"
const MediaTypeImageManifest untyped string = "application/vnd.oci.image.manifest.v1+json"
const MediaTypeImageIndex untyped string = "application/vnd.oci.image.index.v1+json"

The content layer format is up to the artifact author and may utilize standard MIME types or custom formats.

### Helm OCI Support Analysis

Helm defines a new artifact type as defined by the [OCI Artifacts project](https://github.com/opencontainers/artifacts/blob/main/artifact-authors.md).

The defined config mediaType is:

// ConfigMediaType is the reserved media type for the Helm chart manifest config
ConfigMediaType = "application/vnd.cncf.helm.config.v1+json"

It also defines 3 layers types for its content

 // ChartLayerMediaType is the reserved media type for Helm chart package content
 ChartLayerMediaType = "application/vnd.cncf.helm.chart.content.v1.tar+gzip"

 // LegacyChartLayerMediaType is the legacy reserved media type for Helm chart package content.
 LegacyChartLayerMediaType = "application/tar+gzip"

 // ProvLayerMediaType is the reserved media type for Helm chart provenance files
 ProvLayerMediaType = "application/vnd.cncf.helm.chart.provenance.v1.prov"

The resulting manifest looks like this:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cncf.helm.config.v1+json",
    "digest": "sha256:8ec7c0f2f6860037c19b54c3cfbab48d9b4b21b485a93d87b64690fdb68c2111",
    "size": 117
  },
  "layers": [
    {
      "mediaType": "application/vnd.cncf.helm.chart.content.v1.tar+gzip",
      "digest": "sha256:1b251d38cfe948dfc0a5745b7af5ca574ecb61e52aed10b19039db39af6e1617",
      "size": 2487
    }
  ]
}

The registry client (pkg/registry) is built on top of the ORAS [go client](https://github.com/oras-project/oras-go) library.

#### helm push

$ helm push mychart-0.1.0.tgz oci://localhost:5000/helm-charts

The push subcommand can only be used against .tgz files created ahead of time using `helm package`.

When using helm push to upload a chart to an OCI registry, the reference must be prefixed with the scheme **“oci://”** and must not contain the basename or tag. The scheme is verified when setting up the client.

The registry reference basename is inferred from the chart's name, and the tag is inferred from the chart's semantic version. This is currently a strict requirement

If you have created a [provenance file](https://helm.sh/docs/topics/provenance/) (.prov), and it is present next to the chart .tgz file, it will automatically be uploaded to the registry upon push. This results in an extra layer in the [Helm chart manifest](https://helm.sh/docs/topics/registries/#helm-chart-manifest).

Several oci spec descriptor are created:
- For the archive itself
- media type `ChartLayerMediaType`
- Digest computed from the archive file
- Size of the archive file
- For the chart metadata
- Push load the archive into a `*chart.Chart` to get the chart metadata.
- Serialize the meta data to json
- media type `ConfigMediaType`
- Digest computed from the metadata
- Size of the metadata
- For the provenance file
- media type `ProvLayerMediaType`
- Digest computed from the prov file
- Size of the prov file
- An oci image spec manifest is generated and contains the above descriptors

The final blobs (manifest file, config file, chart archive and prov file) and manifest descriptor are pushed to the registry.

#### helm pull

$ helm pull oci://localhost:5000/helm-charts/mychart --version 0.1.0

Pull download the OCI Artifact corresponding to the chart name with the given version to the given destination (default is .); It can untar the archive if specified.

1. The ORAS library permits specifying media type filters. All Helm defined mediaTypes are allowed + manifest and index mediaTypes from oci image spec.
2. Save the archive file in the target path (+ prov file if requested)

#### helm Install

$ helm install myrelease oci://localhost:5000/helm-charts/mychart --version 0.1.0

1. Install tries to locate the Chart by verifying if the supplied name is an OCI name (start by oci: ).
2. It then downloads the corresponding OCI Artifact and stores it in a repository cache. The archive (.tgz) and prov files are stored. It uses the same filters than helm pull
3. The chart is loaded from the archive.
4. The chart dependencies are checked
a. Try to update dependencies if specified, by downloading them (local, remote repositories and oci registries)
5. Install and returns release information

### Source controller helm charts management

The source controller is a set of different controllers working together to reconcile artifacts from different sources(git, helm repo, s3 buckets).
In particular the helmChart controller’s job is to retrieve chart artifacts from a referenced source (`gitrepository source, bucket source, helmrepository source`).

If the referenced source is a helmrepository
1. Create a ChartRepository, required to download the chart index and charts from the repository.
2. Attempt to load the repository index from the local cache
3. Get the desired chart version
4. Check if any change has occurred and if we need to rebuild the Chart
5. Download the chart archive… Here the method used is:

return r.Client.Get(u.String(), r.Options...)

With the client being

// Client to use while downloading the Index or a chart from the URL.
 Client getter.Getter

Which is used to create the ChartRepository in step 1:

func NewChartRepository(repositoryURL, cachePath string, providers getter.Providers, opts []getter.Option) (*ChartRepository, error) {

Then, we get the client used to download the Chart with:

c, err := providers.ByScheme(u.Scheme)

Now, here are the list of declared providers in the helm source code:

var httpProvider = Provider{
 Schemes: []string{"http", "https"},
 New:     NewHTTPGetter,
}

var ociProvider = Provider{
 Schemes: []string{registry.OCIScheme},
 New:     NewOCIGetter,
}

The current implementation of the source controller, sets only one `provider`:

 getters  = getter.Providers{
 getter.Provider{
     Schemes: []string{"http", "https"},
     New:     getter.NewHTTPGetter,
   },
 }

But, if we would add the `oci provider`, our helmChart controller could download helm oci artifacts from registries.

6. Build the chart, package it and persist it locally.

Otherwise we expect a tarball from git or bucket source
1. Untar the chart archive
2. Build the chart, and resolve dependencies
3. package it and persist it locally.

### [Add support for using an OCI image as source Proposal](https://github.com/fluxcd/source-controller/pull/450)

This proposed implementation creates a new CRD `ocirepositories.source.toolkit.fluxcd.io` with a new Kind `OCIRepository`.

An `OCIRepository` declaration is defined as:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: OCIRepository
metadata:
  name: podinfo-deploy
  namespace: flux-system
spec:
  interval: 10m
  timeout: 1m
  url: ghcr.io/stefanprodan/podinfo-deploy
  ref:
    # one of
    tag: "latest"
    digest: "sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2"
    semver: "1.x"
  auth:
    # one of
    secretRef:
      name: regcred
    serviceAccountName: reg
  ignore: ".git, .out"
  suspend: true

The google `go-containerregistry` library is used to communicate with registries. This library supports [oci images and docker media types](https://pkg.go.dev/github.com/google/go-containerregistry@v0.8.0/pkg/v1/types#MediaType).

In order to reconcile an OCIRepository, the controller does the following
1. Generate the credentials to interact with the registry. The docker authentication behavior is emulated as much as possible here.
1. The secret in `Spec.Authentication.Secret` is loaded
2. If a`Spec.Authentication` was not provided, the reconciler uses the DefaultKeychain which checks the following path in order
1. First, check `$HOME/.docker/config.json`
2. If `$HOME/.docker/config.json` isn't found, check `$DOCKER_CONFIG` (if set)
3. If neither was found, look for Podman's auth at `$XDG_RUNTIME_DIR/containers/auth.json` and attempt to load it as a Docker config
3. The serviceAccount in `Spec.Auth.ServiceAccountName` is loaded. If not provided, the default service account will be used.
4. The reconciler tries to get `ImagePullSecrets` from all sources and put them in an authentication keychain.
2. The reconciler tries to construct a reference with the `Spec.URL` and in order:
1. `Spec.Ref.Digest`
2. `Spec.Ref.Semver`
1. Fetch the list of tags from the registry
2. Discard tags not satisfying our semver constraint
3. Sort them in reverse order to get latest on first index
4. Keep tags[0]
3. `Spec.Ref.Tag`
4. Latest
3. Get access to the remote [image](https://pkg.go.dev/github.com/google/go-containerregistry@v0.8.0/pkg/v1#Image) from the given reference
4. [Create a local archive (.tar.gz)](https://pkg.go.dev/github.com/google/go-containerregistry@v0.8.0/pkg/v1/mutate#Extract) with the image manifest digest as name, containing the image's flattened filesystem.
5. Create a symlink of the archive named “latest.tar.gz”.

The consumers can then retrieve this artifact to consume from the filesystem.

## Proposal
We think that the API proposed in [#450](https://github.com/fluxcd/source-controller/pull/450) is a good base.

Instead of an `OCIRepository` custom resources, we propose to split it into two crs, `OCIRegistry` and `OCIArtifact` with the following APIs:

type OCIRegistrySpec struct {
    
	// URL is a reference to an image in a remote registry
	// +required
	URL string `json:"url"`

	// The credentials to use to pull and monitor for changes, defaults
	// to anonymous access.
	// +optional
	Authentication *OCIRepositoryAuth `json:"auth,omitempty"`
}

type OCIRepositoryAuth struct {

	// SecretRef contains the secret name containing the registry login
	// credentials to resolve image metadata.
	// The secret must be of type kubernetes.io/dockerconfigjson.
	// +optional
	SecretRef *meta.LocalObjectReference `json:"secretRef,omitempty"`

	// ServiceAccountName is the name of the Kubernetes ServiceAccount used to authenticate
	// the image pull if the service account has attached pull secrets. For more information:
	// https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account
	// +optional
	ServiceAccountName string `json:"serviceAccountName,omitempty"`
}

type OCIArtifactSpec struct {
    
    // Pull artifacts using this provider.
	// +required
	OCIRegistryRef meta.LocalObjectReference `json:"ociRegistryRef"`
    
	// The OCI reference to pull and monitor for changes, defaults to
	// latest tag.
	// +optional
	Reference *OCIRepositoryRef `json:"ref,omitempty"`

	// The interval at which to check for image updates.
	// +required
	Interval metav1.Duration `json:"interval"`

	// The timeout for remote OCI Repository operations like pulling, defaults to 20s.
	// +kubebuilder:default="20s"
	// +optional
	Timeout *metav1.Duration `json:"timeout,omitempty"`

	// Ignore overrides the set of excluded patterns in the .sourceignore format
	// (which is the same as .gitignore). If not provided, a default will be used,
	// consult the documentation for your version to find out what those are.
	// +optional
	Ignore *string `json:"ignore,omitempty"`

	// This flag tells the controller to suspend the reconciliation of this source.
	// +optional
	Suspend bool `json:"suspend,omitempty"`
}

type OCIRepositoryRef struct {

	// Digest is the image digest to pull, takes precedence over SemVer.
	// Value should be in the form sha256:cbbf2f9a99b47fc460d422812b6a5adff7dfee951d8fa2e4a98caa0382cfbdbf
	// +optional
	Digest string `json:"digest,omitempty"`

	// SemVer is the range of tags to pull selecting the latest within
	// the range, takes precedence over Tag.
	// +optional
	SemVer string `json:"semver,omitempty"`

	// Tag is the image tag to pull, defaults to latest.
	// +kubebuilder:default:=latest
	// +optional
	Tag string `json:"tag,omitempty"`
}

And an `OCIArtifact` declaration would be:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: OCIRegistry
metadata:
  name: ghcr-podinfo
  namespace: flux-system
spec:
  url: ghcr.io/stefanprodan/podinfo-deploy
  # one of secretRef or serviceAccountName
  auth:
    secretRef:
      name: regcred
    serviceAccountName: reg

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: OCIArtifact
metadata:
  name: podinfo-deploy
  namespace: flux-system
spec:
  ociRegistryRef:
    name: ghcr-podinfo
  interval: 10m
  timeout: 1m
  url: ghcr.io/stefanprodan/podinfo-deploy
  ref:
    # one of
    tag: "latest"
    digest: "sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2"
    semver: "1.x"
  # one of secretRef or serviceAccountName
  secretRef:
    name: regcred
  serviceAccountName: reg
  ignore: ".git, .out"
  suspend: true

for HelmChart, we would have something like:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: my-chart
spec:
  interval: 5m
  chart:
    spec:
      chart: my-chart
      version: '4.0.x'
      sourceRef:
        kind: OCIRegistry
        name: ghcr-podinfo
        namespace: flux-system
      interval: 1m

We propose using ORAS to communicate with registries. It would permit having the same behavior when dealing with helm charts or other contents.

We would use it to manage plain kubernetes files. Here an example of how to push pull an archive containing kubernetes files:

⋊> ~/f/podinfo on dev ⨯ oras push localhost:5000/podinfo:v2 \
                                   --manifest-config config.json:application/vnd.oci.image.config.v1+json \       
                                   ./kustomize.tgz:application/vnd.oci.image.layer.v1.tar+gzip
                                   
Uploading 8e0d7e95d362 kustomize.tgz
Pushed localhost:5000/podinfo:v2
Digest: sha256:45cb027606d34e1e47c521d52a218c5a1982eb142e3cc1d12a6e0dde5dcaa9c2

⋊> ~ oras pull localhost:5000/podinfo:v2                            
Downloaded 8e0d7e95d362 kustomize.tgz
Pulled localhost:5000/podinfo:v2
Digest: sha256:45cb027606d34e1e47c521d52a218c5a1982eb142e3cc1d12a6e0dde5dcaa9c2

⋊> ~ tar xvf kustomize.tgz                                       
x kustomize/
x kustomize/config.yaml.encrypted
x kustomize/deployment.yaml
x kustomize/kustomization.yaml
x kustomize/token.encrypted
x kustomize/service.yaml
x kustomize/hpa.yaml
x kustomize/vault.encrypted
x kustomize/secret.yaml

#### Source controller

##### OCIArtifact reconciliation
The ociArtifact controller will retrieve an artifact based on the url and credentials provided by the reference `OCIRegistry`. The credentials will be in a kubernetes secret as in the [#450](https://github.com/fluxcd/source-controller/pull/450) and retrieved from `OCIRegistry.Spec.Authentication.Secret`.

The controller will first pull the manifest to see if it is an OCI Image manifest or an OCI Helm Artifact.

It will then update it's artifact based on the following strategy in order:
1. if `Spec.Ref.Digest` is supplied, pull it and compare to last cached artifact
2. if `Spec.Ref.Semver` is supplied
1. Fetch the list of tags from the registry
2. Discard tags not satisfying our semver constraint
3. Sort them in reverse order to get latest on first index
4. Keep tags[0]
5. pull the tag and compare to last cached artifact
3. if `Spec.Ref.Tag` is supplied, pull it and compare to last cached artifact
4. Pull `latest` and compare to last cached artifact

Store the artifact locally for dependent controllers to consume.

If we have an helm artifact, we will return an event stating that the OCI Artifact type is not supported.

##### Helm Artifacts reconciliation
The helmChart controller will retrieve an artifact based on the url and credentials provided by the reference `OCIRegistry`. The credentials will be in a kubernetes secret as in the [#450](https://github.com/fluxcd/source-controller/pull/450) and retrieved from `OCIRegistry.Spec.Authentication.Secret`.

The controller will first pull the manifest to see if it is an OCI Image manifest or an OCI Helm Artifact.

It will then update it's artifact based on the following strategy in order:
1. if `Spec.Ref.Digest` is supplied, pull it and compare to last cached artifact
2. if `Spec.Ref.Semver` is supplied
1. Fetch the list of tags from the registry
2. Discard tags not satisfying our semver constraint
3. Sort them in reverse order to get latest on first index
4. Keep tags[0]
5. pull the tag and compare to last cached artifact
3. if `Spec.Ref.Tag` is supplied, pull it and compare to last cached artifact
4. Pull `latest` and compare to last cached artifact

Store the artifact locally for dependent controllers to consume.

If we have a non helm artifact, we will return an event stating that the OCI Artifact type is not supported.

##### Dealing with dependencies

The OCI Registry client relies on `manifest.config.mediaType` to know what kind of artifact it is allowed to process. The client pull of artifacts layers from supported types.

The `OCIArtifact` and `HelmChart` reconciler are expecting the archive to be in the first layer of the Artifact, and will discard all the other layers. The dependent controllers will resolve the artifact dependencies, as it is a domain specific problem.

For exeample, if we have a Helm Artifact, the dependent helmChart controller will pull the artifact, extract the Chart and then resolve the chart dependencies.

#### Flux CLI

There will be a specific flux command (built on top of ORAS) to help users push to their preferred OCI Registry

⋊> ~ flux push localhost:5000/podinfo:v1 -path ./podinfo/kustomize --username souleb

The artifact will be of type `application/vnd.oci.image.config.v1+json`. The directory pointed to by `path` will be archived and compressed to `tar+gzip`. The layer media type will be `application/vnd.oci.image.layer.v1.tar+gzip`.

This will ease adoption of this new feature.

## Design Considerations

## Alternatives

We could keep the `OCIRepository` proposal and tie OCIArtifact and Registries specification. That is considered unpractical, as we would like to be able to reuse a registry spec for different artifact types with different reconcilers.

Instead of reusing the `oci image` mediaType to adress the need to store kubernetes yaml declaration, we could register our own artifact type. This is also considered un practical, as declaring a type has to go through the IANA process and flux is not the owner of those type as helm is for helm artifacts for example.