owned this note
owned this note
Published
Linked with GitHub
# Helm repos working group
Every Wednesday at 8:30 - 9:00am PST
https://zoom.us/j/696660622 <--join here
## Meeting Notes: Feburary 6, 2019
PR open on Helm for registry support:
https://github.com/helm/helm/pull/5243
(please see for latest UX) ^
OCI Spec for how things are stored on the filesystem:
https://github.com/opencontainers/image-spec/blob/master/image-layout.md
Is there a lib we can use for this piece? ^
Helm as "opinonated CNAB builder" concept (Simon from Docker):
https://github.com/helm/helm/issues/5242
Auth - still need to chase down, but should be able to borrow from Docker CLI package.
Manifest spec - what have we agreed on thus far? What needs agreeing on?
### Save before push
Steve: do we need to do `helm chart save` prior to `helm chart push`? ?
### Singing
Fisher: Save should sign as well?
Farina: tuf+notary vs PGP in Helm 3. This does not occur locally.
### Lib for storage
https://godoc.org/github.com/containers/image/oci/layout#NewReference
## Meeting Notes: January 30, 2019
.ics for OCI dev call
https://calendar.google.com/calendar/ical/linuxfoundation.org_i0sado0i37eknar51vsu8md5hg%40group.calendar.google.com/public/basic.ics
Passive cache?
Credential helpers
- call aws API to get token
ECR may already support this:
https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-manifest-formats.html
Making things transparent in commands
https://github.com/helm/helm/compare/dev-v3...jdolitsky:feat-v3/oci-registry-support?expand=1
annotations hsould be namespaced
https://github.com/opencontainers/image-spec/blob/master/annotations.md
```
org.opencontainers.image.version version of the packaged software
The version MAY match a label or tag in the source code repository
version MAY be Semantic versioning-compatible
org.opencontainers.image.ref.name Name of the reference for a target (string).
```
```
$ h3 chart list
REF NAME VERSION DIGEST SIZE CREATED
localhost:5000/myrepo:1 wordpress 5.1.2 8224586 18.1 KiB 1 second
```
```
$ curl -s -H 'Accept: application/vnd.oci.image.manifest.v1+json' http://localhost:5000/v2/stable/chartmuseum/manifests/latest | jq
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
"size": 2
},
"layers": [
{
"mediaType": "application/vnd.cncf.helm.chart.meta.v1+json",
"digest": "sha256:d0771a67245b0f50366fdd195d0bfb33c7c665809a1bce294e9ee5e1a9603df2",
"size": 444,
"annotations": {
"org.opencontainers.image.title": "chart-meta.json"
}
},
{
"mediaType": "application/vnd.cncf.helm.chart.content.v1+tar",
"digest": "sha256:a287514ce1d58d0e1020d2e486a4ff13184a9f24885b3c7fec1f744cb7af194b",
"size": 9295,
"annotations": {
"chart.name": "chartmuseum",
"chart.version": "1.8.4",
"org.opencontainers.image.title": "chart-content.tgz"
}
}
]
}
```
```
$ tree /Users/jdolitsky/.helm/registry
/Users/jdolitsky/.helm/registry
├── blobs
│ ├── content
│ │ └── sha256
│ │ ├── 82
│ │ │ └── 24586842c560dcbe3f98acd34aef243bb30233126af62efd3b2a82e4f3cae9
│ │ └── 93
│ │ └── b08215f5386233f1139013dd9fe173920f3d653d4eceea258adfa156f8f03e
│ └── meta
│ └── sha256
│ ├── 2c
│ │ ├── 017c46f229ef5faf021d54c2ca6
│ │ └── 017c46f229ef5faf021d54c2ca6df862169e4314ccdf324ee6faa23ebc585f
│ └── 33
│ └── 44059bb81c49cc6f2599a379da0a6c14313cf969f7b821aca18e489ba3991b
├── charts
│ ├── wordpress
│ │ └── versions
│ │ └── 5.1.2
│ └── zetcd
│ └── versions
│ └── 0.1.9
└── refs
└── localhost:5000
├── myrepo
│ └── tags
└── stable
└── zetcd
└── tags
└── latest
├── chart -> /Users/jdolitsky/.helm/registry/charts/zetcd/versions/0.1.9
├── content -> /Users/jdolitsky/.helm/registry/blobs/content/sha256/93/b08215f5386233f1139013dd9fe173920f3d653d4eceea258adfa156f8f03e
└── meta -> /Users/jdolitsky/.helm/registry/blobs/meta/sha256/33/44059bb81c49cc6f2599a379da0a6c14313cf969f7b821aca18e489ba3991b
22 directories, 10 files
```
## Meeting Notes: January 23, 2019
oras/strawman
- need to leverage annoations in oras
docker-app/CNAB should leverage annoations more effectively
- Matt Farina to take action
Rules for annotations
https://github.com/opencontainers/image-spec/blob/e562b04403929d582d449ae5386ff79dd7961a11/annotations.md
Need to get something merged into OCI spec to support Helm
Are we good with AuthN/AuthZ? Without using the Docker CLI?
Priorities with OCI:
1. Annoations/Spec/Strawman
2. Search
3. Auth
Containerd may take care of Authorization flow for us
How will we store version? Vs. a tag
Should we support pushing v1 charts?
We need a short proposal to helm/community
* v2 charts cannot go in v1 repos
* changes to chart manifest
* how do we handle v1 -> v2?
"My understanding is that the consensus is a hybrid of CNAB to OCI and ORAS:
- Using media types is preferred to duck typing with annotations (aligned with what ORAS does)
- Using existing high-level OCI structures is preferred to adding a new structure (aligned with CNAB to OCI)
- Exposing metadata via standardised annotations will allow registries to provide a better UX (aligned with CNAB to OCI)"
-Chris
Jimmy to submit "official" strawman proposal
Steve: do we want to push values as separate layers?
Fisher: this may be interesting with chart dependencies
*"Registries To Become Cloud Native Artifact Stores"* - Steve
https://hackmd.io/Jk2XCLP2S2y8AfdXJdRLrw
We do not want to change content of a saved version
Actions:
* Follow up with jimmy on OCI proposal
* How to handle legacy charts, Chart.yaml
* What is the UX ?
* Continue to go after search OCI (steve)
* Josh to verify things needed w auth
## Meeting Notes: January 16, 2019
Helm client progress (gif):
https://github.com/helm/helm/compare/dev-v3...jdolitsky:feat-v3/oci-registry-support
How do we feel about this UX? ^
Differences between **oras** and **cnab-to-oci**?
https://github.com/shizhMSFT/oras
- puts everything in layers
- overloads OCI media-type
https://github.com/docker/cnab-to-oci
- Re-use existing thing
- surface as much info as possible to registry itself
- map bundle.json to OCI index
- example: https://github.com/docker/cnab-to-oci#example
- OCI index not supported by Docker hub at the moment
- https://github.com/docker/cnab-to-oci/blob/master/remotes/push.go#L41-L50
- does not require vendor buy-in
- technically sound and no extra work
```
{
"schemaVersion": 1,
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:d59a1aa7866258751a261bae525a1842c7ff0662d4f34a355d5f36826abc0341",
"size": 315,
"annotations": {
"io.cnab.type": "config"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:196d12cf6ab19273823e700516e98eb1910b03b17840f9d5509f03858484d321",
"size": 506,
"annotations": {
"io.cnab.type": "invocation"
}
},
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"digest": "sha256:6bb891430fb6e2d3b4db41fd1f7ece08c5fc769d8f4823ec33c7c7ba99679213",
"size": 507,
"annotations": {
"io.cnab.component_name": "image-1",
"io.cnab.original_name": "nginx:2.12",
"io.cnab.type": "component"
}
}
],
"annotations": {
"io.cnab.keywords": "[\"keyword1\",\"keyword2\"]",
"io.cnab.runtime_version": "v1.0.0-WD",
"io.docker.app.format": "cnab",
"io.docker.type": "app",
"org.opencontainers.image.authors": "[{\"name\":\"docker\",\"email\":\"docker@docker.com\",\"url\":\"docker.com\"}]",
"org.opencontainers.image.description": "description",
"org.opencontainers.image.title": "my-app",
"org.opencontainers.image.version": "0.1.0"
}
}
```
`annotations` is "short-term" solution for this, but works today.
Docker: if we need to support another mediatype, need to get the spec to support it. Long timeline. Vendors may block mimetypes.
Jimmy: way we solve is top-level manifest mediatype stays the same, individual blobs have media-types
Jimmy: dependencies between types (Helm chart refs docker images in same registry). Future discussion.
Docker: close to end of Q1 for CNAB support
Thick vs Thin bundle - where do the images exist that the bundle references?
Steve: we dont need vendor buy-in, thats what OCI is
From a spec perspective - no whitelist/blacklist is required. Can be added upon registry customization.
"clearing house" - specify media type in central location
Helm Chart.yaml:
```
apiVersion: v1
description: Host your own Helm Chart Repository
name: chartmuseum
version: 1.8.4
appVersion: 0.8.1
home: https://github.com/helm/chartmuseum
icon: https://raw.githubusercontent.com/helm/chartmuseum/master/logo2.png
keywords:
- chartmuseum
- helm
- charts repo
```
*"Registries To Become Cloud Native Artifact Stores"* - Steve
https://hackmd.io/Jk2XCLP2S2y8AfdXJdRLrw
Summarize our message for the OCI dev call later today (4pm EST):
* 2 proposals: gain feedback, which is better given tradeoffs
* Simple requirements:
* metatdata that desc what the object in repository is
* opaque object(s) that the client needs to be able to ID
## Meeting Notes: January 9, 2019
### Last weeks action
> * Whats the right place to contribute this? (oras)
* Steve to talk to containerd/oci folks
Steve: thinking deislabs. Unsure why this functionality is needed. "Staging" ground.
Farina: Maybe more appropriate in containerd. containerd are the "building blocks" for the client side
Butcher: For background, the GitHub deislabs org is where MS Azure puts projects whose intent is to go into a foundation. (And is thus highly collborative, and non-MS specific)
Farina: containerd maintainers: https://docs.google.com/spreadsheets/d/1Pr8cyp8RLrNGx9WBAgQvBzUUmqyOv69R7QAFKhacJEM/edit#gid=262035321
> * Mimetype filtering - talk to oci folks, what is appropriate way to filter vs not filter?
* Jimmy
Farina: massive thread on OCI dev list: https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/U49AZRFo3tk
Steve: potentially a few more projects around this to land in deislabs
Farina: need to define why do those that operate services need to limit things?
Farina: important to keep things on open governance
Steve: not trying to make any of this MSFT-
Butcher: in deislabs, each project can manage its self in its own way
> * Helm 3 community updates w basic changes for registries
* Josh
Josh: Unable to get to this, want to mock out UX first
> * Pull request against helm dev-v3 branch
* Josh
Josh: in progress, will show demo
> * Search API conversation
* Steve/Matt
Farina: this has crept into the OCI thread above
Steve: this may go into the same deislabs repo. Stephen says tried this w catalog API and the vendors ended up doing something custom for them
Farina: who are the people we can talk to related to Docker Hub/distribution to discuss?
young man: FWIW: Deislabs is not meant to be a "home" for anything, ever. It's meant to be a place outside of "formal Microsoft" to do work UNTIL it is placed outside in the appropriate governed body. If it doesn't get placed within a reasonable period of time, it's gone, moved to whomever wants it. Just want people to understand what that org is for from our POV....
> * Put this call on Helm calendar
* Josh
Josh: done: https://calendar.google.com/calendar/embed?src=s5anaqbm9kda435dnh5r8lj1l8%40group.calendar.google.com&ctz=America%2FChicago
The OCI calendar can be found as an ICS file at https://calendar.google.com/calendar/ical/linuxfoundation.org_i0sado0i37eknar51vsu8md5hg%40group.calendar.google.com/public/basic.ics
### `oras` v0.3.1 released
Link: https://github.com/shizhMSFT/oras
```
$ gofish install oras
==> Installing oras...
🐠 oras 0.3.1: installed in 1.779228743s
```
### Proposed Helm CLI UX (demo)
#### Discussion
Fisher: metadata in package vs not in package
Farina: how does the translation work between .tgz charts and this?
WHO IS YOUNG MAN??
#### For Chart Publishers
Tag chart directory and store in local cache (`~/.helm/registry`):
```
helm tag mychart/ localhost:5000/mychart:0.1.0
```
List all charts in local cache:
```
helm charts
```
Publish a chart from cache to remote:
```
helm push localhost:5000/mychart:0.1.0
```
#### For Chart Consumers
Download chart from remote:
```
helm pull localhost:5000/mychart:0.1.0
```
Install chart into Kubernetes cluster:
```
helm install myrelease localhost:5000/mychart:0.1.0
```
### OCI meetings
Every 2 weeks at 2pm PST
Calendar: https://calendar.google.com/calendar/embed?src=linuxfoundation.org_i0sado0i37eknar51vsu8md5hg%40group.calendar.google.com&ctz=America%2FMexico_City
IRC #opencontainers
### This week's action items:
* Meeting moved to 1 hr
* Move oras to deislabs
* ?
* What is the correct way to break down the chart into layers?
* Josh to work w Jimmy
* Helm 3 community updates w basic changes for registries, dev-v3 PR
* Josh
## Meeting Notes: January 2, 2019
[shizhMSFT/oras](https://github.com/shizhMSFT/oras) - Go lib to push/pull any files to/from any registry with OCI image support
Working branch for Helm 3 repo changes: https://github.com/helm/helm/compare/dev-v3...jdolitsky:feat-v3/oci-registry-support?expand=1
How do we get all registries to support new mimetype(s)?
Search/retrieve by mimetype - using Helm, Duffle
Where do we make this conversation happen during the week? OCI IRC chatroom
Freenode #opencontainers
https://www.irccloud.com/
Action items:
* Whats the right place to contribute this? (oras)
* Steve to talk to containerd/oci folks
* Mimetype filtering - talk to oci folks, what is appropriate way to filter vs not filter?
* Jimmy
* Helm 3 community updates w basic changes for registries
* Josh
* Pull request against helm dev-v3 branch
* Josh
* Search API conversation
* Steve/Matt
* Put this call on Helm calendar
* Josh
## Meeting Notes: December 21, 2018
Goal - have a reference implementation of the [Helm Repos and Container Registry Convergence Proposal](https://github.com/helm/community/blob/3689b3202e35361274241dc4ec188e1e6f1a2e53/proposals/helm-repo-container-registry-convergence/readme.md)
Speicfically; a Helm 3.0 client, that uses an OCI compliant registry to `login`, `push`, `pull` charts.
A user could `docker run OCI distribution`, with a Helm 3.0 client and `push`, `pull`, `upgrade` charts to a k8 deployment.
### Discussion on [storing different artifact types](https://hackmd.io/Jk2XCLP2S2y8AfdXJdRLrw)
Discussed seperating the pushing of different artifact types (mimeTypes) to the same repo, from how the different artifact types are signed.
The OCI index doesn't have to assume all artifacts are signed as a group.
Agreed to seperate this out, and deal with signing and artifact pushing as layered concepts. Signing would be done in the various clients. It makes the client code a bit more complex, but it protects the user experience.
### Action Items
- Will start with docker/distribution to test out OCI generic artifact requrements
- Will experiement with the UX outlined in the [Helm Repos and Container Registry Convergence Proposal](https://github.com/helm/community/blob/3689b3202e35361274241dc4ec188e1e6f1a2e53/proposals/helm-repo-container-registry-convergence/readme.md)
- Sajay will get Josh the ACR Helm artifact conversion code, to move from the server to the helm client.
- Validate distribution has mediatype opened up
- Josh may get some work done over the break
- Setup a slack channel to consolidate the various threads of discussions
## Meeting Notes: December 19, 2018
Helm and CNAB - can we use the same OCI mechanism across both?
### Helm 3.0 Registry MVP
- Authentication
- Basic Auth in scope
- 2FA can be deferred, or PRs submitted by cloud providers on a common API
- Chart Storage
- Chart Retrieval
- Chart Dependency Resolution, supporting version ranges
- version ranges may be scoped to a cient side tag retrieval, with better support in a subsequent version
### Post 3.0
- 2FA
- Channels
^ should document this
### Questions for Helm 3 MVP
- Can different artifact types with the same name be placed in same repo?
Auth stuff - part of OCI?
- yes, supports MFA
2 things we need for Helm 3.0 (dont want OCI work to block Helm)
- UX changes in CLI
- Detection of repo type, fall back to index.yaml (is there a way to detect OCI?)
Searching for stuff:
- what do we need for this?
Action items:
- Scope code changes (Josh)
- Login element (docker hub, quay, etc)
- GET/PUT APIs, see what we can do
- see what needs to change in spec vs code
- CNAB convert to OCI https://github.com/docker/cnab-to-oci
- Changes to OCI? What is the contribution process for getting changes? (Matt Farina)
### Chart Museum
To make forward progress, discussion on leveraging the brand and awareness of Chart Museum. Chart Museum will add distribtuion to it's installation, so any user can `helm install` Chart Museum, which will add distribution, getting chart registries. Cloud providers can simply follow the OCI spec to get the same capabilities on thier registry infrastructure.
### Helm 3 Registry Support - Success metric
By pushing the needed requirements down to OCI, a Helm 3.0 client should be able to pull Helm Charts from:
- quay.io
- docker hub
- azure container registry (acr)
- google container registry (gcr)
- aws container registry (ecr)
- Chart Museum
- Harbor
## Meeting Notes: December 5, 2018
Agenda:
- Discuss changes required to enable https://github.com/helm/community/pull/65 which is based on what was discussed last week.
- CNAB - what can parts can we make common?
CNAB repos:
https://github.com/deislabs/cnab-spec/blob/master/200-repositories.md
CNAB -
* storing in distribution a first class thing
* smart vs simple protocol
* what did docker folks think of this?
* this is derived from git's http protocol
* docker search
* no common way to do this across registries
Questions
* Anything written up on distribution extensions?
* Should add issue to distribution spec to allow for this (Jimmy)
* Distribution should move to OCI
* What needs to happen?
* spec and implmentation hardocded to contain which artifact types
* make it a dynamic property
* Docker v2 is closer to this, but need to support more than just Docker-related artifacts
* Needs to start in specs
Harbor -
* Built on docker distribution/chartmuseum/clair
* Start talking to them about this. Mailing list?
* Intro and deep dive at kubecon
#65
* anything else needed, breaking changes?
* none in UX of CLI
* exported public function should allow for indirection
* wpengine etc imports Helm pkg
Next steps - move on docker spec
* Steve will help us get in contact
* Chris, COO of CNCF is ee. directr of OCI
---
## Meeting Notes: November 28, 2018
Agenda:
Discuss the UX (Helm CLI semantics etc.) for working with repos in Helm 3. Once we have this solidified, we can start getting into technical details to achieve this.
* What do we take away from AppRegistry and Docker UX?
* https://github.com/app-registry/appr/blob/master/Documentation/quick-start.md
* How will we continue to support Helm 2 repos with these changes?
* If we go towards a "registry" approach, every chart becomes an addressable URL. Do we still need the "helm repo" subcommand?
Proposal to converge helm repositories with docker registries
https://github.com/helm/community/pull/55
Helm 3: push, login, capabilities api
https://github.com/helm/community/pull/57
---
App Registry, little diff than docker:
`docker pull NAME[:TAG|@DIGEST]`
Trying to avoid "latest"
Channels popular concept for Quay users
Tag "locking" - unable to modify tag for digest
### Commmands
#### Pull
Helm 2:
```
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm fetch stable/mysql --version 0.10.2
```
Helm 3:
```
helm pull stable/mysql@0.10.2
helm pull charts.myhelm.io@0.10.2
helm repo add kubernetes-charts https://kubernetes-charts.storage.googleapis.com
helm pull kubernetes-charts/mysql:0.10.2
helm fetch kubernetes-charts/mysql --version 0.10.2 # <- ? back-compat for cli
helm repo add oldstyle https://oldcharts.myhelm.io
helm repo add newstyle https://newcharts.myhelm.io
# Should work the same, does work behind the scenes
helm pull oldstyle/mysql:0.10.2
helm pull newstyle/mysql:0.10.2 --version 0.10.3
# Appr
helm registry pull localhost:4506/ant31:>=4.4.5
helm registry pull localhost:4506/ant31 --version '>=4.4.5'
# farina style
helm pull newstyle/mysql --query '>=4.4.5'
```
If `/index.yaml` exists, treat it as Helm 2 repo
`helm repo update` is no-op on URLs that do not look like a Helm 2 repo?
Detection mechanism: is this old-style repo or container registry
Using registry, some users not relying on information in `Chart.yaml`. Content vs metadata.
#### Search
Helm 2:
```
helm search stable/mysql
```
Helm 3:
Remote query to get list of tags
Does rate limiting come into play?
```
?
```
Appr
```
helm registry search $domain $query
```
#### Push
Helm 2:
```
(n/a)
```
Helm 3:
```
helm push charts.myhelm.io@0.10.2
```
#### Install
Helm 2:
```
helm install stable/mysql --version 0.10.2
```
Helm 3:
```
helm install charts.myhelm.io@0.10.2
```
### Consensus
* keep `helm repo add` for aliasing
* Remove `--version` flag from commands, come up with contract on how we derive version from URL
* Example: `charts.myhelm.io:0.10.2` -> `0.10.2`
### Need to address
* Metadata in `Chart.yaml` - duplicated?
* How will we determine a repo is oldstyle vs newstyle?
* Adding immutiblity (channels)
## Links
App Registry UX
https://github.com/app-registry/appr/blob/master/Documentation/quick-start.md
Proposal to converge helm repositories with docker registries
https://github.com/helm/community/pull/55
Helm 3: push, login, capabilities api
https://github.com/helm/community/pull/57
