---
title: Container plugin meeting
tags: Container plugin, Minutes
---
## 3 months planning ()

## 3 months planing backlog

* integration with security scanning clients (Clair)
* Performance work(sync,push,copy operations)
* helm support
* oci artifacts support

## MISC deadlines
* add pulp https://github.com/opencontainers/oci-conformance/tree/master/distribution-spec + enable OCI tests in CI
* AI open a task to add OCI tests in CI
* AI open a PR to add pulp registry to the list
* add keycloack support
* plan OAuth2

## Food for thought
* now that we have async push, does it make sense to keep the following limitations?
* unable to add content to push repos. We are already adding tags, why not enable adding images also?
* 2 repo types. Can we consolidate into one repo type? Create a repo and push into it and next sync content into it
* if 'mirror' option was enabled during sync, it will remove everything what was pushed before into that repo - this can be unexpected or undesireble
* in push repo type there is a safeguard to not remove repo_version since it can bring repo into a corrupted state. Mirror repo type allows repo-versions removal
* this seems like a big and invasive change - rollback repo_versions would not work properly for pushed content
* If import/export needs to be compatible across versions, it's even worse.
* rbac implications?
* 1to1 push-repo type and distribution, sync repo type can distributed by N distributions. that's ok because content comes in through repo unlike in push operation through distribution
* token authentication
* Our tokens may introduce one additional unnecessary authentication step
* we are not giving the tokens on a small set of servers to server a large set of download servers; instead we even check all the permissions again when the user hands in the token
* also it is not the kind of API-token people keep asking for
* maybe we can get rid of it

## Agenda template
### Action Items

# moved to https://hackmd.io/mrVY2804QnS1_xjQcx0p1g
-
### Upcoming
- we need to document Roles https://github.com/pulp/pulp_container/issues/641
- We need to release 2.8 and 2.10 for katello

### March 21
- sync pipeline refactor PR from Matthias is ready for review
- CI is failing due change in core. Investigation is ongoing - Matthias and Brian are on it
- ipanoved releaseed 2.11
- ipanova submitted PR for sig policy docs
- Lubos submitted PR for signature tests
- Matthias refactores sometests to use pytest
- Lubos is investigating CI intermittent failure related to tag
- we got some bugs reported from a user who upgrated to 2.11
* ipanova is working on the gpg issue
* we need to document Roles https://github.com/pulp/pulp_container/issues/641

### March 14th

- asked Tanya to re-fresh what has been decided/proposed w/r/t testing sync of signatures from a sigstore https://hackmd.io/7EHldi72RhSCr3An67Rnxg#Testing-related-to-sigstore
- Signing policy configuration - what workflow scenarios to document?
* too many combinations, let's have some discussion directly on the PR
- plan to release 2.11 this week
- refactor sync pipeline there is a bug only in one runner not sure it's related
- https://github.com/pulp/pulp_container/runs/5512356553?check_suite_focus=true
* it seems like the order of list of blobs changed
- FYI Matthias is refactoring some rbac tests
- https://github.com/pulp/pulp_container/pull/632

### March 7th
- matthias is working on refactor sync pipeline
- it's failing in a strange way currently
- https://github.com/pulp/pulp_container/pull/608 let's take a look at the test failures
- i'd like to cut 2.11 release. Waiting on remote user and push of manifest list PRs
- ipanova will work on azure bug |https://bugzilla.redhat.com/show_bug.cgi?id=2026151

### February 28

- some customers still seem to have issue with manifest_id null during sync - ipanova is looking into this
- https://github.com/pulp/pulp_container/pull/605 we can remove some unnecessary db reset connnection calls
- enable push with remote user auth - PR is up from Lubos
- what can we focus on next?
* we should write some tests for the signing feature
* sync of signatures is still in question how write tests --> ask Tanya to re-fresh what has been decided/proposed

### February 21

- PRs need review - repo blob mount and manifest list push
- [matthias] will look whether signature code path needs any adjustments for the roles work

### February 14

- Roles PR is ready for review
- moved away from Dockerhub on our CI, should we also propagate this change to other branches?
* 2.5, 2.8, 2.9, 2.10
- Enable docker push in katello https://github.com/pulp/pulp_container/issues/558 needs help with steps to reproduce
- merging signing branch into main
- doing various backports for the 2.5, 2.8, 2.9 and 2.10 branches + release

### February 7th
- think of what to do with directly assigned rbac perms
* copy-pasta form pulpcore meeting notes:
* Idea: manage command to report “unmigrated” permissions and let the admin assign the roles via api
* Create the command in pulp_container codebase
* Matthias has found a way how to identidy directly assigned rbac perms and translate them into roles during the migration
* PR is ready for review
- CI reaches limit on docker pull
* tests need changes, pulp-smash needs a setting to account for user/pass
* [dkliban] this might not be possible for PRs because secrets are not available there for security issues. will investigate.
* Not done yet, moving to the next week
* [matthias] will look into tests whether it is possible to download content less frequently
* Lubos created PoC to move to Github Package Registry.
* Does not support schema converstion, is it a concern?
* Does not evaluate accept headers sent from the client
* https://github.com/pulp/pulp_container/pull/563
- needs a volunteer to enable docker push in katello https://github.com/pulp/pulp_container/issues/558
- can we create push repo ahead of push?

### January 31

Regrets: ipanova, x9c4.

- testing registry
* outcome - use some deprecated repo from RH registry and add basic signature assertions in the tests
* need to find one,
* [deprecated repos](https://catalog.redhat.com/software/containers/search?include_deprecated=1&p=1&release_categories=Deprecated)
* long term - stand up in CI a small sigstore proxy to variously pass through valid and invalid signatures?
* no, work with the real registry, and just mock data for bad signatures

- CI reaches limit on docker pull
* tests need changes, pulp-smash needs a setting to account for user/pass
* [dkliban] this might not be possible for PRs because secrets are not available there for security issues. will investigate.
* Not done yet, moving to the next week
* [matthias] will look into tests whether it is possible to download content less frequently
* [ipanova][done] look whether it is possible to have a robot account or re-purpose one of our accounts
* service account does not seem to be different from regular account except for granting it read-only perms https://docs.docker.com/docker-hub/service-accounts/#creating-a-new-service-account TLDR; we can re-purpose one of the existing accounts
* take Tanya's
* look into whether we can use GitHub registry so we're not dependant on dockerhub?
* AI: lmjachky

### January 24

- https://github.com/pulp/pulp_container/pull/546#issuecomment-1016795721 ci reaches limit on docker pull
* tests need changes, pulp-smash needs a setting to account for user/pass
* [dkliban] this might not be possible for PRs because secrets are not available there for security issues. will investigate.
* [matthias] will look into tests whether it is possible to download content less frequently
* [ipanova] look whether it is possible to have a robot account or re-purpose one of our accounts

- roles RBAC migration
- translating the auto-generated groups may not be sufficient
- add-permission needs to be translated to creator role
- I'd like to have a review on the role layout before continuing writing the migration
- what to do with directly assigned permissions? maybe ask on pulpcore meeting for more insight

### January 17

- refactor of sync pipeline
- roles RBAC
* look into tests and ensure that they user roles
* CI is green, needs review - Tanya will take a look
* (DATA-)Migrations for permissions to roles are missing
- redis caching PR - needs re-review
- signing epic
* sync from extentions API PR ready
* push of signature - in progress
* still figuring out how to test proper integration with sigstore
- "Bringing pulp_ansible and pulp_container together for all the things"
- RHUI is going to add pulp-container
- bz solved with a hotfix patch

### January 10
- bz escalation https://bugzilla.redhat.com/show_bug.cgi?id=2026277 might need to hop on this
- signing epic
* signature serve/pull PR is up
* let's merge the branch after this PR?
- testing signature? testing repo on RH registry - Tanya, any updates?
- redis caching PR needs review
* Ina started to look
- roles RBAC
- https://github.com/pulp/pulp_container/pull/403
- down to 2 failing tests (again?) around seeing content
- push repository perms are always checked via the distributions role assignment (default policy)

### December 6

* signing service WIP
* sync of signatures WIP
* how should we test this? we need a registry with the sigstore
* https://bugzilla.redhat.com/show_bug.cgi?id=2026277 we should switch to use content.resolution() in the sync pipeline
* 2 backport requests from galaxy, we need to release before shutdown
* redis caching PR, close to be done
*
### November 29

* 2.9.1 and 2.8.2 releases are out
* we have community contribution that fixes an S3 bug, should we backport this into earlier release?https://github.com/pulp/pulp_container/pull/437
* ask galaxy
* signing work
* separate branch for development: "signing"
* signature model/viewset/serializer PR is ready https://github.com/pulp/pulp_container/pull/439
* sync signatures WIP
* downloaders question https://github.com/pulp/pulp_container/pull/436#discussion_r757065877
* sign content from within registry WIP
* `add-signing-service` pulpcore command needs adjustments so `script` field is optional
* roles work
* PR needs review, it is ready? https://github.com/pulp/pulp_container/pull/403
* CI is broken, need to fix
* migrate groups into roles https://pulp.plan.io/issues/9572#note-1
* any blockers?
* docker hub pull limit rate - look into Team plan pricing

### November 22

* AH needs a new 2.8.2 release https://pulp.plan.io/versions/314
* Azure backport would require a pulpcore release
* ipanova will do the release
* Signing work, who can contribute?
* Tanya, Ina, Lubos?
* https://hackmd.io/7EHldi72RhSCr3An67Rnxg
* Roles work, needs review/testing for the 3.17 release
* https://github.com/pulp/pulp_container/pull/403
* Tanya, Ina

### October 18
* content app sync_to_async https://github.com/pulp/pulp_container/pull/418/files thanks Matthias
* Azure support https://github.com/pulp/pulp_container/pull/415 - added, thanks to @fao. Ci is failing due to ^
* Container signing epic in progress https://pulp.plan.io/issues/9502
* Roles WIP https://github.com/pulp/pulp_container/pull/403
* Enable cache https://pulp.plan.io/issues/9500 needs a volunteer
* lubos

### October 11

* Pulp runs out of DB connections https://pulp.plan.io/issues/9454
* Azure support for pulp-container https://pulp.plan.io/issues/9488
* wait on @fao to add azurite to dev env
* Container signing

### October 4, 2021
* roles
* What is the impact on pulp_container?
* Can we try to migrate the "groups" into roles?
* In a migration
* In a voluntarily called service script
* Pulp runs out of DB connections https://pulp.plan.io/issues/9454
* Azure backend for pulp-container https://github.com/pulp/pulp_container/pull/406

### September 20

* refactor sync pipeline so it does not have Interrelate stage - AI @matthias to file a ticket
* went trough issues/stories/PRs

### September 13
* RBAC roles. Checkin meeting scheduled later this week
* 2.8.1 is out with tags race fix
* needed changes for 3.16 compat https://pulp.plan.io/issues/9385 + other PRs are up already
* drf token is deprioritized and on hold for now
* performance investigation - meeting with Brian B. later this week
* import/export for push repos - on hold for now
* long term planning - should look into supporting OAuth2; add option to have keycloack as pulp_token_server( keycloack has special protocol for the registry);

### August 30, 2021
* pulpcore 3.15 requires a new release
* https://pulp.plan.io/issues/9292
* We should interrelate before saving those entities [x9c4]

### August 2, 2021
* DRF token (Who can work on this?) https://pulp.plan.io/issues/9115
* @dennis will look into this once he is done with his current work in progress
* RBAC roles - bump to the next meeting
* 2.8.0 release [x9c4]
* waiting on the last Pr to be merged

### June 26, 2021

* should we adjust/recommend adjusting nginx client_max_body_size because of podman push https://pulp.plan.io/issues/8839
* @ipanova file a docs ticket [done]
* token drf auth https://pulp.plan.io/issues/9115
* needs a separate plugin to manage tokens - @dkliban to ask @gerrod if he can do that
* perormance results - gunicorn worker timeout during upload
* capture this in the docs reccomendations [done]
* readinees for 3.15 compat release
* https://pulp.plan.io/issues/9134 touch()
* @matthias will pick this up
### June 7, 2021

AI review
*
Pulp 3:
* import export
* https://pulp.plan.io/issues/7297
Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### MAY 17, 2021

AI Review

Pulp 3:
* Community central demo - Container topics
* Import/Export

Pulp 2:

### MAY 10, 2021

AI review
*

Pulp 3:
* https://pulp.plan.io/issues/7795
* This feature has been asked for by a user
* re-claim disk space https://pulp.plan.io/issues/8313#note-9
* import export work

Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### date, 2021

AI review
*
Pulp 3:
* discussed 2.6.0 milestone
Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159
*
### April 12, 2021

AI review
*
Pulp 3:
* now that we have async push, does it make sense to keep the following limitations?
* unable to add content to push repos. We are already adding tags, why not enable adding images also?
* 2 repo types. Can we consolidate into one repo type? Create a repo and push into it and next sync in content into it
* this seems like a big and invasive change - rollback would not work properly for pushed content
*
Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

## Agenda template

### March 22, 2021

AI review
*
Pulp 3:
* 429 sync/async api
* how podman behaves vs docker
Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159
*

### March 8, 2021

AI review
*
Pulp 3:
* release 2.1.1 [mdellweg]

Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### March 1, 2021

AI review
*
Pulp 3:
* handle 429 during sync- pulpcore topic
* removal of push repository versions - ipanova file issue
* content trut and signing
* sync/async api discussions - mdellweg to proceed
* schema conversion bug - lubos to fix
* push out the rbac+basic auth feature out 2.4.0 milestone
* ipanova pick up the last issue from the milestone
Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### Sync/Async API discussion, February 24

Ideas:
1. trigger async task with 202 redirect, wait few seconds, issue 429 to the client
* explore if the server can tell to the client when to come back
2. Acquire locks, use transaction, release locks. Still issue 429 to the client

### February 15, 2021

AI review
*
Pulp 3:
* need a release 2.3.1 to get in the get_user_model pr
- https://github.com/pulp/pulp_container/pull/236
- catalog endpoint
Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159
*

### February 8, 2021

AI review
*
Pulp 3:
* Cetrero censeo authentificare per signo esse delendam.

Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### February 1, 2021

AI review
*
Pulp 3:
* group permission PR
* locks and sync/async apis

Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### Jan 25, 2021

AI review
*
Pulp 3:
* Can we use the access policy framework to describe push and pull operations?
* https://github.com/pulp/pulp_container/pull/211
* decouple pull/view push/change
* https://github.com/pulp/pulp_container/pull/208
* Sync/async and locks
* https://github.com/pulp/pulp_container/pull/210
* Token
* Groups PR

Pulp 2:
*

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### Jan 11, 2021

AI review
*
Pulp 3:
* RBAC for push repositories through the registry api
* Automatic permission assignment is handled in a post_save hook of the model; Works automatically for all models with an associated RBACed NamedModelViewset
* The actual registry api endpoints require a valid token that is sufficient to decide on the permission
* The missing link is the token generation endpoint
* It must check for permissions of the repo/distro/namespace and grant scoped push/pull
* It should probably follow the access_policy of the distribution viewset

AI - ipanova will open a story - RBAC for token_auth disabled DONE
AI- ipanova will open a story - teach token to always use up-to-date policy
AI- dkliban will start doc with tests plans - DONE

Pulp 2:

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### Dec 14, 2020

AI review
* AI open a task to add OCI tests in CI--> Q1
* AI open a PR to add pulp registry to the list of registries that support OCI
* https://github.com/opencontainers/oci-conformance/tree/master/distribution-spec
* review meeting notes AH+Tower meeting https://docs.google.com/document/d/1f4fNBlZAJRn7zPSL4ugaWdc3NUDuv9DtRaqxZjP_Xrk/edit
* full registry api + RBAC + token is needed by end of Jan
* stress testing and coverage for push api
Pulp 3:
* OCI image builder follow up with AH
* dkliban schedule a meeting
Pulp 2:
* pulp 2 syncing from pulp 3 bug - https://pulp.plan.io/issues/7923

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### Dec 7, 2020

AI review
* AI open a task to add OCI tests in CI--> Q1
* AI open a PR to add pulp registry to the list of registries that support OCI
* https://github.com/opencontainers/oci-conformance/tree/master/distribution-spec
* review meeting notes AH+Tower meeting https://docs.google.com/document/d/1f4fNBlZAJRn7zPSL4ugaWdc3NUDuv9DtRaqxZjP_Xrk/edit
* full registry api + RBAC + token is needed by end of Jan
* stress testing and coverage for push api
Pulp 3:
* OCI image builder folow up with AH
* dkliban schedule a meeting
* FIPS
* ipanova will file a task
* RBAC
* ipanova will file task to add rbac for distributions
Pulp 2:
* pulp 2 syncing from pulp 3 bug - https://pulp.plan.io/issues/7923

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### November 23 date, 2020

AI review
* AI open a task to add OCI tests in CI--> Q1
* AI open a PR to add pulp registry to the list of registries that support OCI
* review meeting notes AH+Tower meeting https://docs.google.com/document/d/1f4fNBlZAJRn7zPSL4ugaWdc3NUDuv9DtRaqxZjP_Xrk/edit
Pulp 3:
* prioritize move from travis to gha
* rbac for repos depends on rbac for remotes. ipanova to unblock mdellweg
* reviewed opened PRs and untriaged bugs
Pulp 2:

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### November 9, 2020

AI review

Pulp 3:
* add pulp https://github.com/opencontainers/oci-conformance/tree/master/distribution-spec + enable OCI tests in CI
* AI open a task to add OCI tests in CI--> Q1
* AI open a PR to add pulp registry to the list
* https://pulp.plan.io/issues/7805 immutable tags
* https://pulp.plan.io/issues/7795 associate remote with repo
* https://pulp.plan.io/issues/7790 re-upload of artifacts

Pulp 2:

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### November 2, 2020

AI review

Pulp 3:
* OCI image builder - dkliban will open PR to re-enable tests and make it work against the s3 deployments
* ipanove will start on RBAC for remotes
* mdellweg will address comments and finalize namespace PR
* work in progress on collecting usecases from AH/Tower for the test coverage

Pulp 2:

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### October 19, 2020

Pulp 3:
* Namespaces
* proposal - merge into pulp_container repo and not wait anymore
* let's do that
* Matthias will look into race condition issue get_or_create()
* RBAC for repos and remotes, add to the sprint
* https://pulp.plan.io/issues/7706
* https://pulp.plan.io/issues/7707
* 3 months planning
* RBAC
* FIPS

Pulp 2:

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### October 12, 2020

AI review

Pulp 3:
* Namespaces: design decision around namespace less distribution.
* https://github.com/pulp/pulp_container/pull/158#issuecomment-698328942
* waiting on feedback from pulp-dev list
* plan to ask jsherill for feedback
* does it make sense to make it master/detail if merging the functionality into the pulpcore repo?
* We plan to start defining RBAC work for "everything but namespaces"
* start with repo and remote
* AI open 2 tickets
* remove authors file
* https://github.com/pulp/pulp_container/blob/master/AUTHORS
* AI ipanova will submite PR and ask Brian on legal implications in case he is aware
* 3 months planning
* focus on RBAC

Pulp 2:

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### September 21, 2020

AI review

Pulp 3:
* Namespaces https://pulp.plan.io/issues/7089
* check how podman clients work with repos without slash
* always force a namespace or allow none? Allow none.
* allow library namespace as default or not? Drop this.
* namespace without /
* image name 0+ / ( mongo or mongo/test)
* namespace is part of relative path of distribution
* RBAC considerations:
* namespaces can allow creation of new repos via push or deny
* users own namespace will be created on the fly if neccessary
* user cannot create other namespace via push
* OCI image builder
* should we switch to Push container repo and use push api instead of pulp api? https://github.com/pulp/pulp_container/blob/master/pulp_container/app/tasks/builder.py#L77
* https://github.com/pulp/pulp_container/blob/master/pulp_container/app/tasks/builder.py#L135
* Add docs

Pulp 2:

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159
*
### September 2, 2020

AI review

Pulp 3:
* https://pulp.plan.io/issues/7419 docker notary
* Namespaces https://pulp.plan.io/issues/7089
* check how podman clients work with repos without slash
* always force a namespace or allow none?
* allow library namespace as default or not?
* namespace without /
* image name 0+ / ( mongo or mongo/test)
* namespace is part of relative path of distribution

Pulp 2:

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159

### August 24, 2020

AI review

Pulp 3:
1. retention policy https://pulp.plan.io/issues/7358
We do not provide any way on how to remove content from Push repo
Docker API provides DELETE on blobs/manifests endpoints
2. rollback for push repos https://pulp.plan.io/issues/7357
3. As a user I can push content to a repo but make it available later https://pulp.plan.io/issues/7286
4. As a user I can hide tags from a repo when distributing content
5. As a user I can sync manifests from a remote registry by digest
6. As a user I can mirror a subset of platforms https://pulp.plan.io/issues/7379
7. Namesfpaces
Usecases:
1. As a user I can create a repository within a namespace
registry_path = namespace/repo_name
* ~~to support multi-tenancy~~
2. As a user, a distribution has a namespace
3. As a user, a distribution and a repository can have the same namespace (pulp_container push repo)

Pulp 2:

Open PRs:
* https://github.com/pulp/pulp_container/pulls

Un-triaged bugs:
* https://pulp.plan.io/projects/pulp_container/issues?query_id=159