owned this note
owned this note
Published
Linked with GitHub
# FreeBSD OCI Runtime Extension WG
###### tags: `oci` `working group` `freebsd`
Every other Friday beginning May 24 at time: 1730 GMT (1200 EST; 900 PST; 1800 CET; 1900 CEST; 0300 AEST; 0100 CST)
> damn timezones
- [Conference URL](https://us06web.zoom.us/j/86975374751?pwd=kNtu4rsMmqsx70aWdV8IkTXpq5LqPN.1) with embedded passcode
- One tap mobile
[++13092053325](++13092053325,,86975374751#,,,,*488978#) US (New York)
Passcode: 488978
Find your local number: https://us06web.zoom.us/u/kdX1cGEIDk
## May 2, 2024
### Attendees:
- Greg
- Doug
- Nour
- Ed
- Samuel Karp
### Note Taker:
- Greg
### Actionable Agenda Items:
- Need to confirm this time will work for all participants or find a new time
- Reschedule tent. Fri May 24 12:30 eastern - GW confirm with Bjorn
### Presentation/Discussion Agenda Items:
- DFR - updated the OCI-Jails proposal - added a table to show how baseline Jails parameters map to the spec
- Topic: how to define devfs parameters
- in Devfs, you can tell it which predefined ruleset to use to determine which devices to expose using a built-in ruleset, exposes the bare minomum.
- either add new ones
- or modify the ruleset
- want to be able to add devices like GPU inside the jail
- DFR has added as a type of sudo mount option in Jails to add extra rules to the devfs baseline ruleset
- first mount devfs then you can add to it using the sudo option
- not a very elegant solution. it's convenient
- QUESTION - how do we define the devfs once we have the runtime spec extension?
- EM - agrees sudo is not the best
- How complex will the devfs rules be?
- DFR - fairly simple stuff - e.g. in linux I want to unhide [ ] and mount a tempfs
- for non NVIDIA gpus unhide devdrm
- this is convenient since no other place to put the rules, but would like a better place?
- Linux is different since they do not have devfs
- SK - in Linux thre's a device list - cgroups: https://github.com/opencontainers/runtime-spec/blob/main/config-linux.md#devices
- in podman, this happens in podman, not in the runtime. But since the runtime is where we substantiate devfs, we need a solution
- Maybe it's adding an extra stanza to the FBSD part of the config
- **could be in the CLI
- container engine (say podman) does the policy and then the policy is turned into a set of actions for the runtime to implement. This may keep it tidier to keep it in the config.**
- This would work across container reboots
- one option (not preferred) is to create a ruleset on the fly and you give it a number
- Still interestedin more user stories - @c5mx3H7dR66_kNeuXhhrGA re-up the call for user stories
- DFR will address the issues that have come in
- To set up the network state for a container, DFR using bare min of network plugins
- covers podman and crio, could be used for containerd
- Now podman is moving in a different direction - they are going with NetAvark - it's very fast, and podman wants it to be the only model - slightly relaxed on timeline. We need a port of NetAvark
- someone needs to understand rust, needs to understand netlink, and then do the work to get NetAvark to work on FBSD. will involve reaching out to rust standard library
- in podman 5, CNI support is still there - opt in at compile time. not sure, but prob removed by podman 6
- EM - if we ask them not to remove, it will stay for a short while, they'll keep longer if we say we have a plan to add netavark to FBSD
- Nour: Would like to help on the netavark port and it will be a growth and learning opportunity
- DCH - there are others who will help and review. Ed can help review as well
- DFR - create issue in rust standard library to start as first step
- second approach podman
- SK - netlink useful to have
- DCH, based on DFR's notes building containers from scratch and how to use repositories (local, Oracle Cloud, and one other one) - hoping to get signed container releases from public registries with v 14.1. Working on getting signed images come out of RE
- what about an official FreeBSD registry? (Nour) - this is an open discussion, and there is a potential to have it in dockerhub and also some open source options - Zot is one DFR has been working wiht - very open and collaborative and a nice system. Maybe a FreeBSD port for Zot would be good? Zot project build FBSD binaries
- Cosign process - we make signatures for the images. sigs hosted by an webserver. put sigs in the correct place, publish config files in the right place. as long as we can sign during hte build process, we can have all images checked against the sigstore
## April 15&18, 2024
### Attendees:
- Sam Karp (4/15)
- Doug Rabson (4/15)
- Bjorn (4/15)
- Greg Wallace (4/15 & 18)
- @dch (4/18)
- Ed Maste (4/18)
- Mohammad Noureldin (Nour) (4/18)
### Note Taker:
- Greg
### Actionable Agenda Items:
- In-person Meeting at Container Plumbing Days
### Presentation/Discussion Agenda Items:
- Sam, Doug, Bjorn, and Greg spent about ab hour on Monday, 4/15 discussion how to describe jails in a way that is expressive enough to cover as many use cases as possible.
- TL;DR - Agreement to puruse a flat(ter) description that keeps the Jails docs as the canonical source.
- @DFR has taken the action to begin creating a table with the list of things in the Jail parameters and map each to some place in the OCI spec
- Either someplace in the common spec
- Or in the FreeBSD extension
- In addition, we need to write an implementation
- Also must create "What is a full example of everything we can specify"
- translation semantics from image to runtime and from linux to FreeBSD, at least as recommendations/guidance
- Complete discussion notes can be found here: https://docs.google.com/document/d/1AwJX-iTHsfykvppNHCy89DWi6m0BJmMfyEDXvr2x64A/edit?usp=sharing
- Thursday April 18 Greg met with Nour and Ed and Dave to provide an update on the Monday in person meeting.
## April 4, 2024
### Attendees:
- @dch
- Greg Wallace
- Ed Maste
- Mohammad Noureldin (Nour)
### Note Taker:
- Greg
### Actionable Agenda Items:
- Who's taking notes?
- Greg will
- Time for meeting at Container Plumbing Days?
- Our next scheduled meeting is on April 18, which is during open source summit seattle. plan is to keep the meeting and use it to brief everyone on meetings during container plumbing days (april 15)
-
### Presentation/Discussion Agenda Items:
- Determine if there will be a Container Plumbing Days at Open Source Summit Europe in Vienna
- Review of Requirements Open Issues
- joh-ku potentially out of scope for the runtime spec, but this is very valid for what conatiners on FBSD on the whole need to facilitate, but not part of the runtime spec
- Nour - the last line may be applicable to the runtime. @DFR thoughts?
- gizahNL - DCH - key is where are resources defined and allocated. in Jails it's done outside Jail. in OCI, definition is in the container, and execution in in runtime.
- "Proposal A" matches the jails approach: https://github.com/opencontainers/wg-freebsd-runtime/blob/simple-jail/docs/proposals/PROPOSAL_A.md
so, vnet jails would continue to exist
- DCH - from a user perspective, as sysadmin I want to ____
- what goes where?
- networking - CNI - does it exist today? is it set up seperately or is it set up in the container world
- Doug created a plugin for CNI as part of podman work for container-based jails on FBSD
- dch understanding does it with pf, but you still need to set up redirects for pf and netanchor. you still need to write the ifconfig for network devices. do we want the containers to do this kind of work then the runtime needs to manage this metadata. questions like which subnet, etc you want it to connect to. maybe we can delegate this to the plugin...
- Nour: in docker, you have network management features and you can attaches different networks to different containers. Greg: can we use podman network management capabilities?
- is network management governed by any specification?
- Yes - https://github.com/dfr/plugins?tab=readme-ov-file
- https://www.cni.dev/
- @dch has a set of [working notes] covering using Doug's tools
- setup of podman & local registry
- fetching & running existing containers
- creating new containers using Doug's existing [freebsd-images](https://github.com/dfr/freebsd-images) scripts
- using an external registry
- Need to get this officially hosted, and come up with a roadmap that lines up to the FBSD release schedule. Should be a small number of trusted sources
- Nour happy to help out with this. @dch will do discussion on Jails IRC. We want to be able to distribute official FreeBSD images and this requires something like AWS "gallery" and they also have official images. E.g. there is an official Node.js image that can be pulled from the docker and AWS registries
- @dch wants to know how the image can be trusted if it's hosted on 3rdparty infrastructure?
- this will be driven by how much work is there to do within Foundation & ClusterAdmin
- @mnour For example, https://hub.docker.com/_/python you can look at this to see how you determine that this is trusted - there is a process to get the "Official Docker image" mark
- @mnour has reviewed that and he was not quite correct, more information can be found in [Docker Hub Trusted Content](https://docs.docker.com/trusted-content/)
- we need to define what it means for us for the image to be "official"
- @dch - it's about provenance. for every FBSD release, there's a signature published out-of-band on the FreeBSD website, and the announce@ mailing list, signed with PGP key by re@ team members. If this sort of signature can be attached to published images, and is easily verified, then we can in principle host images in multiple places
- this is in part handled by the OCI specification - part of the image and runtime specs - image distribution spec signs the images
[working notes]: https://docs.skunkwerks.at/s/fUiAmi4pE#
## March 21, 2024
### Attendees:
- @dch
- @dfr
- Greg Wallace
- Ed Maste
- Mohammad Noureldin (Nour)
### Note Taker:
- Greg
### Actionable Agenda Items:
- Who's taking notes?
- Greg will
- any thoughts on scope?
- e.g. inherit vs. alias vs. vnet, routing/bridge/netgraph setup, ZFS (snap, clone, block dedup?)
> More detailed questions below.
- see https://github.com/opencontainers/wg-freebsd-runtime
### Presentation/Discussion Agenda Items:
- Went around and did introductions and why each is interested in this area
- Having good OCI support on FreeBSD may help in heterogeneous Linux/FreeBSD environments
- first topic
- looking at github - requirements.md should be a list of use cases, maybe grouped
- lets' first get a consensus view on use cases
- please submit PRs if you have changes or additions to the use cases
- TO DO: Let's put out the request for input
- We can use Slack for async, and this meeting for sync
- Second - we need to describe Jails
- [Simple jails](https://github.com/opencontainers/wg-freebsd-runtime/tree/simple-jail) see [proposal A](https://github.com/opencontainers/wg-freebsd-runtime/blob/simple-jail/docs/proposals/PROPOSAL_A.md)
- proposal on a branch
- this proposal is secure
- it is simple and flat versus layered
- network/address control is out of scope for OCI runtime
- container networking plugins take instructions form the container engine and do what they need to set it up on the network
- discussion around configuring and declaring container engine port mapping, using podman and this from docker was provided as an example
```
docker run --rm -it --network local.dimanex.com -p 127.0.0.1:9002:9002 \
--network-alias=api.localhost.dimanex.com \
--name dimanex-api --volume "$(pwd):/src-repo" \
--volume "$HOME/workspaces/dimanex/mvn_home:/mvn_home" \
--workdir /src-repo dimanex/arm64v8/jvm-java-8-run-base:1.0.0 \
./mvnw -e -Dmaven.repo.local=/mvn_home/repository spring-boot:run
```
- splitting of functional pieces from a former monolith driven by K8s. Now have a more modular ecosystem that provides more choice for users and developers
- Next steps we need to describe jail in a way that is expressive enough to cover as many use cases as possible. Need to get the whole group together, including Sam, to discuss containers and jails
- try to get meeting with Sam prior to Cloud Plumbing Days. If not, then discuss this topic there
- Goal: by end of CPD we would like to know which direction we are going in.
- Doug will convert his branch into a PR so others can review/comment
- How handle topics like networking, mounts (specified as cross ___ in specification) including ZFS, runtime instructions
- keep it freeform and feel free to add a section to use cases
- with mounts you get options (read only, NFS v4, etc) dfr added sudo option for ZFS rules that gives a straightforward way for the engine to translate sudo options into rules in Jails. this can be found in OCI Jails code in `mount.cpp`
- we need a way to describe ZFS in the container
- Netgraph networks - complex network setups - are these also out of scope?
> [name=crest] Sounds like it should be handled by a CNI plugin?
> [name=Greg Wallace]
> this can be isolated inside the container networking plugin itself, so you can use netgraph, bridges, whatever
- What are the ideas around implementing this runtime extension?
- we don't need to wait for the spec to be published before we start testing and implementing
- Linux containers have a PID, FreeBSD are identified by a JID (also 32bit int)? Does it make sense to put the JID into the PID field? What would be the implications?
- Should the OCI runtime spec assume that jails are always persistent to split create and start or should the runtime keep a placeholder (child) process in an otherwise empty ephemeral jail?
- Is there a collection of "reference configurations" that should be expressable?
- Foreground vs Background: FreeBSD jails (as configured via jail.conf) are mostly run detached in the background. It's my impression that most (Linux) container orchestrators conceptually run containers in the foreground. Does the runtime spec say anything on this and if so what?
- Which parts of the process execution environment are expected to the specified via an OCI runtime configuation file? Is any of it implicit and unconfigurable (yet different from FreeBSD's default)?
- What CNI/CSI plugins are required on top of the runtime spec?
- to support a single lab server?
- a development system?
- a CI/CD pipeline?
- to interface with commonly available network and storage envs?
- to support "production" cluster?
- Workflows instead of technical features?
- Let's also talk about Cloud Plumbing Days and how we can use that time to meet and make progress
- _add another agenda item_
### Notes:
- _add your notes_