FreeBSD runtime extension WG

# FreeBSD OCI Runtime Extension WG ###### tags: `oci` `working group` `freebsd` Every other Monday at 1200 EST Join Zoom Meeting https://us06web.zoom.us/j/89400251044?pwd=bthdC5BOwQlahSaKlCxOFKSJ5PZba7.1 Meeting ID: 894 0025 1044 Passcode: 600421 --- One tap mobile +13092053325,,89400251044#,,,,*600421# US +13126266799,,89400251044#,,,,*600421# US (Chicago) Find your local number: https://us06web.zoom.us/u/kz5GNMf0Z # September 8, 2025 - Alice - Y - Bjorn - N - Carmi - - Doug - - Dave/dch - - Ed Maste - Y - Jan Bramkamp - Y - Matías - - Sam - ## Updates: * OCI Runtime Extension Pull Request is submitted and undergoing review: https://github.com/opencontainers/runtime-spec/pull/1286 * There has been a request for reference implementations for ocijail and runj. * A reference implementation was provided for ocijail but we don't have one for runj yet. * Not having a reference implementation for runj has led to FreeBSD being removed from the 1.3 update for OCI spec. We believe that the next version is a year away. * Unclear if we can get it added back - timeline? * UPDATE: upon checking, it looks like FreeBSD got added back. * 1.3.0 https://github.com/opencontainers/runtime-spec/issues/1295 * 1.4.0 https://github.com/opencontainers/runc/issues/4875 * [Comment from Brandon Mitchell on Slack](https://opencontainers.slack.com/archives/C06HF6D0GBV/p1757260682116549?thread_ts=1757237429.902739&cid=C06HF6D0GBV) "The runc app follows the runtime spec, and they have a plan to make a runc release which is forcing a runtime spec release. Some things may get bumped to the next runtime spec release so they don't block the runc release." * The release date for runc 1.4.0 is October 31, 2025. * for reference, runj is using an older copy of the runtime spec: https://github.com/samuelkarp/runj/blob/main/runtimespec/config.go * Possibly worth thinking - is the runj/spec conflict to be considered a runj compliance issue (i.e. we consider runj is not compliant) and ask for the spec to be moved forward with as it is. What was the intention from Sam? Presumably to test the spec and if any updates could be made to help runj run better, they should be added. NOTE: It's also possible to have runtime extensions on the spec. Could it be captured in an issue rather than a pull request on runj. # August 25, 2025 * https://www.linkedin.com/posts/ianwevans_freebsd-gpu-opensource-activity-7362629327881916416-NOQo * some comments about containers on this post * https://github.com/apple/containerization shared by Greg W # July 28, 2025 - Alice - Y - Bjorn - N - Carmi - - Doug - - Dave/dch - - Ed Maste - - Jan Bramkamp - - Matías - - Sam - ## Updates: * OCI Runtime Extension Pull Request is submitted and undergoing review: https://github.com/opencontainers/runtime-spec/pull/1286 * Sam Karp has reviewed and approved the PR. There is some feedback that Doug needs to address, however he has approved the PR merge. * Remind Doug that Greg Wallace's offer to document the project still stands. - Done. * Alice cannot attend the next two calls. * Replacement call * Cloud Native Container Technologies * Stated goals. Identifying necessary next steps to building up the ecosystem to be able to make FreeBSD a first-class container OS e.g. orchestration, creating a library of useful images (top 10-50 from Linux and making the equivalent for FreeBSD) and tie this in with build materials projects. * Host the meeting under FreeBSD not OCI. * Start with same time and day. * First call on the 8th September 2025. * Where to publicise? In the announcement of the OCI runtime spec merge, in the jails mailing list, ## Actions previous calls * Doug (not Alice) will email Sam - done ## AOB * Add item here ## Actions this call * Doug is going to address Sam's comments this week. Before Thursday when Carmi joins the OCI developer call. # July 14, 2025 - Alice - Y - Bjorn - N - Carmi - Y - Doug - Y - Dave/dch - N - Ed Maste - Y - Jan Bramkamp - Y - Matías - N - Sam - N ## Updates: * OCI Runtime Extension Pull Request is submitted and undergoing review: https://github.com/opencontainers/runtime-spec/pull/1286 * Alice and Carmi went to the developer call a few weeks ago. And Carmi has attended them all. They were very nice but were not able to help there and then. They recommended reaching out async as not all developers for the runtime extension always attend that call. There has since been some progress but it looks like we just need Sam Karp to give the final nod whether in the call or async. * Carmi - Doug and Carmi started to explore what are the next most useful things that we want to work on. CRI-O stuff is nearly ready but blocked behind the PR. Similar with K8s but not as advanced. These are all the parts needed to make a functioning system for cloud based containers on FreeBSD. * How do we support work that will build on top of this? Getting people to adopt FreeBSD as a supported OS, developing tooling through working groups like this one. * perhaps office hours on the OCI spec (for FreeBSD) would also be helpful. * A beginners guide to cloud native containers on FreeBSD may also be a good next step. * Doug has been writing scripts to spin up a k8s demo https://github.com/dfr/kubernetes-demo * very demo-ware * still working on it * fine to share ## Actions previous calls * Alice & Carmi to attend the dev call on Thursday to champion the review. DONE. ## AOB * Add item here ## Actions this call * Doug will email Sam (not Alice) # June 16, 2025 - Alice - Y - Bjorn - - Carmi - Y - Doug - Y - Dave/dch - - Ed Maste - - Jan Bramkamp - Y - Matías - - Sam - ## Updates: * OCI Runtime Extension Pull Request is submitted and undergoing review: https://github.com/opencontainers/runtime-spec/pull/1286 * Has been difficult to attend due to scheduling. * Doug will take another look at the feedback * Alice & Carmi to attend the dev call on Thursday to champion the review. * Apple’s Swift based Replace for Docker. Open source with a BSD compatible license. They also announced that FreeBSD was now a first class supported platform. https://developer.apple.com/videos/play/wwdc2025/346 Check out Meet Containerization from #wwdc25 ## Actions previous calls * Add item here ## AOB * Add item here ## Actions this call * Alice & Carmi to attend the dev call on Thursday to champion the review. # June 2, 2025 - Alice - Y - Carmi - Y - Doug - Y - Dave/dch - - Sam - - Ed Maste - - Matías - - Bjorn - - Jan Bramkamp - Updates: * OCI Runtime Extension Pull Request is submitted and undergoing review: https://github.com/opencontainers/runtime-spec/pull/1286 * Doug has figured out which call to attend to advocate for it, and aims to attend soon. (It's weekly.) * In future, some additional spec details will be added for more features like ZFS dataset handling. * The details of the OCI images will be in the 14.3 release announcement. * Alice absences * Fri Jun 20 - Fri Jul 4 inclusive. * Missing planned call on June 30th * Mon Aug 1 - Fri Aug 15 inclusive. * Missing planned call on August 11th * Mon Aug 25th - UK public holiday. * Missing planned call on August 25th # May 19, 2025 - Alice - Y - Carmi - Y - Doug - Y - Dave/dch - Y - Sam - Y - Ed Maste - N - Matías - Y - Bjorn - - Jan Bramkamp - ## Notes Updates: * OCI Runtime Extension Pull Request is submitted and undergoing review: https://github.com/opencontainers/runtime-spec/pull/1286 * Some feedback, including on wording. * Some questions about jails namespace semantics to make it make sense to those less familiar with jails. Particularly networking behaviours like inheriting IP addresses. * To merge the PR, it has to be accepted by at least 2 people who own the repo. It could be useful for Doug to attend the OCI weekly dev call and add it to the agenda. * Some of the feedback suggests making it a little clearer (at least considering making it more visible) what the underlying must/should requirements are for jails. * Images now public on: https://hub.docker.com/u/freebsd is public and the github equivalent https://github.com/orgs/freebsd/packages * Still need to document how to create and tag an image. * Also some additional networking functionality. * Using hooks to mount datasets for a container. Seems a partial solution due to some challenges about differentiating between containers. * Our naming scheme is now here: https://docs.skunkwerks.at/s/67b0XqVUV and will be our tags going forwards. ## Actions previous calls * Reviews from anyone. ### AOB * What would be a good announcement that the Foundation can help to get out there? * Suitable for a demo (dch). Could we get a linux containers whizz who can help review the docs and demos from Dave's work. Dave already sending things to Mark. Carmi can help testing things on other hardware if needed. * What do we do with this call once the PR is merged? * Will the spec need extending? * We could make this a cloud native containers call * K8S build? Doug hasn't been working on this for a while. Goal would be to make a FreeBSD Port for kubelet and kube-ed. ### Actions this call * Add item here # May 5, 2025 * Doug - Y * Carmi - Y * Sam - Y ## Notes ### OCI runtime extension * Doug opened https://github.com/opencontainers/runtime-spec/pull/1286 * This would complete the working group once it is merged # April 21, 2025 - Alice - N - Carmi - Y - Doug - Y - Dave/dch - N - Sam - Y - Ed Maste - Y - Matías - Y - Bjorn - N - Jan Bramkamp - N ## Notes ### OCI runtime extension * Doug update * Runtime spec. * Other ### Actions previous calls * Once the PR is published, if anyone can take a look that would be helpful. ### AOB * add item here ### Actions this call * add item here # April 07, 2025 - Alice - Y - Carmi - Y - Doug - Y - Dave/dch - Y - Sam - - Ed Maste - Y - Matías - - Bjorn - - Jan Bramkamp - ## Notes ### OCI runtime extension * Doug update * Runtime spec. * Close to making a pull request. Tidying up and want to give it another read through. Make PR this week. * Once it's published, if anyone can take a look that would be helpful. * Looking forward to feedback from upstream (OCI) * Other * We met with Sam K last week. He doesn't have much time to keep on supporting runj. He would like to change the model for contributing/ownership to stop it from being stagnant. Perhaps add it under containerd. * Doug offered to work on it to make it conform to the FreeBSD runtime spec. * It would be nice to have more functionality in runj. ### Actions previous calls * Sam K to review the Proposal. DONE. * Bjorn to review the Proposal. DONE. He had some queries about camel/snake case which can be cleared up after the PR is raised. ### AOB * Swift should soon be supported on FreeBSD. Carmi will let us know. * Sam still considering future of runj. * Dave, by the next meeting he hopes to the publishing commands to distribute of cloud images to various clouds. ### Actions this call * Once the PR is published, if anyone can take a look that would be helpful. * Add action # March 24, 2025 - Alice - N - Carmi - Y - Doug - Y - Dave/dch - Y - Sam - Y - Ed Maste - Y - Matías - - Bjorn - - Jan Bramkamp - ## Notes ### OCI runtime extension * Doug update * Runtime spec * Other ### Actions previous calls * Sam K to review the Proposal. * Bjorn to review the Proposal. ### AOB * Nour has stepped back from FreeBSD work indefinitely. Alice and Nour were collaborating on building some joined-up momentum on Cloud Native Tech work for FreeBSD. Alice can't do this alone. Does anyone have any interest in helping (or how to find someone who wants to)? ### Actions this call * Doug & dch coordinating effort outside of this call # March 10, 2025 - Alice - N - Carmi - - Doug - N - Dave/dch - - Sam - N - Ed Maste - N - Matías - - Bjorn - N - Jan Bramkamp - N ## Notes * Carmi * working with Doug on getting an existing system on board with podman * a vanilla system is easy, but what are the missing pieces * Matias * discussions on generic API management * users, storage, networking * Matías is going to implement something in Django as an example * Carmi will write some use cases ### OCI runtime extension * Doug update * Runtime spec * Other ### Actions previous calls * Sam K to review the Proposal. * Bjorn to review the Proposal. ### AOB * Nour has stepped back from FreeBSD work indefinitely. Alice and Nour were collaborating on building some joined-up momentum on Cloud Native Tech work for FreeBSD. Alice can't do this alone. Does anyone have any interest in helping (or how to find someone who wants to)? Netavark - ### Actions for this call * Another host needed for call on March 24th - Alice is on vacation. # February 24, 2025 - Alice - Y - Carmi - Y - Nour - Y - Doug - Y - Dave/dch - - Sam -Y - Ed Maste - Y - Matías - - Bjorn - - Jan Bramkamp - Y ## Notes ### OCI runtime extension * Doug's update * Been working on Podman updates (not directly related). * FreeBSD support for cri-o - the PR that has been open for a year has finally been accepted upstream !!!! Big step towards K8s working. Now just a few things need adding until that can work. * What do we want to do about upstreaming jail-specific functionality? * Runtime spec - do we update the helper lib - yes, we should. It's needed for changes to the spec for podman. Nour said he can help. Also needed, unit tests. ### Actions from last call * Sam K to review the Proposal. Still to do. * Carmi to review the Proposal for technical feedback.Done. All looks good. * Bjorn to review the Proposal. * Nour also reviewed it. Had a couple of questions. ### Actions for this call * Sam K to review the Proposal. * Another host needed for the next call - Alice has a clash. Nour has volunteered. ### AOB * Nour and Alice have added more issues into the project repo for Cloud Tech https://github.com/orgs/FreeBSDCloudTech/projects/2 * Dave(dch) and Nour to meet soon (Wednesday, 26 Feb). * Nour also had a chat with Bjorn about questions Bjorn raised during last call over CNI vs Netavark and why Podman decided to move away from CNI in favor of Netavark amongst other things related to containers networking on FreeBSD. * Sam - KubeCon EU / meet in London, April 1-4? Perhaps in an evening. * Alice - Yes (day is better than evening so I can get home) * Doug - Yes (lives in London) * Carmi - Hopefully (tbc) * Nour - Hopefully (tbc) # February 10, 2025 - Alice - Y - Carmi - Y - Nour - Y - Doug - Y - Dave - Y - Sam - Y - Ed Maste - Y - Matías - Y - Bjorn - Y ## Notes ### OCI runtime extension * Doug update * Carmi has been reviewing the doc. Language/grammar all good. Would like a technical review before raising the PR upstream. * Bjorn was able to read it through, will put a PR for a couple of typos etc. Is camelCasing a considered decision as it's more likely to show up in UI context? (Doug: the rest of the spec uses it. Bjorn, may be worth reviewing again when doing the upstream PR to see whether underscores/snake_case are better.) * Sam, please could you also review? * Sam: there has been some change to the network approach which would allow some jail-specific functionality to be upstreamed into the runtime extension. * He did get one person asking if FIBs are supported but feel it may be out of scope for this proposal (which seeks to cover base use cases only). * Matías can share the PR ID if it's ready. Doug should have the first draft for this group only, ready soon. ### Actions from last call: * Carmi will review the PROPOSAL_A.md file before next call. * Doug will start to draft the Pull Request and incorporate Carmi's feedback. * Alice: ask Sam to share the bash script that is in use for other projects had used. * Samuel Karp "Unfortunately I don't think one exists. I did a bit of looking but the bash scripts in use are mostly for pulling rather than pushing." * Alice: To follow up with Bjorn Neergaard about attending these calls. * Sends apologies as work has been so busy. Hopes to attend today. * Matías: Message re: next release -> The release plan is tracked in https://github.com/opencontainers/runtime-spec/issues/1274 We haven't received a PR for supporting FreeBSD. Probably it can be safely added in vNext, if its implementability is confirmed with runj or something else. ### AOB * Carmi - bug question * Nour created https://github.com/FreeBSDCloudTech and Alice created project https://github.com/orgs/FreeBSDCloudTech/projects/2 that we want to populate with work items to start to build more engagement. * Sam - KubeCon EU? ### Actions from this call: * Sam and Bjorn to share networking proposals/use cases as they have some different references that the other might find useful. * Sam will aim to review at a technical level this week. * Bjorn will also do a deeper dive this week (hopefully) to refer to the jails man page. * Capture request for FIB on the proj-cloudtech board? # January 27, 2025 ### Attendees - Alice - Yes - Carmi - Yes - Nour - Yes - Doug - Yes - Dave - No - Sam - - Ed Maste - Yes - Matías - Yes ## Notes OCI runtime extension * Doug: update on preparing a draft PR for review on https://github.com/opencontainers/wg-freebsd-runtime * PR merged https://github.com/opencontainers/wg-freebsd-runtime/pull/8 - the original proposal which was wordsmithed/amended a bit. * Is anyone available to review/sense check the current proposal https://github.com/opencontainers/wg-freebsd-runtime/blob/main/docs/proposals/PROPOSAL_A.md * There may be a vote at some point. * In a different (runtime spec repo) make a PR there with the changes integrated. * Matias took an action to find out the schedule for the next release. Has tried posting to the relevant mailing list but it's not yet showing up (is it being spam filtered?). Could contact directly the maintainers (we know their email addresses) unless this is against etiquette. Or possibly raise an issue in the runtime repo? Or ask a quetion in the Slack chat (OCI slack, General channel). * Please continue to test and log issues. Any Other Business * Alice & Nour would like to have a discussion and hopefully green light on Nour's proposal to create a GH Organization to collect all repos about cloud technology into one place, and also act as a place for a project repo that will help to increase visibility and engagement on the work. ## Actions * Carmi will review the PROPOSAL_A.md file before next call. * Doug will start to draft the Pull Request and incorporate Carmi's feedback. * Alice: Sam to share the bash script that is in use for other projects had used. * Alice: To follow up with Bjorn Neergaard about attending these calls. # January 13, 2025 ### Attendees - Alice - Yes - Carmi - No - Nour - Yes - Doug - Yes - Dave - Yes - Sam - Yes - Ed Maste - Yes - Matías - Yes ## Notes * Focus this quarter on getting the FreeBSD extension to the OCI runtime spec into PR and accepted. 1) Take the proposal that is in a PR, and commit that to the working group repo, and ask people to read through and iterate on it. GH tools are there to help facilitate that process - please comment/propose changes! We want to get this as functional as possible first time. We also need to put Podman and ocijail changes in at the same time and these need to be in harmony with the spec. 2) Then when we reach consensus, prepare it as a PR against the spec itself. * Review cycles upstream. We would like to catch the next OCI runtime spec release. Need to check with the release team on when that will be and what the process is. https://github.com/opencontainers/runtime-spec/tags * https://github.com/opencontainers/runtime-spec/blob/main/RELEASES.md * Alice reached out to Bjorn Neergaard to ask him to attend calls in Q1. * If time: * Dave: building/uploading images without podman - is this a fool's errand? * Wanted to find a way to build and upload images without a large toolchain (no podman), have been using Buildah and cURL, some reservations (Doug) will look and feed back to Dave. Maybe Skopeo would be a suitable tool. Sam may have an existing tool/bash script that could help. * https://stackoverflow.com/questions/59841918/docker-private-registry-image-upload * Multi-arch images, what is different about them? You have to build all the arch images individually, then add them to an index that is an additional json document of OCI formats. * Nour requests a review of his video shared in Slack and also can be accessed here: [FreeBSD OCI Containers: State of The Union - BSD NL Nov 2024 Day Event](https://bsdnl.nl/video/2024/bsdnl_Nov_2024__fbsd_oci_containers_sotu.mp4) ACTIONS * Matías will ask release team if they have a date planned for the next release. * Sam to share the bash script that is in use for other projects had used. * Doug/Dave had agreed review Nour's video when the link is provided. * EVERYONE to keep an eye on the GH notification for Doug's draft PR. ## December 16, 2024 ### Attendees - Alice - Yes - Carmi - Yes - Nour - Yes - Doug - Yes - Dave - Yes - Sam - Yes - Ed Maste - Yes - Matías - Yes ## Notes - Cancel Dec 30th meeting? Yes - Dave's micro-update - Chasing gitadm/clusteradm to get keys to GitHub to be able to upload images. He has signed up to Docker Hub and has connected with the person who secured the namespace (Bartek). - the published release images include cpu architecture type, but inherits by default from the build host, which is incorrect. - this blocks producing multi-architecture images - but cosmetically makes no difference - - k8s follow-on group - what is a good moment to do that? First get a draft mod to the runtime spec to include FreeBSD and get it accepted/merged into the spec. Not far off being ready, needs a last pass, tidy, and then can be submitted. https://github.com/opencontainers/wg-freebsd-runtime/pulls - Releases include, artifact & checksum. Wanting to let the user cross-check/verify what they get from us, and on the container registry. You can fetch a checksum from a container registry. Then can extract it using the image id as the hash of the image config. - unpack tarballed OCI image - yields `index.json` - which yields - `jq .manifests[].digest index.json` "sha256:b98b8f5438a07e000a156ecb216529d8dfa191448ac3dc1f6add9848b4461d05" - this final b98... checksum is what the user would fetch - its the sha256 of the top level blob `blogs/sha256/b98...` which includes the dependent layers required by the image - Nour - did you get the recording of the talk you gave? Not yet. ## December 02, 2024 ### Attendees - Alice - Yes - Carmi - Yes - Nour - Yes - Doug - Yes - Dave - Yes - Sam - Yes - Ed Maste - Yes - Matías - Yes ## Notes * (Dave) Naming containers on GitHub, is there a convention? * Doug: It's OK if we don't get it right first time. Still experimental. The original thought was that the latest tag would be helpful. Maybe we follow Ports naming conventions: the latest patch of the oldest supported branch. * Sam: "latest" can be confusing because it can be understood by different people in different ways. Windows had a similar problem. Can we avoid it entirely? We could still perhaps find a way to embed compatibility data if needed. * Doug: we can use names that refer to either the specific version or the lowest supported. * Nour: this is an example from Node: arm64v8/node:20.15.0-bookworm-slim (you have to go to their docs to get the full explanation) * Carmi: is minimal referencing OS distro? * What is in Static container, e.g. Scratch? In ours, the SSL certs, password file, minimal termcat file. Only suitable for running an SSL application with static links. Dynamic allows dynamic linking, but neither supports shell. * Nour has released the slides for his talk: * https://bsdnl.nl/slides/bsdnl202402-Mohammad_Noureldin-FreeBSD_OCI_Containers_SoTU.pdf * And the whole list of presentations that day: https://bsdnl.nl/slides/ * For the video recordings we are still working on getting separate approvals from presenters and after that preparing the recordings to be shared online. Will share once this has been done. ### Chat paste ## November 18, 2024 ### Attendees - Alice - Yes - Carmi - Yes - Nour - Yes - Doug - Yes - Dave - No - Sam - No - Ed Maste - Yes - Matías - Yes ## Notes - Wrap up for 6-week test period for Podman on FreeBSD testing - Published [Podman on FreeBSD - End of test period report](https://github.com/oci-playground/freebsd-podman-testing/blob/main/End-of-test-report-Nov-2024.md) - Plus [blog "Advancing Cloud Native Containers on FreeBSD: Podman Testing Highlights"](https://freebsdfoundation.org/blog/advancing-cloud-native-containers-on-freebsd-podman-testing-highlights/) - Please join the [FreeBSD Mailing list for jails](https://lists.freebsd.org/archives/freebsd-jail/) where we will share the latest progress as we continue async to the end of the year. - Nour - [OCI container images are now being published](https://lists.freebsd.org/archives/freebsd-stable/2024-November/002528.html) - OCI image archives are being built now. 15 current and 14.2 stable contain these. Some problems on the 14 branch, Doug will look into it. - From our (BSD NL - https://bsdnl.nl) side, we held our BSD NL Day Event last weekend, Saturday 09-11-2024 where I gave a presentation to help bringing more attention to this WG and hope by that we get more people joining the cause. - Link to presentation? Hope to share in near future. - Roadmap/path towards for CCTLs (to build and launch a container), set the SYSV. - spec needs to be agreed in this group, there is a PR open that Sam has given some feedback on, needs another round of review. - some work in Podman needed to effect the change. It constructs the config file. - If time, can we go over [Issues](https://github.com/oci-playground/freebsd-podman-testing/issues) on the test repo? ## November 04, 2024 ### Attendees - Alice - No - Carmi - Yes - Nour - Yes - Doug - Yes - Dave - Yes - Sam - Yes - Ed Maste - Yes - Matías - Yes ## Notes - Nour general and netavark status updates: - netavark port testing still in progress. - Preparing a talk for [BSD NL Day event - Nov 2024](https://bsdnl.nl/). - Got in touch with some people over Mastodon and directed them to this group so we hopefully see more and more people joining. ## October 21, 2024 ### Attendees - Alice - Yes - Carmi - Yes - Nour - Yes - Doug - No - Dave - No - Sam - Yes - Ed Maste - Yes - Matías - Yes ## Notes - Podman testing update - Alice is writing up a report to add to the repo and share on the mailing list. Request for Doug's input on the details. - Alice will also invite people from the testing to join this call. - Nour status update on netavark: - Done with compilation errors. - Will spend a couple more days testing before pushing changes for (hopefully a final) review by both dfr@ and emaste@. - Matías has been trying out what's possible with the Podman on FreeBSD work. He had been looking for a way to orchestrate cloud native containers in a sustainable way, and for real-world use cases. He is also capturing some blueprints: container files, config files and resources in a GH repo. Also will test some GH actions to get it to build automatically, at least to see how far it is possible. Ideally, would see end goal to build a port and share back with the community. - *Watch the call recording (about 15m into the call) for the demo/walkthrough* - https://us06web.zoom.us/rec/share/dvkToeQReZoe-MbtiWEELT7fsoDYJMgJ7JM4h6C9LMrpj7qyhB5nW5DR4XgozO6G.vTGYyK7HQU3iwFnp - Passcode: k$Z6v6N$ ### Chat paste Messages sent during the meeting will be visible to all meeting participants You (21 Oct 2024, 17:03) https://hackmd.io/hq_NOVL4RZS7xYYMqfJ6-A https://hackmd.io/hq_NOVL4RZS7xYYMqfJ6-A Samuel Karp (21 Oct 2024, 17:14) Thanks! I don't think I'm a giant 😆 Ed Maste (21 Oct 2024, 17:15) Me too 🙂 I'm glad you're here presenting! Samuel Karp (21 Oct 2024, 17:32) I have to drop, unfortunately. Thanks for the discussion and demo! 17:42 - Meeting ended: 42m ## October 07, 2024 ### Attendees - Alice Yes - Carmi Yes - Nour Yes - Doug Yes - Dave Yes - Sam Yes - Ed Maste Yes ### Note Takers: - Alice, dch ### Notes - Podman testing update - this is the last week - good call last week with Alice Sowerby, Nour (Mohammed Noureldin), Matias Pizarro, Nathanael Lierly, Doug Rabson in attendance. - Matias is a new test participant. He is starting his own business and has employees who will be using a FreeBSD desktop, plus client machines to run FreeBSD for data science/ML. - We will be extending the testing opportunity. Need to agree on what that will look like. - Keep the repo open and look at the issues async, perhaps until new year. - Alice will be writing up a short report on the test project. - One of our test participants has started submitting PRs to Podman upstream. https://github.com/containers/storage/pull/2123#pullrequestreview-2352332720 - Likely to start using FreeBSD Jails mailing list for more OCI comms. - Update on Doug's Jail proposal PR for Sam's review. - There are a few items still being discussed. Close to being resolved. One is ZFS ruleset, where to document it. - Update on Doug's change to the build process to include container images. https://reviews.freebsd.org/D46759 - Ed also commented on this. - Close to being resolved/approved. - Dave will be testing this soon. Has some questions about package base build intricacies - Baptiste can probably help. - would be nice(tm) to make patch releases available not just for podman/OCI - no consensus yet from BSDcan about how to handle this with mirrors - Dave added https://reviews.freebsd.org/D46975 which frees up "OCI" name in release tooling for Doug to use - Update on Dave's tutorial for EuroBSDcon. How did it go? - I had some problems with the image build stage of OCI stuff, Doug already told me how to fix it - other than that, good. lots of interest. - Could be worth trying to invite people from tutorials next time :) - Dave has a few questions about networking, and ZFS which can wait until Doug is back from his break. - Carmi - just got test system up, requesting some time to help troubleshoot. Graphics card challenge. Will arrange a time. - Nour status update on netavark: - Down to the very last error which is more of the need to reimplement bridge network driver to be based on vnet in place of linux network namespace. Also learning more about jails networking. Hope to finish this week. ### Chat paste You (7 Oct 2024, 17:04) https://hackmd.io/hq_NOVL4RZS7xYYMqfJ6-A Doug Rabson (7 Oct 2024, 17:08) https://github.com/containers/storage/pull/2123#pullrequestreview-2352332720 ## September 23, 2024 ### Attendees - Alice Yes/~~No~~ - Carmi ~~Yes~~/No - Nour ~~Yes~~/No - Greg ~~Yes~~/No - Doug Yes/~~No~~ - Dave ~~Yes~~/No - Sam ~~Yes~~/No ### Note Takers: - Alice ### Notes - Podman testing update - Please help promote on socials, handy text here: https://github.com/oci-playground/freebsd-podman-testing/blob/main/promo-text.md - Testers: - joh-ku (GH name, who is this?) - Jonas Everaert (on Podman GitHub) - Bretton Vine - Anton Whalley - Nathanael Lierly - Mohammed Noureldin - We have found a few bugs/problems which has been helpful. - Sam will share the call for participants - Alice will get the email addresses for the current participants to see whether they have any feedback or thoughts. - Alice will ask Foundation to bump the call for participants on socials. - Doug will look at Jail proposal pull request that Sam made. - Doug is making a change to the build process to include container images. https://reviews.freebsd.org/D46759 ### Chat paste Doug Rabson (23 Sep 2024, 17:18) https://github.com/containers/podman/discussions/24023#discussioncomment-10705046 You (23 Sep 2024, 17:23) https://github.com/oci-playground/freebsd-podman-testing/blob/main/promo-text.md Doug Rabson (23 Sep 2024, 17:26) https://github.com/containers/podman/discussions/24023#discussioncomment-10705046 https://reviews.freebsd.org/D46759 ## September 09, 2024 ### Attendees - Alice - Carmi - Nour - Greg - Doug - Dave - Sam ### Note Takers: - Alice ### Notes - Podman testing update - First office hours. Next one is tomorrow. - Had a few issues raised. One person trying to make podman pods. Discovered that it wasn't installing catatonic init tool by default, which is needed for that. - Doug has been fixing bugs found in testing. Creating a working version of kubeadm. - Polishing jail proposal for working group and implementing in podman. We will need Sam Karp to review the proposal. - Dave is planning on updating his tutorial for Euro BSD con. - https://www.youtube.com/@opencontainers/videos ### Chat paste 17:00 - Meeting started Messages sent during the meeting will be visible to all meeting participants Mohammad Noureldin (9 Sep 2024, 17:02) Dave, man! long time no see, hope all is well ? dch (9 Sep 2024, 17:03) yup :-) busy just back from holiday, and kids off to school again. You (9 Sep 2024, 17:03) https://hackmd.io/hq_NOVL4RZS7xYYMqfJ6-A 17:18 - Meeting ended: 18m ## August 26, 2024 ### Attendees - Alice - Carmi - Ed - Doug - Nour - Greg ### Note Takers: - Alice ### Notes - Do we need a separate call for K8s people? - This OCI runtime working group is just for defining things relating to the OCI runtime, but the K8s group would be concerned with things on top of it. They may not be entirely sequential in terms of timing. In this call we would talk about runtime detail and also its context, but not the details of other K8s things. We could start a K8s working group later if needed. - Doug adding OCI images to FreeBSD release process. Will submit it once the current release is done. - CRI-O, Doug has a PR in review. It's taking a while, but the delays are on their side. The PR is CRI-O on top of ocijail. It will enable containers and pods under CRI-O which enables k8s. - Tara Stella raised some bug reports on the Podman build while building images. The Podman port has been updated with the fixes. hopefully that will be accepted before the test period starts. - Podman testing is ready to promote. There will be a weekly call during the period Sep 2 - Oct 11 to support testers, please see the OCI calendar for timing. We can focus on the runtime after the testing project is completed. https://github.com/oci-playground/freebsd-podman-testing/blob/main/README.md - NOTE: Podman testing call on Sep 17 - Alice will be unavailable that day. - Nour's update: - Still progressing on the directions/suggestions from dfr@ and emaste@ from last call(s) - Took time to study dfr@ work for CNI Plugins to know more about what we support and what not - Need (a bit) more time to wrap things up - Pushing to wrap things up so hopefully we can include it in the testing initiative - Can we get our containers working on other OSs? E.g. MacOS? - There is a PR which makes it possible to create a FreeBSD VM. It's theoretically possible, probably needs more work. - In the meantime you can run Podman remote as a daemon on a FreeBSD host and remote in from a Mac. - Carmi has a set of machines coming from AMD to run a cluster on FreeBSD. ## August 12, 2024 ### Attendees - Alice - Carmi - Doug Rabson - Ed Maste - Sam ### Note Takers: - Alice ### Notes - OCI Playground - Content for test project is ready for review - Doug has reviewed, will PR it. - Bug, feature request, misc - If we want to do rootless containers we would need a major release if it changes the syscall API. Would not likely be in release 15 at this point. - Nour's update (not present): - Progressing on the directions/suggestions from dfr@ and emaste@ from last call - For which I am learning about more features of The Rust Programing Language - Whenever I have something ready will share with both dfr@ and emaste@ - ETA before next scheduled call - Alice to follow up with Carmi re: Swift email. - Kubeadm is the missing gap to be able to use K8s with FreeBSD. Not currently ported, and is still a manual build. - OCI is a good first step to supporting k8s, however it would be good to look at the longer roadmap to that. - Doug = image formats are shared across platforms - Container runtime - Working with K8s upstream (e.g. SIG Node/SIG Storage) will be required. - Plan so far. 1. Changes to CRI-O engine committed upstream. 2. Then work with SIG Node - Sam: Doug's PR to the WG repo. Is it complete? - Doug = couple of last small changes. - Doug: working on FreeBSD release tooling updated to include OCI image builds. Hoping to be ready to get this merged soon with support of release team. With the intention of having the release team to include the images as release artifacts. (Not hosted on Dockerhub.) ## July 29, 2024 ### Attendees - Alice - Greg - Nour - Doug - Ed - Johannes - Sam ### Note Takers: - Alice - Greg ### Notes - [Alice] Apologies to Sam for not spotting him in the Zoom waiting room. I have now removed the waiting room function for this call. - [Alice] Podman Testing - Sam has kindly created the FreeBSD Podman Test repo https://github.com/oci-playground/freebsd-podman-testing - Who would like access to this while it's private? - Let me know if anyone can help me create the test docs etc to go in it. I have drafts that need some input. - Participants - people with experience of Podman/Docker. We can document any FBSD differences if they are not so familiar. - Review Podman tesitng Beta README - Ed: "I think we want to update the runj faq as well -- "when we start working on an OCI jail specification..." - Participant screening review - NetAvark PR review - How do we replicate iptables? - Netlink discussion - Meeting time - As the recent meetings have been around 30 mins in length, and the meeting today started at 8:30am PT, we propose to start it at 9am PT and limit it to 30 mins total going forward and see how we get on. ## July 15, 2024 ### Attendees: - Greg - Alice - Ed - Doug - Nour - Sam is waiting to be let into zoom ### Note Takers: - Greg - Alice ### Notes: - Thanks for shifting times to Mondays! - Intro Alice - Podman PID - [Working document here](https://docs.google.com/document/d/1VXi2_Y-4K5RwA_kaNsD5gEIfOPTzIIL4nNJ0pj3guxQ/edit?usp=sharing) - ASKS - I need someone to help me make test resources (docs, FAQs, participant screener). - I need your help to identify and bring in participants - NB: not quite ready to invite, but please think of possibles. - I need a decision on whether the “beta” should be run in a fork on https://github.com/oci-playground/ - pros and cons? NB. Sam K has access to the GH Org. - DR - Podman is seperate from OCI runtime extension, but does inform it. In playground - How to install FBSD - what kind of feedback we want, and where? how we will use it. We don't want the issues from testers on Podman project. We could have issues in a repo in playgroun - that would work. If issues emerge that affect wider podman, DR can halp bring them upstream - Nour: any benefit to having this under FBSD repo? AS: there is a visibility advantage (maybe small) to doing it on OCI to people focused on and thinking about containerized workloads - Running containers on FBSD - how to test - Can someone accept Greg's PR on the ReadMe? - Nour: Progress on fixing compile issue of netavark - One error left related to socaddressnl. Draft PR today or tomorrow - Nour focusing on errors, but on all issues related to netlink. DR - yes, maybe just explain this in note to Rust libc ## July 5, 2024 ### Attendees: - Greg - Doug - Nour - Carmi ### Note Taker: - Greg ### Notes: - Please complete this poll so we can identify a non-Friday time slot that works for folks. https://doodle.com/meeting/participate/id/b4Y7WBgd The week picked is just illustrative. Please consider times and days that work for you every 2 weeks. Alice has not put any slots on Fridays as she is not available to host the calls then. If you have any questions, please reach out to Alice in the OCI Slack or email alice@freebsdfoundation.org - I just saw this recent article and thought it might provide an interesting deployment scenario: https://it-notes.dragas.net/2024/07/04/from-cloud-chaos-to-freebsd-efficiency/ - Of interest to this group is this section of the deployment description: - A bhyve VM with Alpine Linux - in my opinion, the best distribution for running Docker containers. Do we really need systemd just to launch Docker? They mainly use it as a pre-production test bench, connected via VPN to their company LAN. It is the core of their “online” development, i.e., outside their computers. It has 32GB of RAM, 200GB of disk (obviously bhyve is configured with NVMe drivers), and 4 cores assigned. What would this deployment look like in N months when the FreeBSD OCI Runtime Extension is fully developed and available? - Roadmap - V0 Getting container image that works - we have this in place, with caveat that getting containerd improvement would be really nice - V1 get base official images there - v2: engage with available image infra and take the most commonly-used images and provide FBSD versions - v3: find some way to teach port tree to build images - DR: Still working on the cri-o contribution. Their CI is a bit tricky to navigate. - PR for nerdctl for netwokring support is stalled. this is very important for containerd. Something Nour can work on after netavark - we want to get the mobi tooling to work with containerd - Carmi has been supporting a video game, AAA title that launched with all infra on AWS. Company that launched has run out of money. Carmi's firm acquired rights and wants to reduce AWS spend. Migrating everything off. Using it as a possible launch project - plan is to bring game live in October. 4 player versus one player - HS kids go to a video store and end up in one of the horror movies. so it's 4 kids vs a monster. Halloween seems like a good re-launch time. Started puling containers to see which ones they can / can't build FBSD images for. If they can get it all built natively on FBSD, they will. If they have to they will use some of the Linux compatibility layer. It's a K8s cluster. All that is easy, and all the mongo stuff should be fine. - Questions: Is it insane to think they can launch on FBSD in October? to do so, have to have something reasonably stable in Sept - Second, where are we in terms of timeframe to get somehting reliable enopugh. - DR - K8s is the least mature part of stack RN. DR has a working prototype, but hasn't started thinkning about review. So, it would be using a branch, not official - At initial launch it will be relatively small. They can build it however they want. - Controllers are easy. Kubelet is the tougher part. Kubeproxy is one way to do K8s networking bits. Has prototype. The biggest unknown is persistent volumes. Linux K8s relies on specific semantics of linux bindmounts that fbsd doesn't have. SO need to find a solution to this. - the only persistent storage they have is on the DB side. - DR: can provision volumes with simple plugin. then mount volume in host space - and here we need to rely on linux bind mounts. should we replicate in fbsd, or find a better way to do it? Windows has a special proxy to do this. it runs on host, pass to proxy and uses that interface. There are workarounds for now - Carmi has to manage infra and wants it to be as little Linux as possible - What about Longhorn? - DR needs to read up on longhorn - Cloud native storage. There is a K8s engine and interface. RN it's all linux stuff. It uses underlying storage that you provide. It provides interface to cluster side. interesting b/c it is disconnected from underlying host - DR will look into it. Looks like distributed block storage, which is somewhat better. - It supports iscsi - - DR - this is supported well as far as DR is familiar - Nour: Has learned what he needs to know about the compilation errors so hopefully in the next week or so will have a PR to get netlink into the libc in rust. Will share PR with Doug first. This will complete the support to get us on par with Linux. Is there anyone from our side looking at Rust? - DR: Go to the spark64 channel and ask who to talk to about FBSD and rust - Carmi: manageability especially with a GUI. Anyone looing at a more generic server side thing to talk to the APIs for pf, etc. Problem is that there is not a consistent way to manage things. So, trying to come up with a standard management model that works with config AND API. - Nour: TrueBSD had a similar idea. they had xml-rpc API. - DR: TrueNAS has some similar API - Carmi: we could say "For all systems in FreeBSD, all configuration must be doable through an API to be included" - TO DO: MAYBE WE NEED A MANAGAGEABILITY WORKING GROUP IN CORE to define the problem statement. ## June 21, 2024 ### Attendees: - Greg - Carmi Weinzweig - former CTO at 20th Centurey Fox, then Paramount. Long tme FreeBSD. - Nour - Doug - ### Note Taker: - Greg ### Notes: - Carmi would like Containerized server side Swift. Also interested in bhyve for same reason. Manageability in general for FreeBSD is a pain point. Lots of point solutions (pf, truenas, ...) for native FreeBSD. Would like a GUI and command line for native FreeBSD. GUI for things I don't do alot, command line for things I do all the time. GUI and command line need to be in sync so using one doesn't break the other. - UPDATE: GREG JUST SAW THIS WHICH SOUNDS LIKE IT MIGHT FIT THE BILL: https://gyptazy.ch/blog/clonos-an-alternative-to-proxmox-based-on-freebsd/ - QUICK UPDATE FROM DR: we have a reasonably solid port of podman stack. (CW uses podman on mac) - not quite ready for prod since still early, and the project isn't quite ready to provide supported base images as part of RE - We have port of containerd - this is less usable, espec networking. It lacks developer time. DR is able to work on podman. We are looking for feedback on thepodman stack. What breaks, what's missing, how's the experience - DR updated the Jails proposal fleshed out. Sketched device management - DR close to getting baseline container support into Cri-o. That project moves deliberately. They have a large test environment, so pr is taking some time. Once merged this gives enough functionality for K8s to work. This means k8s control plane will run natively on FreeBSD. - Alice - we should track this and once merged reach out to DR to interview for a write up - DR: need to make this clean enough to review and then connecting with K8s community to get this into their tree - the K8s kublet and kube proxy ports for FreeBSD into their tree - Carmi: Quesiton - how do we do cloud native storage on FreeBSD. Same for distributed file storage. How do we do it? what about longhorn? - DR: K8s has a bunch of plugins to manage storage. can be simple like truenas via agent to mount and use storage. its pretty flexible. What's missing is the use case. Lots of people use fbsd for storage, but it's vendors. isilon for example. We need a customer to work with us to pick a solution (ceph e.g.) to say this is the scale we need and commit to the feedback look to get it stable. - storage is sort of the NEXT step. - TO DO - in Enterprise WG - what do you do for storage, traditional and cloud native? we need to ask this. - DR - also a potential for an OCI runtime for bhyve, but relatively low on the priority - it is doable - VM bsed container runtimes: https://github.com/opencontainers/runtime-spec/blob/main/implementations.md#runtime-virtual-machine - When do we want to start promoting? - Ampere has offered to support if we would like to do a quick dog and pony and/or blog - DR - we are at a place where we really want people trying things to see what works. It is ready PRE production. We can do a call for testing. - TO DO: we need to put together a call for testing. - How is the documentation - Podman has a great set of man pages but DR has not added platform specific notes for FBSD. This would be a big job. - A large fraction of Linux docs for Podman and docker applies to FBSD. Where things work, they work the same - What we could use is a compnainion doc - list of features that are committed to. If there's things you need not on here let us know. - Carmi: We need kernel support for non-privaleged use. - DR: it's a man power issue. No one working on it. we need help here. If we get feedback that adding non-priv use is key, maybe the Foundaiton can support. - And alot of people do their research and found in the past that FreeBSD doesn't do containers. So we need them to look again. - Carmi: Hashicorp nomad talks about FreeBSD and they support FreeBSD. we should reach out to them - DR: motivation - use the features we've had for 20 yrs and make them accessible to people familiar with podman and docker. - Carmi: if I want a docker container, postgres with all the things I need for HA cluster, I can just go. Building up these repositories of blessed images. - Once we've built the first instance, it's easier to recruit others to build out the runtimes for the things they need. - Netavark: Nour - coming slow but steady. Working with Rust a bit. Learning about rust. He did build netazark locally on FBSD. only a few errors. most is missing definitions. DR noticed this too. it should be possible to get a pr out for libc to add these things. Nour learning so he can do the pr. - Nour also working on image compatibility. he shared on slack. when they say "runtime" they mean at the podman level. so may not be super relevant to us. there are two schools. extend or a new definition altogether. Brandon lkes idea of extending image manifest. This might take 1+ yrs to get this out. So, for now, we can use the version key. we can define this as a fbsd api version if we want. So, Nour will observe, but no action required. He will continue focusing on netavark. CC DR on any PRs ## June 7, 2024 ### Attendees: - Greg - Nour - Doug - Ed - Sam - Dave ### Note Taker: - Greg ### Actionable Agenda Items: - review and provide feedback on 'Mapping from jail(8) config file' here: - Looking for something that lets us describe the jail - pairing - and other knobs - SK intermediate level? - DR: having the extra level would help group things? - SK: from container plumbing - an example of a fully written out example with a config - DR: will add as follow-on to this PR - DR: going in right direction? - SK - yes. Needs a bit of polish - Defining Devfs rules to define things in a container - DR: currently doing it in a rough way. Planning to propose another field parallel to jail to allow rules to apply to container jail. Rules that will be applied to the contianer devfs - SK - makes sense, but not a jails expert - purpose is to allow to carefully expose exrta devices from the host. in podman its done with a command line flag "in addition I want access to foo". in FBSD that's done with devfs rules. Should that go into this proposal or a seperate one? - Here is an example (removed some "#" to avoid hackmd formatting): # /etc/devfs.rules (the defaults) [system=10] add path 'usb/*' mode 0660 group operator Allow operators access to usb devices for keyboardio flashing [operator_usb=5] add path usbctl mode 0660 group operator add path 'usb/*' mode 0660 group operator add path 'ugen*' mode 0660 group operator [devfsrules_jail=5] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path fuse hide add path zfs hide - Looks like there is precedent for this to go under the Platform object. This is how they do it in Linux - All three put resources in the platform sections - Decision - keep it in the platform proposal. DR next steps - go through proposal A, address comments, improve examples, add seciton to talk about DevFS - DCH - help? DR: yes! - Other topics? - DR: from BSDCan, there is consensus in RE that FBSD should build images as part of release process. DR will be helping the RE team. Some parts of the process in the source tree, some parts are not public about release building. DR get into this and figure out the best places to build osme simple images (3 types). Static, Dynamic, with a shell. single arch images. publish as tarballs in OCI format. then work on how to publish in a more normal way. at this point, we will pull together the multi-arch image list concept so we can have a single name across all archs - after the fact you can pull together multiple images after build time. You can download oci archives, pull into a local registry using simple commands, and then create an image list based on these - SK: this is well-supported in the OCI image spec. you can also (thinking about application-level container images) have a multi-OS image; think about "nginx" that automatically resolves to Linux, Windows, or FreeBSD depending on the system you pull it to - DCH: discussion on signed images? DR: Building as part of FBSD RE provides sufficiennt trust initially. we can look into this in future. Maybe work with dockerhub to get into their verified image program. - SK - OS vendors can produce a tar and that gets built through bashbrew into a container image. we can find this in docker. This should be doable on the dockerhub side. They do it as a one-off for windows. You open a PR to the dockerhub repo. SEE: https://github.com/docker-library/official-images/blob/master/NEW-IMAGE-CHECKLIST.md - This is how the Ubuntu images are defined: https://github.com/docker-library/official-images/blob/master/library/ubuntu - DCH will take a look at this. - Tianon (sp?) is person who runs this. SK can connect us with them. Also can start this conversation on OCI slack, tag Tianon, he will jump in. - Nour: Keep eye on image compatibility WG? - DR: yes, we should watch, but probably not critical path right now. - SK: For windows, image is tied to underlying version on host. This is one of the reasons why doing the FreeBSD WG makes sense. - we need to be able to tell, if you're running a certain version underslying, what kind of container version can you use - DCH: mybe: - $ dch@f01 sysctl kern |grep osrel kern.osrelease: 14.1-RC1 kern.osreldate: 1401000 $ sysctl kern |grep osrel kern.osrelease: 15.0-CURRENT kern.osreldate: 1500019 - Discussion: - virtiofs - imp@ indicated there's an implementation somewhere. Greg ask imp@ - fuse over virtiofs - main use case is for desktop use - Doug was involved with podman machine to create a VM using a carefully chosen image with a specicif image. So, on Mac, can use podman command as if it's a Linux machine. they use 9pfs for this. The Mac people only want to support virtiofs. So podman exploring moving to virtiofs. - Sam - is this for bhyve? - Yes and to stand up a specific image, podman uses 9p, but perhaps they should move to virtiofs. - Nour: AWS elastic container - we may need this here also - ## May 24, 2024 ### Attendees: - Greg - Doug - Nour - Ed - Bjoern - DCH ### Note Taker: - Greg ### Actionable Agenda Items: - Update on status from Doug and Nour ### Presentation/Discussion Agenda Items: - DFR: Been updating Ports infra to Podman 5 - Nour: Been working on [netavark](https://github.com/containers/netavark) support for [Podman](https://github.com/containers/podman) on FreeBSD - There will be a pkg base package set to produce images in 14.1. Goal is to get images into the release cycle - at BSDCan - how tio wire images into release process and what RE wants - Bjoern - should be pretty straight forward. Can be a in a registry or bunch of OCI files - DFR tends to use buildah. - Discussion about signatures. Bjoern (https://github.com/containers/image/blob/main/docs/containers-certs.d.5.md) - Nour and Dave looking into [GPG](https://github.com/containers/podman/blob/main/docs/tutorials/image_signing.md) key, works with CRIO. Sign image with [GPG](https://github.com/containers/podman/blob/main/docs/tutorials/image_signing.md) key, configure podman to confirm sig with public part of key <- this is a Red Hat ism, but doesn't necessarily work with other K8s models - Bjoern - important to define VNET architecture - DFR: skopio can make public private key pair, podman upload to sigstore, then with configs verify the key, so can be done as part of FBSD package. This would allow moving away from GPG. - Sigstore allows you to use standard PKI - Or FBSD project infra could provide a registry, then this would imbue trust - This is what Docker uses today - sign an OCI layout and participate in an official registry (https://github.com/opencontainers/image-spec/blob/main/image-layout.md) - DCH - - first step is just the file and RE can sign with GPG and this provides plenty of trust - next step is sigstore adding to build chain - then, make these images available for YOLO developers. For this, use existing FBSD GitHub registry - Bjoern: Worth noting: if you ever want e.g. `FROM freebsd:14-RELEASE` to work, participating in Docker Official Images is the way to go - RH no longer suggests overriding the default registry in registry.conf due to the risk of dependency confusion https://github.com/containers/image/blob/main/docs/containers-registries.conf.5.md#note-risk-of-using-unqualified-image-names - Sigstore is encoding signatures into the registry itself (so they can be redistributed) - Docker Official Images are another option - Human Review by Docker and they work upstream - https://github.com/docker-library/official-images - Expectectation in container image world is that they are rolling releases with regular updates - DFR - we need to support multi-arch builds - How should we describe a Jail? We'd like to move conversation forward on this. - How do we create a jails structure that allows for network - Bjoern - container networking is not part of containerization, but not specific in the OCI runtime spec. Container networking is typically very similar to VNET on FBSD - Runtime has to ask for a new vnet when constructing the jail. Beyond that, we run CNI plugins that create epairs and bridges - Good would be polished networking library that allow speople to do what they want depending on which orchestrator they are using - a developer was working on the nerdctl FreeBSD port that hooks in the CNI - DFR has a very minimal port of CNI that replaces IPtables with PF (CNI v2 Mike Zappa at MS is working on it) TO DO: - DFR would like: Feedback on the mapping table and suggestions on what, if anything, Sam would like to change - How do we define ABI Compatibility. Vector selection of ABI - how will it work? Need default logic to select base image - multi-platform images are technically called the OCI index: - https://github.com/opencontainers/image-spec/blob/main/image-index.md - https://github.com/containerd/platforms/ - https://github.com/containerd/platforms/blob/main/defaults_freebsd.go ## May 2, 2024 ### Attendees: - Greg - Doug - Nour - Ed - Samuel Karp ### Note Taker: - Greg ### Actionable Agenda Items: - Need to confirm this time will work for all participants or find a new time - Reschedule tent. Fri May 24 12:30 eastern - GW confirm with Bjorn ### Presentation/Discussion Agenda Items: - DFR - updated the OCI-Jails proposal - added a table to show how baseline Jails parameters map to the spec - Topic: how to define devfs parameters - in Devfs, you can tell it which predefined ruleset to use to determine which devices to expose using a built-in ruleset, exposes the bare minomum. - either add new ones - or modify the ruleset - want to be able to add devices like GPU inside the jail - DFR has added as a type of pseudo mount option in Jails to add extra rules to the devfs baseline ruleset - first mount devfs then you can add to it using the sudo option - not a very elegant solution. it's convenient - QUESTION - how do we define the devfs once we have the runtime spec extension? - EM - agrees pseudo is not the best - How complex will the devfs rules be? - DFR - fairly simple stuff - e.g. in linux I want to unhide [ ] and mount a tempfs - for non NVIDIA gpus unhide devdrm - this is convenient since no other place to put the rules, but would like a better place? - Linux is different since they do not have devfs - SK - in Linux thre's a device list - cgroups: https://github.com/opencontainers/runtime-spec/blob/main/config-linux.md#devices - in podman, this happens in podman, not in the runtime. But since the runtime is where we substantiate devfs, we need a solution - Maybe it's adding an extra stanza to the FBSD part of the config - **could be in the CLI - container engine (say podman) does the policy and then the policy is turned into a set of actions for the runtime to implement. This may keep it tidier to keep it in the config.** - This would work across container reboots - one option (not preferred) is to create a ruleset on the fly and you give it a number - Still interestedin more user stories - @c5mx3H7dR66_kNeuXhhrGA re-up the call for user stories - DFR will address the issues that have come in - To set up the network state for a container, DFR using bare min of CNI plugins - covers podman and crio, could be used for containerd - Now podman is moving in a different direction - they are going with NetAvark - it's very fast, and podman wants it to be the only model - slightly relaxed on timeline. We need a port of NetAvark - someone needs to understand rust, needs to understand netlink, and then do the work to get NetAvark to work on FBSD. will involve reaching out to rust standard library - in podman 5, CNI support is still there - opt in at compile time. not sure, but prob removed by podman 6 - EM - if we ask them not to remove, it will stay for a short while, they'll keep longer if we say we have a plan to add netavark to FBSD - Nour: Would like to help on the netavark port and it will be a growth and learning opportunity - DCH - there are others who will help and review. Ed can help review as well - DFR - create issue in rust standard library to start as first step - second approach podman - SK - netlink useful to have - DCH, based on DFR's notes building containers from scratch and how to use repositories (local, Oracle Cloud, and one other one) - hoping to get signed container releases from public registries with v 14.1. Working on getting signed images come out of RE - what about an official FreeBSD registry? (Nour) - this is an open discussion, and there is a potential to have it in dockerhub and also some open source options - Zot is one DFR has been working wiht - very open and collaborative and a nice system. Maybe a FreeBSD port for Zot would be good? Zot project build FBSD binaries - Cosign Sigstore process - we make signatures for the images. sigs hosted by an webserver. put sigs in the correct place, publish config files in the right place. as long as we can sign during hte build process, we can have all images checked against the sigstore. - Cosign is another mechanism for image signing that stores the signatures in the image registry. ## April 15&18, 2024 ### Attendees: - Sam Karp (4/15) - Doug Rabson (4/15) - Bjorn (4/15) - Greg Wallace (4/15 & 18) - @dch (4/18) - Ed Maste (4/18) - Mohammad Noureldin (Nour) (4/18) ### Note Taker: - Greg ### Actionable Agenda Items: - In-person Meeting at Container Plumbing Days ### Presentation/Discussion Agenda Items: - Sam, Doug, Bjorn, and Greg spent about ab hour on Monday, 4/15 discussion how to describe jails in a way that is expressive enough to cover as many use cases as possible. - TL;DR - Agreement to puruse a flat(ter) description that keeps the Jails docs as the canonical source. - @DFR has taken the action to begin creating a table with the list of things in the Jail parameters and map each to some place in the OCI spec - Either someplace in the common spec - Or in the FreeBSD extension - In addition, we need to write an implementation - Also must create "What is a full example of everything we can specify" - translation semantics from image to runtime and from linux to FreeBSD, at least as recommendations/guidance - Complete discussion notes can be found here: https://docs.google.com/document/d/1AwJX-iTHsfykvppNHCy89DWi6m0BJmMfyEDXvr2x64A/edit?usp=sharing - Thursday April 18 Greg met with Nour and Ed and Dave to provide an update on the Monday in person meeting. ## April 4, 2024 ### Attendees: - @dch - Greg Wallace - Ed Maste - Mohammad Noureldin (Nour) ### Note Taker: - Greg ### Actionable Agenda Items: - Who's taking notes? - Greg will - Time for meeting at Container Plumbing Days? - Our next scheduled meeting is on April 18, which is during open source summit seattle. plan is to keep the meeting and use it to brief everyone on meetings during container plumbing days (april 15) - ### Presentation/Discussion Agenda Items: - Determine if there will be a Container Plumbing Days at Open Source Summit Europe in Vienna - Review of Requirements Open Issues - joh-ku potentially out of scope for the runtime spec, but this is very valid for what conatiners on FBSD on the whole need to facilitate, but not part of the runtime spec - Nour - the last line may be applicable to the runtime. @DFR thoughts? - gizahNL - DCH - key is where are resources defined and allocated. in Jails it's done outside Jail. in OCI, definition is in the container, and execution in in runtime. - "Proposal A" matches the jails approach: https://github.com/opencontainers/wg-freebsd-runtime/blob/simple-jail/docs/proposals/PROPOSAL_A.md so, vnet jails would continue to exist - DCH - from a user perspective, as sysadmin I want to ____ - what goes where? - networking - CNI - does it exist today? is it set up seperately or is it set up in the container world - Doug created a plugin for CNI as part of podman work for container-based jails on FBSD - dch understanding does it with pf, but you still need to set up redirects for pf and netanchor. you still need to write the ifconfig for network devices. do we want the containers to do this kind of work then the runtime needs to manage this metadata. questions like which subnet, etc you want it to connect to. maybe we can delegate this to the plugin... - Nour: in docker, you have network management features and you can attaches different networks to different containers. Greg: can we use podman network management capabilities? - is network management governed by any specification? - Yes - https://github.com/dfr/plugins?tab=readme-ov-file - https://www.cni.dev/ - @dch has a set of [working notes] covering using Doug's tools - setup of podman & local registry - fetching & running existing containers - creating new containers using Doug's existing [freebsd-images](https://github.com/dfr/freebsd-images) scripts - using an external registry - Need to get this officially hosted, and come up with a roadmap that lines up to the FBSD release schedule. Should be a small number of trusted sources - Nour happy to help out with this. @dch will do discussion on Jails IRC. We want to be able to distribute official FreeBSD images and this requires something like AWS "gallery" and they also have official images. E.g. there is an official Node.js image that can be pulled from the docker and AWS registries - @dch wants to know how the image can be trusted if it's hosted on 3rdparty infrastructure? - this will be driven by how much work is there to do within Foundation & ClusterAdmin - @mnour For example, https://hub.docker.com/_/python you can look at this to see how you determine that this is trusted - there is a process to get the "Official Docker image" mark - @mnour has reviewed that and he was not quite correct, more information can be found in [Docker Hub Trusted Content](https://docs.docker.com/trusted-content/) - we need to define what it means for us for the image to be "official" - @dch - it's about provenance. for every FBSD release, there's a signature published out-of-band on the FreeBSD website, and the announce@ mailing list, signed with PGP key by re@ team members. If this sort of signature can be attached to published images, and is easily verified, then we can in principle host images in multiple places - this is in part handled by the OCI specification - part of the image and runtime specs - image distribution spec signs the images [working notes]: https://docs.skunkwerks.at/s/fUiAmi4pE# ## March 21, 2024 ### Attendees: - @dch - @dfr - Greg Wallace - Ed Maste - Mohammad Noureldin (Nour) ### Note Taker: - Greg ### Actionable Agenda Items: - Who's taking notes? - Greg will - any thoughts on scope? - e.g. inherit vs. alias vs. vnet, routing/bridge/netgraph setup, ZFS (snap, clone, block dedup?) > More detailed questions below. - see https://github.com/opencontainers/wg-freebsd-runtime ### Presentation/Discussion Agenda Items: - Went around and did introductions and why each is interested in this area - Having good OCI support on FreeBSD may help in heterogeneous Linux/FreeBSD environments - first topic - looking at github - requirements.md should be a list of use cases, maybe grouped - lets' first get a consensus view on use cases - please submit PRs if you have changes or additions to the use cases - TO DO: Let's put out the request for input - We can use Slack for async, and this meeting for sync - Second - we need to describe Jails - [Simple jails](https://github.com/opencontainers/wg-freebsd-runtime/tree/simple-jail) see [proposal A](https://github.com/opencontainers/wg-freebsd-runtime/blob/simple-jail/docs/proposals/PROPOSAL_A.md) - proposal on a branch - this proposal is secure - it is simple and flat versus layered - network/address control is out of scope for OCI runtime - container networking plugins take instructions form the container engine and do what they need to set it up on the network - discussion around configuring and declaring container engine port mapping, using podman and this from docker was provided as an example ``` docker run --rm -it --network local.dimanex.com -p 127.0.0.1:9002:9002 \ --network-alias=api.localhost.dimanex.com \ --name dimanex-api --volume "$(pwd):/src-repo" \ --volume "$HOME/workspaces/dimanex/mvn_home:/mvn_home" \ --workdir /src-repo dimanex/arm64v8/jvm-java-8-run-base:1.0.0 \ ./mvnw -e -Dmaven.repo.local=/mvn_home/repository spring-boot:run ``` - splitting of functional pieces from a former monolith driven by K8s. Now have a more modular ecosystem that provides more choice for users and developers - Next steps we need to describe jail in a way that is expressive enough to cover as many use cases as possible. Need to get the whole group together, including Sam, to discuss containers and jails - try to get meeting with Sam prior to Cloud Plumbing Days. If not, then discuss this topic there - Goal: by end of CPD we would like to know which direction we are going in. - Doug will convert his branch into a PR so others can review/comment - How handle topics like networking, mounts (specified as cross ___ in specification) including ZFS, runtime instructions - keep it freeform and feel free to add a section to use cases - with mounts you get options (read only, NFS v4, etc) dfr added sudo option for ZFS rules that gives a straightforward way for the engine to translate sudo options into rules in Jails. this can be found in OCI Jails code in `mount.cpp` - we need a way to describe ZFS in the container - Netgraph networks - complex network setups - are these also out of scope? > [name=crest] Sounds like it should be handled by a CNI plugin? > [name=Greg Wallace] > this can be isolated inside the container networking plugin itself, so you can use netgraph, bridges, whatever - What are the ideas around implementing this runtime extension? - we don't need to wait for the spec to be published before we start testing and implementing - Linux containers have a PID, FreeBSD are identified by a JID (also 32bit int)? Does it make sense to put the JID into the PID field? What would be the implications? - Should the OCI runtime spec assume that jails are always persistent to split create and start or should the runtime keep a placeholder (child) process in an otherwise empty ephemeral jail? - Is there a collection of "reference configurations" that should be expressable? - Foreground vs Background: FreeBSD jails (as configured via jail.conf) are mostly run detached in the background. It's my impression that most (Linux) container orchestrators conceptually run containers in the foreground. Does the runtime spec say anything on this and if so what? - Which parts of the process execution environment are expected to the specified via an OCI runtime configuation file? Is any of it implicit and unconfigurable (yet different from FreeBSD's default)? - What CNI/CSI plugins are required on top of the runtime spec? - to support a single lab server? - a development system? - a CI/CD pipeline? - to interface with commonly available network and storage envs? - to support "production" cluster? - Workflows instead of technical features? - Let's also talk about Cloud Plumbing Days and how we can use that time to meet and make progress - _add another agenda item_ ### Notes: - _add your notes_

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.