# OSBuild

## Timeline for 4.15: 2 months of development

1. Complete the integration of osbuild as part of our pipeline for the creation of metal and qemu images

- Merge in osbuild support into COSA
- optional via image.yaml configuration setting
- Create a bootupd stage in OSBuild (should cover bootloader install for x86_64, ppc64le, aarch64)
- Multi-arch support (we need to create a new card for this)
- all kola tests passing
- Decided to only switch our pipelines to use OSBuild for building once 4.16 branching happens

2. As an experienced customer you can install osbuild on a classic RHEL system and use it to create an image using a container as input (Tech Preview)

- on a classic RHEL system:
- we need to make sure necessary OSBuild PRs land and can get released into RHEL
- work with Achilleas to figure out what lead time we need to get PRs into a release in RHEL
- worst case scenario we have them checkout from upstream and run from that
- for tech preview: could probably get away with x86_64 only
- create documentation that can be shared with customers
- would be good to get this into openshift official documentation
- will take some time to get things reviewed probably
- work with Jesse to find out what dates we need to hit to land it in 4.15 documentation
- write openshift enhancement to convey value of this effort

Strech goals:

3. Start the discussion around Image Builder integration
4. Investigate building cloud images

- how information like platforms.yaml and image.yaml get consumed and/or passed around in the future
- the answer to this could involve some of the higher level abstractions in IB like osbuild-composer
-

External dependencies (at the moment):

- Multi-arch hardware move
- Renata working on AWS (aarch64) and IBMCloud PowerVS (ppc64le)
- Dusty at ATO and OpenShift Commons week of 10/16
- Presentation about FCOS and RHCOS coreos layering
- Project Sagano
- Not necessarily a bit detractor of attention, but could receive higher priority than OSBuild work at some point

Timeline for 4.15:

- Sprint 243
- COS-2351 [spike] Investigate building a disk image (raw/qcow2) from a container image
- COS-2479 Allow both ostree commit and container image as input
- Sprint 244 - ends on 11/10
- COS-2388 Figure out how to run osbuild in our CoreOS pipelines
- COS-2517 support conditional osbuild building in the pipelines
- COS-2432 Create bootupd stage
- COS-2490 create openshift enhancement to introduce custom image builds from OCI
- COS-2495 Investigate unit tests for OSBuild
- Sprint 245 - ends on 12/01 # END OF FEATURE FREEZE - 4.16 branching happens - enable OSBuild in 4.16
- COS-2488 confirm multi-arch support for building images
- COS-2452 Add the platform bits to grub2 stage
- COS-2389 Figure out future of image.yaml
- aleph-version stage or no stage in OSBuild?
- composefs?
- Sprint 246 - ends on 12/22
- COS-2516 switch pipeline to using osbuild for building qcow2 and raw image
- Sprint 247 - ends on 01/12
- COS-2489 create tech preview documentation for advanced customers
- Sprint 248 - ends on 02/02
- COS-2469 When not deploying via container still include container layers refs

## Build steps

osbuild --store osbuild_store --output-directory osbuild_output  fedora-ostree-container.json

build:          81b166dce1322f833c43b47acf9edcce9968344d71da29cce4271f0c8ee01e2b
ostree-tree:    fe275a4972d845c61c90bb8db4c6a8cb34bedc7d25db2d79125e5a3140197eb8
ostree-commit:  7b53b3797bcf3841c932bf45dd62b6f9239a0afe4229f3129871c50eb7644035
container-tree: 9c55a1952cc14e210a6c9b908331dcc72efc7fdd8cbfe82539166b820a87daf4
container-ostree:       75922087e9ebf66243b5f47143306a2536298001c2b3d0ee53c6eb4867a5f2fd
container:      1f1ca1f78a0431c7f0da473a289f04733e3e3c0267f41e6c4ef1384185bab003

osbuild --json fedora-ostree-container.json --export 1f1ca1f78a0431c7f0da473a289f04733e3e3c0267f41e6c4ef1384185bab003 --output-directory out

osbuild --json fedora-ostree-container.json --export container  --output-directory out_container

### Create json file

 osbuild-mpp fedora-ostree-container.mpp.json fedora-ostree-container.json

There are 2 versions of the manifest. See: https://jenkins-rhcos.apps.ocp-virt.prod.psi.redhat.com/securityRealm/commenceLogin?from=%2F

Cosa provides a container image oci-archive osbuild gets this oci-archive and turns to a bootable image

Some links:

https://github.com/osbuild/images/blob/main/HACKING.md
https://sigs.centos.org/automotive/building/
https://www.osbuild.org/guides/developer-guide/projects/osbuild/index.html

The difference between a stage and an assembler is that the former takes
a read-write filesystem-tree and performs a certain modification to it,
whereas the latter takes a read-only filesystem tree and produces an
output artifact.

## Remove`root=LABEL=root` from image
It is added during:

ostree admin deploy fedora:fedora/x86_64/coreos/stable --karg=root=LABEL=root'

--- a/stages/org.osbuild.ostree.deploy
+++ b/stages/org.osbuild.ostree.deploy
@@ -115,7 +115,6 @@ def main(tree, options):

     if rootfs:
         rootfs_id = make_fs_identifier(rootfs)
-        kargs += [f"--karg=root={rootfs_id}"]

     for opt in kopts:
         kargs += [f"--karg-append={opt}"]

## Testing failing:

non-exclusive-test-bucket-0/ext.config.files.aleph-version
non-exclusive-test-bucket-0/ext.config.disks.root-boot-ro
non-exclusive-test-bucket-0/ext.config.files.root-immutable-bit 
non-exclusive-test-bucket-0/ext.config.files.console-config
ext.config.boot.grub2-install
coreos.ignition.failure
ext.config.butane.grub-users
ext.config.rpm-ostree.kernel-replace
ext.config.reboot

## Issues:

ext.config.reboot:
the bootuuid.cfg file added by coreos-boot-edit.sh, is missing for the efi boot config.

ext.config.butane.grub-users:
Missing grub_users setting in new/old BLS config
cat /boot/loader.0/entries/ostree-1-fedora-coreos.conf | grep grub
cat /boot/loader.0/entries/ostree-2-fedora-coreos.conf | grep grub

ext.config.boot.grub2-install:
https://github.com/coreos/fedora-coreos-config/blob/0c63b1c3db769718f7e087bd23f9c91ec144c909/tests/kola/boot/grub2-install

fatal 'file /boot/grub2/i386-pc/core.img doesn'\''t exist'
echo 'file /boot/grub2/i386-pc/core.img doesn'\''t exist'
file /boot/grub2/i386-pc/core.img doesn't exist

## create_disk specific stuff

- aleph file
- bootupd stuff
- bootuuid stuff
- console settings stuff

## Grub config

Generated in [create_disk.sh](https://github.com/coreos/coreos-assembler/blob/main/src/create_disk.sh#L399-L450)

 case "$arch" in
08:59:00  + install_uefi
08:59:00  + unshare -m /usr/bin/bootupctl backend install --src-root=/tmp/rootfs/ostree/deploy/rhcos/deploy/8f13acb63a245a78b40e01c00966b3644d0587da61b86ac28fe208fe2918b7d2.0 /tmp/rootfs
08:59:00  + local target_efi=/tmp/rootfs/boot/efi
08:59:00  + local grubefi
08:59:00  ++ find /tmp/rootfs/boot/efi/EFI/ -maxdepth 1 -type d
08:59:00  ++ grep -v BOOT
08:59:00  + grubefi='/tmp/rootfs/boot/efi/EFI/
08:59:00  /tmp/rootfs/boot/efi/EFI/redhat'
08:59:00  + local vendor_id=redhat
08:59:00  + local vendordir=/tmp/rootfs/boot/efi/EFI/redhat
08:59:00  + mkdir -p /tmp/rootfs/boot/efi/EFI/redhat
08:59:00  + cat
08:59:00  + install_grub_cfg
08:59:00  + mkdir -p /tmp/rootfs/boot/grub2
08:59:00  + chmod 0700 /tmp/rootfs/boot/grub2
08:59:00  + printf '%s\n' 'set pager=1
08:59:00  # petitboot doesn'\''t support -e and doesn'\''t support an empty path part
08:59:00  if [ -d (md/md-boot)/grub2 ]; then
08:59:00  + sed -E 's@(^# CONSOLE-SETTINGS-START$)@\1\nserial --speed=115200\nterminal_input serial console\nterminal_output serial console@'
08:59:00    # fcct currently creates /boot RAID with superblock 1.0, which allows
08:59:00    # component partitions to be read directly as filesystems.  This is
08:59:00    # necessary because transposefs doesn'\''t yet rerun grub2-install on BIOS,
08:59:00    # so GRUB still expects /boot to be a partition on the first disk.
08:59:00    #
08:59:00    # There are two consequences:
08:59:00    # 1. On BIOS and UEFI, the search command might pick an individual RAID
08:59:00    #    component, but we want it to use the full RAID in case there are bad
08:59:00    #    sectors etc.  The undocumented --hint option is supposed to support
08:59:00    #    this sort of override, but it doesn'\''t seem to work, so we set $boot
08:59:00    #    directly.
08:59:00    # 2. On BIOS, the "normal" module has already been loaded from an
08:59:00    #    individual RAID component, and $prefix still points there.  We want
08:59:00    #    future module loads to come from the RAID, so we reset $prefix.
08:59:00    #    (On UEFI, the stub grub.cfg has already set $prefix properly.)
08:59:00    set boot=md/md-boot
08:59:00    set prefix=($boot)/grub2
08:59:00  else
08:59:00    if [ -f ${config_directory}/bootuuid.cfg ]; then
08:59:00      source ${config_directory}/bootuuid.cfg
08:59:00    fi
08:59:00    if [ -n "${BOOT_UUID}" ]; then
08:59:00      search --fs-uuid "${BOOT_UUID}" --set boot --no-floppy
08:59:00    else
08:59:00      search --label boot --set boot --no-floppy
08:59:00    fi
08:59:00  fi
08:59:00  set root=$boot
08:59:00  
08:59:00  if [ -f ${config_directory}/grubenv ]; then
08:59:00    load_env -f ${config_directory}/grubenv
08:59:00  elif [ -s $prefix/grubenv ]; then
08:59:00    load_env
08:59:00  fi
08:59:00  
08:59:00  if [ x"${feature_menuentry_id}" = xy ]; then
08:59:00    menuentry_id_option="--id"
08:59:00  else
08:59:00    menuentry_id_option=""
08:59:00  fi




ostree config --repo /tmp/rootfs/ostree/repo set sysroot.bls-append-except-default 'grub_users=""'

# Unknowns

make sure that we get all info needed for disk image creation into the .ociarchive
create_disk bits
platforms.yaml bits
handling of grub password
using the generated qcow2 as input for later stages.

# Questions for the OSBuild team

- We've made some patches to some of the existing osbuild stages to allow for an FCOS container image as input. Is it okay to have these patches and just use a conditional flag to indicate a container image input or is it preferred to create a separate stage?
- https://github.com/lukewarmtemp/osbuild/pull/1
- Considering higher levels (image builder, osbuild-composer)
- [skip] The Skopeo stage copies a container image to a local container image. We then deploy FCOS from this local container image. The problem with this is that when you run `rpm-ostree upgrade` on the system, it will not be able to upgrade since it's pointing to a local image, not a remote image as its source of truth. We're thinking of manually modifying the ostree origin file to change the remote back to the remote container image by either making another patch or new stage. What are your thoughts on this approach and do you have other suggestions?
- We don't need to ask this because it's resolved by --target-imgref flag
- Ask about copying files from /boot from "host" context
- https://github.com/dustymabe/osbuild/commit/a93da2fec4ca308e73e5e385c0272a4f140e9061
- Ask about making one parameter optional for the deploy stage
- https://github.com/dustymabe/osbuild/commit/9e80eaf79dbf02d5ee73b40ccbe6c0d66d685755
- Ask about being able to tease out pipeline defnition/information from OCI archive provided by the user
- OCI archive is our source of truth. We'd like a user to be able to provide an OCI archive and say they want e.g. an "Openstack QCOW" and just get the image.
- How do we evolve the Image Builder blueprints to add what we need / disable/refuse what we don't want to support
- https://www.osbuild.org/guides/image-builder-on-premises/blueprint-reference.html
- Ask about path to end user success in hosted Image Builder
- If our teams do the lower level OSBuild work is there someone from their team side to integrate it into the higher level image builder hosted service that is customer facing.