# KNI + CoreOS Notes (26-May-2020)

## Attendees
- Colin Walters
- Jonathan Lebon
- Kiran Thyagaraja
- Stephen Benjamin
- Micah Abbott

walters: describing `coreos-installer iso embed` functionality
stephen: running Ignition twice? on installer and in real root?
walters: yes
walters: which config source is being used?
stephen: mounted disk option; not metadata service
stephen: ironic boots BM host with inspector agent; gathers introspection data; waits for ironic to tell it to do something
walters: do you use pxe?
stephen: we do; configurable via IPMI etc. can use virtual media as well, but ironic has constraints around this; will be able to do introspection in a container eventually; sounds like we can use virtual media, but not sure how pxe will work?
walters: we provide pxe artifacts; our initramfs is huge and is breaking s390x
stephen: will openshift-install need to create two ignition assets?
walters: imagining IPI workflow would contain the generation of all the necessary assets;
jlebon: how are net args generated per machine?
stephen: Many customers have homogenous hardware, all networks the same. don't support per machine configs (i.e. disabling interfaces, etc)
jlebon: wherever the logic exists for generating the machine specific ignition would...???
stephen: ???
walters: we should ship a CLI/service that would be able to generate ignition configs with static networking, etc and would give bootable ISOs or USB or something
kiran: what about the kickstart approach? providing ks cfg to the ISO, i.e. "answer file"
jlebon: the "wrapper" ignition config embedded in iso has: systemd unit to run installer, ignition config for networking on the real host, and a pointer config to the remainder of the ignition config
walters: people want diskless installs; we support this
stephen: the iso work sounds like it solves our needs
walters: we are making this for 4.6 and it is available internally now
stephen: will other IPI solutions use the ISO?
walters: clouds will use cloud stuff; vmware maybe
stephen: we'd probably move to the live ISO as part of 4.7
jlebon: doesn't support ISO but does support PXE?
stephen: ironic is designed to write things out to disk; making it do something else is more complicated; need to sort out the best workflow with ironic folks
jlebon: same functionality in live PXE as exists in live ISO
stephen: have to discuss with ironic folks for both live ISO/PXE cases
walters: what is current customer using?
stephen: all BMC based; issue with the DHCP serial problem
walters: what are the easy fixes that we can provide that are not the live ISO? no one likes the workaround of popping open the image and changing kargs
jlebon: kargs always win; unless you use `--copy-network` from the installer; we could write a tool to open the image and changing kargs and even inject the ignition config
stephen: that would be useful; would require automation somewhere; we cache RHCOS image in cluster, could write automation to call the tool; we see customers going to production with 4.4; more official coreos tool would make them happier than virsh-edit
walters: maybe kbase article to lend it official support?
stephen: might be better than our "workaround"
kiran: what about day 2? stub ignition is sacred, used to feed new nodes later. what happens on day 2?
stephen: provisioning on day 2 would be the same; day 1 is ironic, day 2 is cluster api
kiran: ???
walters: new nodes boot off stub config and MCS servers newly rendered config
stephen: how will new nodes disable DHCP on NICs on new nodes, day 2?
walters: that lives in live ISO. the complex networking config gets copied into the initramfs as part of the coreos-installer experience.
stephen: MCS generates the ???
walters: `coreos-installer iso embed` will be used to embed ignition + generate custom ISOs
kiran: we need to work that into our automation
stephen: minimum scale for our customers is e.g. 100-200 hosts
jlebon: need to determine what is exactly machine specific. could provide a script that can make the determination. i.e. lookup table of MAC to ip address for static IP assignment.
walters: you can use live ISO to pull a container. could answer question of what physical disk should i install to? could be container or shell script, can be templated to make ignition configs
stephen: ???
walters: if we had machine specific machine configs, that would solve problems. but how do we identify machines? TPMs? mac addresses? there is a large gap from fundamentals to scaling to cluster, etc
stephen: we should have conversations with installer and cluster api and mao
jlebon: ???
kiran: only two pools: master and worker
stephen: bare metal have more specific requirements than vmware, etc
walters: ignition config that MCO applies is not exclusive, have to support external data
stephen: better understanding of live ISO; kbase might be good enough; not sure what else we could tell them
walters: working on conditional networking in Ignition
jlebon: if we flatten ignition configs, we don't need networking in initramfs
walters: solution #2 is conditional networking + ignition flattening
walters: SDN rules are for unprivileged access; if we have root on the cluster then...well, we could use it
stephen: rules to allow machine api to access MCS by default?
walters: if we support cracking open images and editing kargs, we need to revalidate this in 4.6 when the default kargs are removed

**plan**: check with customer about kbase approach; inform them about live ISO work in 4.6