---
tags: Agenda
---
# Design Meetings

[TOC]

## Administrative
### [Recordings ](https://hackmd.io/CvuF8MzmR9KPqyAePnilPQ)
### Old Etherpad https://etherpad.openstack.org/p/Airship_OpenDesignDiscussions

### Design Needed - Issues List
| Priority | Issues List |
| - | - |
| Critical | https://github.com/airshipit/airshipctl/issues?q=is%3Aopen+is%3Aissue+milestone%3Av2.0++label%3Apriority%2Fcritical+label%3A%22design+needed%22+|
| Medium | https://github.com/airshipit/airshipctl/issues?q=is%3Aopen+is%3Aissue+milestone%3Av2.0++label%3Apriority%2Fmedium+label%3A%22design+needed%22+ |
| Low | https://github.com/airshipit/airshipctl/issues?q=is%3Aopen+is%3Aissue+milestone%3Av2.0++label%3Apriority%2Flow+label%3A%22design+needed%22+ |

---

## Tuesday Dec 1, 2020

### PXE boot issue with new ironic python agent

The current ironic images have an issue where the ironic-inspector ramdisk is using a different network interface than the one it was PXE booted from -- resulting in not being able to report status back to Ironic, which is listening on the PXE network. This has manifested in multiple bare metal deployments having multiple NICs.

Sai & Konstantine found a workaround -- downgrading the ipa package in the ironic image.

* Do we need to create an issue w/ Ironic for this
* How can we bake this into / use our custom Ironic image
* Let's use this as an opportunity to align / chart a course w.r.t. ironic image overall
* We have an image, but so far it's very light & we're not using it in our deployment manifests: https://github.com/airshipit/images/tree/master/ironic
* We should bake all scripts into the image rather than overriding them in, e.g. https://github.com/airshipit/airshipctl/tree/master/manifests/function/baremetal-operator/entrypoint
* An "isolinux.bin file is not found" ironic misconfiguration error has been reported - maybe address this at the same time. https://github.com/airshipit/airshipctl/issues/420

### Secret stringData & last-applied-configuration
We are using `stringData` as a human-friendly way to author Secret data, with the understanding that Kubernetes will change it into a b64-enc `data` (which it does).

However the kubectl library also adds a `kubectl.kubernetes.io/last-applied-configuration` annotation, which unfortunately includes the cleartext `secretData`, which kind of defeats the point.

Is there some trick we can employ to change this behavior, or do we need to switch from `stringData` to `data`? The ReplacementTransformer has b64-enc capabilities that should make this easier to deal with than it was.

### Align phase list-related work
* This one (in progress) is for `phase list <planName>` proper:
* This one is for a `plan describe <planName>`
* https://github.com/airshipit/airshipctl/issues/394
* Seems to include `phase list` functionality
* Should we simply output the plan description in `phase list`?
* Do we have an issue for listing out phase plans?
* Could do via `phase list` (no `planName`: implicit "all plans")
* Could do via `plan list`

## Thursday Nov 19, 2020

### Revisit SOPS encryption subcommand
The GPG team pointed us to documentation that the SOPS library we're using for encryption is not intended for external use.
https://lists.gnupg.org/pipermail/gnupg-users/2020-November/064320.html
https://godoc.org/go.mozilla.org/sops/v3

Let's confirm whether we're comfortable using that library, and if we need to pivot to using the sops binary, what the best integration approach is.

**Alternative**
Implement encryption as a krm-function (as we have for Templater and ReplacementTransformer here https://github.com/airshipit/airshipctl/tree/master/krm-functions) and use generic container executor to run it as a phase or as a subcommand

### What is airshipctl cluster status ?
Follow discussion here https://hackmd.io/eiVvMiCwR5KsZSKGWjxlfQ

## Tuesday Nov 17, 2020
### Review list of functions for refactoring analysis
https://hackmd.io/cxcxX6ypQbCNifjJbDJnrg

## Thursday Nov 12, 2020
Follow up to Matt McEuen's question on removing taint from script https://github.com/airshipit/airshipctl/blob/master/tools/deployment/31_deploy_initinfra_target_node.sh#L22-L26

On testing found that ironic, helm-controller, source-controller, capi components, cert-manager etc was not comming up. For fixing ironic, helm-controller, source-controller, capi components etc we can add the toleration on the yaml or patch them with kustomize. But for cert-manager since its brought up by clusterctl, we will have to add cert-manager as a function and then deploy it via init-infra phase.
WIP Commit: https://review.opendev.org#/c/762186/

From Chat:
https://github.com/kubernetes-sigs/kustomize/issues/3095

New Issues :
1. Add a function for cert-manager, to give more control over its deployment config (like tolerations), rather than relying on clusterctl to manage it for us: https://github.com/airshipit/airshipctl/issues/408
2. Add patches to all infrastructure components to apply tolerations for master nodes, to allow them to run on master nodes. Remove the kubectl toleration application from our deployment scripts. https://github.com/airshipit/airshipctl/issues/406
3. Add patches to all infrastructure components to apply nodeSelectors for master nodes, to constrain them away from worker nodes. https://github.com/airshipit/airshipctl/issues/407
4. New issue to refactor functions to follow the pattern below.
a. If a function uses no copied upstream base, leave out `[function]/upstream/`
b. If a function is *only* an upstream base, put it under `[function]/upstream/` and have a thin passthrough `[function]/kustomization.yaml`

@startsalt
{
{T
 /
 + manifests/
 ++ function/
 +++ capi/
 ++++ vx.y.z/
 +++++ upstream/ | Put the upstream here
 +++++ replacements/
 +++++ patches1..n.yaml | Airship specific patches
 +++++ kustomization.yaml | Apply our patches on top of upstream
 }
}
@endsalt

### What is airshipctl cluster status ?
Continue discussion here https://hackmd.io/eiVvMiCwR5KsZSKGWjxlfQ

New Issues :
* Update executor definition to introduce. astatus interface: https://github.com/airshipit/airshipctl/issues/409
* Update different executors to implement status interface [EPIC]: https://github.com/airshipit/airshipctl/issues/410
* Add a new Phase status subcommand: https://github.com/airshipit/airshipctl/issues/411
* Add a new PhasePlan status subcommand: https://github.com/airshipit/airshipctl/issues/412

## Tuesday Nov 10, 2020

Please review design proposal for BMH -> Node label synchronization in metal-3: https://github.com/metal3-io/metal3-docs/pull/149

### Alternative for sops+gpg - airshipctl secret ...
* https://github.com/mozilla/sops

* SOPS seems like the best alternative, ..
* SOPS issue https://github.com/mozilla/sops/issues/767
* We wait for this to be solved?
* Implement a non gpg Client for Sops *tree.GenerateDataKeyWithKeyServices([]keyservice.KeyServiceClient{keySvc})
* Utilize the SOPS container from airshipctl.
* Is there a personal vault approach?
* Would it make sense to see if that can be integrated with airshipctl
* Standing a personal vault in a container?

### What is airshipctl cluster status ?
Follow discussion here https://hackmd.io/eiVvMiCwR5KsZSKGWjxlfQ

## Thursday Nov 5, 2020

### Not Assigned, Not Reasearch, Not Ready for Review issues needing Design
https://github.com/airshipit/airshipctl/issues?q=is%3Aopen+is%3Aissue+label%3A%22design+needed%22+no%3Aassignee+-label%3Aresearch+-label%3A%22ready+for+review%22+

### Ability to support multiple phaseplans within the same manifest context. [#385](https://github.com/airshipit/airshipctl/issues/385)
~~Do we simple have a phasePlan entryt in metadata?

https://github.com/airshipit/airshipctl/blob/master/manifests/metadata.yaml

phase:
  path: manifests/phases
  docEntryPointPrefix: manifests/site/test-site

~~
airshipctl phase list|describe|run
Phase Plans...
***airshipctl plan list|describe|run***
This would behave the same way as phases command. But act upon the phaseplan documents instead.

Need new isses for airshipctl phase describe and run commands.

What should be the "well curated", Phase Plans.
How many phaseplans do we need to define in treasuremao?
Hoow do we optimize phases utilization while reducing duplication impactign ci/cd.

### Investigate post-beta release approach [#354 ](https://github.com/airshipit/airshipctl/issues/354)

* Release Management Lifecycle
* Issue create the dumm or entryt point github actions release.yaml
* see https://github.com/fluxcd/flux2/blob/main/.github/workflows/release.yaml

## Tuesday Oct 27, 2020
### GPG Sops integration issues with gate
* https://review.opendev.org/#/c/758392
* https://review.opendev.org/#/c/758707/
Issues :

* Redesign inplementation of secret decrypt/encrypt
* Fix gating with current design / create mock tests to bypass gpg issues.
* Example ps's that have failed because of gpg sops.
* Issue: https://github.com/airshipit/airshipctl/issues/378

### Patchset failing:
* https://review.opendev.org/#/c/759763/

From chat:

* https://github.com/ProtonMail/gopenpgp
* https://console.cloud.google.com/gcr/images/kpt-functions/GLOBAL/sops?gcrImageListsize=30
* https://github.com/GoogleContainerTools/kpt-functions-catalog/blob/master/functions/ts/src/sops.ts

## Thursday Oct 22, 2020
### Host Config Operator Design
Discuss updates needed for Host Config Operator.
Code: https://github.com/airshipit/hostconfig-operator
Issues: https://github.com/airshipit/hostconfig-operator/issues
* Should we leverage [#1](https://github.com/airshipit/hostconfig-operator/issues/1) to encompass the design changes or create a new issue?

### Container execution from phases

Issues:
* https://github.com/airshipit/airshipctl/issues/369
* https://github.com/airshipit/airshipctl/issues/202

Options: [doc](https://docs.google.com/document/d/1qiao8ApYCavndwhVGuSE4588vAJLXyWmvfXQSOuxUJE/edit#heading=h.qdvgn5twc4uw)

The [demo](https://github.com/aodinokov/noctl-airship-poc) of the first approach and its [screencast](https://drive.google.com/file/d/1f4ZRZqce6NkVY1xvOjeaWwApKkUKGBoc/view?usp=sharing)

### Bootstrap Container execution from phases (Sidney)
Discussion about two design proposals for this feature.
Description of designs can be found [here](https://hackmd.io/Ah1CRbxETLCsUMhdncHERw?view)

### Airship PTG Discussion / Reminder

* Next Wed/Thurs, 8-12 central
* Focus will be on Post-v2 scope, but all design/community topics welcome
* Please add anything to discuss to the agenda:
* https://etherpad.opendev.org/p/wallaby-ptg-airship
* Wed 8-9 will be new dev onboarding
* Overall PTG info & agenda (don't forget to register for free)
* http://ptg.openstack.org/ptg.html

## Tuesday Oct 20, 2020
### HWCC Profile Evaluation (Rajat, Ashu, Matt)

HWCC Profiles are evaluated by the HWCC at the time the profiles are applied, but are only re-checked when the profile resource changes, not when the node status changes. I.e., the node resources must be in Ready state prior to applying profiles.

In the CS below that is accomodated by separating out worker provisioning into separate phases.

Do we want to enhance the HWCC to re-evaluate profiles when node statuses change?
* https://review.opendev.org/#/c/748421/9/tools/deployment/34_deploy_worker_node.sh@37

### Finish Review of Image Builder/Host Config spreadsheet?
https://docs.google.com/spreadsheets/d/1BQRadxOOzvRq8C6j3fe4xQ1f4AjSCfAyWG73U15pIho/edit#gid=0
In reviewing can we determine if the item is already supported by the selected solution or if we need an issue to add support?

### Boostrap Container/Ephemeral Cluster with Phase Run (Sidney)
* Discuss the design for decoupling airshipctl command from provider's bootstrap container
* How to support command concurrency
* How to improve command usability

## Thursday Oct 15, 2020
### Review of Image Builder/Host Config spreadsheet?
https://docs.google.com/spreadsheets/d/1BQRadxOOzvRq8C6j3fe4xQ1f4AjSCfAyWG73U15pIho/edit#gid=0
In reviewing can we determine if the item is already supported by the selected solution or if we need an issue to add support?

### Discussion about the Labeling of Hosts Day 2
Proposal:
* notion of label BMH/Provider Host and cascade upwards to the appropriate nodes
* set of machines (MachineDeployment/machine sets) , update the labels.. Not available through at the moment.
* Currently only supported via redeployment.
* Security issue with kubelet labeling nodes.

* Generic solution : New Generic Labeling Operator
* Running on management cluster
* Can access any workload cluster it manages.
* Has kubeconfig/secret for each cluster.
* Should eventually be a component of CAPI operator
* Would work for multiple providers.
* Where do the labels belong/ to cascade up from... or as the source for the operator
* Should support both:
* BMH's
* MachineDeployment Sets..
* Upstream discussions in CAPI: https://github.com/kubernetes-sigs/cluster-api/issues/493

### Pros & Cons of go<->shell

For the container bootstrap work discussed last week, there are pros & cons around using go, CLI tools, or both to do things like:
* parsingYAML input (go has an advantage)
* program logic & maintainability (go has an advantage)
* interfacing with public clouds (provider CLI tools have an advantage)

The patchsets are currently set up so that a go program is the entrypoint to the containers, and it exec's out to run the commands. Using go libraries for these CLIs may be an option; is it worth it? Other thoughts?

Changes:
* https://review.opendev.org/#/c/752298
* https://review.opendev.org/#/c/748537

### Discussion about Ironic and progress with Redfish

Discussed earlier about bridging current gaps within Ironic for Redfish compliance.
Here's a document in progress
https://hackmd.io/YDLMNIq2SCexddXHbXwosQ

### CAPI related proposal we might need to follow
* [Cluster API Bootstrap Reporting ](https://docs.google.com/document/d/1FVRxo9toKSUmvKIUFFzPFhnFrfdR9s7S6Bl4shovNlg/edit#heading=h.3mwmvwsf4jyi)
* [Management cluster operator](
https://docs.google.com/document/d/1ZsusF5c9pYxseuaKxTpctI5aUDqzl0sdCW4xxDbLm3k/edit#heading=h.wdfs5v5gumb8)

## Tuesday Oct 13, 2020

### Remember PTG Wed Oct 28, 29

* Open Infra PTG Oct 26-30: http://ptg.openstack.org/ptg.html
* Airship agenda (please add topics): https://etherpad.opendev.org/p/wallaby-ptg-airship
* Airship sessions in Cactus room 13-17 UTC (8-12 US Central), Wednesday & Thursday
* First hour of Wednesday session will be New Developer Onboarding
* Dont forget to register (free)! https://october2020ptg.eventbrite.com

### Airshipctl persisting data

**Kubeconfig** for target cluster. Problem is especially vivid during public cloud deployments, capz, capg but in bmh deployments as well:
- **BMH dployments**: We deploy target cluster via ephemeral cluster, where do we store the kubeconfig for the target cluster? You can get a secret from ephmeral cluster with target cluster kubeconfig, however once ephemeral cluster is gone, kubeconfig is gone with it? Is it a manual job to commit it into a document model, or should we require an external storage to save it, and teach airshipctl to use it if needed? In current scenario we predefine the certificates and keys, and we **DO** know the ip address of the server, but that requires that user **MUST** generate certs beforehand, that is a manual labor.
- **Public clouds and CAPD**: Even when we can predefine the certs ands keys, we cant predict the IP address of the API server, which is part of kubeconfig, so we get it from parent ephemeral cluster (KIND), but once you tear it down, you don't have the IP, and airshipctl has no means to connect to it. Of course, we do have an ability, to supply our own kubeconfig via flag, and airshipctl will use it and will connect to it. But the kubeconfig is not persisted anywhere, should we simply rely on user and expect him to commit the kubeconfig to the manifest? should that be automated, or should this be a general bootstrap phase, which ends in kubeconfig being commited to persistent storage automatically that can be reused later?

discussion at https://hackmd.io/qC7PZYxWSaqC3PZVp4d4iw

### Deduplication of network definitions #315
Look at https://review.opendev.org/#/c/749611/ for implementation details

### Airship session in 2020 virtual Open Infrastructure Summit.
https://www.airshipit.org/blog/airship-featured-at-virtual-open-infrastructure-summit-in-october/

## Thursday Oct 8, 2020

### Discuss bootstrap approach for Azure, GCP etc
Defined in https://review.opendev.org/#/c/737864
And https://review.opendev.org/#/c/748537/
Questions:
* airshipctl and public clouds: priority (do we need this right now?) and level of support. If it's needed ASAP - Can we have it properly architectured first to comply with our current architecture? (see some big design concerns below)
* Why do we need a separate sub-command instead of new executor?

Some documentation https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_420/755398/20/check/openstack-tox-docs/4204487/docs/phases.html

* Why do we need to extend the config module with bootstrap container options(we moved them to document model a while ago)?
* Why do we need new container interface for bootstraping a cluster? can not we use 'Kubernetes in Docker' a.k.a 'kind' as an ephemeral cluster?
Proposal:
1. Use KIND cluster instead of ephemeral node for cloud providers
2. Implement executor (and phase) for cloud providers
3. Use clusterctl executor as is against KIND cluster (no additional code required)

### Regarding Image Builder, Have we consider existing tooling, in particular: https://www.packer.io ?
- It can be coupled with things like ansible to build custom images.
- They also support cloud specific image generation. If we ever went down the route of deploying images customized for a particular cloud, say azure or gcp, packer has cloud specific plugins.

### Airshipctl persisting data

**Kubeconfig** for target cluster. Problem is especially vivid during public cloud deployments, capz, capg but in bmh deployments as well:
- **BMH dployments**: We deploy target cluster via ephemeral cluster, where do we store the kubeconfig for the target cluster? You can get a secret from ephmeral cluster with target cluster kubeconfig, however once ephemeral cluster is gone, kubeconfig is gone with it? Is it a manual job to commit it into a document model, or should we require an external storage to save it, and teach airshipctl to use it if needed? In current scenario we predefine the certificates and keys, and we **DO** know the ip address of the server, but that requires that user **MUST** generate certs beforehand, that is a manual labor.
- **Public clouds and CAPD**: Even when we can predefine the certs ands keys, we cant predict the IP address of the API server, which is part of kubeconfig, so we get it from parent ephemeral cluster (KIND), but once you tear it down, you don't have the IP, and airshipctl has no means to connect to it. Of course, we do have an ability, to supply our own kubeconfig via flag, and airshipctl will use it and will connect to it. But the kubeconfig is not persisted anywhere, should we simply rely on user and expect him to commit the kubeconfig to the manifest? should that be automated, or should this be a general bootstrap phase, which ends in kubeconfig being commited to persistent storage automatically that can be reused later?

## Tuesday Oct 6, 2020

### Generation & rotation of site-level secrets (Sirisha)
[airshipctl secrets generate command design doc ](https://hackmd.io/oXLb7IMGRtGUBpewo7spUA?view)
* Secret catalog structure - one or more than one?
* Incorporating generated secrets into phase kustomizations
* Generating secrets other than Secrets
* Design doc: https://hackmd.io/oXLb7IMGRtGUBpewo7spUA?view

Secrets generate discussion notes , or details https://hackmd.io/Mx3X6U6lQ8GLEW4ISgbDoA
-
## Thursday Oct 1, 2020

### [Image Builder Declarative Discussion](https://hackmd.io/6CgeJKqVQJ6vpT2DC5mx6A)

### Generation & rotation of site-level secrets (Sririsha)
Secret catalog structure - one or more than one?
Incorporating generated secrets into phase kustomizations
Generating secrets other than Secrets

### Airshipui - issue grooming & next steps (Andy Schiefelbein)
* Phase render & apply steps (https://github.com/airshipit/airshipui/issues/37) is the most actionable issue as of right now
* Needing use cases for what a user wants to see as a landing page
* Needing use cases for wanted user interactions

* https://github.com/airshipit/airshipctl/issues/359 exposing hosts for use with baremetal
* https://review.opendev.org/#/c/755380/ A first attempt at exposing the info

### HostConfig Operator Discussion Continuation

## Tuesday Sept 29, 2020

*Airship Tuesday Design Meeting-20200929 1305-1*
***Password***: dPEJMDw6
***Recording***: https://attcorp.webex.com/recordingservice/sites/attcorp/recording/playback/24f6248b93164a4b9077c3b2b94285b4

### Using **relative paths as entry points for Phases**
Impact of that , is what about empty site level phases, do they need to be explicitly created
Idea of a airshiopctl phase run --inherited < meaning get the phase from the type)
airshipctl to generate phase documents implicitly form the type associated with the site.
Options that have been discussed in slack :

manifests:
dummy_manifest:
primaryRepositoryName: primary
repositories:
primary:
checkout:
branch: ${AIRSHIP_CONFIG_PRIMARY_REPO_BRANCH}
force: false
remoteRef: ""
tag: ""
url: ${AIRSHIP_CONFIG_PRIMARY_REPO_URL}
metadataPath: manifests/metadata.yaml
targetPath: ${AIRSHIP_CONFIG_MANIFEST_DIRECTORY}
subPath:

apiVersion: airshipit.org/v1alpha1
kind: Phase
metadata:
name: bootstrap
config:
executorRef:
apiVersion: airshipit.org/v1alpha1
kind: ImageConfiguration
name: isogen
documentEntryPoint: airshipctl/manifests/site/test-site/ephemeral/bootstrap

How do I inplicitly figure out the site specific endpoint path:
**Option I**
- Combination of TargetPath +subPath in manifest document imply the location of the site
- con about this approach is an implicit expectation on the site author for where the site phase doccument live.
Example, in airship config:
``subPath: manifests/site/test-site``
``targetPath: /home/matt/airshipctl``
And then in the Phase resources:
``documentEntryPoint: target/initinfra``

**Option II**
Introduce a new value in the metadata.yaml
Airship Config --> Manifest --> metadataPath --> metadata.yaml has entry for location of phase or site entry point.

Repos are been clone to :
targetPath+<PhaseRepositoryName>

Example, in metadata.yaml:
If metadata.yaml is defined at the type level, this might look like :
``documentEntryPointRoot: manifests/type/mytype/myphases``

If metadata.yaml is been defined at the site level then:
``documentEntryPointRoot: manifests/site/test-site``

Code will use TargetPath + <PhaseRepositoryName> + documentEntryPointRoot to identify entryPoint.

And then in the Phase resources:
``documentEntryPoint: target/initinfra``
Example, in metadata.yaml:
``phaseBase: ../treasuremap/manifests/type/cruiser``
And then in the Phase resources:
``documentEntryPoint: target/initinfra``

**Option III**
Use PrimaryRepositoryName, which is already part of the manifest to indicate which repository holds phases and prepend it's name to entrypoint, example: primaryRepositoryName: primary
example result:
``primaryRepositoryName: primary``
``DocumentEntryPoint: manifests/site/test-site``
URL: /root/airshipctl
dirName = airshipctl
``kustomizeRoot: airshipctl/manfiests/site/test-site``

Outcome:
- Change Airship Config Manifest , replace PrimaryRepositoryName for PhaseRepositoryName
- Introduce a new value in metadata.yaml called documentEntryPointRoot that will identify the kustomize entrypoint for the site where the phases can be found. **Phase run ** will use *TargetPath + <PhaseRepositoryName> + documentEntryPointRoot* to identify kustomization entryPoint.
- Need to move Phases and Phase Plan under a ***type*** in the airshipctl manifests.
- Assumption is metadata.yaml is usually defined at the type level ,and comes as part of that repo. But since we can define in the airship.config.manifest then it can be custom at the site level when appropriate.

### Discussion of Firmware config extension and update on configuration moulds - Noor
I talked with Richard about this, and have some things to discuss
* **Timeframe for configuration** moulds is at best 2021
* Finish ironic implementation, still working on specification
* gopher cloud
* Update metal3
*
How do we close the RedFish gaps in the current Bios/Firmware/Raid implementation:
- Ironic PTG about this RedFish point.