Design Meetings

--- tags: Agenda --- # Design Meetings [TOC] ## Meeting Links Can be found here https://wiki.openstack.org/wiki/Airship#Get_in_Touch ## Archive Agenda/notes from prior to 2021-04-01 can be found [here](https://hackmd.io/OuI_aOfXQzCjE5NEsvDfIw?both). ## Troubleshooting Guide & FAQs HackMDs **Purpose:** provide a more accessible, flexible & dynamic way of capturing troubleshooting information & frequently asked questions.Depending on the amount of content (or lack thereof) these may be combined in the future. https://hackmd.io/Nbc4XF6mQBmutMX_FEs51Q https://hackmd.io/jIr3An6MT5C2xAQbKR3qoA :::success Feel free to add content to the pages. Thanks! ::: ## Administrative ### [Recordings ](https://hackmd.io/CvuF8MzmR9KPqyAePnilPQ) ### Old Etherpad https://etherpad.openstack.org/p/Airship_OpenDesignDiscussions ### Design Needed - Issues List | Priority | Issues List | | - | - | | Critical | https://github.com/airshipit/airshipctl/issues?q=is%3Aopen+is%3Aissue+milestone%3Av2.0++label%3Apriority%2Fcritical+label%3A%22design+needed%22+| | Medium | https://github.com/airshipit/airshipctl/issues?q=is%3Aopen+is%3Aissue+milestone%3Av2.0++label%3Apriority%2Fmedium+label%3A%22design+needed%22+ | | Low | https://github.com/airshipit/airshipctl/issues?q=is%3Aopen+is%3Aissue+milestone%3Av2.0++label%3Apriority%2Flow+label%3A%22design+needed%22+ | ## Thursday August 5, 2021 ### Upgrade capm3/bmo/ironic deployment [continued] (James) [#554] Some new features from upstream, are we interested in leveraging any/all of them? Referring to [CAPM# releases](https://github.com/metal3-io/cluster-api-provider-metal3/releases) * ironic TLS and basic auth * ironic keepalived * bmo live iso image support Manifest pattern for upstream dependency: * copy vs reference to upstream manifests ### Kernel/Driver/Package upgrade (Sreejith) [#603, #604, #605] There are various issues created for uplifting kernel/Driver/packages in both airshipctl and treasuremap. its mentioned to use image-builder for this purpose. when we use image-builder, we will have to do os reinstallation on all the nodes and if we want to perform this multiple time a year, it consumes lot of time. Also i have found that UUID of the disk is getting changed after reinstallation which may cause problem with ceph cluster. cant we use hostconfig-operator to perform the kernal/driver/package upgrade and then use an updated image when performing distro upgrade? ### Ironic boot intrface using Redfish AS2 uses pxe for booting. Ironic has Redfish as an option for the boot interface in its configuration. Are there any known issues or concerns to use Redfish for booting? ## Thursday July 29, 2021 ### Per cluster-type Image-builder configuration using kpt approach (Alexey, as per internal preliminary discussion) We can do this per type: * pull image-builder similarly to how we did with SIP/Vino: [example](https://review.opendev.org/c/airship/treasuremap/+/802615/1/manifests/type/multi-tenant/image-builder/Kptfile). * update image-builder config if needed: [example](https://review.opendev.org/c/airship/treasuremap/+/802615/1/manifests/type/multi-tenant/image-builder/manifests/rootfs/multistrap-vars.yaml#51). * Next time when we need to pull changes from upstream - just execute `kpt pkg update [@<commit-id>]` from image-builder dir and kpt will do 3way merge - it will be possible to see what new changes were introduced and if they are in the conflict with local changes. Once conflicts are resolved - just commit the changes: [example](https://review.opendev.org/c/airship/treasuremap/+/802627). Needed changes: * Makefile in treasuremap has to be updated in order to build images per type: [example](https://review.opendev.org/c/airship/treasuremap/+/802615/1/Makefile). Manifests have to be updated as well to use the right tag of the image (TBD) * Image-builder has to be adjusted a bit to work with kpt: https://review.opendev.org/c/airship/images/+/802626 * POST Gating has to be updated to build images on merge (TBD). ### Upgrade kpt to v1.0.0-beta.x (Matt F) v1.0.0-beta brings some breaking changes for our current design. This relates to [#598](https://github.com/airshipit/airshipctl/issues/598) * concept of local package with upstream dependencies is gone; `dependencies` key in Kptfile has been deprecated * Kptfiles must define an `upstream` repository, or they will be assumed to be dependent subpackages with a parent higher up in the directory tree that provides the root upstream repository * local changes must be committed via git before performing an update How this affects current workflow: * No top-level Kptfile that can be easily modified to update pkg version. Each individual package's Kptfile must be edited (e.g. manifests/function/flux/base/upstream/policies/Kptfile) * Cannot simply run `kpt pkg update .` from root of function (e.g. flux/helm-controller) anymore. Packages must be updated independently, with git commits before and after each update Possible solution: * Replace function's top-level Kptfile with a script (`update.sh`?) that runs `kpt pkg update upstream/<pkgname>` for everything in each function's `upstream` directory, and makes the necessary intermediate local commits between pkg updates ### Upgrade capm3/bmo/ironic deployment [#554] Some new features from upstream, are we interested in leveraging any/all of them? Referring to [CAPM# releases](https://github.com/metal3-io/cluster-api-provider-metal3/releases) * ironic TLS and basic auth * ironic keepalived * bmo live iso image support Manifest pattern for upstream dependency: * copy vs reference to upstream manifests ## Thursday July 15, 2021 #### *Carryovers from 7/8 meeting ->* ### K8s v1.21 upgrade & general uplift approach (Andrew) When we had the v1.20 uplift scramble, we discussed the need to establish an ongoing k8s uplift cadence to keep Airship current, certified & in conformance. This is intially for greenfield deployments, i.e. what version of K8s are we deploying out of the box. Brownfield upgrades will be handled under a different set of issues, but will need to bring the findings of that back to ensure we have a commmon approach. * With v1.20, the approach was to upgrade the version for each provider. We want to have a more general approach, if possible for v1.21 & going forward. * Determine the timeframe by when we need to uplift the k8s version in Airshipctl (i.e. how many months after a release should we uplift?). The k8s release process is fairly mature at this point, though there do tend to be several point releases per main version release. * Should we bake conformance testing into the version upgrades? Separate issues for testing? Testing tasks: * Perform conformance testing (Anuket and/or CNCF) for Bare Metal against the uplifted version * Perform conformance testing against at least one other provider (maybe all, includes: Docker, Azure, GCP & Openstack) * Report uplift/conformance testing to CNCF to maintain certification Updated the below issue to be more v1.21 focused with the more general approach. https://github.com/airshipit/airshipctl/issues/589 v1.22 info https://github.com/kubernetes/sig-release/tree/master/releases/release-1.22 Some future topics for brownfield upgrades once we've worked the spikes: * Determine impacts/process for brownfield bare metal upgrades via CAPI. * Determine position on backwards compatibility with prior releases (Do we support? If so, how many?) ### #491 Redesign airship cluster status command (Vladimir Kozhukalov) https://github.com/airshipit/airshipctl/issues/491 This feature was dependent on a different feature. Here is an old discussion https://hackmd.io/BbFyJRKGRQiuXYJduPhu4Q Current architecture cannot support this command. Vladimir's comments from the issue: > Airship phase engine is stateless, we don't use any operator to deal with phases/phase plans. We don't deploy phase CRs anywhere. We also don't store phase run results/errors anywhere. A user who runs a phase plan can see which of the phases were successful and thus cluster status. But we can not have a separate command to gather the cluster status since we don't have such a place from where the status could be collected. > Discuss path forward and if we should close this issue for now. ### #597 Airship specific implementation of KRM Function Specification https://github.com/airshipit/airshipctl/issues/597 > At the moment Airship uses two types of generic containers: krm (kyaml/runfn) and airship (airship specific docker client). kyaml/runfn has some limitations that do not allow to use it everywhere. For example, sometimes we want a container to be run from a privileged user. Also kyaml/runfn does not provide some of the functionality we potentially would like to have (e.g. running timeout). > Let's - Get rid of Airship specific docker client - Copy the code from kyaml/runfn and modify it to cover Airship specific needs (just like what KPT did) #### *New Items ->* ### SIP#19 Configurable HAProxy in SIP (Manoj Alva) This is related to https://github.com/airshipit/sip/issues/19 and PS https://review.opendev.org/c/airship/sip/+/799161 is put in place. Discussion needed on the following item. * The attributes of HAProxy configuration to be considered. The PS currently considers ==timeout== parameters only [HAProxy Config Ref] https://cbonte.github.io/haproxy-dconv/2.0/configuration.html ### #545 Generic container timeout validation & enforcement (Manoj Alva) For the requirement "provide a compliance mechanism to validate the timeout has been acknowledged & action has been taken", need help on the scope covering this issue. Are the requirements targetted at ensuring e2e testing of the timeout support implemented via #544 possibly via Ginkgo framework ? ## Thursday July 8, 2021 ### Component Uplift Review Ensure we have all the major components accounted for, and determine if we have any gaps. v2.2 milestone issues list: https://github.com/airshipit/airshipctl/issues?q=is%3Aopen+is%3Aissue+milestone%3Av2.2 K8s uplift to v1.20 / v1.21 > * https://github.com/airshipit/airshipctl/issues/555 < Bare Metal * https://github.com/airshipit/airshipctl/issues/556 < Docker * https://github.com/airshipit/airshipctl/issues/569 < Azure * https://github.com/airshipit/airshipctl/issues/570 < GCP * https://github.com/airshipit/airshipctl/issues/571 < OpenStack * Vulnerability issue: * https://github.com/airshipit/airshipctl/issues/559 < etcd - should be resolved by uplift CAPM3, BMO & Ironic to v0.4.2 -- needs Still is still against capi v1apha3. What Ironic OS? * https://github.com/airshipit/airshipctl/issues/554 * Vulernability issues: * https://github.com/airshipit/airshipctl/issues/558 < dependent on upstream issue https://github.com/metal3-io/ironic-image/issues/266 * https://github.com/airshipit/airshipctl/issues/560 < kube-rbac-proxy - should be resolved by uplift CAPM3, BMO & Ironic to v0.5.0 -- needs to wait .. Still is still against capi v1apha3 BMO. and Ironic are now separated. Maybe we can drive versions ? What Ironic OS is targetted? CAPI to v0.4 & CAPM3 to 0.5.0 * https://github.com/airshipit/airshipctl/issues/518 < need upstream to drop first. Arvinder updated today, now need CAPM3 v0.5.0 upgrade to fully implement. CAPI (and docker Provider ) to v1alpha4 uplift [ NEW ISSUE ] [NEW ISSUE] Explore KPT upgrade options 0.37 vs. v1.0 [NEW ISSUE] Kustomize Upgrade to the latest version (v4.2.0) Clusterctl binary as KRM function: * https://github.com/airshipit/airshipctl/issues/568 < removes CAPI dependencies & positions for the future moved to v2.1 Sonobuoy to v0.51 * https://github.com/airshipit/airshipctl/issues/557 < believe this is already being used, just need patchset reviewed & merged. v2.1 issue just needs review iLO Redfish API * https://github.com/airshipit/airshipctl/issues/540 < Updated yesterday to v2.1 ## Thursday June 24, 2021 ### Incorporating Image Builder manifests into Treasuremap (Need: Matt M., Craig, Pallav) * Today, this part of our declarative intent is really just captured as `image-builder` defaults * How should we integrate it w/ Treasuremap? * Need a pattern that makes sense for operators * Should we use catalogues to build type/site-specific catalogue info into config files on qcows: * For Example: https://review.opendev.org/c/airship/airshipctl/+/775035/45/manifests/function/ephemeral/chrony-secret.yaml :::info Craig : "I would suggest we re-use the pattern of parent+child zuul jobs that we currently use. The parent Zuul job is defined upstream images repo, and allows for child job (like we have downstream) to override any needed parameters for image building. This would be a good pattern as well to follow for other container image customizations (e.g., the same pattern would permit operator customization of airshipctl)" ::: ### airshipctl secret generate encryptionkey, airshipctl cluster rotate-sa-token, airshipctl cluster check-certificate-expiration commands discussion (Ruslan A.) * rotate-sa-token, check-certificate-expiration - both commands doesn't work at all since they were never used in test deployments and they use old-style API; since we have an upcoming upgrade of k8s dependecies in airshipctl - it would be diffucult to upgrade these commands to make them work; * secret generate encryptionkey was introduced 2 years ago and it's not clear what's its purpose in current curcumstances, it just generate secure random string variable length; * Do we really need above commands? If so, shouldn't they be as phases, not the core airshipctl functionality? * https://github.com/airshipit/airshipctl/issues/588 ## Tuesday June 22, 2021 ### Generating secrets for subclusters * it's not implemented in treasuremap yet * there are some params that may be a good candidates for that: [ssh keys](https://github.com/airshipit/treasuremap/blob/master/manifests/type/airship-core/target/generator/secret-template.yaml#L58) and [dex clientSecret](https://github.com/airshipit/treasuremap/blob/master/manifests/type/airship-core/target/generator/secret-template.yaml#L63) * Questions: * should we do that? (seems like we have to if there are no other approaches) * what master key should we use? from target? or create gpg per subcluster? (probably, per subcluster - we don't want to allow any subcluster users to decrypt target cluster creds, right?) * should target cluster admin have access to that subcluster master keys? (probably yes, because target cluster admin will also do day1 for subcluster, right?) Alternativly we may even avoid storing everything in git- instead follow ClusterAPI approach - where we keep everything in target cluster? Probably it would be great to get a 'big picture' of how encryption/decryption will work for the subcluster scenario. Let's make it together? :) **ISSUE** : Define an integration with system to manage GpG Key Management system that provides RBAC, distribution etc,. I.e. Such as Admision Controller Mutating Admission. Controller that injects keys as needed. ### VINO namespace issue. Vino creates BMHs in a single namespace, currently in the same as vino-manager runs (vino-system). Cluster-API CAPM3 right now requires that BMHs reside in the same namespace as m3m objects and hence KCP. So KCP needs to be in the same namespace as BMHs. With the VINO design, we can only specify 'count' per vino flavor(nodeset), so even if we add namespace field to VINO CR nodeset, it is still going to be one namespace per whole vino nodeset infrastructure. For example, if we have nodset called `control-plane` with count=1, and 40 nodes on the site, we will end up with 40 masters in single namespace. ### FAQs page Similar to Troubleshooting Guide, the documentation tema has created a FAQs page to allow for community input to develop content that may go into https://docs.airshipit.org https://hackmd.io/jIr3An6MT5C2xAQbKR3qoA ## Thursday June 17, 2021 ### airshipctl exits with error when expanding controlplane nodes * https://github.com/airshipit/airshipctl/issues/525 * It blocks the bare metal phase plan STL2 testing. Is it 2.1? * Labeld as designed needed. ### Managing Gatekeeper Policy Constraint Templates & Constraints in Treasuremap (cont'd) * Are there any treasuremap policies we might want to enforce i.e. What policies do we start with? * Restrictions on image sources? * Restrictions on helm chart repositories? * Any other standards for treasuremap deployments? * Any PodSecurityPolicy implementations needed in treasuremap? * Talk about how we might handle Audit violations in TM * I would say some basic ones : * https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/general/allowedrepos * https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/general/containerlimits * https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/general/imagedigests ## Tuesday June 15, 2021 ### Use clusterctl as a binary inside of KRM function instead of calling API (Ruslan A.) Discussion of the issue - https://github.com/airshipit/airshipctl/issues/568 PoC patchset - https://review.opendev.org/c/airship/airshipctl/+/793701 Diagram - https://drive.google.com/file/d/1lqTW4ALAKOJcCTCYvMEh9C7RyWxMGwME/view ### Managing Gatekeeper Policy Constraint Templates & Constraints in Treasuremap (Larry) * Design discussion for https://github.com/airshipit/treasuremap/issues/174 * Definition of the Policy == Constraint Template * e.g. https://github.com/open-policy-agent/gatekeeper-library/blob/master/library/pod-security-policy/users/template.yaml manifests/function/gatekeeper/policies/ manifests/function/gatekeeper/policies/<policy-name> manifests/function/gatekeeper/policies/<policy-name>/ manifests/function/gatekeeper/policies/<policy-name>/kustomization.yaml manifests/function/gatekeeper/policies/<policy-name>/template e.g. https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/pod-security-policy/users * Instance of a Policy * e.g https://github.com/open-policy-agent/gatekeeper-library/blob/master/library/pod-security-policy/users/samples/psp-pods-allowed-user-ranges/constraint.yaml manifests/function/gatekeeper/policies/instances/ manifests/function/gatekeeper/policies/instances/<instance-of-policy-x-name> manifests/function/gatekeeper/policies/instances/<instance-of-policy-x-name>/kustomization.yaml manifests/function/gatekeeper/policies/instances/<instance-of-policy-x-name>/constraint.yaml manifests/function/gatekeeper/policies/instances/<instance-of-policy-x-name>/replacements/... || TBD if we use catalogue info for defining the constraints * How do we define a collection of policies as a group that menas something. e.g. PodSecurityPolicy ... manifests/composite/gatekeeper/<name of policy group> manifests/composite/gatekeeper/<name of policy group>/kustomization.yaml .. Uses Instance of policy as resources. manifests/composite/gatekeeper/<name of policy group>/replacements/kustomization.yaml * When do we deliver the Policies Will keep this as a TBD, expect we might need to deliver policies in multiple phases, yet to be determined. * Installing gatekeepr is init infra phase, ... whatever helm "thingie" * Explore using this for policy validation : https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/master/functions/go/gatekeeper ### Bake Helm Charts in Helm-Chart-Collator (Sidney S.) Some of service deployment relying on (Flux) Helm operator is still pulling charts from the public repository. Should all Helm charts used by airshipctl be "baked" in the Helm Chart Collator Docker image, so charts are not exposed to the public? (Sean) This is being worked here: https://github.com/airshipit/treasuremap/issues/162 ## Thursday June 10, 2021 ### Discuss different approaches to Day2 upgrade operations (Vladimir S., Alexey O.) Several approaches based on Ceph/Rook upgrade examples Presentation in this hackmd note https://hackmd.io/0Sw53doBSwiOzgfzYrfmRQ ## Tuesday June 8, 2021 ### RAID implementation (JT & Matt) ### NextGen secret generation (Alexey) (Following up our conversation started [here](https://hackmd.io/QiEksO4fRk-MnBjwBFaAkQ#Tuesday-Apr-27-2021)) UPD: **we decided to proceed with that changes in 2.2** Implementation is here: https://review.opendev.org/c/airship/airshipctl/+/794887 Requires kustomize 4.x: that's why based on https://review.opendev.org/c/airship/airshipctl/+/794269/ Supports: 1. Generation with different timeperiods (e.g. once, monthly, yearly etc.). Includes `forced` regeneration by different period, e.g. regenerate all `yearly` secrets. Template contains info about how ofter to regenerate each group. 2. All secrets: generated and externally provided - **are in 1 single file** 3. `Pinning` of secret - this secret will be considered as manually (exnternally provided) and won't be regenerated 4. All modifications are done with 1 phase `secret-update` + [file](https://review.opendev.org/c/airship/airshipctl/+/794887/13/manifests/site/test-site/target/encrypted/updater/import.yaml) that should contain a patch that is getting merged to the final secrets file. The document structure can be strictly defined - we can switch from VariableCatalogue to something like SecretCatalogue with CRD and that will allow validation. See example: https://review.opendev.org/c/airship/airshipctl/+/794887/13/manifests/site/test-site/target/encrypted/results/secrets.yaml Each group has date of last update. Here is a [template example](https://review.opendev.org/c/airship/airshipctl/+/794887/13/manifests/type/gating/target/regenerate-secrets/template.yaml). The implementation introduced `functions` and `modules` for templater. Function defined in module can be called with `include` function (definition was taken from helm). Module is a document that contains function definitions. E.g. [this file](https://review.opendev.org/c/airship/airshipctl/+/794887/13/manifests/function/templater-helpers/secret-generator/lib.yaml) is inlcuded [here](https://review.opendev.org/c/airship/airshipctl/+/794887/13/manifests/site/test-site/target/encrypted/updater/kustomization.yaml#4) and contains implementation of function `group` that does the main magic on understanding what to regenerate, what to import and etc. ## Thursday June 3, 2021 ### Treasuremap to include Gatekeeper/OPA? (Bryan) Does it make sense to include [Gatekeeper](https://kubernetes.io/blog/2019/08/06/opa-gatekeeper-policy-and-governance-for-kubernetes/) (and [here](https://open-policy-agent.github.io/gatekeeper/website/docs/howto/)) as a "core platform" level policy agent in the reference implementation? If so, what are the resiliency and deployed features that would be considered fundamental to the implementation? Is the [basic installation](https://open-policy-agent.github.io/gatekeeper/website/docs/install#deploying-via-helm) sufficient? Does this provide a real path to the support of deprecation of PSP, or too early to say? * [PodSecurityPolicy Deprecation: Past, Present, and Future](https://kubernetes.io/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) * [PodSecurity admission (PodSecurityPolicy replacement) #2579](https://github.com/kubernetes/enhancements/issues/2579) Create some issues : * Issue 1 - Define gatekeepr function in treasuremap/ identify proper phase/type etc where it should be deployed. * Issue 2 - Bare policies as part of deployment that demonstrate gatekeeper is properly configured and working. * Issue 3 - Policies that enforce PSP like rules. Esssentially remove PSP's. ### Base Images (Andrii, MattF, MattM) We have a [story](https://github.com/airshipit/airshipctl/issues/514) for switching KRM function base images away from dockerhub. The idea was to switch to an alternative non-dockerhub base if there was a good alternative to alpine, and maintain our own mirror in quay for alpine if not. It's slightly more nuanced than that; a few things to consider: - Alpine has a strong security stance. Do we want to keep with Alpine for that reason alone? - We already have downstream overrides of these base images to a downstream Alpine build. Changing upstream requires changing downstream. - We could get more cross-distro compatibility if we use `#!/bin/bash`. However, Alpine doesn't have bash, so if we want Alpine that won't work. - We could get more cross-distro compatibility if we refactored package management (`apk` vs `apt` vs `yum`) into a shared base image. In that case we are back to maintaining our own base images, however. - Expecting compatibility between base images is tough: - Alpine is weird because it doesn't have `/bin/bash` - minideb is weird because `/bin/sh` is `dash` Matt has a patchset switching to minideb [here](https://review.opendev.org/c/airship/airshipctl/+/790865), which changes both shell conventions (bash-like vs dash-like) and package manager (`apk`->`apt`). ### Uplifting BMO/CAPM3/Ironic, CAPI & the CAPI Management Operator (Andrew - Arvinder) **Issues:** [#554](https://github.com/airshipit/airshipctl/issues/554) & [#518](https://github.com/airshipit/airshipctl/issues/518) **Timing:** v1alpha4 is currently available for BMO/CAPM3/Ironic. v1alpha4 for CAPI is a month or so out. There aren't dependencies between the two uplifts though an additional uplift of CAPM3 will be required when shifting to CAPI v1alpha4. **Approach:** Let's leave #554 for the BMO, CAPM3 & Ironic uplift. Let's create a new issue or revise #518 for CAPI uplift to v1alpha4 when its available (and required CAPM3 uplift). **Discussion:** Do we want to shift to using CAPI Management Operator at some point? https://hackmd.io/qdTfhNj8RSuQOM0QZL0JOA#CAPI-Management-Cluster-Operator ## Tuesday June 1, 2021 ### Using AGE as an alternative to PGP in SOPS (Matt, Alexey) Pronounced "ah-gay". A recently-merged option in SOPS, recommended by the SOPS community: "age is a simple, modern, and secure tool for encrypting files. It's recommended to use age over PGP, if possible." It has a more compact representation - encrypted data: ```age1yt3tfqlfrwdwx0z0ynwplcr6qxcxfaqycuprpmy89nr83ltx74tqdpszlw``` Public key: ```AGE-SECRET-KEY-1NJT5YCS2LWU4V4QAJQ6R4JNU7LXPDX602DZ9NUFANVU5GDTGUWCQ5T59M6``` Is it something we should consider adopting? * https://github.com/GoogleContainerTools/kpt-functions-catalog/pull/241 * https://github.com/mozilla/sops#encrypting-using-age * https://github.com/FiloSottile/age ## Thursday May 27, 2021 ### Dex HelmRelease - LDAP Patch locationn (Sidney S.) A while ago it was decided to split the Dex/API server configuration from Dex/LDAP, the former relying on replacement rules while the latter kustomized through Strategic Merge patch. The split was done with Dex/LDAP patch being implemented in *treasuremap/manifests/type/**airship-core**/target/workload/dex-aio* folder. As *multi-tenant* type will need similar patch, where would be the best place to implement this patch where it is shared between **airship-core**, **multi-tenant**, etc. *Could it be under composite/utility (new) where all utility patch would be added, starting with dex/ldap patch?* >REMINDER: this patch is composed of kustomization.yaml and the patch as shown below. kustomization.yaml ```yaml= apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../../../../function/dex-aio patchesStrategicMerge: - dex-aio-helm-patch.yaml ``` dex-aio-helm-patch.yaml ```yaml= apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: dex-aio namespace: default spec: values: params: ldap: bind_password: "your LDAP bind password" name: "LDAP TEST SERVICES" config: host: "your LDAP FQDN" port: 636 bind_dn: "your LDAP bind username" bind_pw_env: LDAP_BIND_PW username_prompt: SSO Username user_search: base_dn: dc=testservices,dc=test,dc=com filter: "(objectClass=person)" username: cn idAttr: cn emailAttr: name nameAttr: name group_search: base_dn: ou=groups,dc=testservices,dc=test,dc=com filter: "(objectClass=group)" userMatchers: userAttr: DN groupAttr: member nameAttr: cn ``` ### Upgrade to kustomize 4.1.3 issues (Alexey O.) Description of issues ``` kustomize 4.1.3 upgrade was difficult. It's possible to understand by the number of different options I've tried: https://review.opendev.org/c/airship/airshipctl/+/788582 The main issue appeared because of this commit in kustomize: https://github.com/kubernetes-sigs/kustomize/commit/a2871181fe7e01b42741019c49d12d1ca24602a3 . They switched from github.com/go-openapi to k8s.io/kube-openapi and they use k8s.io/kube-openapi/pkg/validation/spec module. (this module appreared recently - when k8s.io/kube-openapi already switched to gnostic 0.4.1+) There is another module called github.com/googleapis/gnostic that is used by variaty of components including k8s.io/kube-openapi and as well as such modules as k8s.io/apiextensions-apiserver, k8s.io/apimachinery, k8s.io/apiserver, k8s.io/client-go and k8s.io/kubectl. The mainteiners of gnostic have made a breaking change in API here: https://github.com/google/gnostic/commit/896953e6749863beec38e27029c804e88c3144b8: All versions <=0.4.0 had a module github.com/googleapis/gnostic/OpenAPIv2 and started >=0.4.1 github.com/googleapis/gnostic/openapiv2 The problem was that k8s.io/kube-openapi/pkg/validation/spec was added AFTER k8s.io/kube-openapi switched to github.com/googleapis/gnostic >0.4.1. That is possible to see that it references github.com/googleapis/gnostic/openapiv2. That means that if we switching to kustomize newer/or commit a2871181fe7e01b42741019c49d12d1ca24602a3, we have only a choice to use github.com/googleapis/gnostic >=0.4.1. k8s modules (k8s.io/apiextensions-apiserver, k8s.io/apimachinery, k8s.io/apiserver, k8s.io/client-go and k8s.io/kubectl) switched to github.com/googleapis/gnostic >=0.4.1 starting version 0.18.0 Airshipctl is currently using Cluster-Api 0.3.13 (v1alpha3). The latest from v1alpha3 is 0.3.16 (see branch release 0.3) Unfortunately the whole branch is compatible only with k8s components version 0.17.9 and if we switch to 0.18 it requires changes. There are 2 more branches: master and 0.4, that are compatible with k8s compontents versions 0.18+. But there are not released 0.4 branch. Since we want to stick with stable version, we should use 0.3 versions. So, here is dependency chain: if we switch to the latest kustomize components, we need to switch to kube-openapi that references gnostic >= 4.1 and that automatically switches to k8s components >=0.18.0, but these components are not compatible with the latest stable releases of Cluster-API. To make that switch possible I've created a special version of kube-openapi that works with gnostic 4.0: https://github.com/aodinokov/airshipctl_3rdparty/commit/7e53cf79bd173bab65b996a4e5769987806e9d17#diff-8ecbe9b28adc79fb26a4c43a7eb214fa923a24551a741669d1ba010101de6022 if we replace: k8s.io/kube-openapi => github.com/aodinokov/airshipctl_3rdparty/kube-openapi/95288971da7e v0.0.0-20210524214255-1b8a0f8bc487 it's possible to keep k8s components as is (0.17.4 and 0.17.9) and don't touch cluster-api as well. But there is another issue that is described here: https://github.com/kubernetes-sigs/kustomize/blob/1eb77a6cab06d5cdedf6b0e5ccbeb42a2e2010b4/cmd/depprobcheck/README.md k8s.io/cli-runtime@v0.20.4 and older depend on sigs.k8s.io/kustomize@v2.0.3+incompatible but if we're taking newer k8s.io/kube-openapi it can't be compiled. Here are the changes that have to be done https://github.com/aodinokov/airshipctl_3rdparty/commit/5ee7556e95f25866e33dfdb8e6efdacd4de3f152 + https://github.com/aodinokov/airshipctl_3rdparty/commit/1b8a0f8bc487d96e5a0b36eddcbbda88883ac3e7 This modified version should be referenced in the replacement: sigs.k8s.io/kustomize v2.0.3+incompatible => github.com/aodinokov/airshipctl_3rdparty/kustomize/a6f65144121d v0.0.0-20210524214255-1b8a0f8bc487 Going forward: Everything will work if we switch to the latest versions: e.g github.com/googleapis/gnostic 0.5.5 latest k8s.io/kube-openapi k8s components 0.21.0+ The problem is that it causes update of cluster-api to unstable versions: e.g.: https://review.opendev.org/c/airship/airshipctl/+/788582/29/go.mod It causes the issue for our case: [airshipctl] 2021/05/22 08:32:01 opendev.org/airship/airshipctl/pkg/clusterctl/client/client.go:104: Starting cluster-api initiation [airshipctl] 2021/05/22 08:32:01 opendev.org/airship/airshipctl/pkg/events/processor.go:60: Received error on event channel {unsupported management cluster server version: v1.18.19 - minimum required version is v1.19.1} Error events received on channel, errors are: [unsupported management cluster server version: v1.18.19 - minimum required version is v1.19.1] Here is that constant: https://github.com/kubernetes-sigs/cluster-api/blob/bfc6f80add5c21b8dc2b704951f42bc14708ebc4/cmd/clusterctl/client/cluster/client.go#L35 If we are creating a special version where we change this constant back, it still doesn't work with (See https://review.opendev.org/c/airship/airshipctl/+/788582/38): [airshipctl] 2021/05/23 08:16:38 opendev.org/airship/airshipctl/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/capi/v0.3.7 [airshipctl] 2021/05/23 08:16:38 opendev.org/airship/airshipctl/pkg/events/processor.go:60: Received error on event channel {failed to read "metadata.yaml" from the repository for provider "cluster-api": document filtered by selector [Group="clusterctl.cluster.x-k8s.io", Version="v1alpha3", Kind="Metadata"] found no documents} Error events received on channel, errors are: [failed to read "metadata.yaml" from the repository for provider "cluster-api": document filtered by selector [Group="clusterctl.cluster.x-k8s.io", Version="v1alpha3", Kind="Metadata"] found no documents] That version of cluster-api isn't released and may be unstable. That's why the simplest and easiest way of using latest kustomize - is with additional 3rd-party modified modules: https://github.com/aodinokov/airshipctl_3rdparty. We will need to create that repo in airshipctl.. or maybe use a directory in airshipctl, but it will make issues for linter. Once the new version of Cluster-Api is released that will support 0.21.0+ versions of k8s components, we'll need to switch to it. In addition it will be necessary to upgrade k8s that we deploy at least to v1.19.1 ``` Short-term solution: upgrade kustomize to 4.1.2 (this will be much easier) Long-term solution (preliminary): Cluster-api 0.4.x release is planned to June-July (may slip further though). Once it's release our plan will be: upgrade our gating to k8s at lease 1.19.1 (0.4 has that [dependency](https://github.com/kubernetes-sigs/cluster-api/blob/bfc6f80add5c21b8dc2b704951f42bc14708ebc4/cmd/clusterctl/client/cluster/client.go#L35)).And AFTER that try to upgrade to k8s modules in go.mod to 0.21.0. That should resoulve all dependency issues. ## Tuesday May 25, 2021 ### Clusterctl move & the CA The target cluster CA that is used for generating Dex certs is currently generated on the ephemeral cluster, then `clusterctl moved` over to target. It is created in the `default` NS, so that's where it winds up; can we create it in a more appropriate namespace, or is `default` special somehow? - `clusterctl move` moves from ephemeral NS X to target NS X - The KubeadmControlPlane that creates it is put in the default NS - clusterctl move can only move resources from a single NS at a time - `clusterctl move` has an option to change the target NS (don't need to use this) - We want to be consistent in naming between target & subclusters - E.g. `target-infra`, `lma-infra`, etc - Then, just put the resources for the target cluster in the ephemeral cluseter's `target-infra` NS, and they'll end up where they should go - Need issue: implement the above and make sure it works ### Enabling Physical Disks and Controllers parameters for RAID Configuration - Zainub/Mahnoor This is related to the demo represented by Noor, last year, related to RAID configurations in Metal3. https://review.opendev.org/c/airship/airshipctl/+/749043/6/manifests/function/hardwareprofile-example/hardwareprofile.yaml Support of RAID configurations for Baremetal Servers has been added. The link for this is: https://github.com/metal3-io/baremetal-operator/pull/292 Furthermore, we want to extend BMH for Disk names and RAID Controllers. While doing this, we came to know we need to extend BMH first, as Ironic API does not contain such information right now. https://github.com/metal3-io/baremetal-operator/issues/206 airship-discuss@lists.airshipit.org [Snia Swordfish](https://www.snia.org/forums/smi/swordfish) - Cross platform redfish extension for managing disk things - Does M3 or Ironic make use of this / plan to? - not sure, will follow up - Look at this for inspiration for how we want Airship manifests to look How will M3 handle bad disks when doing raid config? - We may want to be tolerant of some level of disk failure - E.g. a rule like "I need Xgb" - We'll need to understand how Ironic and Metal3 will handle this scenario, and then build on top how we want Airship to behave. Zainub will follow up with the M3 community. ## Thursday May 20, 2021 ### Generic Container timeout implementation details https://github.com/airshipit/airshipctl/issues/544 ### Function-specific catalogues (Matt, Sidney) Let's talk about our approach to function-specific catalogues. 1. What kinds of config do we want to target to normal patches, vs. config we want to put in catalogues? - Site-specific stuff should go in catalogues - Highly duplicated stuff should go in catalogues 2. Do we want to scope catalogues to individual functions, or broader categories? - Eg: we added a `networking-ha` catalogue for VIP configuration, rather than an `ingress` (function name) catalogue. I think we've added related data to that catalogue since. Dex example: **https://review.opendev.org/c/airship/treasuremap/+/791835/4/manifests/site/test-site/target/catalogues/dex-aio.yaml** Catalogue Conventions https://hackmd.io/HM-CNNIuRIm2MseaL523eA ### Using AGE as an alternative to PGP in SOPS (Matt, Alexey) Pronounced "ah-gay". A recently-merged option in SOPS, recommended by the SOPS community: "age is a simple, modern, and secure tool for encrypting files. It's recommended to use age over PGP, if possible." It has a more compact representation - encrypted data: ```age1yt3tfqlfrwdwx0z0ynwplcr6qxcxfaqycuprpmy89nr83ltx74tqdpszlw``` Public key: ```AGE-SECRET-KEY-1NJT5YCS2LWU4V4QAJQ6R4JNU7LXPDX602DZ9NUFANVU5GDTGUWCQ5T59M6``` Is it something we should consider adopting? * https://github.com/GoogleContainerTools/kpt-functions-catalog/pull/241 * https://github.com/mozilla/sops#encrypting-using-age * https://github.com/FiloSottile/age ## Tuesday May 18, 2021 ### Target-state PhasePlans We currently have phase plans named `phasePlan` and `iso`. We should true that up for the v2.1 release. * We'd discussed greenfield vs brownfield before. Do we want to call those something like `deploy` and `upgrade`? * An alternative which would avoid duplicate phase lists between the two phase groups would be `ephemeral` vs `deploy` or `target`, where ephemeral takes you up through clusterctl initing the target cluster (upgrade and greenfield look the same after that) * Vlad's work to change scripting into generic containers means that we have VM-specific phases too: `virsh-eject-cdrom-images`, `virsh-destroy-vms`. * We have subcluster-specific phases, but I believe we still need to create subcluster-specific PhasePlans for them Conclusion: create `deploy`, `deploy-virt`, and `manage-secrets` phasePlans at the type level for now. In the future we can add (as needed) things like `upgrade`, `update`, `release` (for generating qcows), `rotate-secrets`, etc etc. ### Can we archive the old content in this agenda? HackMD is getting really slow for me :) ## Thursday May 13, 2021 ### Discuss Design for [Make executors respect timeouts (airshipctl #533, v2.1)]( https://github.com/airshipit/airshipctl/issues/533) - Ony KubernetsApply respect timeout. - solution might need to be executor specific : - **Generic Container** - issue 1 - Generic container implementation, interface agreement discussed in the call. - issue 2 - Wait interface aggreement verification and enforcement will be a future Issue. - **ClusterCtl** - Issue on the clusterctl community. - [Clusterctl should wait for providers to be availabe.](https://github.com/kubernetes-sigs/cluster-api/issues/4474) ### Troubleshooting Guide HackMD Purpose: provide a more accessible, flexible & dynamic way of capturing troubleshooting information. https://hackmd.io/Nbc4XF6mQBmutMX_FEs51Q We can gauge usage & see if there's value in transposing this into our formal documentation suite or if this is sufficient. Secondary topic, do we have a comprehensive list of all errors produced by airshipctl? ## Tuesday May 11, 2021 ### Discuss Design for [Make executors respect timeouts (airshipctl #533, v2.1)]( https://github.com/airshipit/airshipctl/issues/533) ### Discuss next steps for KRM function gating/version management Below are notes from a smaller group meeting for a proposed approach for https://github.com/airshipit/airshipctl/issues/524). - Decided against moving KRM functions out of airshipctl and into their own repo(s). This reduces some complexity in version management. KRM image versions would be managed to match the Airship release version. Recommended approach: Task 1: - Update airshipctl to build KRM function images and tag locally with some local tag (not latest) - Update all references in KRM functions in manifests to reference local tagged images so that all gating will use current KRM function code - Update treasuremap the same way to use the local tags so that the versions of the KRM functions will match the airshipctl version specified by ${AIRSHIPCTL_REF}. - During release process, KRM functions would be pinned with same version as airshipctl and release script would update manifests with pinned version. - Treasuremap manifests would also be updated with pinned version of KRM functions during release process. - Downstream sites could follow similar process or pin to specific versions. Task 2: - Externalize the KRM function image versions such that versions are only specified in one place. See https://github.com/airshipit/airshipctl/issues/524#issuecomment-833004449 - https://review.opendev.org/c/airship/airshipctl/+/790507 ### Generate user guide command links (Sirisha Gopigiri) Related to https://github.com/airshipit/airshipctl/issues/281 PS: https://review.opendev.org/c/airship/airshipctl/+/789775 * PS address the issue? * Do we need to add custom help template to all commands? This would update the golden help output. ## Tuesday May 4, 2021 ### CLI documentation (Sirisha Gopigiri) Documentation structure in issue https://github.com/airshipit/airshipctl/issues/280 To render the documents properly in https://docs.airshipit.org/airshipctl/cli/airshipctl.html proposing two approaches which approach to take? * https://review.opendev.org/c/airship/airshipctl/+/784358/20 - Implemented custom code to generate the docs appropriate to docs.airshipit.org website * https://review.opendev.org/c/airship/airshipctl/+/789250 - Custom code + cobra.doc.GenReSTTreeCustom function used ### Generate CA certificate/Secret from a known authority (Sidney S.) The API server/OIDC authenticator plugin is configured with a CA certificate. When using a CA generated without being signed by a known authority I get the error **Unable to connect to the server: x509: certificate signed by unknown authority**. - What is the mechanism to auto generate a CA signed by a known authority and keep it secure? - How can the generated signed CA be secured, e.g., SOPS encryption? ### Splitting up the KRM toolbox image (Vlad/Matt) https://review.opendev.org/c/airship/images/+/786664 ## Thursday Apr 29, 2021 ### Lifecycle management of Airship KRM functions (Sean, Matt) The new `:v2` container tag is a moving tag, but it only moves when a git tag is pushed to the repo. * How frequently / under what conditions do we want to push a tag to update the `:v2` tag? * Probably more frequently than our "big" releases * How do we want to test/validate the updated container? * Test via a DNM patchset using `:latest` functions? * At the moment `:v2` is 22 days old * Base image changes for RT have broken `:latest` * How do we improve KRM function gating: https://github.com/airshipit/airshipctl/issues/524 * Can we externalize KRM function versions: https://github.com/airshipit/airshipctl/issues/419#issuecomment-829267689 ..... ### CLI documentation (Kostiantyn Kalynovskyi) Documentation structure in issue https://github.com/airshipit/airshipctl/issues/280 We have a patch set with a script that changes MarkDown format to ReST: * https://review.opendev.org/c/airship/airshipctl/+/784358/20 * I've put -2; please help understand if I am wrong here. * The scirpt inside is complex and has no tests: hard to support and understand * Why not use doc.GenReSTTree() function that generates rst documentation in a similar fassion as it was done before. ### Generate CA certificate/Secret from a known authority (Sidney S.) The API server/OIDC authenticator plugin is configured with a CA certificate. When using a CA generated without being signed by a known authority I get the error **Unable to connect to the server: x509: certificate signed by unknown authority**. What is the mechanism to auto generate a CA signed by a known authority and keep it secure? ### Continued pleas for assistance with Troubleshooting Guide - Andrew K Lots of opportunities to contribute! ## Tuesday Apr 27, 2021 ### Validate ecnryption/decrypt design of externally provided secrets (Alexey O.) * working patchset: https://review.opendev.org/c/airship/airshipctl/+/786286 * Discussion image https://go.gliffy.com/go/publish/13490574 * Dex/Ldap - will be the first user of that (Sidney) * It would be great to discuss and simplify that if possible * There may be additional requirements for encryption we didn't set before: we may want to be able to override generated secrets with manually set. In that case even this approach won't work and we instead/in addition should use encrypted SMP. The only issue with 'in addition' is - it will be one more call of SOPs for decryption. On another hand - encrypted SMP approach has an issue - it won't be possible to render without decryption. * Consider simplification on overrides (maybe get rid of them by default?) * Nice to have - move manifests/site/test-site/target/generator/results/decrypt-secrets/ to functions (functions/sops/gpg_decrypt) or type, so we don't copy it everywhere. ## Thursday Apr 15, 2021 ### Airship 2.0 Troubleshooting Guide Continued - Andrew K * WIP patchset - https://review.opendev.org/c/airship/docs/+/786062 * What do we want to target for v2.1? This will be evolutionary. * Collaborators needed * Related question: do we log what phase is being executed when it starts/ends? ### FQDN resolution - Andrii O. * Some software require fqdn resolution from hostname. /etc/hosts and systemd-resolved config need to be populated accordingly during the deployment. * Arvinder: * I see three possible approaches to doing this: 1. host-config operator: this is best when updates need to happen dynamically on existing K Nodes. 2. pre/post kubeadm hooks in KubeadmConfigSpec: you can write custom scripts that for example patch the /etc/hosts accordingly. For example, you can use KubeadmConfigSpec.Files to create one or more files in say the /etc/host.d directory and then during PreKubeadmCommand run something along the lines of `cat /etc/hosts.d/*.conf > /etc/hosts` 3. Metal3DataTemplate: the above KubeadmConfig approach applies the same configuration across all nodes in an KCP or MD. Metal3DataTemplate provides more flexibility by allowing some of the configuration to be Node specific. https://github.com/metal3-io/metal3-docs/blob/master/design/metadata-handling.md :::warning We are already using #3 for networking etc, its unclear to me that this helps with the original question. FQDN/Systemd changes etc. ::: ## Tuesday Apr 13, 2021 ### Hostconfig-operator integration (Sreejith P) While integrating HCO with treasuremap, found that we need to annotate the secret on to nodes and we also need to have a specific label. what would be the best way to annotate nodes. Also would it be best to add a mechanism to override the default labels in HCO via manifests. - Metal3 Feature for Label Cascading: https://github.com/airshipit/airshipctl/issues/377 ### Replacing into multiple targets (Reddy / Matt) There are use cases where we need to ReplacementTransform the same source data into multiple target paths -- e.g. replacing an IP address into many network policy rules. It would be helpful for the RT to support this natively. Some options, 1) add a `targets` list as an alternative to `target` * Lets open an ISSUE to support this. Improve Transformer to support Target as Slice : https://github.com/airshipit/airshipctl/blob/7998615a7b5847c367c29874641c8422157ebb52/pkg/document/plugin/replacement/transformer.go#L77 2) allow for global substring replacements across a document, in addition to the path-based substring replacement we have ISSUE : Open thi sfor a future priority. specify a patteern that does not hae a specific target. ### PTG later this month * Thurs, Apr 22 1300 UTC (8:00 AM CDT) - 1700 UTC (12:00 PM CDT) * Fri, Apr 23 1300 UTC (8:00 AM CDT) - 1700 UTC (12:00 PM CDT) * Open agenda: https://etherpad.opendev.org/p/xena-ptg-airship * Registration (free): https://april2021-ptg.eventbrite.com ### Need to discuss document pull command behavior (Kozhukalov) Related issues * https://github.com/airshipit/airshipctl/issues/416 * https://github.com/airshipit/airshipctl/issues/417 Patch is ready for review (probably needs rebasing) * https://review.opendev.org/c/airship/airshipctl/+/767571 Patch implements two things * Default target path is ${HOME}/.airship/<manifest_name> (i.e. ${HOME}/.airship/default) * If target path is not defined in the config for a manifest, then the current directory is used to pull the manifests repo ## Thursday Apr 8, 2021 ### Discuss the document-validation solution for airshipctl (Ruslan A.) https://hackmd.io/t2mxDiB3TdGXI8B6gDtA-Q ### Discuss "dex-aio" Implementation Short-/Long-term (Sidney S.) Approach to discuss described in https://hackmd.io/bdPFHBBSQy-IrpPe1U9itg ### Align on approach to troubleshooting guide (Andrew K) Use the life cycle states as a high level framework: - Initialize State - Prepare State - Physical Validation - Bootstrap State - Target Cluster Lifecycle - Target Workload Lifecycle - Sub-clusters (separate state or combine with Target?) - Day 2 Operations Proposed approach would be to ~~to list the phases/steps within each higher level lifecycle state & then~~ reference the relevant troubleshooting areas listed below within the life cycle states. Generally speaking here's what you need to look at within a phase (base on executor, do x, y, z). Troubleshooting areas: - Manifests: - Documents, - Kustomize, - Kustomize-KRM-Plugins - Our troubleshooting for Replacememt, Sops, - Running phases: How to debug a failed phase, where to start, which logs to read > Focus from the phase perspective. - Identify Phase - UUnderstand Phase.yaml - Identify Executor - Understand Executor.yaml - Given. theexecutor type ; - Different guidance - is it generic - which one, - sops, gpg,kubeval, image builder, etc - is it k8s - is it clusterctl - Cluster-API & Kubernetes Deployment: grouped together as the k8s deployment is done by Cluster-API - Proxy settings - Networking: is this too broad/complex/specific to individual use cases? - Helm Charts: Helm Operator & Helm Chart Collator debugging - Image Builder: base image generation debugging, ISO/QCOW generation & application debugging - - Host Config Operator (may be a future topic) - Sub-Clusters? - Services/Applications - i.e. CEPH - DEX - LMA stuff - ... Point to their documentation Assuming our other documentation will provide details on what each phase does. Would it make sense to incorporate troubleshooting into the deployment guide so you have a one stop shop or keep it separate so it's not cluttering up the deployment guide? We created an issue for this quite awhile back [#328](https://github.com/airshipit/airshipctl/issues/328). It references this TM v1 debug report script as a potential starting point. Is this still valid? https://github.com/airshipit/treasuremap/blob/master/tools/gate/debug-report.sh Next Steps: - Create WIP Patchset with documentation framework - Identify SMEs for each troubleshooting area who can help contribute - Try to get a basic framework & some initial content by the EOM for v2.1 & continue to build on it ## Tuesday Apr 6, 2021 ### Discuss the document-validation solution for airshipctl (Ruslan A.) Review the followings commits https://review.opendev.org/q/topic:%22add-validation-phases%22+(status:open) ### Discuss "dex-aio" certifcate generated by Cert-Manager (Sidney S.) Approach to discuss described in https://hackmd.io/bdPFHBBSQy-IrpPe1U9itg ## Thursday Apr 1, 2021 ### Discuss the rook-ceph cluster implementation (Vladimir S.) Review the initial commit https://review.opendev.org/c/airship/treasuremap/+/784184 , Discuss rook-ceph components which should be deployed by default, Discuss the further downstream/WHARF work - Set failure domain to host by default - ## Place for scripts such as waiting, that are currently in tools/deployment. (KKalynovskyi) We have new pattern of waiting and adding new scripts to gate/deployments: https://review.opendev.org/c/airship/airshipctl/+/782520 as example, we placed script here, but now its test-site specific, we need a place to be shared between every site: https://github.com/airshipit/airshipctl/tree/master/manifests/site/test-site/phases/helpers

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.