Try   HackMD

Secret Generate and Regenerate

Table of Contents

  1. Problem statement
  2. Proposal
  3. Usecases

Problem statement

airshipctl secret encrypt and airsipctl secret decrypt commands are used to encrypt/decrypt the secrets already available. Adding airshipctl secret generate helps in generating passphrases and certificates as kuberenetes secrets. It would also encrypt the secrets. With --regenerate-all the secrets which are marked as regenerate true can be regenerated again. The operator can then use airshipctl phase run to apply the newly generated secrets.

The passphrase based secrets that are generated would be imported to another existing CRD object using Replacementtransformer. In this case the operator would define a replacement transformer object to replace the generated passphrase secret in the targetted CRD object.

Proposed Commands

This below command would allow operators to generate all passphrases for a new site, or to generate only newly added passphrase catalogue entries for that site without regenerating existing passphrases.
airshipctl secret generate passphrase

To regenerate passphrases for a site the operator can use the below command. The passphrases which are flagged disable_regenerate: true will not be recreated again using the below comamnd.
airshipctl secret generate passphrase --regenerate-all

The below command would allow operators to generate the CA certificates for a new site, or to generate only newly added CA certificate catalogue entries for that site without regenerating existing certificates.
airshipctl secret generate certificate

To regenerate CA certificates for a site the operator can use the below command. Certificates which are flagged as disable_regenerate: true in the catalogue document will not be recreated again.
airshipctl secret generate certificate --regenerate-all

Proposal

To generate the required secret passphrases/certificates we would propose a secretGenerator catalog yaml file which contains the details about what secret passphrases/certificates needs to be created. This catalog file also holds the information about the regeneration, encryption flags and other details.

Passphrase Command

Introduction

The command will generate kubernetes secrets with passwords/uuids as data. These kubernetes kind: Secrets yaml files data will be used by the operator to define a replacement transformer which would replace the data in the desired kubernetes object. Please note that these secrets are used for replacement purpose and will not be created on the kubernetes cluster itself.

To generate the passphrases we would use the existing airshipctl secret generate masterpassphrase command. The command will be enhanced to take regex parameter as input so that the generated passphrase matches the regex.

By default the disable_encrypt: false flag is set to false, so the kubernetes secrets generated will be encrypted by default. Under the hood we would use the airshipctl secret encrypt command to encrypt the secrets. So the pre-requisite would be to set the gpg keys in the $HOME/.airship/config file using airshipctl config set-encryption command.

Whenever airshipctl secret generate passphrase or airshipctl secret generate passphrase --regenerate-all command is triggered, the current context's metadata yaml file is read for catalog yaml file details.

Example of metadata.yaml file

phase:   # this exists today in metadata.yaml
    path: manifests/phase
passphrases:
    - path: airshipctl/manifests/site/test-site/generation-source/openstack.yaml # Catalogue file location
      secrets_path: airshipctl/manifests/site/test-site/generation-secrets # Generated secrets and kustomization.yaml file will be placed here
    - path: airshipctl/manifests/site/test-site/generation-source/bmc.yaml # Another example
      secrets_path: airshipctl/manifests/site/test-site/generation-passphrases

Here is an example of the catalog yaml and the expected kubernetes secret outputs once the command has executed successfully.

Catalog File with passphrase details

Location:
$HOME/airshipctl/manifests/site/test-site/generation-source/openstack.yaml

Contents:

apiVersion: secrets.airshipit.org/v1alpha1
kind: secretGenerator
metadata:
  name: secret-catalog
data:
  passphrases:
    # This will generate a Secret named "helm-release-password" in:
    #     <secrets_path>/
    - name: 'helm-release-password'
      # disable_encrypt: false  # by default
      disable_regenerate: true # default is false
      data:
        - key: password
        - key: username
    - name: 'helm-release-test-password'
      disable_encrypt: false
      data: 
        - key: password
          regex: "^Xy[a-f][6-9][@#$%&]*$" # This regex will be used when creating the passphrase
    - name: 'helm-release-testing-password'
      disable_encrypt: false
      data: 
        - key: password
          value: "airship" # This value would be base64 encoded in the secret
    - name: 'helm-release-example-password'
      stringData: 
        - key: password
          value: "testing" # This value will not be encoded will just be encrypted directly

Sequence Diagram

Generate and Regenerate command

Output Secrets

Once the command gets executed it would generates the secret yaml files in the secrets_path specified and also a kustomization file invoking the generated secrets will be created.

The expectation is that the kustomization file in the corresponding phase will be edited by the operator to add this secrets_path as the resource in the kustomization file. The transformer section also needs to be changed by operator to call replacement folder if not present. And the replacement rules should also be created by the operator in the replacement folder.

If the secrets are encrypted then the sops plugin should be invoked before the replacement transformer which would decrypt the secrets. Please note the same gpg keys should be used for decryption which were set in the $HOME/.airship/config when the generate command was triggered.

The ReplacementTransformer will be enhanced to decode the secrets before they get replaced. The decoding feature will remove the base64 encoding applied on the secrets.

The required secret yaml files will be created in the secrets_path defined.

Please note that the airshipit.org/deploy-k8s: "false" annotation will be specified for all the secrets which means the secret will just be used to replace some fields in the target reference object (here HelmRelease) and will not be actually created in the kubernetes cluster itself.

Kustomization File

Location:
$HOME/airshipctl/manifests/site/test-site/generation-secrets/kustomization.yaml

Contents:

resources:
  - helm-release-password.yaml
  - helm-release-test-password.yaml
Secret YAML files

Sample secret yaml files that would be generated using the command.

  • helm-release-password

    Location: $HOME/airshipctl/manifests/site/test-site/generation-secrets/helm-release-password.yaml

    Contents:

    ​​​​apiVersion: v1
​​​​kind: Secret
​​​​metadata:
​​​​name: helm-release-password
​​​​annotations:
​​​​  airshipit.org/encrypt: "true"
​​​​  airshipit.org/regenerate: "false"
​​​​labels:
​​​​  airshipit.org/deploy-k8s: "false"
​​​​data:
​​​​  username: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
​​​​  password: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
​​​​sops:
​​​​  kms: []
​​​​  gcp_kms: []
​​​​  azure_kv: []
​​​​  hc_vault: []
​​​​  lastmodified: '2020-09-09T09:08:15Z'
​​​​  mac: ENC[AES256_GCM,data:pYzWql1bZWQ6s1hBub3XhChA5OpQzK5r+yYO6glsl5fhgdO6Ck/gveoSP2utFNWXS786M3gcR6F1DjWqMZp6kJ5Qm3yO1oyx9Inxsd2DxqP8CFXhqo0xape9c1SIpRnc/TdMfsu+iyyK4smtak2mibHzc6mfOKt/FJ03lONis2I=,iv:YrNZ6qp1hyLKMQXHHSstOqKro2EAKwSPWKsuH1Hyshg=,tag:VFwddodT5UmsDoZ1qgnN3Q==,type:str]
​​​​  pgp:
​​​​    - created_at: '2020-09-09T09:08:15Z'
​​​​      enc: |
​​​​        -----BEGIN PGP MESSAGE-----
​​​​        hQEMA0Xswh95z+O7AQf/QrfvLfWtAitMmJULsaHPoETrDXpOQ4dK6CHntONguVVx
​​​​        JxO91dUhGtNd88OQAOEIvU7Ycag3+bEKDjQxadRY/ig0FgWIbv5OeH5mFkvNDooh
​​​​        ysH5KSJ5mQfFyIKwxk4kxbWWLfTEuX+MyxSBPW7jESJ8534nOhB2/znWe984rASX
​​​​        +Edg6J85oJhWhqg3/yTSSGyrpgcdF0CEpUhPYBl9FLRJQI23x+jqgFB2AQ40xNRu
​​​​        wUTNwumTk2hI9zJpwOhoY/XqJhGMplm5IgwTiY1rwzDOgbPK2WtKgfbm91kifEi0
​​​​        O7I3G9qOUNNnFkc1VBBRrXRnqW5QbktntIxDnpH51tJeAb2d4DLk/CuhvBWmTtLb
​​​​        /GsxhEt8IJvu9JHEbQKJLUG9oHwzRZewTyAYtKTJDzXLXZUYvozZ0j3aAyfW48O9
​​​​        XgyYuGjz9TPWFdYvNO8dntL2Kdq+xZfobflFgBiZmg==
​​​​        =ZELQ
​​​​        -----END PGP MESSAGE-----
​​​​      fp: 681E3A89EB1DAFD36EB883120A73BB48E26694D8
​​​​  encrypted_regex: ^data
​​​​  version: 3.6.0

  • helm-release-test-password

    Location: $HOME/airshipctl/manifests/site/test-site/generation-secrets/helm-release-test-password.yaml

    Contents:

    ​​​​apiVersion: v1
​​​​kind: Secret
​​​​metadata:
​​​​name: helm-release-test-password
​​​​annotations:
​​​​  airshipit.org/encrypt: "false"
​​​​labels:
​​​​  airshipit.org/deploy-k8s: "false"
​​​​data:
​​​​  password: "Xyb8z#df"

  • helm-release-testing-password

    Location: $HOME/airshipctl/manifests/site/test-site/generation-secrets/helm-release-testing-password.yaml

    Contents:

    ​​​​apiVersion: v1
​​​​kind: Secret
​​​​metadata:
​​​​name: helm-release-testing-password
​​​​annotations:
​​​​  airshipit.org/encrypt: "false"
​​​​labels:
​​​​  airshipit.org/deploy-k8s: "false"
​​​​data:
​​​​  password: "YWlyc2hpcA=="

  • helm-release-example-password

    Location: $HOME/airshipctl/manifests/site/test-site/generation-secrets/helm-release-example-password.yaml

    Contents:

    ​​​​apiVersion: v1
​​​​kind: Secret
​​​​metadata:
​​​​name: helm-release-example-password
​​​​annotations:
​​​​  airshipit.org/encrypt: "true"
​​​​labels:
​​​​  airshipit.org/deploy-k8s: "false"
​​​​stringData:
​​​​   password: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
​​​​sops:
​​​​  kms: []
​​​​  gcp_kms: []
​​​​  azure_kv: []
​​​​  hc_vault: []
​​​​  lastmodified: '2020-09-09T09:08:15Z'
​​​​  mac: ENC[AES256_GCM,data:pYzWql1bZWQ6s1hBub3XhChA5OpQzK5r+yYO6glsl5fhgdO6Ck/gveoSP2utFNWXS786M3gcR6F1DjWqMZp6kJ5Qm3yO1oyx9Inxsd2DxqP8CFXhqo0xape9c1SIpRnc/TdMfsu+iyyK4smtak2mibHzc6mfOKt/FJ03lONis2I=,iv:YrNZ6qp1hyLKMQXHHSstOqKro2EAKwSPWKsuH1Hyshg=,tag:VFwddodT5UmsDoZ1qgnN3Q==,type:str]
​​​​  pgp:
​​​​    - created_at: '2020-09-09T09:08:15Z'
​​​​      enc: |
​​​​        -----BEGIN PGP MESSAGE-----
​​​​        hQEMA0Xswh95z+O7AQf/QrfvLfWtAitMmJULsaHPoETrDXpOQ4dK6CHntONguVVx
​​​​        JxO91dUhGtNd88OQAOEIvU7Ycag3+bEKDjQxadRY/ig0FgWIbv5OeH5mFkvNDooh
​​​​        ysH5KSJ5mQfFyIKwxk4kxbWWLfTEuX+MyxSBPW7jESJ8534nOhB2/znWe984rASX
​​​​        +Edg6J85oJhWhqg3/yTSSGyrpgcdF0CEpUhPYBl9FLRJQI23x+jqgFB2AQ40xNRu
​​​​        wUTNwumTk2hI9zJpwOhoY/XqJhGMplm5IgwTiY1rwzDOgbPK2WtKgfbm91kifEi0
​​​​        O7I3G9qOUNNnFkc1VBBRrXRnqW5QbktntIxDnpH51tJeAb2d4DLk/CuhvBWmTtLb
​​​​        /GsxhEt8IJvu9JHEbQKJLUG9oHwzRZewTyAYtKTJDzXLXZUYvozZ0j3aAyfW48O9
​​​​        XgyYuGjz9TPWFdYvNO8dntL2Kdq+xZfobflFgBiZmg==
​​​​        =ZELQ
​​​​        -----END PGP MESSAGE-----
​​​​      fp: 681E3A89EB1DAFD36EB883120A73BB48E26694D8
​​​​  encrypted_regex: ^data
​​​​  version: 3.6.0

Example changes that are expected by the user/operator:

For example:
Replacement rules should be created in a file like <function>/replacements/secrets.yaml in the replacements folder.

Kustomization file for a phase after editing would be as follows:

# kustomization.yaml for `workload` phase
resources:
  - ../../../../type/airship-core/target/workload
  - ../catalogues
  - ../generated-secrets # New line of config
transformers:
  - sops.yaml # New line of config, decrypt the generated Secrets first
  # Note: this line will exist already, and will pull in the new replacements
  - ../../../../type/airship-core/target/workload/replacements

Phase run Sequence Diagram

After secrets are generated below is the sequence of flows showing how the secrets would be applied.

Or once the secrets are generated the operator can store them in github/document manifest repository which can be pulled by other user and run the airshipctl phase run command directly.

Phase run command sequence

The below is flow of execution once the operator updates the kustomization.yaml file, add replacements and sops.yaml(for decryption).

Certificate Command

Introduction

The command will generate kubernetes tls secrets. These kubernetes kind: Secrets yaml files can be directly applied on the kubernetes cluster. Using this command the CA certificates for CAPI components(capi, capo, capd, capz, capm3, capg), kubernetes components and service account keypairs can be created.

Whenever airshipctl secret generate certificate or airshipctl secret generate certificate --regenerate-all command is triggered, the current context's metadata.yaml is searched for the catalogue information for certificate details.

By default the disable_encrypt: false flag is set to false, so the kubernetes secrets generated will be encrypted by default. Under the hood we would use the airshipctl secret encrypt command to encrypt the secrets. So the pre-requisite would be to set the gpg generated keys in the $HOME/.airship/config file using airshipctl config set-encryption command.

Here is an example of the catalog yaml and the expected kubernetes secret outputs once the certificate command has executed successfully.

Example of metadata.yaml file

phase:   # this exists today in metadata.yaml
    path: manifests/phase
certificates:
    - path: airshipctl/manifests/site/test-site/generation-source/certificate.yaml # Catalogue file location
      certificate_path: airshipctl/manifests/site/test-site/generation-certificate # Generated certificates and kustomization.yaml file will be placed here
    - path: airshipctl/manifests/site/test-site/generation-source/test.yaml
      certificate_path: airshipctl/manifests/site/test-site/generation-certificate-secrets # Another example
Catalog File with certificate details

Location:
$HOME/airshipctl/manifests/site/test-site/generation-source/certificate.yaml

Contents:

apiVersion: secrets.airshipit.org/v1alpha1
kind: secretGenerator
metadata:
  name: secret-certitificate-catalog
data:
  certificates:
    cluster_certificate_authorities:
      - description: 'capi ca certificate' # Certs generated using openssl commands
        document_name: capi_ca
        days: 365
        disable_encrypt: false #defaults to false
        common_name: capi_system
      - description: 'capd ca certificate' #   Certs generated using openssl commands
        document_name: capd_ca
        days: 365
        disable_encrypt: true
        common_name: capd_system
      - description: 'capm3 ca certificate'
        document_name: capm3_ca
        disable_regenerate: true # defaults to false
        tls.key: $HOME/tls.key
        tls.crt: $HOME/tls.crt
    kubernetes_certificate_authorities:
      - workloadCluster: dtc #Target cluster name
        days: 365 # Days of validity - default 365
        description: 'kubernetes ETCd, CA , Proxy certificates'
        common_name: kubernetescerts
        type: k8s_certs
        disable_encrypt : false # default false
      - workloadCluster: dtc
        days: 365
        description: 'kubernetes SA certificates'
        type: key_pairs
        tls.key: $HOME/tls.key
        tls.crt: $HOME/tls.crt

Sequence Diagram

Please note: airshipctl secret certificate –regenerate-all regenerates the CA certificate, and it is expected that if the regenerated CA certificates are applied to the existing cluster the certificates using the CA has also to be regenerated.

Generate and Regenerate command

Output Secrets

The expectation is that the command generates the secret yaml files in the certificate_path specified and a corresponding kustomization file created. It would be the responsiblity of the user to invoke this kustomization file from the corresponding phase run's kustomization file (like initinfra or workload or controlplane) that is of interest. And if the secrets are encrypted then the user should also invoke the sops plugin in kustomization file to perform the decrpytion of certificates before they get applied on the cluster. Please note the same gpg keys should be used for decryption which were set in the $HOME/.airship/config when the generate command was triggered.

It will be the operator responsibility to invoke the certificates kustomization as resource in the corresponding phase kustomization file.

Kustomization File

Location:
$HOME/airshipctl/manifests/site/test-site/generation-certificate/kustomization.yaml

Contents:

resources:
  - capi-ca.yaml
  - capd-ca.yaml
  - capm3-ca.yaml
  - test-site-ca.yaml
  - test-site-proxy.yaml
  - test-site-etcd.yaml
  - test-site-sa.yaml
CAPI CA Secret

Certificate secret generated with both key and cert details encrypted.

Location:
$HOME/airshipctl/manifests/site/test-site/generation-certificate/capi-ca.yaml

Contents:

apiVersion: v1
kind: Secret
metadata:
  annotations:
        airshipit.org/encrypt: "true"
   name: capi-ca
type: kubernetes.io/tls
data:
 tls.crt: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
 tls.key: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    lastmodified: '2020-09-09T09:08:15Z'
    mac: ENC[AES256_GCM,data:pYzWql1bZWQ6s1hBub3XhChA5OpQzK5r+yYO6glsl5fhgdO6Ck/gveoSP2utFNWXS786M3gcR6F1DjWqMZp6kJ5Qm3yO1oyx9Inxsd2DxqP8CFXhqo0xape9c1SIpRnc/TdMfsu+iyyK4smtak2mibHzc6mfOKt/FJ03lONis2I=,iv:YrNZ6qp1hyLKMQXHHSstOqKro2EAKwSPWKsuH1Hyshg=,tag:VFwddodT5UmsDoZ1qgnN3Q==,type:str]
    pgp:
    -   created_at: '2020-09-09T09:08:15Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQEMA0Xswh95z+O7AQf/QrfvLfWtAitMmJULsaHPoETrDXpOQ4dK6CHntONguVVx
            JxO91dUhGtNd88OQAOEIvU7Ycag3+bEKDjQxadRY/ig0FgWIbv5OeH5mFkvNDooh
            ysH5KSJ5mQfFyIKwxk4kxbWWLfTEuX+MyxSBPW7jESJ8534nOhB2/znWe984rASX
            +Edg6J85oJhWhqg3/yTSSGyrpgcdF0CEpUhPYBl9FLRJQI23x+jqgFB2AQ40xNRu
            wUTNwumTk2hI9zJpwOhoY/XqJhGMplm5IgwTiY1rwzDOgbPK2WtKgfbm91kifEi0
            O7I3G9qOUNNnFkc1VBBRrXRnqW5QbktntIxDnpH51tJeAb2d4DLk/CuhvBWmTtLb
            /GsxhEt8IJvu9JHEbQKJLUG9oHwzRZewTyAYtKTJDzXLXZUYvozZ0j3aAyfW48O9
            XgyYuGjz9TPWFdYvNO8dntL2Kdq+xZfobflFgBiZmg==
            =ZELQ
            -----END PGP MESSAGE-----
        fp: 681E3A89EB1DAFD36EB883120A73BB48E26694D8
    encrypted_regex: ^data
    version: 3.6.0
CAPD CA Secret

Certificate secret generated with both key and cert details without any encryption, as encrypt is set to false in the certificate.yaml file.

Location:
$HOME/airshipctl/manifests/site/test-site/generation-certificate/capd-ca.yaml

Contents:

apiVersion: v1
kind: Secret
metadata:
   name: capd-ca
type: kubernetes.io/tls
data:
 tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURWRENDQWp5Z0F3SUJBZ0lVTUNwc09vRXhyRzdnRTVMOVJSamdnT01UOG53d0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09TM1ZpWlhKdVpYUmxjeUJCVUVrd0hoY05NakF3T1RFMU1ERXdORE0zV2hjTgpNekF3T1RFek1ERXdORE0zV2pBWk1SY3dGUVlEVlFRRERBNUxkV0psY201bGRHVnpJRUZRU1RDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUtBZFo0UWJHZmlLTExpTXNHcFJKS3d5ZkRGWVI5U0MKbGtVb3hlTU1BZVBkeVNNU0paTTlFMFBOaDM5TUtTVjNSZDRIZWt1eGdHK3J4em83WmcrZU1aY1hyNFk3ektQMwo1SW0vaERkMm1TYThsMEkxZTRwV3B0Z25vZjdvRWJpSXVIU2YxQmRhMU4wWm1EUUdtckxyQnFOZFE3c1BVenNWCllPejZVUFZlamNIeEFjMXBvMWZsQXYrWVNZejVXa28wRVRnTXZYRGtxT0hrWFc1WnhPcHBVbiszOVpvWTZMK3gKVmUwUHFQdHlmSVZ1M3dtcnZFNGd4SmxtWEk3dUxmdzZONHpwS2RuK0k0K1RJRWF5aE1EMWRRenNwQzRMM0IrcApYcHFPMWNWM2ZKMlBycS9mNU14SnIxWTVHUTZlQlZyTGVod1ZWTEhEMzF3ZWFpZ3UzeStyM3RVQ0F3RUFBYU9CCmt6Q0JrREFkQmdOVkhRNEVGZ1FVT1d5YTNFd2J5c25UUy9ZajFWTEtjMGh4aDRvd1ZBWURWUjBqQkUwd1M0QVUKT1d5YTNFd2J5c25UUy9ZajFWTEtjMGh4aDRxaEhhUWJNQmt4RnpBVkJnTlZCQU1NRGt0MVltVnlibVYwWlhNZwpRVkJKZ2hRd0ttdzZnVEdzYnVBVGt2MUZHT0NBNHhQeWZEQU1CZ05WSFJNRUJUQURBUUgvTUFzR0ExVWREd1FFCkF3SUJCakFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBTVp1U2tJbTdQdlA4MW5HSjlYOVZFOFVZTVdDSU5GMEEKYit1UURFaHRGc0dxdnZFZHhQcURUWUpwdlF1SUJlOVd0cmlWRzh0MENIL1NnZ0g2TlJod0wyYkJwMm5WaEFVVwphK3hZL1RpTmMzUEl5RHNFeEY3VHVENGJzaW1BQUJTZ2ZtbXRxV1dqajRyOStodS9vZ09jLzQyYk9JT0JWbHNkCi9VNzBiR3dZQjU5QXgvL2dIWVJmVDl3L3p0VHBvY2tzdEhhSjZsVDd5SFlqYUkzaU5EWnZNSnFRSWNxME4vTEMKcVBjWjBWQXBMUTZRUHRpMWpVSzBGM1VlZEF6TVc3ZFF4NkV3Qjd5UHo4NWdZS3ZJdWdyaStrc2YwbGMyeHVDRwpXTGg2YjFNWk9Cc1NZNkppVHpSUUpYdXNCRUdaTGN5VkRJSEU3Y0Q4NWhOQmZpdDAvejFmZlE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
 tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2QUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktZd2dnU2lBZ0VBQW9JQkFRQ2dIV2VFR3huNGlpeTQKakxCcVVTU3NNbnd4V0VmVWdwWkZLTVhqREFIajNja2pFaVdUUFJORHpZZC9UQ2tsZDBYZUIzcExzWUJ2cThjNgpPMllQbmpHWEY2K0dPOHlqOStTSnY0UTNkcGttdkpkQ05YdUtWcWJZSjZIKzZCRzRpTGgwbjlRWFd0VGRHWmcwCkJwcXk2d2FqWFVPN0QxTTdGV0RzK2xEMVhvM0I4UUhOYWFOWDVRTC9tRW1NK1ZwS05CRTRETDF3NUtqaDVGMXUKV2NUcWFWSi90L1dhR09pL3NWWHRENmo3Y255RmJ0OEpxN3hPSU1TWlpseU83aTM4T2plTTZTblovaU9Qa3lCRwpzb1RBOVhVTTdLUXVDOXdmcVY2YWp0WEZkM3lkajY2djMrVE1TYTlXT1JrT25nVmF5M29jRlZTeHc5OWNIbW9vCkx0OHZxOTdWQWdNQkFBRUNnZ0VBWjNRNFUySlRlSVNHK3NOa3BYMUNiY1M4L0FFbmdFYlVJMkdCNHY3NkphcEMKOE5jajBpdnZTNnI3OXFOV0hyQWZRNk9mUUZNelFuUkNhUHpDS0NzMXJZT1BWUE5FZVZtTm4vZFB6YXBpc0dYQQpjZll1bWFiOWJNTEc1L1k0cFB3cCtxamVtQ3lIUjBqblVBNUlYSHlCTUlMdFpXczBnd09BT2Y1TzJ3dTZHbW5CCnlDbk5FaFFjbmZUdnN0MVBINUlTeEZ0TjFtODlnQ2orNEhsdHkybUdOdUNEczZ2QnZyWmEyeEJRZUtTcTl1eXEKY1dsdDkrRzI2ZmZFRmw0UEwwWit2Y0wrVDZLd0I4NFl2ampUd25pNEMzb1BmV3lZeEpkSjF0U3NFK1lSM3NIOApEdmlobmMwak00c0Fxb3hrMk5UV2kvNHpUUkppdytRcGhyRnNJWFNvUVFLQmdRRFNRaU44djFEUFk4VzdJUkFXClp6MTN5SWdrRTJHZXUzc2tFMkhweDN6ZFdMZWdmazF6SW42eUtpN0RzVTZhV2xjQURpMm5mU3BVSCtZQ2M0dVkKcmlnZ0xFMG9iZHJLVEFGdm1YYjA1WFRhR3FtcFE4TGxiWmxMTXVycXp0aG5BWmN3M2NySmxMb25kNkZISW9icAp3QU50K2p5a3pTaGs1T3MwSi9XajZiL3prUUtCZ1FEQzhxTkdndWVIbGw5QVNKeHJmM2hTeGJXWFhhTnJqMTJGCkEwNHlnNTFkbHg1QksweUVsZ3VZQ3RtZW5mZWgySWw5U1E3Yng4Z0Z5OWdWeDI5d1Z4eFp2b0xMaU16WU16RGMKWU5UTW9ETEtITjA3dHZ6ZDVza3piWFA3cFUwc25SMDV2RkJ2SHZtTHhWVlkvVHd6ZTdpZDVlNGtISTRWRmUxWQpnUDBYV0ZITkJRS0JnQTRXWGxoU1hUQzRCNXlGRjVYWXJ2YWltZlNJMCthVnV5ZHNvUWZQMU43anZkSGtCSDV0ClZqM0xzN3hxMmRCZnN5cU95S0pMTVpYWFdVcmF3UVNtem90eFRHNGtCaCs5dmU3alFtUWdKNWNoYURLdUZwWFcKcFFtenpLZVUya3owZjFQSDJIbHZISlhlWHhEc0VFd0RFSGZDNTJOSFY2aUM2ZnRobmdTd2VhcnhBb0dBRVd0Ywo2NUFHNERhdmpDN3d5eW80dGl5MGJUSVF5Q3VuVDV0Y0FXZUJTRHVZbUhvbC9ETHNGa25oNkNwMVZpRGpLQzYvCkJTUjAydys3M3paUzN0YnAwWnNVVk51RWNrMGdzSkIyYzFKZE4zSWMwcGtuUHl6QURiaGFCTUpnZ3Z3SEFJR3oKTGpxMlVhYndXV05IWGRKUVRNdWUyOXN4VnZEK3BFbmlVNU93dTRFQ2dZQmptRkRMZEZuT0VtOGlOZEFNVFM0SQpXM0VnckgvQVhsZ0dIOTVJVElKZCtuTmM3QUJQcktYUGEzTTlPRytCTUlpVlJZZ2JMVitXWU1hbHUvZWFJeWFXClR3TGpxaW5nak8rQXhKWFJkT1M3NGpOamFSWGtmM0RWQ1F3WUd0T0xqdE1qRVhmTWdORnViQXZ4TFVOUmZFblIKa09nR0pWQkgwSyttcVVaNVFiVWUwZz09Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
CAPM3 CA Secret

Certificate secret generated based on the tls.key and tls.crt files specified in the secretGenerate catalog yaml and also encrypted.

Location:
$HOME/airshipctl/manifests/site/test-site/generation-certificate/capm3-ca.yaml

Contents:

apiVersion: v1
kind: Secret
metadata:
  annotations:
        airshipit.org/encrypt: "true"
   name: capm3-ca
type: kubernetes.io/tls
data:
 tls.crt: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
 tls.key: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    lastmodified: '2020-09-09T09:08:15Z'
    mac: ENC[AES256_GCM,data:pYzWql1bZWQ6s1hBub3XhChA5OpQzK5r+yYO6glsl5fhgdO6Ck/gveoSP2utFNWXS786M3gcR6F1DjWqMZp6kJ5Qm3yO1oyx9Inxsd2DxqP8CFXhqo0xape9c1SIpRnc/TdMfsu+iyyK4smtak2mibHzc6mfOKt/FJ03lONis2I=,iv:YrNZ6qp1hyLKMQXHHSstOqKro2EAKwSPWKsuH1Hyshg=,tag:VFwddodT5UmsDoZ1qgnN3Q==,type:str]
    pgp:
    -   created_at: '2020-09-09T09:08:15Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQEMA0Xswh95z+O7AQf/QrfvLfWtAitMmJULsaHPoETrDXpOQ4dK6CHntONguVVx
            JxO91dUhGtNd88OQAOEIvU7Ycag3+bEKDjQxadRY/ig0FgWIbv5OeH5mFkvNDooh
            ysH5KSJ5mQfFyIKwxk4kxbWWLfTEuX+MyxSBPW7jESJ8534nOhB2/znWe984rASX
            +Edg6J85oJhWhqg3/yTSSGyrpgcdF0CEpUhPYBl9FLRJQI23x+jqgFB2AQ40xNRu
            wUTNwumTk2hI9zJpwOhoY/XqJhGMplm5IgwTiY1rwzDOgbPK2WtKgfbm91kifEi0
            O7I3G9qOUNNnFkc1VBBRrXRnqW5QbktntIxDnpH51tJeAb2d4DLk/CuhvBWmTtLb
            /GsxhEt8IJvu9JHEbQKJLUG9oHwzRZewTyAYtKTJDzXLXZUYvozZ0j3aAyfW48O9
            XgyYuGjz9TPWFdYvNO8dntL2Kdq+xZfobflFgBiZmg==
            =ZELQ
            -----END PGP MESSAGE-----
        fp: 681E3A89EB1DAFD36EB883120A73BB48E26694D8
    encrypted_regex: ^data
    version: 3.6.0
k8s Certificate Secret

Certificate secret generated with both key and cert details encrypted. There are 3 secrets created for the workload cluster these are the ca, proxy and etcd secrets. The names of the secrets are preffixed by the target workload cluster name.

Location:
$HOME/airshipctl/manifests/site/test-site/generation-certificate

  1. CA File:

    Location:
    $HOME/airshipctl/manifests/site/test-site/generation-certificate/test-site-ca.yaml

    Contents:

    ​​​​apiVersion: v1
​​​​kind: Secret
​​​​metadata:
​​​​   name: test-site-ca
​​​​   annotations:
​​​​        airshipit.org/encrypt: "true"
​​​​type: kubernetes.io/tls
​​​​data:
​​​​ tls.crt:     ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
​​​​ tls.key: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
​​​​sops:
​​​​    kms: []
​​​​    gcp_kms: []
​​​​    azure_kv: []
​​​​    hc_vault: []
​​​​    lastmodified: '2020-09-09T09:08:15Z'
​​​​    mac: ENC[AES256_GCM,data:pYzWql1bZWQ6s1hBub3XhChA5OpQzK5r+yYO6glsl5fhgdO6Ck/gveoSP2utFNWXS786M3gcR6F1DjWqMZp6kJ5Qm3yO1oyx9Inxsd2DxqP8CFXhqo0xape9c1SIpRnc/TdMfsu+iyyK4smtak2mibHzc6mfOKt/FJ03lONis2I=,iv:YrNZ6qp1hyLKMQXHHSstOqKro2EAKwSPWKsuH1Hyshg=,tag:VFwddodT5UmsDoZ1qgnN3Q==,type:str]
​​​​    pgp:
​​​​    -   created_at: '2020-09-09T09:08:15Z'
​​​​        enc: |
​​​​            -----BEGIN PGP MESSAGE-----

​​​​        hQEMA0Xswh95z+O7AQf/QrfvLfWtAitMmJULsaHPoETrDXpOQ4dK6CHntONguVVx
​​​​        JxO91dUhGtNd88OQAOEIvU7Ycag3+bEKDjQxadRY/ig0FgWIbv5OeH5mFkvNDooh
​​​​        ysH5KSJ5mQfFyIKwxk4kxbWWLfTEuX+MyxSBPW7jESJ8534nOhB2/znWe984rASX
​​​​        +Edg6J85oJhWhqg3/yTSSGyrpgcdF0CEpUhPYBl9FLRJQI23x+jqgFB2AQ40xNRu
​​​​        wUTNwumTk2hI9zJpwOhoY/XqJhGMplm5IgwTiY1rwzDOgbPK2WtKgfbm91kifEi0
​​​​        O7I3G9qOUNNnFkc1VBBRrXRnqW5QbktntIxDnpH51tJeAb2d4DLk/CuhvBWmTtLb
​​​​        /GsxhEt8IJvu9JHEbQKJLUG9oHwzRZewTyAYtKTJDzXLXZUYvozZ0j3aAyfW48O9
​​​​        XgyYuGjz9TPWFdYvNO8dntL2Kdq+xZfobflFgBiZmg==
​​​​        =ZELQ
​​​​        -----END PGP MESSAGE-----
​​​​        fp: 681E3A89EB1DAFD36EB883120A73BB48E26694D8
​​​​    encrypted_regex: ^data
​​​​    version: 3.6.0

  2. Proxy File:

    Location:
    $HOME/airshipctl/manifests/site/test-site/generation-certificate/test-site-proxy.yaml

    Contents:

    ​​​​apiVersion: v1
​​​​kind: Secret
​​​​metadata:
​​​​   name: test-site-proxy
​​​​   annotations:
​​​​        airshipit.org/encrypt: "true"
​​​​type: kubernetes.io/tls
​​​​data:
​​​​ tls.crt: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
​​​​ tls.key: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
​​​​sops:
​​​​    kms: []
​​​​    gcp_kms: []
​​​​    azure_kv: []
​​​​    hc_vault: []
​​​​    lastmodified: '2020-09-09T09:08:15Z'
​​​​    mac: ENC[AES256_GCM,data:pYzWql1bZWQ6s1hBub3XhChA5OpQzK5r+yYO6glsl5fhgdO6Ck/gveoSP2utFNWXS786M3gcR6F1DjWqMZp6kJ5Qm3yO1oyx9Inxsd2DxqP8CFXhqo0xape9c1SIpRnc/TdMfsu+iyyK4smtak2mibHzc6mfOKt/FJ03lONis2I=,iv:YrNZ6qp1hyLKMQXHHSstOqKro2EAKwSPWKsuH1Hyshg=,tag:VFwddodT5UmsDoZ1qgnN3Q==,type:str]
​​​​    pgp:
​​​​    -   created_at: '2020-09-09T09:08:15Z'
​​​​        enc: |
​​​​            -----BEGIN PGP MESSAGE-----

​​​​        hQEMA0Xswh95z+O7AQf/QrfvLfWtAitMmJULsaHPoETrDXpOQ4dK6CHntONguVVx
​​​​        JxO91dUhGtNd88OQAOEIvU7Ycag3+bEKDjQxadRY/ig0FgWIbv5OeH5mFkvNDooh
​​​​        ysH5KSJ5mQfFyIKwxk4kxbWWLfTEuX+MyxSBPW7jESJ8534nOhB2/znWe984rASX
​​​​        +Edg6J85oJhWhqg3/yTSSGyrpgcdF0CEpUhPYBl9FLRJQI23x+jqgFB2AQ40xNRu
​​​​        wUTNwumTk2hI9zJpwOhoY/XqJhGMplm5IgwTiY1rwzDOgbPK2WtKgfbm91kifEi0
​​​​        O7I3G9qOUNNnFkc1VBBRrXRnqW5QbktntIxDnpH51tJeAb2d4DLk/CuhvBWmTtLb
​​​​        /GsxhEt8IJvu9JHEbQKJLUG9oHwzRZewTyAYtKTJDzXLXZUYvozZ0j3aAyfW48O9
​​​​        XgyYuGjz9TPWFdYvNO8dntL2Kdq+xZfobflFgBiZmg==
​​​​        =ZELQ
​​​​        -----END PGP MESSAGE-----
​​​​        fp: 681E3A89EB1DAFD36EB883120A73BB48E26694D8
​​​​    encrypted_regex: ^data
​​​​    version: 3.6.0

  3. Etcd File

    Location:
    $HOME/airshipctl/manifests/site/test-site/generation-certificate/test-site-etcd.yaml

    Contents:

    ​​​​apiVersion: v1
​​​​kind: Secret
​​​​metadata:
​​​​   name: test-site-etcd
​​​​   annotations:
​​​​        airshipit.org/encrypt: "true"
​​​​type: kubernetes.io/tls
​​​​data:
​​​​ tls.crt: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
​​​​ tls.key: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
​​​​sops:
​​​​    kms: []
​​​​    gcp_kms: []
​​​​    azure_kv: []
​​​​    hc_vault: []
​​​​    lastmodified: '2020-09-09T09:08:15Z'
​​​​    mac: ENC[AES256_GCM,data:pYzWql1bZWQ6s1hBub3XhChA5OpQzK5r+yYO6glsl5fhgdO6Ck/gveoSP2utFNWXS786M3gcR6F1DjWqMZp6kJ5Qm3yO1oyx9Inxsd2DxqP8CFXhqo0xape9c1SIpRnc/TdMfsu+iyyK4smtak2mibHzc6mfOKt/FJ03lONis2I=,iv:YrNZ6qp1hyLKMQXHHSstOqKro2EAKwSPWKsuH1Hyshg=,tag:VFwddodT5UmsDoZ1qgnN3Q==,type:str]
​​​​    pgp:
​​​​    -   created_at: '2020-09-09T09:08:15Z'
​​​​        enc: |
​​​​        -----BEGIN PGP MESSAGE-----

​​​​        hQEMA0Xswh95z+O7AQf/QrfvLfWtAitMmJULsaHPoETrDXpOQ4dK6CHntONguVVx
​​​​        JxO91dUhGtNd88OQAOEIvU7Ycag3+bEKDjQxadRY/ig0FgWIbv5OeH5mFkvNDooh
​​​​        ysH5KSJ5mQfFyIKwxk4kxbWWLfTEuX+MyxSBPW7jESJ8534nOhB2/znWe984rASX
​​​​        +Edg6J85oJhWhqg3/yTSSGyrpgcdF0CEpUhPYBl9FLRJQI23x+jqgFB2AQ40xNRu
​​​​        wUTNwumTk2hI9zJpwOhoY/XqJhGMplm5IgwTiY1rwzDOgbPK2WtKgfbm91kifEi0
​​​​        O7I3G9qOUNNnFkc1VBBRrXRnqW5QbktntIxDnpH51tJeAb2d4DLk/CuhvBWmTtLb
​​​​        /GsxhEt8IJvu9JHEbQKJLUG9oHwzRZewTyAYtKTJDzXLXZUYvozZ0j3aAyfW48O9
​​​​        XgyYuGjz9TPWFdYvNO8dntL2Kdq+xZfobflFgBiZmg==
​​​​        =ZELQ
​​​​        -----END PGP MESSAGE-----
​​​​        fp: 681E3A89EB1DAFD36EB883120A73BB48E26694D8
​​​​    encrypted_regex: ^data
​​​​    version: 3.6.0
SA CA Secret Certificate secret generated with both public and private keys encrypted. This service account with public and private keys is used by the kubernetes controller manager.

Location:
$HOME/airshipctl/manifests/site/test-site/generation-certificate/test-site-sa.yaml

Contents:

apiVersion: v1
kind: Secret
metadata:
   name: test-site-sa
  annotations:
        airshipit.org/encrypt: "true"
type: kubernetes.io/tls
data:
 tls.crt: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
 tls.key: ENC[AES256_GCM,data:UA9BH2/Lhqsuekfe2VEFsw==,iv:pA1LkhM9O1WUfyB6f4MqW+ywDQrVeZOUmD8DNsK9gq0=,tag:9DVSISNEgibN3qs3STTN4w==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    lastmodified: '2020-09-09T09:08:15Z'
    mac: ENC[AES256_GCM,data:pYzWql1bZWQ6s1hBub3XhChA5OpQzK5r+yYO6glsl5fhgdO6Ck/gveoSP2utFNWXS786M3gcR6F1DjWqMZp6kJ5Qm3yO1oyx9Inxsd2DxqP8CFXhqo0xape9c1SIpRnc/TdMfsu+iyyK4smtak2mibHzc6mfOKt/FJ03lONis2I=,iv:YrNZ6qp1hyLKMQXHHSstOqKro2EAKwSPWKsuH1Hyshg=,tag:VFwddodT5UmsDoZ1qgnN3Q==,type:str]
    pgp:
    -   created_at: '2020-09-09T09:08:15Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQEMA0Xswh95z+O7AQf/QrfvLfWtAitMmJULsaHPoETrDXpOQ4dK6CHntONguVVx
            JxO91dUhGtNd88OQAOEIvU7Ycag3+bEKDjQxadRY/ig0FgWIbv5OeH5mFkvNDooh
            ysH5KSJ5mQfFyIKwxk4kxbWWLfTEuX+MyxSBPW7jESJ8534nOhB2/znWe984rASX
            +Edg6J85oJhWhqg3/yTSSGyrpgcdF0CEpUhPYBl9FLRJQI23x+jqgFB2AQ40xNRu
            wUTNwumTk2hI9zJpwOhoY/XqJhGMplm5IgwTiY1rwzDOgbPK2WtKgfbm91kifEi0
            O7I3G9qOUNNnFkc1VBBRrXRnqW5QbktntIxDnpH51tJeAb2d4DLk/CuhvBWmTtLb
            /GsxhEt8IJvu9JHEbQKJLUG9oHwzRZewTyAYtKTJDzXLXZUYvozZ0j3aAyfW48O9
            XgyYuGjz9TPWFdYvNO8dntL2Kdq+xZfobflFgBiZmg==
            =ZELQ
            -----END PGP MESSAGE-----
        fp: 681E3A89EB1DAFD36EB883120A73BB48E26694D8
    encrypted_regex: ^data
    version: 3.6.0

Phase run Sequence Diagram

After certificate secrets are generated below is the sequence of flows showing how the certificate secrets would be applied to the cluster.

Please note: If the CA is regenrated and applied on the cluster, it would be the responsiblity of the operator to rotate the certificates as well.

  1. For Management: Forcebily rotate certificates
  2. For Workload: Rotate Machine templates and KCP
Phase run sequence

The below is flow of execution once the operator updates the kustomization.yaml file. And executes airshipctl phase run command.

Usecases

  1. Secret Generate Certificate Usecase: This can be used to generate external certificates for capi, capck, capbk and capz/capd/capo/capm3/capg components. It also facilitates creation of certificate secrets for the kubernetes components in the target workload cluster.
  2. Secret Generate Passphrases Usecase: This command can be used to generate generic secrets in the kubernetes manifest files, these can be secrets related to the applications that would be deployed on the kubernetes cluster.

Changes to existing code

  1. Enchance airshipctl secret generate masterpassphrase command to accept regex, to generate passphrases based on regex specified.
  2. Add decoding functionality to the ReplacementTransformer plugin so that the secretes would be decoded before they get replaced.

Some Known conflicts

  1. If the secrets are manually created by the operator or if already exists in the secrets_path or certificate_path location when the corresponding command is triggered, those secrets will not be overridden. If the commands are executed will --regenerate-all flag and has regenerate: true only then the secrets will be over-written.
  2. In case of certificates if the command is triggered with --regenerate-all flag and the secret source certificates are not present tls.key and tls.crt then previously generated certificates will be retained.
  3. If the same certificate secret is created in some other location in the mainfest file by the operator manually, the phase run will fail with conflicts.
Last changed by 
 
sirishagopigiri
8
365
Published on HackMD