# SIT OPEN CLUSTER PROVISION STEPS INSTRUCTION

## 1.CI CODE

1.1 Add new environment: tkg-open-sit on conf.json

1.2 Add new domain name: open-sit.mcp.mli.com.corp in helm\values\mcp\m2\cicd\tkg-core-sit\environment\k8s\values_env.yaml

1.3 Add secretPath for istio TLS certificate:helm\values\mcp\m2\cicd\tkg-core-sit\environment\istio\cluster_values_comp.yaml

1.4 Add static IP for istio: helm\values\mcp\m2\cicd\tkg-core-sit\environment\istio\cluster_values_comp.yaml

1.5 Add static IP for coredns: helm\values\mcp\m2\cicd\tkg-core-sit\components\coredns\values.yaml

## 2.JENKINS JOB

1.1 Add new environment: tkg-open-sit into associated Jenkins job:

build_component

deploy_component

build_deploy

build_deploy_release

3.AUTH TKG Cluster(ie. tkg-open-sit) WITH VAULT [Refer To ... ](#Auth with TKGm)

<a name="Auth with TKGm"></a>

Reference:

https://craftech.io/blog/manage-your-kubernetes-secrets-with-hashicorp-vault/


#### 1>Deploy 1st component: `external-secrets`

#### 2>Define Environment Variable

Execute commands on one of **build server** which have:

- kubeconfig file created for target K8s cluster
- vault command installed

///  Set VAULT_SA_NAME to the service account:external-secrets 
$ export VAULT_SA_NAME=$(kubectl -n external-secrets get sa external-secrets -o jsonpath="{.secrets[*]['name']}") 

$ echo $VAULT_SA_NAME
external-secrets-token-gnjt5

/// Set SA_CA_CRT to the PEM encoded CA cert used to talk to Kubernetes API
$ export SA_CA_CRT=$(kubectl -n external-secrets get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo) 

/// Set SA_JWT_TOKEN value to the service account JWT used to access the TokenReview API
$ export SA_JWT_TOKEN=$(kubectl -n external-secrets get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo) 

/// Set K8S_HOST(API_SERVER AND PORT)
$ export K8S_HOST=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.server}')

/// Set VAULT server's URL
export VAULT_ADDR='http://vault_server:port'

/// Set VAULT_TOKEN (!Revoke token immediately when Auth with TKGm is enabled)
export VAULT_TOKEN=hvs.b1jNejvNaHQkNfu8VHIHdiCY

#### 3>Configuration

##### 1>Enable Kubernetes auth method on the same build server with “VAULT_ADDR” and "VAULT_TOKEN" en

vault auth enable --path="tkg-open-sit" kubernetes
vault auth list
Path          Type          Accessor                    Description
----          ----          --------                    -----------
approle/      approle       auth_approle_df733460       n/a
cluster_name/ kubernetes    auth_kubernetes_05200431    n/a
token/        token         auth_token_9d0ec1db         token based credentials
userpass/     userpass      auth_userpass_a63d8367      n/a

##### 2>Use the `/config` endpoint to configure Vault to talk to Kubernetes. For the list of available configuration options, please see the API documentation.

$ vault write auth/tkg-open-sit/config  token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT" namespace="external-secrets"

##### 3>Create a named role

$ vault write auth/tkg-open-sit/role/tkg-open-sit-role bound_service_account_names=external-secrets   bound_service_account_namespaces=external-secrets   policies=tkg-open-sit-read  ttl=24h

$ vault read auth/tkg-open-sit/role/tkg-open-sit-role
Key                                 Value
---                                 -----
alias_name_source                   serviceaccount_uid
bound_service_account_names         [external-secrets]
bound_service_account_namespaces    [external-secrets]
policies                            [tkg-core-sit-read]
token_bound_cidrs                   []
token_explicit_max_ttl              0s
token_max_ttl                       0s
token_no_default_policy             false
token_num_uses                      0
token_period                        0s
token_policies                      [tkg-core-sit-read]
token_ttl                           24h
token_type                          default
ttl                                 24h

This role authorizes the "**vault-auth**" capable of service account:<u>external-secrets</u> in the namespace:**external-secrets** and it gives it the named policy:sit-tanzu4

# kubectl get sa external-secrets -n external-secrets
NAME               SECRETS   AGE
external-secrets   1         5d1h

# kubectl describe sa external-secrets -n external-secrets
Name:                external-secrets
Namespace:           external-secrets
Labels:              app.kubernetes.io/instance=external-secrets
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=external-secrets
                     helm.sh/chart=external-secrets-0.1.0
Annotations:         meta.helm.sh/release-name: external-secrets
                     meta.helm.sh/release-namespace: external-secrets
Image pull secrets:  registry-secret (not found)
Mountable secrets:   external-secrets-token-cvxcz
Tokens:              external-secrets-token-cvxcz
Events:              <none>

Policies - regulates the access level (when adding a new policy, add the ticket for which we create it).

$ cat cluster_name-kv-read.hcl
#JIRA-Ticket-Number
path "secret2/metadata/rnd/cicd/new-core/sit/cluster_name/*" {
  capabilities = [ "list", "read" ]
}
path "secret2/data/rnd/cicd/new-core/sit/cluster_name/*" {
  capabilities = [ "list", "read" ]
}
#Certificate Read
path "secret2/metadata/infra/ssl/cluster_name.eisgroup.com/*" {
  capabilities = [ "list", "read" ]
}
path "secret2/data/infra/ssl/cluster_name.eisgroup.com/*" {
  capabilities = [ "list", "read" ]
}

Apply policies to role:

$ vault policy write tkg-core-sit-read tkg-core-sit-read.hcl
$ vault policy read cluster_name-kv-read

#JIRA-Ticket-Number
path "secret2/metadata/rnd/cicd/new-core/sit/cluster_name/*" {
  capabilities = [ "list", "read" ]
}
path "secret2/data/rnd/cicd/new-core/sit/cluster_name/*" {
  capabilities = [ "list", "read" ]
}
#Certificate Read
path "secret2/metadata/infra/ssl/cluster_name.eisgroup.com/*" {
  capabilities = [ "list", "read" ]
}
path "secret2/data/infra/ssl/cluster_name.eisgroup.com/*" {
  capabilities = [ "list", "read" ]
}

##### 4>Update Vault parameter in <project>/cicd/genesis/<cluster>/environment/k8s helm_values

secretBackend:
  backendType: vault
  vaultMountPoint: cluster_name (kube-cluster-name from step 1)
  vaultRole: cluster_name-role (role name from step 3)