## AKS Cluster Initial Configuration
### Prerequisites
This document assumes that you already have the following infrastructure built:

- A Kubernetes cluster running on Azure, managed by AKS
- RBAC-Enabled
- Advanced Networking enabled
- Azure App Service Certificate
- Key Vault secrets value
- An Azure Application Gateway v2
- In the same VNet as AKS, different Subnet
- Standard SKU (minimum)
- A Managed Identity resource

### Cluster RBAC
There are different ways to authenticate with and secure Kubernetes clusters. Using role-based access controls (RBAC), you can grant users or groups access to only the resources they need. With AKS, you can further enhance the security and permissions structure by using Azure Active Directory.

With AAD-integrated AKS clusters, you can grant users or groups access to Kubernetes resources within a namespace or across the cluster. To obtain a `kubectl` configuration context, a user can run the `az aks get-credentials` command. When a user then interacts with the AKS cluster with `kubectl`, they are prompted to sign in with their Azure AD credentials. This approach provides a single source for user account management and password credentials. The user can only access the resources as defined by the cluster administrator. Cluster admins can also configure Kubernetes role-based access control (RBAC) based on a user's identity or directory group membership.

To provide granular filtering of the actions that users can perform, Kubernetes uses role-based access controls (RBAC). This control mechanism lets you assign users, or groups of users, permission to do things like creating or modify resources or view logs from running application workloads. These permissions can be scoped to a single namespace, or granted across the entire AKS cluster. Kubernetes RBAC enables us to create roles to define permissions, and then assign those roles to users/groups with role bindings.

Azure AD can only be enabled when we build a new RBAC-enabled AKS cluster as part of infrastructure deployment. A one-time configuration for this purpose is required to create few AAD components such as Cluster Service Principal, Cluster AAD Server App, Cluster AAD Client App and the AAD Tenant details. A script is authored For creating these components and populate the values in an Azure Key Vault, and then they will be retrieved and used during the infrastructure deployment to create a RBAC-enabled cluster.

### Setup RBAC
For enabling RBAC on an AKS cluster, there are a few preparatory steps needed before creating the cluster. The script `aks-aad-prerequisite.sh` is prepared for this purpose of creating and registering the required Azure Active Directory Applications for enabling RBAC on the AKS cluster and stores the details *(application id, secret, etc.)* in a Key Vault secrets for further use by IaC deployments when provisioning the cluster; thus the shared Key Vault `vonto-app-kv-ae-<env>` in the shared Resource Group `vonto-app-rgshared-ae-<env>` should exist first.

Inside this scripts there few commands to grant specific permissions to created Application objects which needs Global Admin rights in Azure Active Directory when executing the script. (Subscription Owner does not suffice and is different from AAD Global Admin!)

With the underpinnings of RBAC prerequisites in place, proceeding to the deployment and creation of the Azure resources of the AKS cluster follows in the IaC deployment part.

Once all Azure resources are deployed and the cluster is ready, run the following commands to retrieve the cluster's admin credentials and query its nodes to verify that things are in a good state.

az aks get-credentials --resource-group <aks-resource-group-name> --name <aks-cluster-name> --admin

kubectl get nodes -o wide

### Enabling RBAC for Users
Using the `--admin` parameter gives us full admin access to the cluster, but it hasn't been applied permanently for RBAC to work. To this, the following manifest needs to be applied in the cluster to enable RBAC on a group level for cluster admins.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vonto-cluster-admins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: <azure-active-directory-admin-group-object-id>

Save this in a file and replace the placeholders with the corresponding value from the Azure Active Directory Group which contains all cluster-wide admin users, then apply onto the cluster:

kubectl apply -f admin-group-rbac.yaml

These steps make sure you can use the `kubectl` command-line options.

### Enabling RBAC for the Kubernetes Dashboard
There is a web-based dashboard that can use for managing and monitoring a Kubernetes cluster and that also needs to be RBAC-enabled. Enabling RBAC to access the dashboard involves a service account which has not assigned permissions to yet. The following manifest should be deployed onto the cluster for assigning the required permissions.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

Create a file with this content and apply it onto the cluster

kubectl apply -f dashboard-admins.yaml

Now we can verify that it is possible to open the Kubernetes dashboard by running this command. By using the Azure CLI, everything is automatically piggybacked onto the current authenticated Azure session.

az aks browse --resource-group <aks-resource-group-name> --name <aks-cluster-name>

### Helm
Helm is a tool for managing applications that run in the Kubernetes cluster manager. It is being leveraged to install the application/services packages, as well as third-party package. Helm can be installed and initialised in a RBAC-enabled cluster via following shell commands *(they need to be executed in a cluster's admin context)*:

## Create Tiller Service Account
kubectl create serviceaccount --namespace kube-system tiller-sa

## Create ClusterRoleBinding for the Tiller Serivce Account
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller-sa

## Initialize Helm and Tiller service for the first time only
helm init --tiller-namespace kube-system --service-account tiller-sa

## Update the Helm default repo with latest Charts
helm repo update

The cluster should now be ready to have images deployed.

### Helm Charts
Helm charts are packages of pre-configured Kubernetes resources. A Helm chart describes how to manage a specific application on Kubernetes. It consists of metadata that describes the application, plus the infrastructure needed to operate it in terms of the standard Kubernetes primitives. Each chart references one or more (typically Docker-compatible) container images that contain the application code to be run.

Helm chart contains at least these two elements:

* A description of the package (chart.yml).
* One or more templates, which contains Kubernetes manifest files.

There are two ready-made Helm charts to simplify the deployment of application services used by Vonto App and run them in Kubernetes cluster. Helm charts simply indicate to Kubernetes how to perform the application deployment and how to manage the container clusters.

These charts are stored in the main repository under `/vonto-web/.azure/charts` folder; `vonto-connect-backend` and `vonto-connect-backend`. Each Helm chart has a set of manifests used to deploy the application, plus a values file `values.yaml` that is used to specify the default parameters value when deploying the chart which can be overridden by a user-supplied values file, which can in turn be overridden by individual parameters passed with `--set`.

These Helm charts will be packaged and stored as artifacts as part of the build pipeline, then will be used in the deployment pipeline to deploy the applications to the AKS cluster.

### Azure Container Registry (ACR)

To get running containers inside a Kubernetes cluster, we need a repository for these images. The default public repo is Docker Hub, and images stored in there will be entirely suited for your AKS cluster and available publicly. However, it is not needed to have our images available on the Internet; thus, this requires using a private repository instead of Docker Hub. In the Azure ecosystem, this is delivered by Azure Container Registry (ACR).

A single ACR is deployed by IaC and should suffice for the entire solution regardless of the number of environments we will deploy and run the containers. Upon successful creation of the ACR component, we need to establish the authentication mechanism between AKS and ACR, and there are two different scenarios to achieve this:

**Same AAD Tenant**
If both AKS and ACR resources are provisioned using the same Azure Active Directory, then only step needed is to authenticate the AKS Service Principal against ACR for pulling images. The following script used to grant the required permission and establish the authentication between AKS and ACR in the same AAD tenant:

## Get the Service Principal's id, configured for AKS
$AKS_RESOURCE_GROUP = "<aks-resource-group-name>"
$AKS_CLUSTER_NAME = "<aks-cluster-name>"
$CLIENT_ID=$(az aks show --resource-group $AKS_RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query "servicePrincipalProfile.clientId" --output tsv)

## Get the ACR registry resource id
$ACR_NAME = "<acr-registry-name>"
$ACR_RESOURCE_GROUP = "<acr-registry-reource-group-name>"
$ACR_ID=$(az acr show --name $ACR_NAME --resource-group $ACR_RESOURCE_GROUP --query "id" --output tsv)

## Create role assignment
az role assignment create --assignee $CLIENT_ID --role AcrPull --scope $ACR_ID

**Cross AAD Tenant**
If AKS cluster and ACR are provisioned and live in two different AAD tenant, then there will be another approach to establish the authentication between two resources. To achieve this, an extra Service Principal is required in the same AAD tenant that ACR resource is created, then an extra `secret` of type `docker-registry` needs to be created to store this credentials detail to connect to the private ACR registry and pull the images. The following script will create the extra Service Principal in the source AAD:

ACR_NAME="<acr-registry-name>"
SERVICE_PRINCIPAL_NAME="<service-principal-name-for-aks-acr-authentication>"

## Populate the ACR login server and resource id
ACR_LOGIN_SERVER=$(az acr show --name $ACR_NAME --query loginServer --output tsv)
ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv)

## Create acrpull role assignment with a scope of the ACR resource
SP_PASSWD=$(az ad sp create-for-rbac --name http://$SERVICE_PRINCIPAL_NAME --role acrpull --scopes $ACR_REGISTRY_ID --query password --output tsv)

## Get the Service Principal client id.
CLIENT_ID=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv)

## Output used when creating Kubernetes secret.
echo "Service principal ID: $CLIENT_ID"
echo "Service principal password: $SP_PASSWD"

Then, the `secret` object should be created in the target AKS cluster in the target AAD tenant by the following script:

kubectl create secret docker-registry vonto-acr-auth --docker-server $ACR_LOGIN_SERVER --docker-username $CLIENT_ID --docker-password $SP_PASSWD --docker-email <any-admin-email-address>

This secret name then will be used in the Kubernetes deployments referenced by the `imagePullSecrets` block and set its name property to `vonto-acr-auth` in the deployment manifest.

The AKS cluster should now be in a state where it is ready for pulling images from the ACR registry.

### AAD Pod Identity
The AAD Pod Identity is a component to enable Kubernetes applications to access Azure resources securely with Azure Active Directory.

Using Kubernetes primitives, administrators configure identities and bindings to match pods. Then without any code modifications, your containerised applications can leverage any resource in the cloud that depends on AAD as an identity provider.

Run this command to create the `aad-pod-identity` deployment on a RBAC-enabled cluster:

kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml

Before proceeding to the next step, need to retrieve the `clientId` and `id` values of the Managed Identity. Then the following Kubernetes manifest snippet can be used to deploy a new `AzureIdentity` resource onto the cluster:

apiVersion: "aadpodidentity.k8s.io/v1"
    kind: AzureIdentity
    metadata:
      name: vonto-prd-aad-pod-identity
      annotations:
        aadpodidentity.k8s.io/Behavior: namespaced
    spec:
      type: 0
      ResourceID: /subscriptions/{subscription-id}/resourcegroups/{resourcegroup-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{managed-identity-name}
      ClientID: <clientId>

Save this in a file and replace the placeholders with the corresponding values from the Managed Identity resource and then apply onto the cluster:

kubectl apply -f aad-pod-identity.yaml

Next, need to create the `AzureIdentityBinding` resource in the cluster. For a pod to match an identity binding, it needs a label with the key `aadpodidbinding` whose value is that of the `Selector:` field in the binding.

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: vonto-prd-azure-identity-binding
spec:
  AzureIdentity: <azure-identity-resource-name>
  Selector: azure-identity

Save the following manifest, replace the placeholders with proper values, ensure that the `AzureIdentity` name matches the one in the previous step and deploy onto the cluster:

kubectl apply -f aad-pod-identity-binding.yaml

The AAD Pod Identity consists of the Managed Identity Controller (MIC) deployment, the Node Managed Identity (NMI) daemon set, and several standards and custom resources. The MIC uses the Service Principal credentials stored in the cluster to access Azure resources. This service principal needs `Microsoft.ManagedIdentity/userAssignedIdentities/\*/assign/action` permission on the identity to work with user-assigned MSI. Assign the required permissions with the following command:

az role assignment create --role "Managed Identity Operator" --assignee <aks-service-principal-client-id> --scope <full id of the managed identity resource>

In addition, the User Assigned Managed Identity resource needs the `Reader` role on the resource groups to function properly with the AKS cluster. Obtain the `principalId` of the user-assigned identity by this command:

az identity show -g <identity-resource-group-name> --name <identity-name>

Then provide the `principalId` value to this command for each Resource Group:

az role assignment create --role Reader --assignee <principalId> --scope /subscriptions/{subscription-id}/resourcegroups/{resourcegroup-name}

The identity resource also needs to have access to Application Gateway to apply the configurations according to changes on the ingress resources. Following command can be used to give `Contributor` access to the Application Gateway resource:

az role assignment create --role Contributor --assignee <principalId> --scope <application-gateway-id>

### Application Gateway Ingress Controller
The Application Gateway Ingress Controller (AGIC) is a Kubernetes application, which makes it possible for AKS clusters to leverage Azure's native Application Gateway L7 load-balancer to expose cloud software to the Internet. The App Gateway Ingress Controller (AGIC) is a pod within your Kubernetes cluster. AGIC monitors the Kubernetes Ingress resources and creates and applies Application Gateway configurations based on these.

The AGIC is available as a Helm package. Add its custom repository by this command:

helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update

AGIC communicates with the Kubernetes API server and the Azure Resource Manager, using the AAD Pod Identity to access these APIs.

Below is the AGIC chart values file that needs preparation based on the environment to replace the placeholders:

verbosityLevel: 5

rbac:
    enabled: true

appgw:
    subscriptionId: <application-gateway-subscription-id>
    resourceGroup: <application-gateway-resource-group-name>
    name: <application-gateway-name>
    shared: false

armAuth:
    type: aadPodIdentity
    identityResourceID: <identity-id> 
    identityClientID: <identity-client-id>

aksClusterConfiguration:
    apiServerAddress: <aks-api-server-endpoint>

The `<identity-id>` and `<identity-client-id>` are properties of the Azure AD Identity resource that was setup in the previous section. This command can obtain them:

az identity show -g <identity-resource-group-name> --name <identity-name>

This command can obtain the AKS API server endpoint:

az aks show --resource-group <aks-resource-group-name> --name <aks-name> --query fqdn -o tsv

Once the content is saved in a file and all placeholders are replaced with actual values, we can proceed with installing the AGIC Helm chart in the `default` namespace:

helm install --name azure-agic -n default -f <agic-helm-values.yaml> application-gateway-kubernetes-ingress/ingress-azure

To make sure a successful installation, we can check the logs of the newly deployed Pod for AGIC and verify if it started properly.

### Key vault & Certificates

There are two different Key Vault resources designed and deployed for the solution:

| # | Name | Resource Group | Purpose |
| - | --------------------- | --------------------------- | -------- |
| 1 | vonto-app-kv-ae-<env> | vonto-app-rgshared-ae-<env> | Stores cluster's RBAC and infrastructure related secrets |
| 2 | vontoappakvase | vonto-app-rg-ae-<env> | Stores application configurations & certificates details |

All the secrets in Key Vault #1 must be populated manually, or the script provided above can be used for that, prior to Azure resources deployment.

| Secret name | Description |
| ----------- | ----------- |
| aks-cluster-client-id | AKS Cluster's Service Principal Id |
| aks-cluster-client-secret | AKS Cluster's Service Principal Secret |
| aks-rbac-server-app-id | Id of the AAD Application created for Cluster RBAC Server App |
| aks-rbac-server-app-secret | Secret value of the AAD Application created for Cluster RBAC Server App |
| aks-rbac-client-app-id | Id of the AAD Application created for Cluster RBAC Client App |
| aks-rbac-tenant-id | Tenant Id of the AAD contains all AAD Applications |
| aks-cluster-ssh-key | AKS cluster SSH Key used in creating the cluster |
| postgresql-admin-password | Admin password used for creating Postgres SQL resources |

Key Vault #2 contains multiple secrets related to application services configurations (environment variables), plus few certificates detail required by the application or to enable HTTPS on the services endpoints exposed by the Azure Application Gateway Ingress. This certificate is provided by an Azure App Service certificate resource `vonto-api-ssl-<env>` located in the shared resource group, which needs to be bundled as a full-chained certificate first, then convert into `.pem`/`.crt` files for importing into the Key Vault.

| Secret name | Description |
| ----------- | ----------- |
| connect-backend-api | Contains environment variables of the `connect-backend-api` service |
| connect-backend-tasks | Contains environment variables of the `connect-backend-tasks` service |
| daily-backend-api | Contains environment variables of the `daily-backend-api` service |
| daily-backend-tasks | Contains environment variables of the `daily-backend-tasks` service |
| waf-frontend-ssl-crt | Contains a PEM-encoded x509 full-chained certificate for HTTPS on Application Gateway Ingress |
| waf-frontend-ssl-pem | Contains the private key matching the x509 certificate for HTTPS on Application Gateway Ingress |
| xero-private-pem | Contains client certificate in a PEM-encoded RSA format used to authenticate against Xero |

At present, all these secrets need to be populated manually and following Azure CLI command can be used to populate the secrets from an ENV file:

az keyvault secret set --subscription <subscription-id> --vault-name <keyvault-name> --name <secret-name> --file <env-file> --encoding utf-8

These secret elements then will be used in the deployment pipeline to create/update the equivalent Kubernetes secrets, which then needed in services deployments.

Currently, Kubernetes does not support storing PKCS #12 certificate file format, and any certificates that need to be stored as Kubernetes secret must be in the CRT/PEM keypair format.

az keyvault secret set --subscription <subscription-id> --vault-name <keyvault-name> --name <secret-name> --file <crt-or-pem-file>

These certificates also will be retrieved in deployment pipeline execution to pass to the ingress to enable HTTPS, plus Xero PEM will be mounted as a volume that services read for authentication against Xero.

In the future, any extra certificates used for authentication, signing, etc. should be managed in a similar approach using Azure Key Vault to store and inject it to cluster securely.

---

## Azure DevOps
Microsoft Azure DevOps service is used for setting up an end-to-end CI/CD pipeline to build, publish and deploy the containerised application to ACR and AKS cluster by leveraging docker capabilities enabled on Azure DevOps Hosted Agents.

**Service Connections**
Seceral service connection and endpoints are required for pipelines to connect to external and remote services and environments to execute tasks.

Service connections in Azure Pipelines are created at project scope, defined and managed under each project from Admin settings of the project:[https://dev.azure.com/Vontoapp/Vonto/_settings/adminservices](https://dev.azure.com/Vontoapp/Vonto/_settings/adminservices)

Different Service connections are required in CI/CD pipelines defined for Vonto solution:
* **GitHub:** Used to make the connection to the repository hosted in GitHub.
* **Azure Resource Manager:** Used for connecting to an Azure subscription using a Service Principal, and is required per subscription.
* **Azure Container Registry:** Used for connecting to the ACR resource and is only required single one for a single ACR resource.
* **Generic:** Used for connecting to deployed Vonto services to verify the connectivity and health check at release gates, and is required per environment.

*Note:* Connection to AKS cluster is made using native Azure Resource Manager and Kubernetes Service Connection is not required.

**Variable Groups**
For ease of management and use, Variable Group feature of the Azure DevOps is used to store values that we want to control and make available across pipelines. Variable groups are defined and managed in the Library section under Pipelines.

Only one Variable Group is defined and linked to an existing Key Vault in Azure for Release pipeline and to load and inject the value of the secrets (as described above) into the pipeline during the execution in a secure manner. Because there are separate Key Vault resources defined for each environment, a separate Variable Group per each environment and linked to its Key Vault needs to be created and initialised respectively.

### Build Pipeline
The build pipeline is a YAML based pipeline, and the definition is stored in the main GitHub repository under `/vonto-web/.azure` folder and YAML file `ci-publish-pipeline.yaml`, and the build definition `Publish` is created based on this

The build pipeline has one single stage covering two jobs:
* **Build Application:** Contains all steps required to build and compile the application, package the static site, build and push Docker images.
* **Publish Artifacts:** Contains steps to package and publish Helm charts as artifacts. The overall execution and steps involved in this job are summarised as follows:
-- Create two folders, `assets` and `charts`
-- Copy source asset folder `.azure/assets` from repo to created `asset` folder
-- Install Helm client tool only (used for packaging charts locally)
-- Package and version chart `vonto-connect-backend`
-- Package and version chart `vonto-daily-backend`
-- Publish both folders as artifacts, stored in the internal Azure DevOps pipeline storage

Publish artifacts, including static site packages and Helm charts packages, will be retrieved in release pipelines (via linked artifact) to deploy to different environments.

### Deployment Pipeline
The build pipeline used to set up CI has already built the application and pushed Docker images to the ACR. It also packaged and published two Helm chart as an artifact. In the release pipeline, we will deploy the container images as a Helm application to the AKS cluster.

The deployment pipeline is fully defined and configured in the Azure DevOps project.