![](https://i.imgur.com/nMao5HZ.jpg =50x) DKP 2.2.1 Airgapped Install on vSphere with FIPS and RHEL 8.4

# ![](https://i.imgur.com/nMao5HZ.jpg =50x) DKP 2.2.1 Airgapped Install on vSphere with FIPS and RHEL 8.4 ## VMWare vSphere Prerequisites Before installing, verify that your VMware vSphere Client environment meets the following basic requirements: * Access to a bastion VM, or other network connected host, running vSphere Client version v6.7.x with Update 3 or later version * You must be able to reach the vSphere API endpoint from where the DKP command line interface (CLI) runs. * vSphere account with credentials configured - this account must have Administrator privileges. * For the networking, ensure the hosts and VIP do not overlap * For air-gapped environments, a bastion VM host template with access to a configured Docker registry * Valid vSphere values for the following: * vCenter API server URL * Datacenter name * Zone name that contains ESXi hosts for your cluster’s nodes * Datastore name for the shared storage resource to be used for the VMs in the cluster. * Use of PersistentVolumes in your cluster depends on Cloud Native Storage (CNS), available in vSphere v6.7.x with Update 3 and later versions. CNS depends on this shared Datastore’s configuration. * Datastore URL from the datastore record for the shared datastore you want your cluster to use. * You need this URL value to ensure that the correct Datastore is used when DKP creates VMs for your cluster in vSphere. * Folder name * Base template name, such as base-rhel-8, or base-rhel-7 * Name of a Virtual Network that has DHCP enabled for both air-gapped and non air-gapped environments * Resource Pools - at least one resource pool needed, with every host in the pool having access to shared storage, such as VSAN * Each host in the resource pool needs access to shared storage, such as NFS or VSAN, to make use of MachineDeployments and high-availability control planes. * The default template of 2vCPUs and 8GB of RAM is to small. Instead use the following as minimimum requirements. * (3) Control Plane Nodes * 4 vCPU * 16GB Memory * 80GBs of available storage * (4) Worker Nodes * 8 vCPU * 32GB Memory * 80GB of available storage ## Create an Airgapped Bundle ### Downloads from CSS Log into css-support.d2iq.com Navigate to the DKP 2.2.x section. From the main page select DKP -> DKP Downloads. *Note: If you end up on the Konvoy 2.2.1 page, you will need to reselect the platform at the top of the page to DKP.* List of Files to Download from CSS Page (starting top of page): - [ ] **dkp_v2.2.1_linux_amd64.tar.gz** or **dkp_v2.2.1_darwin_amd64.tar.gz** DKP CLI Binary -- Choose from Linux () or Darwin() based on your platform deployment machine - [ ] **konvoy_image_bundle_v2.2.1_linux_amd64.tar.gz** -- Konvoy Image Bundle - [ ] **kommander-image-bundle-v2.2.1.tar.gz** - Kommander Image Bundle - [ ] **dkp-catalog-applications-image-bundle-v2.2.1.tar.gz** - DKP Catalog Application Images - [ ] **dkp-insights-image-bundle-v2.2.1.tar.gz** - DKP Insights Catalog Application Images - [ ] **dkp-kommander-charts-bundle-v2.2.1.tar.gz** -- Kommander Airgap Helm Charts - [ ] **dkp-catalog-applications-charts-bundle-v2.2.1.tar.gz** - Catalog Airgap Helm Charts - [ ] **dkp-insights-charts-bundle-v2.2.1.tar.gz** -- DKP Insights Airgap Helm Charts - [ ] **kommander-applications-v2.2.1.tar.gz**-- Kommander Airgap Git Repos - [ ] **dkp-catalog-applications-v2.2.1.tar.gz** -- Catalog Airgap Git Repos - [ ] **dkp-insights-v2.2.1.tar.gz** -- DKP Insights Airgap Git Repos - [ ] **konvoy_bootstrap_v2.2.1** -- Airgapped Konvoy Bootstrap Image ### Downloads from Github - [ ] **konvoy-image-bundle-v1.12.0_linux_amd64.tar.gz** - Konvoy Image Builder (KIB) https://github.com/mesosphere/konvoy-image-builder/releases/tag/v1.12.0 ### Downloads from D2IQ Site - [ ] **pip-packages.tar.gz** - Pip Packages https://downloads.d2iq.com/dkp/airgapped/pip-packages/pip-packages.tar.gz - [ ] **1.22.8_redhat_8_x86_64_fips.tar.gz** - OS Packages https://downloads.d2iq.com/dkp/airgapped/os-packages/1.22.8_redhat_8_x86_64_fips.tar.gz - [ ] **1.22.8_images.tar.gz** - Kubernetes Base Images https://downloads.d2iq.com/dkp/airgapped/kubernetes-images/1.22.8_images.tar.gz ### Other Prerequistes - [ ] **docker-ce-20.10.17-3.el8.src.rpm** - Docker https://download.docker.com/linux/rhel/8/source/stable/Packages/docker-ce-20.10.17-3.el8.src.rpm - [ ] **kubectl** - Kubectl 1.21.6 https://dl.k8s.io/release/v1.21.6/bin/linux/amd64/kubectl ## Setup Bastion VM Host ### Create Bastion VM Host Template Create a bastion VM host template for the cluster nodes to use within the air-gapped network. This bastion VM host also needs access to a Docker registry in lieu of an Internet connection for pulling Docker images. The recommended template naming pattern is ```../folder-name/dkp-e2e-bastion-template``` or similar. ### Bastion Host Prerequisites * Docker-CE 19.03+ * Kubectl Remove the default Docker enginge ``` sudo yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine ``` Install DockerCE ``` sudo yum install docker-ce-20.10.17-3.el8.src.rpm sudo systemctl start docker ``` Manage Docker as non-root user ``` sudo systemctl enable --now docker sudo groupadd docker sudo usermod -aG docker $USER ``` Install kubectl ``` sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl ``` ### SSH into Bastion VM Use your credentials to SSH into the bastion VM host with the command: ``` ssh -i </path/to/private_key> <USER>@<BASTION_IP> ``` ### Set Environment Variables for vSphere ``` export VSPHERE_SERVER=your_vCenter_APIserver_URL export VSPHERE_USERNAME=your_vCenter_user_name export VSPHERE_PASSWORD=your_vCenter_password ``` ### Set Environment Variables for Docker Registry Set an environment variable with your registry address: ``` export DOCKER_REGISTRY_ADDRESS=<registry-address>:<registry-port> export DOCKER_REGISTRY_CA=<path to the CA on the bastion host> ``` ### Node Prerequisites For DKP to install completely, you must stop firewalld ``` systemctl status firewalld systemctl stop firewalld ``` Also, please uninstall podman, runc, and buildah ## Build the vSphere OVA Template ### Directory Structure The final directory structure needs to look like the following chart. You will need to create the kib, artifacts, and images directories. ``` |- |- konvoy-bootstrap_v2.2.1.tar | |- konvoy_image_bundle_v2.2.1_linux_amd64.tar.gz | |- dkp-catalog-applications-charts-bundle-v2.2.1.tar.gz | |- dkp-catalog-applications-image-bundle-v2.2.1.tar.gz | |- dkp-catalog-applications-v2.2.1.tar.gz | |- dkp-insights-charts-bundle-v2.2.1.tar.gz | |- dkp-insights-image-bundle-v2.2.1.tar.gz | |- dkp-insights-v2.2.1.tar.gz | |- dkp-kommander-charts-bundle-v2.2.1.tar.gz | |- kommander-applications-v2.2.1.tar.gz | |- kommander-image-bundle-v2.2.1.tar.gz | |- kib/ | | |- konvoy-image-bundle-v1.12.0_linux_amd64.tar.gz | | |- artifacts/ | | | |- 1.22.8_redhat_8_x86_64_fips.tar.gz | | | |- pip-packages.tar.gz | | | |- images/ | | | | |- 1.22.8_images.tar.gz ``` ### Unpack KIB Go into the kib directory and unpack the konvoy image bundle. ``` tar -xvf konvoy-image-bundle-v1.12.0_linux_amd64.tar.gz ``` ### Add vSphere overrides. The following will override the default values in `images/ova/rhel_84.yaml`. ```shell vi overrides/overrides-vcenter.yaml ``` ##### overrides/overrides-vcenter.yaml ```yaml # Add the following fields and values packer: vcenter_server: <vcenter> vsphere_username: <username> vsphere_password: <password> ``` ### (Optional) Add bundle overrides Use the included sample bundle overrides included in the package ```shell vi override-bundles.yaml ``` ##### overrides/overrides-bundles.yaml ```yaml # Update the following fields with the desired OS and or FIPS related references os_packages_local_bundle_file: /tmp/kib/artifacts/1.22.8_redhat_8_x86_64_fips.tar.gz pip_packages_local_bundle_file: /tmp/kib/artifacts/pip-packages.tar.gz images_local_bundle_dir: /tmp/kib/artifacts/images/ # packer: # ssh_username: ``` ##### overrides/overrides-registry.yaml ```yaml image_registries_with_auth: - host: "<registry>" username: "<registry-username>" password: "<registry-password> auth: "" identityToken: "" ``` ### Build the OVA Template ``` ./konvoy-image build images/ova/rhel-84.yaml \ --overrides overrides/overrides-offline-fips.yaml \ --overrides overrides/overrides-vcenter.yaml ``` ## Create Bootstrap ### Load the DKP Bootstrap and Registry Container Images into Docker Registry ``` docker load < konvoy-bootstrap_v2.2.1.tar # Optional docker load < registry_2.tar ``` ### (Optional) Configure Local Docker Registry for Insecure ``` cat <<-EOF | sudo tee /etc/docker/daemon.json { "insecure-registries" : ["$(hostname -I | awk '{print $1}'):5000"] } EOF sudo systemctl restart docker ``` ### Push the base DKP bundles to default registry ``` export REGISTRY_IP=$(hostname -I | awk '{print $1}') ./dkp push image-build \ --??image-build konvoy_image_builder_v2.2.1_linux_amd64.tar.gz \ --to-registry=${REGISTRY_IP}:5000 \ --to-registry-insecure-skip-tls-verify ``` ### Unpack Image Bundles ``` tar -xvf dkp_v2.2.1_linux_amd64.tar.gz tar -xvf kommander-image-bundle-v2.2.1.tar.gz tar -xvf dkp-catalog-applications-image-bundle-v2.2.1.tar.gz tar -xvf dkp-insights-image-bundle-v2.2.1.tar.gz ``` Push each image bundle to the Docker Registry ``` ./dkp push image-bundle --image-bundle konvoy_image_bundle_v2.2.1_linux_amd64.tar.gz --to-registry $DOCKER_REGISTRY_ADDRESS --to-registry-insecure-skip-tls-verify ./dkp push image-bundle --image-bundle kommander-image-bundle-v2.2.1.tar.gz --to-registry $DOCKER_REGISTRY_ADDRESS --to-registry-insecure-skip-tls-verify ./dkp push image-bundle --image-bundle dkp-catalog-applications-image-bundle-v2.2.1.tar.gz --to-registry $DOCKER_REGISTRY_ADDRESS --to-registry-insecure-skip-tls-verify ./dkp push image-bundle --image-bundle dkp-insights-image-bundle-v2.2.1.tar.gz --to-registry $DOCKER_REGISTRY_ADDRESS --to-registry-insecure-skip-tls-verify ``` ## Create Bootstrap ### Docker Load Images ``` docker load < konvoy-bootstrap_v2.2.1.tar ``` ### Create Bootstrap Cluster ``` ./dkp create bootstrap --with-aws-bootstrap-credentials=false ``` The output resembles this example: ``` Creating a bootstrap cluster Initializing new CAPI components ``` Ensure that the CAPV controllers are present with the command: ``` kubectl get pods -n capv-system ``` The output resembles the following: ``` NAME READY STATUS RESTARTS AGE capv-controller-manager-785c5978f-nnfns 1/1 Running 0 13h ``` Refresh the credentials used by the vSphere provider at any time, using the command: ``` dkp update bootstrap credentials vsphere ``` ### Set Environment Variables for vSphere ``` export VSPHERE_SERVER="<vsphere-server>" export VSPHERE_USERNAME="<administrator@vsphere.local>" export VSPHERE_PASSWORD="<password>" export VSPHERE_DATACENTER="<vsphere-datacenter-name>" export VSPHERE_DATASTORE="<data-store-name>" export VSPHERE_FOLDER="<folder-name>" export VSPHERE_RESOURCE_POOL="<resource-pool-name>" export VSPHERE_NETWORK="<network-name>" export VSPHERE_TEMPLATE="<vsphere-template>" export DKP_CLUSTER_NAME="<dkp-cluster-name>" export DKP_CONTROL_PLANE_VIP="<vip-ip-address>" export DKP_CONTROL_PLANE_VIP_INTERFACE="<ethernet-interface-name>" export DKP_METALLB_RANGE="<rangeIpBegin-rangeIpEnd>" export SSH_PRIVATE_KEY="<ssh-private-key>" export SSH_USERNAME="<ssh_username>" ``` If vCenter is using a self-signed certificate ``` openssl s_client -connect ${VSPHERE_SERVER_IP}:${VSPHERE_SERVER_PORT} < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin | export VCENTER_TLS_THUMBPRINT ``` ## Create the Cluster ### Generate the Config.yaml ``` ./dkp create cluster vsphere \ --cluster-name=${DKP_CLUSTER_NAME} \ --network="${VSPHERE_NETWORK}" \ --control-plane-endpoint-host=${DKP_CONTROL_PLANE_VIP} \ --virtual-ip-interface=${DKP_CONTROL_PLANE_VIP_INTERFACE} \ --data-center=${VSPHERE_DATACENTER} \ --data-store=${VSPHERE_DATASTORE} \ --folder=${VSPHERE_FOLDER} \ --server=${VSPHERE_SERVER} \ --resource-pool=${VSPHERE_RESOURCE_POOL} \ --vm-template=${VSPHERE_TEMPLATE} \ --registry-mirror-url=http://${REGISTRY_IP}:5000 \ --tls-thumb-print="${VCENTER_TLS_THUMBPRINT}" \ --dry-run \ -o yaml \ --ssh-public-key-file=${SSH_PRIVATE_KEY} \ --ssh-username=${SSH_USERNAME} > cluster.yaml ``` To adjust the template size for the Control Planes and Worker Pools, edit the cluster.yaml. Update all the desired VSphereMachineTemplate references. ``` vi cluster.yaml ``` ##### cluster.yaml ```yaml= --- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: VSphereMachineTemplate metadata: name: dkp-md-0 namespace: default spec: template: spec: cloneMode: linkedClone datacenter: kompton datastore: vsanDatastore diskGiB: 25 folder: d2iq memoryMiB: 8192 network: devices: - dhcp4: true networkName: VM Network numCPUs: 4 resourcePool: DKP ``` ### Create the DKP Cluster Create the Cluster ``` kubectl create -f <CLUSTER_NAME>.yaml ``` Watch the status of the cluster to completion ``` watch ./dkp describe cluster -c <CLUSTER_NAME> ``` Get the kubeconfig for the DKP cluster ``` ./dkp get kubeconfig -c ${CLUSTER_NAME} > ${CLUSTER_NAME}.conf ``` ### Create the CAPI components on the DKP Cluster ``` ./dkp create capi-components --kubeconfig ${CLUSTER_NAME}.conf ``` ### Configure MetalLB Create MetalLB Configuration YAML File ``` cat << EOF > metallb-conf.yaml apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: default protocol: layer2 addresses: - xx.x.xxx.xx-xx.x.xxx.xx EOF ``` Install MetalLB on the New Cluster ``` kubectl apply -f metallb-conf.yaml ``` ## Install DKP Addons ### Install the DKP Addons ``` ./dkp install kommander \ --airgapped \ --kommander-applications-repository kommander-applications-v2.2.0 \ --charts-bundle dkp-kommander-charts-bundle-v2.2.0.tar.gz \ --charts-bundle dkp-catalog-applications-charts-bundle-v2.2.0.tar.gz \ --charts-bundle dkp-insights-charts-bundle-v2.2.0.tar.gz \ --kubeconfig admin.conf ``` ### Watch the Helm Releases Deploy ``` watch kubectl get hr -A --kubeconfig ${DKP_CLUSTER_NAME}.conf ```