# Airgapped FIPS 2.3.1 RHEL 8.4 on Vsphere **

## vSphere Prerequisites
* RHEL Subscription satalite or airgapped licencse for RHEL 8.4 https://access.redhat.com/solutions/3121571
* access to a RHEL repo or ability to copy packages to the airgappped machines (packages needed bzip2 yum utils on the template)
* Docker 18.04 or greater on the bastion bootstrap host (need packages docker-ce docker-ce-cli containerd.io wget)

## Bastion Node Creation (vSphere VM)

**1. Download RHEL-8.4 boot ISO to local host**

https://developers.redhat.com/content-gateway/file/rhel-8.4-x86_64-boot.iso

You will need to authenticate with your Red Hat Developer credentials. Make sure to keep them handy as you will need them later.

**2. Copy the ISO to a datastore that is acessible by all ESX nodes in the cluster**

**3. Create a "New Virtual Machine":**

from the Vsphere web UI right click on cluster you plan to use for DKP and select Create a "New Virtual Machine"
* Name: rhel-boot-8.4
* Select the "Datacenter" you want to deploy DKP on
* Select the "vCenter Cluster" you want to deploy DKP on
* Select the "Datastore" you plan to use for DKP
* select "ESXI 6.7 and later" compatibility
* select "OS family" Linux
* select "OS Version" Red Hat Enterprise Linux 8 (64-bit)
* Hardware Configuration
* CPU 2
* Memory 8GB
* New Hard disk 20GB
* New CD/DVD Drive Datastore ISO file
* Select Rhel-8.4-x86_64-boot.iso
* select the Connect the CD/DVD Drive box

**4. Once the OS boots open a virtual console and configure the following items:**

* Manual Drive partioning
* delete swap and add free space to /
* Enable the network interface
* Set the time zone
* Add the RHEL subscription
* select minimal install without any additional packages
* Start the install
* Create a new admin user "admin" password "admin"
** The user name can be anything you want admin/admin is the exmple
* allow install to finish then reboot
* allow passwordless sudo for wheel group
* sudo vi /etc/sudoers
* uncomment #
* %wheel ALL=(all) NOPASSWD:ALL
* sudo yum install -y yum-utils bzip2
* fips-mode-setup --enable
* reboot the system
* fips-mode-setup --check
* ensure that tar is install by checking yum install tar

**5. Poweroff the vm and Take a snapshot**

* right click on the base os vm
* select snapshots
* the take snapshot and name it root

## Create OVA Template for Cluster Nodes

**6. Clone the Base-RHEL-OS to a VM**

* Right click on base-rhel-os select clone to vm
* select 4vcpus 8GB RAM and a 80GB vdisk
* make sure to check and adjust MAC adress so there isn't conflict

**7. Install Docker (Only on Bastion Host)**
Add the repo for upstream docker...

sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

**8. Then install Docker and Kubectl**

sudo yum install -y docker-ce docker-ce-cli containerd.io

tar -xvf /root/dkp/dkp-v2.3.1/artifacts/kubectl-1.23.12-0.x86_64.rpm
rpm -ivh /root/dkp/dkp-v2.3.1/artifacts/kubectl-1.23.12-0.x86_64.rpm 

**9. Create directory for DKP CLI**

mkdir /root/dkp 

**11. Get the needed D2iQ Software**
Download/Decompress the airgapped bundle...

cd /root/dkp
wget http://downloads.d2iq.com/dkp/v2.3.1/dkp_airgapped_bundle_v2.3.1_linux_amd64.tar.gz
tar -xvf dkp_airgapped_bundle_v2.3.1_linux_amd64.tar.gz

Deploy your offline Repo using Docker

//Start Registry
docker run -d -p 5000:5000 --restart=always --name registry registry:2
//once locahost registry is created go to the DKp folder and run
docker load -i registry_2.tar
//push image bundle  This will push and set up your offline registry with the containers for konvoy
./dkp push image-bundle --image-bundle konvoy_image_bundle_v2.3.1_linux_amd64.tar --to-registry-insecure-skip-tls-verify --to-registry 172.16.0.181:5000

**12. Create a folder and resource pool in vCenter for DKP cluster**

Folder

Right click on the datacenter
* select New Folder
* select host and cluster folder
* name folder "D2IQ"

Resource Pool

Right click on the vCenter cluster you plan to use for DKP
* select New Resource Pool
* Adjust values if you need to restrict resources for DKP

**13. build template using konvoy image builder**

vi /root/dkp/dkp-v2.3.1/kib/images/ova/rhel-84.yaml

**14. Adjust the packer file for your Vsphere cluster**

---
download_images: true
build_name: "vsphere-rhel-84"
packer_builder_type: "vsphere" 
guestinfo_datasource_slug: "https://raw.githubusercontent.com/vmware/cloud-init-vmware-guestinfo"
guestinfo_datasource_ref: "v1.4.0"
guestinfo_datasource_script: "{{guestinfo_datasource_slug}}/{{guestinfo_datasource_ref}}/install.sh"
packer:
  vcenter_server: "10.0.1.52"
  vsphere_username: "[e-mail]"
  vsphere_password: "Password"
  cluster: "cluster1"
  datacenter: "dc1"
  datastore: "vsanDatastore"
  folder: "d2iq"
  insecure_connection: "true"
  network: "VM Network"
  resource_pool: "D2IQ"
  template: "rhel-boot-8.4"
  vsphere_guest_os_type: "rhel8_64Guest"
  guest_os_type: "rhel8-64"
  # goss params
  distribution: "RHEL"
  distribution_version: "8.4"

**15. Create overides for offline bundle location**
Navigate to the directorey with konvoy-image...

  cd /root/dkp/root/dkp/dkp-v2.3.1/kib
  cp -r ../artifacts .
  

Ensure that the artificats folder is sitting inside the kib folder

...and create the overrides file (you can override the passwordless sudo user w/ the ssh_username and password)

vi /root/dkp/dkp-v2.3.1/kib/overrides-bundles.yaml
os_packages_local_bundle_file: /root/dkp/dkp-v2.3.1/artifacts/1.23.12_redhat_8_x86_64_fips.tar.gz
pip_packages_local_bundle_file: /root/dkp/dkp-v2.3.1/artifacts/pip-packages.tar.gz
images_local_bundle_dir: /root/dkp/dkp-v2.3.1/artifacts/images
containerd_local_bundle_file: /root/dkp/dkp-v2.3.1/artifacts/containerd-1.4.13-d2iq.1-rhel-8.4-x86_64_fips.tar.gz
packer:
  ssh_username: "username you created"
  ssh_password: "password you created"

vi /root/dkp/dkp-v2.3.1/kib/overrides/fips.yaml
---
k8s_image_registry: docker.io/mesosphere

fips:
  enabled: true
  configure: false

build_name_extra: -fips
kubernetes_build_metadata: fips.0
default_image_repo: hub.docker.io/mesosphere
#kubernetes_rpm_repository_url: "https://packages.d2iq.com/konvoy/stable/linux/repos/el/kubernetes-v{{ kubernetes_version }}-fips/x86_64"
#docker_rpm_repository_url: "\
#  https://containerd-fips.s3.us-east-2.amazonaws.com\
#  /{{ ansible_distribution_major_version|int }}\
#  /x86_64"

vi /root/dkp/dkp-v2.3.1/kib/overrides/offline-fips.yaml
# fips os-packages
os_packages_local_bundle_file: "{{ playbook_dir }}/../artifacts/{{ kubernetes_version }}_{{ ansible_distribution|lower }}_{{ ansible_distribution_major_version }}_x86_64_fips.tar.gz"
containerd_local_bundle_file: "{{ playbook_dir }}/../artifacts/{{ containerd_tar_file }}"
pip_packages_local_bundle_file: "{{ playbook_dir }}/../artifacts/pip-packages.tar.gz"
images_local_bundle_dir: "{{ playbook_dir}}/../artifacts/images"

**16. Build template using konvoy image builder**

./konvoy-image build images/ova/rhel-84.yaml --overrides overrides/fips.yaml --overrides overrides/offline-fips.yaml --overrides overrides-bundles.yaml

!add -v6 for troubleshooting at the end of the command

## Create DKP cluster on vSphere

**17. Export your vSphere Environment Variables**
copy the below set of "exports" to a text document in a notepad so that you can modify them and copy/paste into the terminal. Save for later referance or use.

export CLUSTER_NAME=dkp
export CLUSTER_FOLDER=d2iq
export DKP_CONTROLPLANE=172.16.0.50
export VSPHERE_SERVER=172.16.0.15
export VSPHERE_USERNAME="[e-mail]"
export VSPHERE_PASSWORD='Gap$#oba108'
export VSPHERE_NETWORK="VM Network"
export VSPHERE_NETOWRK_INTERFACE="eth0"
export VSPHERE_DATA_CENTER="ICBM"
export VSPHERE_DATA_STORE="flash"
export VSPHERE_RESOURCE_POOL="dkp"
export VSPHERE_KIB_IMAGE="konvoy-rhel-84-fips-1.23.12+fips.0-1666189477"
export SSH_PUBLIC_KEY="/root/.ssh/id_rsa.pub"
export DOCKER_REGISTRY_ADDRESS="http://172.16.0.181:5000"
export CP_NODES=3
export CP_CPU=8
export CP_DISK=120
export CP_MEMORY=16
export WORKER_NODES=4
export WORKER_CPU=8
export WORKER_DISK=120
export WORKER_MEMORY=32

Retrieve the Vsphere Thumbprint

openssl s_client -connect ${VSPHERE_SERVER}:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin

Export the Vsphere THumbprint

export VSPHERE_TLS_THUMBPRINT="86:15:2B:54:27:29:5B:55:4D:67:57:B5:3E:97:31:57:D2:80:A9:C1"

**18. Build the Bootstrap Cluster**

cd /root/dkp/dkp-v2.3.1 
./dkp create bootstrap 

**19. Update Vsphere Credentials**

./dkp update bootstrap credentials wsphere

**20. Create the DKP cluster deplotment yaml**
If you are not using self-signed certificates, remove the last line of the command`--tls-thumb-print=` below

./dkp create cluster vsphere --cluster-name=${CLUSTER_NAME} --network="VM Network" --control-plane-endpoint-host=${DKP_CONTROLPLANE} --virtual-ip-interface=${VSPHERE_NETOWRK_INTERFACE}  --data-center=${VSPHERE_DATA_CENTER} --data-store=${VSPHERE_DATA_STORE} --folder=${CLUSTER_FOLDER} --server=${VSPHERE_SERVER} --ssh-public-key-file=${SSH_PUBLIC_KEY} --resource-pool=${VSPHERE_RESOURCE_POOL} --vm-template=${VSPHERE_KIB_IMAGE} --tls-thumb-print=${VSPHERE_TLS_THUMBPRINT} --registry-mirror-url=${DOCKER_REGISTRY_ADDRESS} --control-plane-cpus=${CP_CPU} --control-plane-disk-size=${CP_DISK} --control-plane-memory=${CP_MEMORY} --control-plane-replicas=${CP_NODES}  --worker-cpus=${WORKER_CPU}  --worker-disk-size=${WORKER_DISK}  --worker-memory=${WORKER_MEMORY}  --worker-replicas=${WORKER_NODES} --dry-run -o yaml > ${CLUSTER_NAME}.yaml

**22. Check out the ${cluster}.yaml**
{In this example its DKP.yaml}

vi ${cluster}.yaml 

**23. Deploy the cluster**

kubectl create -f ${CLUSTER_NAME}.yaml

**24. Watch Your Cluster Build**

watch ./dkp describe cluster -c ${CLUSTER_NAME}

**25. Extract the kubeconfig**

./dkp get kubeconfig -c ${CLUSTER_NAME} > ${CLUSTER_NAME}.conf

**26. Deploy MetalLB Configuration**
Change the IP range to open Ips on your network that are not being used.

cat <<EOF > metallb.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.0.0.50-10.0.0.51
EOF
kubectl --kubeconfig ${CLUSTER_NAME}.conf apply -f metallb.yaml

**27. Pivot the Cluster Controllers
Create CAPI Controllers on Cluster...**

./dkp create capi-components --kubeconfig ${CLUSTER_NAME}.conf

...and move the configuration to the cluster**

./dkp move capi-resources --to-kubeconfig ${CLUSTER_NAME}.conf

You Now have a Self-Managing Kubernetes Cluster deployed on vSphere!

**28. Adjust the Storage class to only allow PVs on a specfic VMware datastore**

kubectl delete sc vsphere-raw-block-sc --kubeconfig ${CLUSTER_NAME}.conf

Create a storage class yaml with the URL of the VMware datastore you want to use
Replace the ds below with the proper ds tag for the vmware storage you are using

vi sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: vsphere-raw-block-sc
provisioner: csi.vsphere.vmware.com
allowVolumeExpansion: true
parameters:
  datastoreurl: "ds:///vmfs/volumes/vsan:5238a205736fdb1f-c71f7ec7a0353662/" 
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Apply the Storage class yaml to create a new default SC

kubectl apply -f sc.yaml --kubeconfig ${DKP_CLUSTER_NAME}.conf

## Kommander Deployment
**29 Deploy Kommander to the DKP Cluster**
Go to DKP folder

cd root/dkp/dkp-v2.3.1

kommander-image-bundle-v2.3.1.tar is double tarred so you need to untar it one time

tar -xvf kommander-image-bundle-v2.3.1.tar

Push the image bundle to your offline registry

./dkp push image-bundle --image-bundle kommander-image-bundle-v2.3.1.tar --to-registry-insecure-skip-tls-verify --to-registry 172.16.0.181:5000

Create a Kommander Air gapped yaml file

./dkp install kommander --init --airgapped > install.yaml

Validate that there is a line that says airgap instlal:true

Time to install DKP UI {kommander}

./dkp install kommander --installer-config install.yaml --kommander-applications-repository kommander-applications-v2.3.1 --charts-bundle dkp-kommander-charts-bundle-v2.3.1.tar.gz --kubeconfig ${CLUSTER_NAME}.conf1

**30 Watch the Helm Releases Deploy**
Wait for all the helm release to turn TRUE

watch kubectl get hr -A --kubeconfig ${CLUSTER_NAME}.conf

**31 Watch the Helm Releases Deploy**
Get the IP and root login for the DKP UI

./dkp open dashboard --kubeconfig=${CLUSTER_NAME}.conf