---
# System prepended metadata

title: Airgapped FIPS 2.3.1 RHEL 8.4 on Vsphere **

---

# Airgapped FIPS 2.3.1 RHEL 8.4 on Vsphere **


## vSphere Prerequisites
* RHEL Subscription satalite or airgapped licencse for RHEL 8.4 https://access.redhat.com/solutions/3121571
* access to a RHEL repo or ability to copy packages to the airgappped machines (packages needed bzip2 yum utils on the template)
* Docker 18.04 or greater on the bastion bootstrap host (need packages docker-ce docker-ce-cli containerd.io wget)


## Bastion Node Creation (vSphere VM)

**1.  Download RHEL-8.4 boot ISO to local host**
```
https://developers.redhat.com/content-gateway/file/rhel-8.4-x86_64-boot.iso
```
You will need to authenticate with your Red Hat Developer credentials.  Make sure to keep them handy as you will need them later.

**2.  Copy the ISO to a datastore that is acessible by all ESX nodes in the cluster**

**3. Create a "New Virtual Machine":**

    from the Vsphere web UI right click on cluster you plan to use for DKP and select Create a "New Virtual Machine"
    * Name: rhel-boot-8.4
    * Select the "Datacenter" you want to deploy DKP on 
    * Select the "vCenter Cluster" you want to deploy DKP on 
    * Select the "Datastore" you plan to use for DKP
    * select "ESXI 6.7 and later" compatibility 
    * select "OS family" Linux 
    * select "OS Version" Red Hat Enterprise Linux 8 (64-bit)
    * Hardware Configuration
        * CPU 2
        * Memory 8GB
        * New Hard disk 20GB
        * New CD/DVD Drive Datastore ISO file 
        * Select Rhel-8.4-x86_64-boot.iso 
        * select the Connect the CD/DVD Drive box

**4.  Once the OS boots open a virtual console and configure the following items:**

    * Manual Drive partioning 
    * delete swap and add free space to /
    * Enable the network interface 
    * Set the time zone 
    * Add the RHEL subscription 
    * select minimal install without any additional packages
    * Start the install 
    * Create a new admin user "admin" password "admin"
    ** The user name can be anything you want admin/admin is the exmple
    * allow install to finish then reboot 
    * allow passwordless sudo for wheel group
        * sudo vi /etc/sudoers 
        * uncomment #
        * %wheel ALL=(all) NOPASSWD:ALL
        * sudo yum install -y yum-utils bzip2
        * fips-mode-setup --enable
        * reboot the system 
        * fips-mode-setup --check
        * ensure that tar is install by checking yum install tar
     


**5.  Poweroff the vm and Take a snapshot**

    * right click on the base os vm 
    * select snapshots 
    * the take snapshot and name it root


## Create OVA Template for Cluster Nodes

**6.  Clone the Base-RHEL-OS to a VM** 

    * Right click on base-rhel-os select clone to vm 
    * select 4vcpus 8GB RAM and a 80GB vdisk 
    * make sure to check and adjust MAC adress so there isn't conflict


**7.  Install Docker (Only on Bastion Host)**
Add the repo for upstream docker...
```
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
```
**8. Then install Docker and Kubectl** 
```
sudo yum install -y docker-ce docker-ce-cli containerd.io

tar -xvf /root/dkp/dkp-v2.3.1/artifacts/kubectl-1.23.12-0.x86_64.rpm
rpm -ivh /root/dkp/dkp-v2.3.1/artifacts/kubectl-1.23.12-0.x86_64.rpm 
```


**9. Create directory for DKP CLI**
```
mkdir /root/dkp 
```
**11.  Get the needed D2iQ Software**
Download/Decompress the airgapped bundle...
```
cd /root/dkp
wget http://downloads.d2iq.com/dkp/v2.3.1/dkp_airgapped_bundle_v2.3.1_linux_amd64.tar.gz
tar -xvf dkp_airgapped_bundle_v2.3.1_linux_amd64.tar.gz
```

Deploy your offline Repo using Docker
```
//Start Registry
docker run -d -p 5000:5000 --restart=always --name registry registry:2
//once locahost registry is created go to the DKp folder and run
docker load -i registry_2.tar
//push image bundle  This will push and set up your offline registry with the containers for konvoy
./dkp push image-bundle --image-bundle konvoy_image_bundle_v2.3.1_linux_amd64.tar --to-registry-insecure-skip-tls-verify --to-registry 172.16.0.181:5000
```


**12.  Create a folder and resource pool in vCenter for DKP cluster**

Folder

    Right click on the datacenter 
    * select New Folder
    * select host and cluster folder
    * name folder "D2IQ"

Resource Pool

    Right click on the vCenter cluster you plan to use for DKP
    * select New Resource Pool
    * Adjust values if you need to restrict resources for DKP


**13.  build template using konvoy image builder**
````

vi /root/dkp/dkp-v2.3.1/kib/images/ova/rhel-84.yaml

````
**14. Adjust the packer file for your Vsphere cluster**
````
---
download_images: true
build_name: "vsphere-rhel-84"
packer_builder_type: "vsphere" 
guestinfo_datasource_slug: "https://raw.githubusercontent.com/vmware/cloud-init-vmware-guestinfo"
guestinfo_datasource_ref: "v1.4.0"
guestinfo_datasource_script: "{{guestinfo_datasource_slug}}/{{guestinfo_datasource_ref}}/install.sh"
packer:
  vcenter_server: "10.0.1.52"
  vsphere_username: "administrator@vsphere.local"
  vsphere_password: "Password"
  cluster: "cluster1"
  datacenter: "dc1"
  datastore: "vsanDatastore"
  folder: "d2iq"
  insecure_connection: "true"
  network: "VM Network"
  resource_pool: "D2IQ"
  template: "rhel-boot-8.4"
  vsphere_guest_os_type: "rhel8_64Guest"
  guest_os_type: "rhel8-64"
  # goss params
  distribution: "RHEL"
  distribution_version: "8.4"
  ````
**15.  Create overides for offline bundle location**
  Navigate to the directorey with konvoy-image...
  ```
  cd /root/dkp/root/dkp/dkp-v2.3.1/kib
  cp -r ../artifacts .
  
  ```
  Ensure that the artificats folder is sitting inside the kib folder
  
  ...and create the overrides file (you can override the passwordless sudo user w/ the ssh_username and password)
```
vi /root/dkp/dkp-v2.3.1/kib/overrides-bundles.yaml
os_packages_local_bundle_file: /root/dkp/dkp-v2.3.1/artifacts/1.23.12_redhat_8_x86_64_fips.tar.gz
pip_packages_local_bundle_file: /root/dkp/dkp-v2.3.1/artifacts/pip-packages.tar.gz
images_local_bundle_dir: /root/dkp/dkp-v2.3.1/artifacts/images
containerd_local_bundle_file: /root/dkp/dkp-v2.3.1/artifacts/containerd-1.4.13-d2iq.1-rhel-8.4-x86_64_fips.tar.gz
packer:
  ssh_username: "username you created"
  ssh_password: "password you created"
```
```
vi /root/dkp/dkp-v2.3.1/kib/overrides/fips.yaml
---
k8s_image_registry: docker.io/mesosphere

fips:
  enabled: true
  configure: false

build_name_extra: -fips
kubernetes_build_metadata: fips.0
default_image_repo: hub.docker.io/mesosphere
#kubernetes_rpm_repository_url: "https://packages.d2iq.com/konvoy/stable/linux/repos/el/kubernetes-v{{ kubernetes_version }}-fips/x86_64"
#docker_rpm_repository_url: "\
#  https://containerd-fips.s3.us-east-2.amazonaws.com\
#  /{{ ansible_distribution_major_version|int }}\
#  /x86_64"
```
```
vi /root/dkp/dkp-v2.3.1/kib/overrides/offline-fips.yaml
# fips os-packages
os_packages_local_bundle_file: "{{ playbook_dir }}/../artifacts/{{ kubernetes_version }}_{{ ansible_distribution|lower }}_{{ ansible_distribution_major_version }}_x86_64_fips.tar.gz"
containerd_local_bundle_file: "{{ playbook_dir }}/../artifacts/{{ containerd_tar_file }}"
pip_packages_local_bundle_file: "{{ playbook_dir }}/../artifacts/pip-packages.tar.gz"
images_local_bundle_dir: "{{ playbook_dir}}/../artifacts/images"
```

**16.  Build template using konvoy image builder**

```
./konvoy-image build images/ova/rhel-84.yaml --overrides overrides/fips.yaml --overrides overrides/offline-fips.yaml --overrides overrides-bundles.yaml
```
!add -v6 for troubleshooting at the end of the command

## Create DKP cluster on vSphere

**17.  Export your vSphere Environment Variables**
copy the below set of "exports" to a text document in a notepad so that you can modify them and copy/paste into the terminal.  Save for later referance or use.
```
export CLUSTER_NAME=dkp
export CLUSTER_FOLDER=d2iq
export DKP_CONTROLPLANE=172.16.0.50
export VSPHERE_SERVER=172.16.0.15
export VSPHERE_USERNAME="administrator@vsphere.local"
export VSPHERE_PASSWORD='Gap$#oba108'
export VSPHERE_NETWORK="VM Network"
export VSPHERE_NETOWRK_INTERFACE="eth0"
export VSPHERE_DATA_CENTER="ICBM"
export VSPHERE_DATA_STORE="flash"
export VSPHERE_RESOURCE_POOL="dkp"
export VSPHERE_KIB_IMAGE="konvoy-rhel-84-fips-1.23.12+fips.0-1666189477"
export SSH_PUBLIC_KEY="/root/.ssh/id_rsa.pub"
export DOCKER_REGISTRY_ADDRESS="http://172.16.0.181:5000"
export CP_NODES=3
export CP_CPU=8
export CP_DISK=120
export CP_MEMORY=16
export WORKER_NODES=4
export WORKER_CPU=8
export WORKER_DISK=120
export WORKER_MEMORY=32
```
Retrieve the Vsphere Thumbprint
```
openssl s_client -connect ${VSPHERE_SERVER}:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
```
Export the Vsphere THumbprint
```
export VSPHERE_TLS_THUMBPRINT="86:15:2B:54:27:29:5B:55:4D:67:57:B5:3E:97:31:57:D2:80:A9:C1"
```
**18. Build the Bootstrap Cluster**
```
cd /root/dkp/dkp-v2.3.1 
./dkp create bootstrap 
```
**19. Update Vsphere Credentials**
```
./dkp update bootstrap credentials wsphere
```


**20.  Create the DKP cluster deplotment yaml**
If you are not using self-signed certificates, remove the last line of the command`--tls-thumb-print=` below

```

./dkp create cluster vsphere --cluster-name=${CLUSTER_NAME} --network="VM Network" --control-plane-endpoint-host=${DKP_CONTROLPLANE} --virtual-ip-interface=${VSPHERE_NETOWRK_INTERFACE}  --data-center=${VSPHERE_DATA_CENTER} --data-store=${VSPHERE_DATA_STORE} --folder=${CLUSTER_FOLDER} --server=${VSPHERE_SERVER} --ssh-public-key-file=${SSH_PUBLIC_KEY} --resource-pool=${VSPHERE_RESOURCE_POOL} --vm-template=${VSPHERE_KIB_IMAGE} --tls-thumb-print=${VSPHERE_TLS_THUMBPRINT} --registry-mirror-url=${DOCKER_REGISTRY_ADDRESS} --control-plane-cpus=${CP_CPU} --control-plane-disk-size=${CP_DISK} --control-plane-memory=${CP_MEMORY} --control-plane-replicas=${CP_NODES}  --worker-cpus=${WORKER_CPU}  --worker-disk-size=${WORKER_DISK}  --worker-memory=${WORKER_MEMORY}  --worker-replicas=${WORKER_NODES} --dry-run -o yaml > ${CLUSTER_NAME}.yaml
```
**22.  Check out the ${cluster}.yaml**
{In this example its DKP.yaml}
```
vi ${cluster}.yaml 

```
**23.  Deploy the cluster** 
```
kubectl create -f ${CLUSTER_NAME}.yaml
```

**24.  Watch Your Cluster Build**
```
watch ./dkp describe cluster -c ${CLUSTER_NAME}
```

**25.  Extract the kubeconfig**
```
./dkp get kubeconfig -c ${CLUSTER_NAME} > ${CLUSTER_NAME}.conf
```

**26.  Deploy MetalLB Configuration**  
Change the IP range to open Ips on your network that are not being used.
```
cat <<EOF > metallb.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.0.0.50-10.0.0.51
EOF
kubectl --kubeconfig ${CLUSTER_NAME}.conf apply -f metallb.yaml
```

**27.  Pivot the Cluster Controllers
Create CAPI Controllers on Cluster...** 
```
./dkp create capi-components --kubeconfig ${CLUSTER_NAME}.conf
```
...and move the configuration to the cluster**
```
./dkp move capi-resources --to-kubeconfig ${CLUSTER_NAME}.conf
```

You Now have a Self-Managing Kubernetes Cluster deployed on vSphere!

**28.  Adjust the Storage class to only allow PVs on a specfic VMware datastore**
```
kubectl delete sc vsphere-raw-block-sc --kubeconfig ${CLUSTER_NAME}.conf
```
Create a storage class yaml with the URL of the VMware datastore you want to use 
Replace the ds below with the proper ds tag for the vmware storage you are using
```
vi sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: vsphere-raw-block-sc
provisioner: csi.vsphere.vmware.com
allowVolumeExpansion: true
parameters:
  datastoreurl: "ds:///vmfs/volumes/vsan:5238a205736fdb1f-c71f7ec7a0353662/" 
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
```
Apply the Storage class yaml to create a new default SC

```
kubectl apply -f sc.yaml --kubeconfig ${DKP_CLUSTER_NAME}.conf
```

## Kommander Deployment
**29  Deploy Kommander to the DKP Cluster**
Go to DKP folder
```
cd root/dkp/dkp-v2.3.1
```
kommander-image-bundle-v2.3.1.tar is double tarred so you need to untar it one time
```
tar -xvf kommander-image-bundle-v2.3.1.tar
```
Push the image bundle to your offline registry
```
./dkp push image-bundle --image-bundle kommander-image-bundle-v2.3.1.tar --to-registry-insecure-skip-tls-verify --to-registry 172.16.0.181:5000
```
Create a Kommander Air gapped yaml file
```
./dkp install kommander --init --airgapped > install.yaml
```
Validate that there is a line that says airgap instlal:true

Time to install DKP UI {kommander}
```
./dkp install kommander --installer-config install.yaml --kommander-applications-repository kommander-applications-v2.3.1 --charts-bundle dkp-kommander-charts-bundle-v2.3.1.tar.gz --kubeconfig ${CLUSTER_NAME}.conf1
```

**30 Watch the Helm Releases Deploy**
Wait for all the helm release to turn TRUE
```
watch kubectl get hr -A --kubeconfig ${CLUSTER_NAME}.conf
```
**31 Watch the Helm Releases Deploy**
Get the IP and root login for the DKP UI
```
./dkp open dashboard --kubeconfig=${CLUSTER_NAME}.conf
```




