# Labops - Openshift Networking with Antrea

###### tags: `labops`, `antrea`, `openshift`

# Table of Content
[ToC]

This script runs the flow of installing Openshift Cluster in vSphere environment as I wrote in [My blog post](https://alvianus.net/posts/2020/08/deploying-openshift-4.5-automatically-on-vsphere/).

# **DNS Record Requirement**
Create following DNS entries using following format. In my example, my cluster name is **ocp-east** and my base domain is **lab01.one**

| Function | Domain Name | IP Address |
| ---------------| ------------------------------------| ----------------|
| API VIP | api.ocp-east.lab01.one | 10.10.10.251 |
| Ingress | *.apps.ocp-east.lab01.one | 10.10.10.252 |

# Prerequisites
The admin box in Labops template has no git installed. So install git first

yum install git -y

Clone the repository

git clone https://github.com/ralvianus/ocp-install
cd ocp-install

Run the shell script

./ocp-install-prep.sh


This script will download and install `openshift-install` tool, create SSH key, getting root vCenter CA certificate, and install it in the installer host.

# Creating `install-config.yaml` File
Create install-config.yaml through installer. Set the compute and control plane parameters based on your environment. The networkType parameter must be antrea. Make sure that apiVIP and ingressVIP are correctly set. Here is a sample install-config.yaml with Antrea as the CNI

mkdir ocp-east
cd ocp-east
openshift-install create install-config

Follow the wizard and fill in the form.


After finishing, you will get `install-config.yaml` file. You need to modify the networking section to antrea like below

networking:
  networkType: antrea
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineCIDR: 192.168.25.0/24
  serviceNetwork:
  - 172.30.0.0/16

Complete `install-config.yaml` is in the appendix

# Creating Manifests Files
You can backup or save the `install-config.yaml` in other place, because this file will be consumed to create the manifests.

openshift-install create manifests

You will see the manifests directory is created.

Download the file below from customerconnect portal:
- VMware Container Networking with Antrea, K8s Operator Manifests
- NSX Interworking Connector & Deployment manifests

The Antrea Operator manifests are the deployment for Antrea Operator and Antrea controller/agent. The manifests contain:
- `namespace.yaml`
- `nsx-cert.yaml`
- `operator.antrea.vmware.com_antreainstalls_crd.yaml`
- `operator.antrea.vmware.com_v1_antreainstall_cr.yaml`
- `operator.yaml`
- `role.yaml`
- `role_binding.yaml`
- `service_account.yaml`

The NSX Interworking manifests are the deployment for NSX Interworking. This is optional when you want to integrate Antrea with NSX. The manifests contain:
- `bootstrap-config.yaml`
- `deregisterjob.yaml`
- `interworking.yaml`
- `inventorycleanup.yaml`
- `ns-label-webhook.yaml`

Unzip the Kubernetes Operator manifest file (deploy.tar.gz) and copy the contents to the openshift install manifests directory

mkdir /path/to/antrea-operator-for-kubernetes 
cd /path/to/antrea-operator-for-kubernetes/ 
tar xvfz deploy.tar.gz 
cp /path/to/antrea-operator-for-kubernets/deploy/openshift ~/ocp-east/manifests


Edit the manifests to add the Antrea and operator images.
- In `operator.yaml`, update the antrea-operator image with the URI of the Antrea operator container image.
- In `operator.antrea.vmware.com_v1_antreainstall_cr.yaml`, change antreaImage to the URI of the Antrea container image.

For Antrea 1.4.0, you can use the container images on VMware distribution Harbor:

**Antrea Images**

projects.registry.vmware.com/antreainterworking/antrea-standard-debian:v1.5.2_vmware.2
projects.registry.vmware.com/antreainterworking/antrea-advanced-debian:v1.5.2_vmware.2
projects.registry.vmware.com/antreainterworking/antrea-ubi:v1.5.2_vmware.2

**Operator Images**

projects.registry.vmware.com/antreainterworking/antrea-operator:v1.5.2_vmware

**Antrea NSX Interworking Images**

projects.registry.vmware.com/antreainterworking/interworking-debian:0.5.0
projects.registry.vmware.com/antreainterworking/interworking-ubuntu:0.5.0
projects.registry.vmware.com/antreainterworking/interworking-photon:0.5.0
projects.registry.vmware.com/antreainterworking/interworking-ubi:0.5.0

# Create Openshift Cluster

openshift-install create cluster

# Install Antrea to NSX Interworking
Note the following prerequisites for Openshift with Antrea cluster and NSX Interworking:

- Antrea 1.4.0 or later
- Antrea Operator for OpenShift manifest files
- Antrea interworking (UBI) image
- OpenShift Cluster with Antrea 1.4.0 or later
- NSX-T 3.2 or later

## Create Principal Identity in NSX-T
Generate the self-signed certificate to be installed as Principal Identity in NSX Manager

CLUSTER_NAME="ocp-east"
openssl genrsa -out $CLUSTER_NAME-private.key 2048
openssl req -new -key $CLUSTER_NAME-private.key -out $CLUSTER_NAME.csr -subj "/C=US/ST=CA/L=Palo Alto/O=VMware/OU=Antrea Cluster/CN=$CLUSTER_NAME"
openssl x509 -req -days 3650 -sha256 -in $CLUSTER_NAME.csr -signkey $CLUSTER_NAME-private.key -out $CLUSTER_NAME.crt

To create a principal identity user:
1. In the NSX Manager UI, click the System tab.
2. Under Settings, navigate to User Management > User Role Assignment.
3. Click Add > Principal Identity with Role.
4. Enter `ocp-east` as a name for the principal identity user.
5. Select the role as Enterprise Admin.
6. In the Node Id text box, enter `ocp-east` as a name for the Antrea container cluster.
7. In the Certificate PEM text area, paste the complete self-signed certificate, which you created earlier. Ensure that the -----BEGIN CERTIFICATE---- and ------END CERTIFICATE----- lines are also pasted in this text box.
8. Click Save.
9. From the left navigation pane, under Settings, click Certificates. Verify that the self-signed certificate of the Antrea container cluster is shown.


## Edit Antrea Configuration yaml Files
Edit the Antrea configuration in the `openshift/operator.antrea.vmware.com_v1_antreainstall_cr.yaml` file. Edit the cluster name and NSXManagers accordingly. Change the `enableInterworking` to true to initiate the deployment.

bootstrapConfig: |
    clusterName: ocp-east
    NSXManagers: [172.16.10.117]
    vhcPath: ""
  antreaImage: projects.registry.vmware.com/antreainterworking/antrea-ubi:v1.5.2_vmware.2
  interworkingImage: projects.registry.vmware.com/antreainterworking/interworking-ubi:0.5.0
  antreaPlatform: openshift
  enableInterworking: true

Edit the `nsx-cert.yaml` file to include the certificate and key used to create Principal Identity. The certificate and key has to be in base64 format, use the following command to generate the base64 format of the certificate and key.

cat ocp-east.crt | base64 -w 0
cat ocp-east-private.crt | base64 -w 0

apiVersion: v1
kind: Namespace
metadata:
  name: vmware-system-antrea
  labels:
    app: antrea-interworking
    openshift.io/run-level: '0'
---
apiVersion: v1
kind: Secret
metadata:
  name: nsx-cert
  namespace: vmware-system-antrea
type: kubernetes.io/tls
data:
  # One line base64 encoded data. Can be generated by command: cat tls.crt | base64 -w 0
  tls.crt: <insert the cert here>
  # One line base64 encoded data. Can be generated by command: cat tls.key | base64 -w 0
  tls.key: <insert the key here>

## Apply the `operator.antrea.vmware.com_v1_antreainstall_cr.yaml` and `nsx-cert.yaml`

oc apply -f operator.antrea.vmware.com_v1_antreainstall_cr.yaml
oc apply -f nsx-cert.yaml

## Check the Deployment

oc get pods -n vmware-system-antrea


After a few seconds, the Antrea cluster should be registered with NSX. You can check the inventory in NSX Manager.

# Appendix
`install-config.yaml`

apiVersion: v1
baseDomain: lab01.one
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 3
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
metadata:
  creationTimestamp: null
  name: ocp-east
networking:
  networkType: antrea
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineCIDR: 192.168.25.0/24
  serviceNetwork:
  - 172.30.0.0/16
platform:
  vsphere:
    apiVIP: 10.10.10.251
    cluster: cmp
    datacenter: lab01
    defaultDatastore: vsanDatastore
    ingressVIP: 10.10.10.252
    network: ocp-east-00
    password: VMware1!SDDC
    username: [e-mail]
    vCenter: vcenter.lab01.one
publish: External
pullSecret: '<pull secret>'
sshKey: |
  <ssh pub key>

# Source

- [Antrea on Openshift Install Steps - VMware Docs](https://docs.vmware.com/en/VMware-Container-Networking-with-Antrea/1.x/vmware_antrea_install/GUID-303EC29B-01CE-4668-A187-2566BD02CFB0.html)
- [Antrea 1.4.0 Release Notes](https://docs.vmware.com/en/VMware-Container-Networking-with-Antrea/1.4.0/rn/VMware-Container-Networking-with-Antrea-Version-140-Release-Notes.html)