# SRIOV POC

## Create a SNC TKG cluster

## Install multus package
1. Add Standard Package Repository

tanzu package repository add tanzu-standard --url projects.registry.vmware.com/tkg/packages/standard/repo:v2.1.0 --namespace tkg-system

2. Create multus values file

---
namespace: kube-system

daemonset:
  args:
    - "--multus-conf-file=auto"
    - "--cni-version=0.3.1"

4. Install multus CNI

tanzu package install multus-cni-pkg -p multus-cni.tanzu.vmware.com --version 3.8.0+vmware.2-tkg.2 -n tkg-system --values-file multus-cni-values.yaml

## VM customizations
1. Power off the VM
2. Add network devices for SR-IOV passthrough networks
3. Power on the VM
4. ssh into node with SRIOV IP
5. tdnf install netmgmt
6. netmgr ip4_address --set --interface eth0 --mode static --addr <eth0 IP> --gateway <default gateway>

## Install sriov device plugin package

1. Label the node

kubectl label node wl-snc-edge-dl8md-4zndk key=value1

2. Delete the default gateway for SRIOV interface

sudo ip route delete default via 10.89.82.253 dev eth1

4. Create a device plugin config file

namespace: kube-system
daemonset:
  args:
    - --log-dir=sriovdp
    - --log-level=10
sriov_nodes_resources:
  key:value1: |
    {
  "resourceList": [
    {
      "resourceName": "sriov_net",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "devices": [
          "1016"
        ]
      }

3. Create a secret

kubectl create secret generic sriov-data-values --from-file deviceconfig  -n tkg-system

4. Create Carvel Package CR

apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
  namespace: tkg-system
  name: sriov-network-device-plugin.community.tanzu.vmware.com.3.3.2
spec:
  refName: sriov-network-device-plugin.community.tanzu.vmware.com
  version: 3.3.2
  releasedAt: 2021-09-30T13:00:00Z
  releaseNotes: "sriov-network-device-plugin 3.3.2 https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/releases/tag/v3.3.2"
  licenses:
    - "Apache 2.0"
  valuesSchema:
    openAPIv3:
      title: sriov-network-device-plugin.community.tanzu.vmware.com.3.3.2 values schema
      properties:
        namespace:
          type: string
          description: Namespace where sriov-network-device-plugin daemonset will be deployed.
          default: kube-system
        daemonset:
          type: object
          description: DaemonSet related configurations.
          required:
            - args
          properties:
            resources:
              type: object
              description: DaemonSet resources settings.
              properties:
                requests:
                  type: object
                  description: Requests for CPU and memory usage.
                  properties:
                    cpu:
                      oneOf:
                        - type: string
                          description: Requests for CPU usage.
                          default: 250m
                        - type: integer
                          description: Requests for CPU usage.
                          default: 2
                    memory:
                      type: string
                      description: Requests for memory usage.
                      default: 40Mi
                limits:
                  type: object
                  description: Limits for CPU and memory usage.
                  properties:
                    cpu:
                      oneOf:
                        - type: string
                          description: Limits for CPU usage.
                          default: 1000m
                        - type: integer
                          description: Limits for CPU usage.
                          default: 1
                    memory:
                      type: string
                      description: Limits for memory usage.
                      default: 200Mi
            args:
              type: array
              description: Args passed via command-line to sriov-network-device-plugin daemonset.
              items:
                type: string
        sriov_nodes_resources:
          type: array
          description: SR-IOV devices lists on each node specified by node labels.
          items:
            type: string
  template:
    spec:
      fetch:
        - imgpkgBundle:
            image: projects.registry.vmware.com/tce/sriov-network-device-plugin@sha256:f9b51698f12448803f5875ba859891ca3f800b3ba6fe43c5e7844e3c738e1b00
      template:
        - ytt:
            paths:
              - config/
        - kbld:
            paths:
              - "-"
              - .imgpkg/images.yml
      deploy:
        - kapp: {}

5. Create a PackageInstall CR

apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: sriov
  namespace: tkg-system
spec:
  packageRef:
    refName: sriov-network-device-plugin.community.tanzu.vmware.com
    versionSelection:
      constraints: 3.3.2
      prereleases: {}
  serviceAccountName: tanzu-cluster-bootstrap-sa
  syncPeriod: 5m0s
  values:
  - secretRef:
      name: sriov-data-values

5. Verify the daemonset and pod for sriov device plugin are running.
6. Verify the node yaml has the required networks added

## Create the pods with SRIOV networks
1. Create a NetworkAttachmentDefinition

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-conf
spec:
  config: '{
        "cniVersion": "0.3.1",
        "type": "host-device",
        "pciBusID": "0000:13:00.0",
        "ipam": {
          "type": "host-local",
          "ranges": [
          [
            {
              "subnet": "192.168.1.0/24"
            }
          ]
          ]
        }
    }'

2. Create a pod

apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: sriov-conf
spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap: TERM INT; sleep infinity & wait"]
    image: alpine
    resources:
      requests:
        intel.com/sriov_net: "1"
      limits:
        intel.com/sriov_net: "1"

## Validations for pods with SR-IOV
| Validation | Result | Comment |
| -------- | -------- | -------- |
| Ping SRIOV IP of a pod with sriov interface from another pod | success | SRIOV IP is pingable |
| Ping SRIOV IP of a pod with eth0 interface from another pod | success | SRIOV IP is not pingable |
| curl PodIP on SRIOV interface | success | service is reachable |

## Observations for Pod behaviour

| Test | Observation |
| -------- | -------- |
| Remove SRIOV network adaptor | Pod is in running state but the SRIOV network IP is not reachable, Pod cannot be deleted gracefully and will fail with error killing pod: failed to "KillPodSandbox" |
| Remove memory reservation | Pod is in running state and the SRIOV network IP is reachable, removing memory reservation might have some performance impact |

## Observations for SNC rolling upgrade
| Test | Result | Comment |
| -------- | -------- | -------- |
| Remove memory reservation(unmanaged pods) | SNC rolling upgrade is succesful, we could see the multus and SRIOV device plugin packages as well but the pods created manually for SRIOV are not there in the new node | We need to manually create the pods again, We need to add the node labels required for SRIOV device plugin daemonset to discover SRIOV networks |
| Remove memory reservation(managed pods) | SNC rolling upgrade is succesful, we could see the multus and SRIOV device plugin packages as well, the pods consuming SRIOV pods would be stuck in containercreating state. | Adding SRIOV network adapter to the VM brings the pods in running state |
| Persistent Volumes | Text | Text |
| CPU pinning / CPU reservation | Text | Text |

## Open questions
1. What is the Telco Edge architecture look like? Is it vSphere or ESXi host? How the networking is between MC and WC?
[Leon] This confluence page contains Telco Edge Architecture https://confluence.eng.vmware.com/display/TELCOPS/RAN+Edge+with+SNC-ESX
The context is vSphere based (not bare metal). Networking between MC and WC depends on customer, but there is bandwidth <300kbps requirement usually.
1. What are the workloads look like? Are these K8s deployments, K8s pods etc. ? Is it a single CNF workload per Edge Single Node cluster?
*[Narendra] Different vendors have different requirements and there could be varying number of deployments/workloads in the same SNC.*
1. How's the downtime introduced for a TKG cluster in TCA? Is it okay if the workload is not funtional till the upgrade/TCA customization is complete?
[Leon] Customer will define the upgrade plan. It's ok for a cell site not functional during upgrade window. Either it completes upgrade in the upgrade window, or rollback to previous state.
1. Are there any TCA libraries for reset customizations? basically for code reusability.
[Leon] we don't have library, but we can discuss how we leverage TCA vmconfig operator to reset customization.
1. Would TCA be able to discover the new node and apply all the customization on the newly created TKG node?
*[Leon] TCA will be able to handle this if the Machine is part of the same KCP*
1. Does a VM on a single ESXi host for a single node cluster requires CPU pinn*ing/CPU reservations?
*[Narendra] This is still required as this is for processes/daemons and nothing to do with VM.*