:::info
:bulb:This is an example of adding NFS to your Harvester cluster as a storage class.
:::

## :question:Purpose
:::success
Add a new storage class to your harvester cluster as an additional storage option to the built in longhorn storage.
:::

## :feet:Pre-Req steps
:::success
Pre-reqs make sure you have the tools and configurations to successfully complete all the main steps.
:::
1. Install Hauler (https://github.com/hauler-dev/hauler)

Linux/Darwin

# installs latest release
curl -sfL https://get.hauler.dev | bash

Homebrew

# installs latest release
brew tap hauler-dev/homebrew-tap
brew install hauler

2. Hauler login

hauler login -u <redacted> -p <redacted> rgcrprod.azurecr.us

3. Carbide Public Key

curl -sfOL https://raw.githubusercontent.com/rancherfederal/carbide-releases/main/carbide-key.pub

4. Install kubectl

# Download the latest stable release for your architecture (e.g., AMD64)
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# Make the binary executable
chmod +x ./kubectl

# Move the executable to a directory in your PATH (e.g., /usr/local/bin)
sudo mv ./kubectl /usr/local/bin/kubectl

# Verify the installation
kubectl version --client

5. Install helm

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4
chmod 700 get_helm.sh
./get_helm.sh
#verify 
helm version

6. Turn on developer extensions
Go to preferences in the upper right then check the box for "Enable Extension developer features"

7. Access embedded rancher
Go to support
Access Embedded Rancher UI

8. Some sort of OCI Container Registry or plan to use Hauler as your temporary registry.
my setup has a Harbor Registry installed at registry.millennium.rebel.systems.

## :feet:Main Steps
:::success
Start
:::

1. Carbide License from RGS
i.e. eyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
2. NFS CSI Image https://portal.ranchercarbide.dev/apps/kubernetes-csi-driver-nfs
3. Grab the hauler command and run on the system with hauler installed

hauler store sync --products apps-kubernetes-csi-driver-nfs=4.12.1 --platform linux/amd64 --key carbide-key.pub

4. Inspect the store and optionally share if you do not have a registry already

hauler store list
#optional if you don't have an existing registry
hauler store serve registry

5. Login to your registry for example
:::info
Update for your environment registry.millennium.rebel.systems is mine
:::

hauler login registry.millennium.rebel.systems --username admin --password Harbor12345

6. Copy to your registry

hauler store copy registry://registry.millennium.rebel.systems --insecure

7. Get kubeconfig
Go to Support bottom left corner of the embeded rancher screen
Download KubeConfig
Copy to server with kubectl
8. Set kubeconfig environment variable (alias if, for example, you are using a self signed cert)

#example location
export KUBECONFIG="~/myHarvester.yaml"
#only needed for untrusted ssl certs
alias kubectl='kubectl --insecure-skip-tls-verify'

9. Validate you can access the right server

kubectl get namespaces

example output

NAME                                     STATUS   AGE
carbide-stigatron-system                 Active   115m
cattle-dashboards                        Active   50d
cattle-fleet-clusters-system             Active   50d
cattle-fleet-local-system                Active   50d
cattle-fleet-system                      Active   50d
cattle-impersonation-system              Active   50d
cattle-logging-system                    Active   50d
cattle-monitoring-system                 Active   50d
cattle-provisioning-capi-system          Active   50d
cattle-system                            Active   50d
cattle-ui-plugin-system                  Active   50d
cert-manager                             Active   50d
cluster-fleet-local-local-1a3d67d0a899   Active   50d
default                                  Active   50d
fleet-default                            Active   50d
fleet-local                              Active   50d
git                                      Active   50d
harbor                                   Active   50d
harvester-public                         Active   50d
harvester-system                         Active   50d
kasm                                     Active   50d
kube-node-lease                          Active   50d
kube-public                              Active   50d
kube-system                              Active   50d
local                                    Active   50d
longhorn-system                          Active   50d
user-s7vr4                               Active   50d

10. Create helmchart for nfs
:::info
Update for your environment
Chart/imageregistry registry.millennium.rebel.systems is mine.
Set the values for your nfs server "server" "share" etc
:::
vim mychart.yaml

Option 1 dedicated registry

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: nfs-csi-driver
  namespace: kube-system
spec:
  imagePullSecrets:
    - name: my-image-pull-secret
  chart: oci://<server hauler is on>:5000/hauler/kubernetes-csi-driver-nfs
  version: 4.12.1
  createNamespace: true
  insecureSkipTLSVerify: true
  targetNamespace: kube-system
  valuesContent: |-
    global:
      imageRegistry: "registry.millennium.rebel.systems"
    storageClass:
      create: true
      name: nfs-csi
      annotations:
        storageclass.kubernetes.io/is-default-class: "false"
      parameters:
        server: 10.0.1.100
        share: /volume1/harvester
    #     subDir:
    #     mountPermissions: "0"
    #     csi.storage.k8s.io/provisioner-secret is only needed for providing mountOptions in DeleteVolume
    #     csi.storage.k8s.io/provisioner-secret-name: "mount-options"
    #     csi.storage.k8s.io/provisioner-secret-namespace: "default"
    #   reclaimPolicy: Delete
    #   volumeBindingMode: Immediate
    #   mountOptions:
    #     - nfsvers=4.1

Option 2 using hauler registry

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: nfs-csi-driver
  namespace: kube-system
spec:
  imagePullSecrets:
    - name: my-image-pull-secret
  chart: oci://<server hauler is on>:5000/hauler/kubernetes-csi-driver-nfs
  version: 4.12.1
  createNamespace: true
  insecureSkipTLSVerify: true
  targetNamespace: kube-system
  valuesContent: |-
    global:
      imageRegistry: "<server hauler is on>:5000"
    storageClass:
      create: true
      name: nfs-csi
      annotations:
        storageclass.kubernetes.io/is-default-class: "false"
      parameters:
        server: 10.0.1.100
        share: /volume1/harvester
            #     subDir:
    #     mountPermissions: "0"
    #     csi.storage.k8s.io/provisioner-secret is only needed for providing mountOptions in DeleteVolume
    #     csi.storage.k8s.io/provisioner-secret-name: "mount-options"
    #     csi.storage.k8s.io/provisioner-secret-namespace: "default"
    #   reclaimPolicy: Delete
    #   volumeBindingMode: Immediate
    #   mountOptions:
    #     - nfsvers=4.1

11. Install nfs helmchart

kubectl apply -f mychart.yaml

12. Validate NFS storage class exists
Go to Harvester
Left menu Advanced
Storage Classes
"nfs-csi" should exist and be active

13. Test using the storage class
Go to volumes -> Create
Pick a namespace
Name the volume
Ensure the storage class is nfs-csi
Set a size
Click create on the bottom
If it ends up bound, that means your storage class (CSI) is working

14. Test an image download
Go to images -> Create
Pick a namespace
Name the image
URL: http://cloud-images.ubuntu.com/releases/jammy/release/ubuntu-22.04-server-cloudimg-amd64.img
Go to storage
Ensure the storage class is "nfs-csi"
Click create on the bottom
If both the volume create and the image create work (that means both the CSI, and the CDI CRD/storageprofile are working)

15. Create a VM with the new storage class and image
Go to Virtual Machines -> Create
Pick a namespace
Name the Vritual Machine
Basics pick cpu/ram and set ssh key if you so choose
Set the volume
Optional add an additonal drive
Set the Network(s)
Click Create
:::warning
The VM status may initially be unschedulable as everything gets created but then should resolve to running in short order
:::
## Completion
:::success
Congratulations you have successfully added a new nfs storage class and validated it!!
:::
If everything was setup correctly you should have a new running vm