Velero issue: support NFS/local-volume/host-path volume as target BSL: https://github.com/vmware-tanzu/velero/issues/8190

vmware-tanzu pr for kopia to lvp https://github.com/vmware-tanzu/velero/pull/8358

# dev aliases

alias ghcr_notag='echo ghcr.io/kaovilai/$(basename $PWD)'
alias current-branch='git branch --show-current'
alias rev-sha-short='git rev-parse --short HEAD'
alias cluster-arch-only='kubectl get nodes -o jsonpath='\''{range .items[0]}{.status.nodeInfo.architecture}{end}'\'
# run this from velero repo dir
alias velero-makecontainer-cluster-arch='make container IMAGE=$(ghcr_notag) VERSION=$(current-branch)-$(rev-sha-short)-$(cluster-arch-only) && docker push $(ghcr_notag):$(current-branch)-$(rev-sha-short)-$(cluster-arch-only) && echo $(ghcr_notag):$(current-branch)-$(rev-sha-short)-$(cluster-arch-only)'
alias velero-goruninstall-node-agent='velero-makecontainer-cluster-arch && go run cmd/velero/velero.go install --use-node-agent --image=$(ghcr_notag):$(current-branch)-$(rev-sha-short)-$(cluster-arch-only) --provider aws --bucket $AWS_BUCKET --prefix velero --plugins velero/velero-plugin-for-aws:latest --secret-file $AWS_SECRET_FILE'
alias kubectl-patch-velero-debug="kubectl patch -n \$VELERO_NS deployment.apps/velero --type=json -p=\"[{\"op\": \"add\", \"path\": \"/spec/template/spec/containers/0/args/-\", \"value\": \"--log-level=debug\"}]\""

# Prereq
`crc start`: [crc.dev](https://crc.dev)

checkout branch https://github.com/vmware-tanzu/velero/pull/8358

export VELERO_NS=openshift-adp
oc delete ns $VELERO_NS --wait
velero-goruninstall-node-agent

echo patch log level to debug
kubectl-patch-velero-debug


echo verify velero deployment contains under args --uploader-type=kopia

echo "apiVersion: v1
kind: ConfigMap
metadata:
  name: local-volume-provider-config
  namespace: $VELERO_NS
  labels:
    velero.io/plugin-config: \"\"
    replicated.com/nfs: ObjectStore
    replicated.com/pvc: ObjectStore
    replicated.com/hostpath: ObjectStore
data:
  # match securityContext of velero pod
  securityContextRunAsUser: \"0\"
  securityContextRunAsGroup: \"0\"
  securityContextFsGroup: \"0\"
  fileserverImage: replicated/local-volume-provider:v0.6.7
" | oc create -f -

sleep 2


velero plugin add replicated/local-volume-provider:v0.6.7 --confirm
oc wait -n $VELERO_NS --for=condition=ready deployment/velero

echo "apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: pvc
  namespace: $VELERO_NS
spec:
  backupSyncPeriod: 2m0s
  provider: replicated.com/pvc
  objectStorage:
    bucket: pvc-snapshots
  config:
    storageSize: 1Gi
" | oc create -f -

oc create -f https://raw.githubusercontent.com/openshift/oadp-operator/refs/heads/master/tests/e2e/sample-applications/minimal-8csivol/minimal-3csivol.yaml

volume should be readwritemany as we will need to access it from velero and node-agent pods
# run backup

velero backup create mongo-backup --storage-location=pvc --default-volumes-to-fs-backup

<details>
<summary>past investigations</summary>
# How [they do it](https://github.com/vmware-tanzu/velero/blob/b5c9921ee89e1b1e03b4958ca6b8098ce2ee1b65/pkg/repository/config/config.go#L61-L90) for restic

They simply mount a volume to restic daemonset, ask that bsl.spec.resticRepoPrefix be pointed to there.. and following velero repo code is responsible for "destination" part. By providing a `/local/path/to/somewhere` this is how restic can copy to local volumes all via BSL definition.

	if repoPrefix := location.Spec.Config["resticRepoPrefix"]; repoPrefix != "" {
		return repoPrefix, nil
	}

	switch backendType {
	case AWSBackend:
		var url string
		// non-AWS, S3-compatible object store
		if s3Url := location.Spec.Config["s3Url"]; s3Url != "" {
			url = strings.TrimSuffix(s3Url, "/")
		} else {
			var err error
			region := location.Spec.Config["region"]
			if region == "" {
				region, err = getAWSBucketRegion(bucket)
			}
			if err != nil {
				return "", errors.Wrapf(err, "failed to detect the region via bucket: %s", bucket)
			}
			url = fmt.Sprintf("s3-%s.amazonaws.com", region)
		}

		return fmt.Sprintf("s3:%s/%s", url, path.Join(bucket, prefix)), nil
	case AzureBackend:
		return fmt.Sprintf("azure:%s:/%s", bucket, prefix), nil
	case GCPBackend:
		return fmt.Sprintf("gs:%s:/%s", bucket, prefix), nil
	}

	return "", errors.Errorf("invalid backend type %s, provider %s", backendType, location.Spec.Provider)

Where kopia destination storage is defined in velero code

TBD
sseago mentions veleropod:~/...
same backing store with s3 overheard (via minio) and without via pvc in lvp

nfs is a later followup

If there is a way to make this work via LVP without velero repo changes?

Y/N

Performance results.

# Note on how Kubernetes volumes works

A kubernetes cluster typically has a control plane and one or more nodes.
Control plane makes scheduling decisions about the cluster and responding to events. It has following [components](https://kubernetes.io/docs/concepts/architecture/).

- kubeapi-server: horizontally scalable front end exposing k8s api
- etcd: HA cluster state
- kube-scheduler: watch for pods with no assigned nodes and select nodes to run them
- kube-controller-manager: a compiled single binary that has many controllers
- cloud-controller-manager: [something something cloud specific](https://kubernetes.io/docs/concepts/architecture/#cloud-controller-manager)

When a pod is created, and kube-scheduler assigns a node to pod.

Node has a component called kubelet which will ensure that containers defined in pod are running.

kubelet has a flag `--root-dir` which is *Directory path for managing kubelet files (volume mounts, etc).* Default is `/var/lib/kubelet`

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go#L348
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_volumes.go
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_getters.go#L159

const (
	DefaultKubeletPodsDirName                = "pods"
    DefaultKubeletVolumesDirName             = "volumes"
)
// getPodVolumesDir returns the full path to the per-pod data directory under
// which volumes are created for the specified pod.  This directory may not
// exist if the pod does not exist.
func (kl *Kubelet) getPodVolumesDir(podUID types.UID) string {
	return filepath.Join(kl.getPodDir(podUID), config.DefaultKubeletVolumesDirName)
}

// getPodDir returns the full path to the per-pod directory for the pod with
// the given UID.
func (kl *Kubelet) getPodDir(podUID types.UID) string {
	return filepath.Join(kl.getPodsDir(), string(podUID))
}

// getPodsDir returns the full path to the directory under which pod
// directories are created.
func (kl *Kubelet) getPodsDir() string {
	return filepath.Join(kl.getRootDir(), config.DefaultKubeletPodsDirName)
}

// getRootDir returns the full path to the directory under which kubelet can
// store data.  These functions are useful to pass interfaces to other modules
// that may need to know where to write data without getting a whole kubelet
// instance.
func (kl *Kubelet) getRootDir() string {
	return kl.rootDirectory
}

From this we can gather that a volume for a given pod UID is in this format

<kubeletRoot>/pods/<pod-uid>/volumes/

Digging further.. a kubelet will [create three data dirs](https://github.com/kubernetes/kubernetes/blob/fccbbf324d9c773fa42d4da279d9667195755a4f/pkg/kubelet/kubelet_pods.go#L979) for a given pod

<kubeletRoot>/pods/<pod-uid>
<kubeletRoot>/pods/<pod-uid>/volumes/
<kubeletRoot>/pods/<pod-uid>/plugins/

then waits for volumes to attach and mount

https://github.com/kubernetes/kubernetes/blob/fccbbf324d9c773fa42d4da279d9667195755a4f/pkg/kubelet/kubelet.go#L1730C1-L1745C74

// The workflow is:
//   - If the pod is being created, record pod worker start latency
//   - Call generateAPIPodStatus to prepare an v1.PodStatus for the pod
//   - If the pod is being seen as running for the first time, record pod
//     start latency
//   - Update the status of the pod in the status manager
//   - Stop the pod's containers if it should not be running due to soft
//     admission
//   - Ensure any background tracking for a runnable pod is started
//   - Create a mirror pod if the pod is a static pod, and does not
//     already have a mirror pod
//   - Create the data directories for the pod if they do not exist
//   - Wait for volumes to attach/mount
//   - Fetch the pull secrets for the pod
//   - Call the container runtime's SyncPod callback
//   - Update the traffic shaping for the pod's ingress and egress limits

https://danielmangum.com/posts/where-does-the-kubelet-mount-volumes/

From this blog

path appears to be

<kubeletRoot>/pods/<pod-uid>/volumes/<driver-name>/<volume-name>/<contents of volume>

## [Minimum Requirements (for Developing and Deploying a CSI driver for Kubernetes)](chrome-extension://pcmpcfapbekmbjjkdalcgopdkipoggdi/_generated_background_page.html#minimum-requirements-for-developing-and-deploying-a-csi-driver-for-kubernetes)

Kubernetes is as minimally prescriptive about packaging and deployment of a CSI Volume Driver as possible.

The only requirements are around how Kubernetes (master and node) components find and communicate with a CSI driver.

Specifically, the following is dictated by Kubernetes regarding CSI:

- Kubelet to CSI Driver Communication
- Kubelet directly issues CSI calls (like `NodeStageVolume`, `NodePublishVolume`, etc.) to CSI drivers via a Unix Domain Socket to mount and unmount volumes.
- Kubelet discovers CSI drivers (and the Unix Domain Socket to use to interact with a CSI driver) via the [kubelet plugin registration mechanism](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/pluginmanager/pluginwatcher/README.md).
- Therefore, all CSI drivers deployed on Kubernetes MUST register themselves using the kubelet plugin registration mechanism on each supported node.
- Master to CSI Driver Communication
- Kubernetes master components do not communicate directly (via a Unix Domain Socket or otherwise) with CSI drivers.
- Kubernetes master components interact only with the Kubernetes API.
- Therefore, CSI drivers that require operations that depend on the Kubernetes API (like volume create, volume attach, volume snapshot, etc.) MUST watch the Kubernetes API and trigger the appropriate CSI operations against it.

Because these requirements are minimally prescriptive, CSI driver developers are free to implement and deploy their drivers as they see fit.

_That said, to ease development and deployment, the mechanism described below is recommended._

## [Recommended Mechanism (for Developing and Deploying a CSI driver for Kubernetes)](chrome-extension://pcmpcfapbekmbjjkdalcgopdkipoggdi/_generated_background_page.html#recommended-mechanism-for-developing-and-deploying-a-csi-driver-for-kubernetes)

The Kubernetes development team has established a "Recommended Mechanism" for developing, deploying, and testing CSI Drivers on Kubernetes. It aims to reduce boilerplate code and simplify the overall process for CSI Driver developers.

This "Recommended Mechanism" makes use of the following components:

- Kubernetes CSI [Sidecar Containers](chrome-extension://pcmpcfapbekmbjjkdalcgopdkipoggdi/_generated_background_page.htmlsidecar-containers.html)
- Kubernetes CSI [objects](chrome-extension://pcmpcfapbekmbjjkdalcgopdkipoggdi/_generated_background_page.htmlcsi-objects.html)
- CSI [Driver Testing](chrome-extension://pcmpcfapbekmbjjkdalcgopdkipoggdi/_generated_background_page.htmltesting-drivers.html) tools

To implement a CSI driver using this mechanism, a CSI driver developer should:

1. Create a containerized application implementing the _Identity_, _Node_, and optionally the _Controller_ services described in the [CSI specification](https://github.com/container-storage-interface/spec/blob/master/spec.md#rpc-interface) (the CSI driver container).
- See [Developing CSI Driver](chrome-extension://pcmpcfapbekmbjjkdalcgopdkipoggdi/_generated_background_page.htmldeveloping.html) for more information.
2. Unit test it using csi-sanity.
- See [Driver - Unit Testing](chrome-extension://pcmpcfapbekmbjjkdalcgopdkipoggdi/_generated_background_page.htmlunit-testing.html) for more information.
3. Define Kubernetes API YAML files that deploy the CSI driver container along with appropriate sidecar containers.
- See [Deploying in Kubernetes](chrome-extension://pcmpcfapbekmbjjkdalcgopdkipoggdi/_generated_background_page.htmldeploying.html) for more information.
4. Deploy the driver on a Kubernetes cluster and run end-to-end functional tests on it.
- See [Driver - Functional Testing](chrome-extension://pcmpcfapbekmbjjkdalcgopdkipoggdi/_generated_background_page.htmlfunctional-testing.html)
</details>