# Analysis of kubelet volume manager to understand what resources and their state would be post NodeUnstage is called for a CSI volume

## Reconciler: pkg/kubelet/volumemanager/reconciler/reconciler.go

The volumemanager reconciler invokes 3 routines, in order, to unmount, mountOrAttach, and detach devices. On analysis this is what is found:
- mountOrAttachVolumes creates a VolumeAttachement resource prior to staging the volume on the node via a NodeStageVolume CSI call
- Volumes are unmounted from a pod using CSI NodeUnpublishVolume calls
- Volumes are then detached from the node invoking a NodeUnstageVolume call
- Finally the VolumeAttachement resource is deleted for the volume on the node

As a result, to determine if a deleted PVC is no longer in use, we could:
- Wait for the PVC kubernetes.io/pvc-protection finalizer to be deleted
- Wait for all VolumeAttachements for the PV bound to the PVC

### Questions:
- Does any of this change across RWX and RWO volumes or Block mode volumes?
- Does the understanding look correct as per the analysis below?

## Code references/tracing to establish the above behaviour:

1. unmountVolumes

  for _, mountedVolume := range rc.actualStateOfWorld.GetAllMountedVolumes() {
		if !rc.desiredStateOfWorld.PodExistsInVolume(mountedVolume.PodName, mountedVolume.VolumeName) {
			err := rc.operationExecutor.UnmountVolume(mountedVolume.MountedVolume, rc.actualStateOfWorld, rc.kubeletPodsDir)
		}
	}

1. rc.operationExecutor.UnmountVolume ---(invokes)---> GenerateUnmountVolumeFunc
- File: pkg/volume/util/operationexecutor/operation_generator.go
- This generates the following order of operations to execute,

      volumeUnmounter.TearDown()
      actualStateOfWorld.MarkVolumeAsUnmounted(volumeToUnmount.PodName, volumeToUnmount.VolumeName)

2. TearDown ---(invokes)---> TearownAt ---(invokes)---> NodeUnpublishVolume
- File: pkg/volume/csi/csi_mounter.go

3. MarkVolumeAsUnmounted ---(invokes)---> DeletePodFromVolume
- File: pkg/kubelet/volumemanager/cache/actual_state_of_world.go
- Remove pod from mountedPods of volume object `asw.attachedVolumes[volumeName]`

2. mountOrAttachVolumes

This basically performs the following in order as needed (conditionally),
- Create VolumeAttachment -> NodeStateVolume -> NodePublishVolume -> NodeUnpublishVolume -> NodeUnstageVolume -> delete VolumeAttachment

3. unmountDetachDevices

This is done in 2 phases, with phase one unstaging the volumes and the next stage deleting the VolumeAttachment:

	for _, attachedVolume := range rc.actualStateOfWorld.GetUnmountedVolumes() {
		if !rc.desiredStateOfWorld.VolumeExists(attachedVolume.VolumeName) &&
			!rc.operationExecutor.IsOperationPending(attachedVolume.VolumeName, nestedpendingoperations.EmptyUniquePodName, nestedpendingoperations.EmptyNodeName) {
			
			if attachedVolume.DeviceMayBeMounted() {
				err := rc.operationExecutor.UnmountDevice(attachedVolume.AttachedVolume, rc.actualStateOfWorld, rc.hostutil)
			} else {
				if rc.controllerAttachDetachEnabled || !attachedVolume.PluginIsAttachable {
					rc.actualStateOfWorld.MarkVolumeAsDetached(attachedVolume.VolumeName, attachedVolume.NodeName)
				} else {
					err := rc.operationExecutor.DetachVolume(attachedVolume.AttachedVolume, false /* verifySafeToDetach */, rc.actualStateOfWorld)
				}
			}
		}
	}

1. UnmountDevice ---(invokes)---> GenerateUnmountDeviceFunc
- This is the first phase to unstage the volume
- This generates the following order of operations to execute,

    volumeDeviceUnmounter.UnmountDevice(deviceMountPath)
    actualStateOfWorld.MarkDeviceAsUnmounted(deviceToDetach.VolumeName)

2. volumeDeviceUnmounter.UnmountDevice ---(invokes)---> NodeUnstageVolume

3. MarkDeviceAsUnmounted ---(invokes)---> SetDeviceMountState
- Marks volume object `asw.attachedVolumes[volumeName]` as unmounted, for detach to proceed in the next iteration

4. DetachVolume ---(invokes)---> GenerateDetachVolumeFunc
- This is the second phase to remove the VolumeAttachment
- This generates the following order of operations to execute,

        volumeDetacher.Detach(volumeName, volumeToDetach.NodeName)
        actualStateOfWorld.MarkVolumeAsDetached(volumeToDetach.VolumeName, volumeToDetach.NodeName)

5. volumeDetacher.Detach
- `c.k8s.StorageV1().VolumeAttachments().Delete(context.TODO(), attachID, metav1.DeleteOptions{})`
- `c.waitForVolumeDetachmentWithLister(volID, attachID, c.watchTimeout)` (analysis **TBD**)

6. MarkVolumeAsDetached
- Deletes the volume from `delete(asw.attachedVolumes, volumeName)`