:::danger
The tool should not be Run when the provisioning is going on or else it will leads to incosistency. Make sure you have no cephfs/RBD provisioner pod is running
:::

$ rados listomapkeys csi.volumes.default --pool=replicapool                
csi.volume.pvc-17d02c9c-8aa7-472c-9c79-b14d9b2dfde3
csi.volume.pvc-27ce73fa-ac5f-4283-b5c6-74d13492f1e5

trace.py will help to check the consistency between the kubernetes (PVC,PV) objects and the backend resources. But it doesnot take care of below tasks which we need to do it as a enhancement

* Trace the VolumeSnapshot and VolumeSnapshotContent
* Identify stale resouces in below areas
* Kubernetes
* PVC,PV
* Snapshot,SnapshotContent
* Ceph cluster
* Rados omap
* Rbd Images in all the pools (Rados namespaces)
* Enhancement (images in trash and also ask user to run rbd trash purge?)
* CephFS subvolumes in different subvolume groups

# Kubernetes Stale resources

## PVC and PV
trace.py already as the functionality to trace the PVC and subvolumes/rbd images we need to update/rewrite trace.py to be more verbose in providing the informations like

example:
* Total PVC/PV found in the cluster
* Total stale PVC or PV
* Example commands to clean it
* If its a stale PV, sample commands to cleanup the rados omap and subvolume/rbd image

## VolumeSnapshot and VolumeSnapshotContent
Add functionalitry similar to PVC and PV but instead of PVC and PV we will track the VS and VSC

example:
* Total VS/VSC found in the cluster
* Total stale VS or VSC
* Example commands to clean it
* If its a stale VSC, sample commands to cleanup the rados omap and subvolume/rbd image
* We might need to have a cluster for Parent subvolume

# Ceph Resouce cleanup

# Rados objects tracking and stale Resouce cleanup

## RBD

### PVC/ PV
cephcsi creates a mapping between the RBD image/subvolume UUID to the PV name and also PV name to the RBD image/subvolume UUID

Note:

The instanceID is default to `default` for the rados objects and this can be configurable when deploying the cephcsi and we need to provide options to pass the instance ID when running the Tool

* List all the pools in the cluster
*`rados lspools`
* List all the namespaces in the cluster
*`rbd namespace ls --pool=replicapool`
* List all the keys in the pool/namespace in `csi.volumes.default` object
* `rados listomapkeys csi.volumes.default --pool=replicapool`
* This list contains the PV names. Check for the PV object (if the object present means the omap is not stale and we dont need to check for the rbd image/subvolume as the validation for that is already done as part of PV tracking)
* `kubectl get pv <pv-name> -A`
* If its a stale Resouces log the command to clean it up
* Print the total number of stale resouces found?
* List all the keys in the pool/namespace
* `rados ls --pool=replicapool`
* Consider the keys starting with `csi.volume.`
* List all the keys and values for Each of the keys found in above command
* `rados listomapvals csi.volume.02d5906d-fe5b-11eb-bc3a-0242ac110004 --pool=replicapool`
* Validate the PV Name found in the above value
* Log the Stale Details and command to cleanup

### Snapshot/SnapshotContent
The above steps remains same for snapshot also but we need to sightly modify the command input

* List all the pools in the cluster `rados lspools`
* List all the namespaces in the cluster
`rbd namespace ls --pool=replicapool`
* List all the keys in the pool/namespace in `csi.snaps.default` object
* `rados listomapkeys csi.snaps.default --pool=replicapool`
* This list contains the VSC names. Check for the VSC object (if the object present means the omap is not stale and we dont need to check for the rbd image/subvolume as the validation for that is already done as part of VS/VSC tracking)
* If its a stale Resouces log the command to clean it up
* Print the total number of stale resouces found?
* List all the keys in the pool/namespace
* `rados ls --pool=replicapool`
* Consider the keys starting with `csi.snap.`
* List all the keys and values for Each of the keys found in above command
* `rados listomapvals csi.snap.02d5906d-fe5b-11eb-bc3a-0242ac110004 --pool=replicapool`
* Validate the VSC Name found in the above value
* Log the Stale Details and command to cleanup

## CephFS

The commands for the rados objects cleanup remains same as for RBD only difference is that we need to pass the rados`namespace` input when running the rados commands

`rados ls --pool=myfs-metadata --namespace=csi`

**Note:** `csi` is the namespace for cephfs rados objects

# Rbd Image cleanup

### Images belong to PV/VSC
* List all the pools in the ceph cluster
* List all the rbd namespace in the pool
* List all the images in the pool/namespace
* Check the image has any entry in the rados omap
* Look for stale rbd snapshots on the image?
* List the total number of stale Resouces found
* List the command to cleanup the resouce

# CephFS cleanup

### Images belong to PV/VSC
* List all the filesystem in the ceph cluster
* List all the subvolume group in the filesystem
* List all the subvolumes in the filesystem and subvolume group
* Check the subvolume has any entry in the rados omap
* List the snapshots on the subvolume
* Check for the snapshot entry in the rados omap
* List the total number of stale Resouces found
* List the command to cleanup the resouce

:::warning
Note that a ceph cluster can be shared with multiple csi drivers or different kubernetes clustes. When doing the resource cleanup you need to make sure the stale resources doesnt belongs to any of the kubernetes clusters
:::