[Jason] I am still advocating for ceph-csi to automatically promote on
ControllerPublishVolume and automatically demote on ControllerUnpublishVolume.

[Madhu] Yes we can do it with ceph-csi

[Jason] If the current primary cluster is "dead" and never demotes, what's the
timeout or operator intervention required to force ceph-csi to issue the
equivalent of rbd mirror image promote --force?

[Madhu] do we need an operator or something else to promotes the images forcelly?
[Jason/Shyam] CSI can promote the image based on the some timeout
- Timeout can lose last bit of data if that is taking time to sync
[Erin] Higher level operator is currently promoting images on the secondary on disasater (PoC setups), which is getting into ceph tools etc. to make it happen
[Shyam] Could we trigger force based on a config, that the higher level operator sets?

[Humble] we have to understand the use case some more from BU side.

[Humble]Can we make use of `watcher` before demoting an image?

[Madhu] if we have watchers we cannot demote the image.

[Humble] How we can do the demotion in case of a network failure?

[Madhu] we need to use volume attach object for demoting

[Madhu] as we are going to create rbd image name based on the pvc name and namespace name, are we going to face any issues if the user create pvc with same name in remote cluster-> as the rbd is promoted in primary cluster hope the user is not able to use the PVC unless its demoted in primary cluster(or there any other issues)

[Madhu] Is there any problem if csi force promotes the images? if yes how we can handle remote cluster dead scenerios?

[Madhu] we are going to promote the image in remote cluster only in case first cluster is dead or in bad state?

[Jason] Is this the end-user expectation? Note that RBD mirroring snapshots are
not the same as user-snapshots and will be auto-deleted by the system when no
longer required.

[Madhu] Is it possible to mirror the kubernetes snapshot as its an rbd image
for us internally?
if only the rbd image is mirror but not its snapshot we
still could create a snapshot from the image during restore operation?

[Shyam] Can image be in demoted state on both clusters?
[Jason] yes

[Shyam] Is last set of changed blocks synced on a demote to the mirror?
[Jason] Yes, on demote a mirror snapshot is created to mirror the last known (good) state, so mirror is all caught up

[Shyam] RBD mirror daemons also have watchers, so the whole scheme of watcher based node stage checks in CSI needs to change
[Jason] Also, watcher in general needs to be removed as such

[Madhu] What happens to remote mirrored image if we delete primary rbd image?
[Jason] Mirrored image is deleted
[Shyam] Any implications for PVC life cycle and mirror lifecycle?

[Madhu] Are user snapshots mirrored?
[Jason] Yes, when a mirror snapshot syncs up to the remote the user snapshots are also synced

[Madhu] If we resize the primary rbd image, will the mirrored rbd image will also be resized?
[Jason] Yes
[Madhu] If resize is invoked on non-primary cluster, what are the consequences?
- We cannot promote it for a resize, as remote site may still be using it
[Shyam] A gitops like workflow may send resize to both clusters, and so a non-primary recieving a resize is feasible
[Jason] As fs resize is required, should it not be in-use on a cluster for resize to be initiated?
[Shyam] I think as it stands resize is split into 2 parts, controler resize and node resize, and the former can be done without an image in-use.
- Need to factor this possibility in