# Replica 1 Non Resilient Pools
## Why Ceph Replica 1 Non Resilient Pools
There are applications that manage their resiliency on the application level. These applications aren’t concerned with data loss since they have multiple instances of the application running on top of independent replicas of the data at the application layer.
So Replica 1 Non Resilient pools provide a storage solution that offers Ceph replica 1 for RBD, avoiding unnecessary replication when not required by the application.
So if you have an application where data resiliency is managed by your application, you can try ceph replica 1 to avoid unnecesssary replication of data.
## Important
**For this to work with Local Storage Operator, We need to have at least one additional usable disks on each node of the cluster. Otherwise after the patch for the feature the replica-1 osds will not come up & storagecluster will forever be in progressing.**
## Setup
Wait for the storagecluster to be ready
```
~ % oc get storagecluster
NAME AGE PHASE EXTERNAL CREATED AT VERSION
ocs-storagecluster 7m30s Ready 2024-02-05T13:54:15Z 4.15.0
```
then Patch the storagecluster to enable Non-Resilient pools
```
~ % oc patch storagecluster ocs-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/managedResources/cephNonResilientPools/enable", "value": true }]'
```
After patching storagecluster would be progressing again,
Wait for the storagecluster to reach the ready stage.
```
~ % oc get storagecluster
NAME AGE PHASE EXTERNAL CREATED AT VERSION
ocs-storagecluster 10m Ready 2024-02-05T13:56:15Z 4.15.0
```
## Inner Details
### CephBlockPools
```
~ % oc get cephblockpools
NAME PHASE
ocs-storagecluster-cephblockpool Ready
ocs-storagecluster-cephblockpool-us-east-1a Ready
ocs-storagecluster-cephblockpool-us-east-1b Ready
ocs-storagecluster-cephblockpool-us-east-1c Ready
```
The 1st one is the default cephblockpool with replica 3, The 3 other pools with the failure domain names at the end are the replica-1 cephblockpools. One replica-1 cephblockpool each is created for each failure domain.
### Storageclasses
```
~ % oc get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer true 104m
gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 104m
gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 104m
ocs-storagecluster-ceph-non-resilient-rbd openshift-storage.rbd.csi.ceph.com Delete WaitForFirstConsumer true 46m
ocs-storagecluster-ceph-rbd openshift-storage.rbd.csi.ceph.com Delete Immediate true 52m
ocs-storagecluster-cephfs openshift-storage.cephfs.csi.ceph.com Delete Immediate true 52m
openshift-storage.noobaa.io openshift-storage.noobaa.io/obc Delete Immediate false 50m
```
ocs-storagecluster-ceph-non-resilient-rbd is the new storageclass for non-resilient topology constrained pools
```
~ % oc get storageclass ocs-storagecluster-ceph-non-resilient-rbd -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
description: 'Ceph Non Resilient Pools : Provides RWO Filesystem volumes, and
RWO and RWX Block volumes'
creationTimestamp: "2022-09-08T15:50:31Z"
name: ocs-storagecluster-ceph-non-resilient-rbd
resourceVersion: "69004"
uid: e59faabe-688a-45fb-bc42-dccac72a4efd
parameters:
clusterID: openshift-storage
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
csi.storage.k8s.io/fstype: ext4
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
imageFeatures: layering,deep-flatten,exclusive-lock,object-map,fast-diff
imageFormat: "2"
pool: ocs-storagecluster-cephblockpool
topologyConstrainedPools: |-
[
{
"poolName": "ocs-storagecluster-cephblockpool-us-east-1b",
"domainSegments": [
{
"domainLabel": "zone",
"value": "us-east-1b"
}
]
},
{
"poolName": "ocs-storagecluster-cephblockpool-us-east-1c",
"domainSegments": [
{
"domainLabel": "zone",
"value": "us-east-1c"
}
]
},
{
"poolName": "ocs-storagecluster-cephblockpool-us-east-1a",
"domainSegments": [
{
"domainLabel": "zone",
"value": "us-east-1a"
}
]
}
]
provisioner: openshift-storage.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
```
### OSD Pods
```
~ % oc get pods | grep osd
rook-ceph-osd-0-6dc76777bc-snhnm 2/2 Running 0 9m50s
rook-ceph-osd-1-768bdfdc4-h5n7k 2/2 Running 0 9m48s
rook-ceph-osd-2-69878645c4-bkdlq 2/2 Running 0 9m37s
rook-ceph-osd-3-64c44d7d76-zfxq9 2/2 Running 0 5m23s
rook-ceph-osd-4-654445b78f-nsgjb 2/2 Running 0 5m23s
rook-ceph-osd-5-5775949f57-vz6jp 2/2 Running 0 5m22s
rook-ceph-osd-prepare-ocs-deviceset-gp2-0-data-0x6t87-59swf 0/1 Completed 0 10m
rook-ceph-osd-prepare-ocs-deviceset-gp2-1-data-0klwr7-bk45t 0/1 Completed 0 10m
rook-ceph-osd-prepare-ocs-deviceset-gp2-2-data-0mk2cz-jx7zv 0/1 Completed 0 10m
rook-ceph-osd-prepare-us-east-1a-data-0l874p-2tc99 0/1 Completed 0 5m44s
rook-ceph-osd-prepare-us-east-1b-data-0q7t7h-swgzk 0/1 Completed 0 5m43s
rook-ceph-osd-prepare-us-east-1c-data-07nv5w-77mmw 0/1 Completed 0 5m44s
```
We can see after patching for replica-1 there are 3 new osd-prepare pods for replica-1 & 3 new osds after that.
### Ceph details
```
sh-5.1$ ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 12.00000 - 12 TiB 302 MiB 199 MiB 0 B 103 MiB 12 TiB 0.00 1.00 - root default
-5 12.00000 - 12 TiB 302 MiB 199 MiB 0 B 103 MiB 12 TiB 0.00 1.00 - region us-east-1
-14 4.00000 - 4 TiB 100 MiB 66 MiB 0 B 34 MiB 4.0 TiB 0.00 1.00 - zone us-east-1a
-13 2.00000 - 2 TiB 58 MiB 32 MiB 0 B 25 MiB 2.0 TiB 0.00 1.15 - host ocs-deviceset-gp3-csi-2-data-05ggvh
2 ssd 2.00000 1.00000 2 TiB 58 MiB 32 MiB 0 B 25 MiB 2.0 TiB 0.00 1.15 52 up osd.2
-41 2.00000 - 2 TiB 43 MiB 33 MiB 0 B 9.1 MiB 2.0 TiB 0.00 0.85 - host us-east-1a-data-0knkjg
3 us-east-1a 2.00000 1.00000 2 TiB 43 MiB 33 MiB 0 B 9.1 MiB 2.0 TiB 0.00 0.85 61 up osd.3
-4 4.00000 - 4 TiB 101 MiB 66 MiB 0 B 34 MiB 4.0 TiB 0.00 1.00 - zone us-east-1b
-3 2.00000 - 2 TiB 71 MiB 46 MiB 0 B 26 MiB 2.0 TiB 0.00 1.42 - host ocs-deviceset-gp3-csi-1-data-0d98jz
0 ssd 2.00000 1.00000 2 TiB 71 MiB 46 MiB 0 B 26 MiB 2.0 TiB 0.00 1.42 57 up osd.0
-51 2.00000 - 2 TiB 29 MiB 21 MiB 0 B 8.6 MiB 2.0 TiB 0.00 0.58 - host us-east-1b-data-0jp5x5
5 us-east-1b 2.00000 1.00000 2 TiB 29 MiB 21 MiB 0 B 8.6 MiB 2.0 TiB 0.00 0.58 57 up osd.5
-10 4.00000 - 4 TiB 101 MiB 67 MiB 0 B 34 MiB 4.0 TiB 0.00 1.01 - zone us-east-1c
-9 2.00000 - 2 TiB 64 MiB 39 MiB 0 B 25 MiB 2.0 TiB 0.00 1.27 - host ocs-deviceset-gp3-csi-0-data-0mgxqb
1 ssd 2.00000 1.00000 2 TiB 64 MiB 39 MiB 0 B 25 MiB 2.0 TiB 0.00 1.27 57 up osd.1
-46 2.00000 - 2 TiB 37 MiB 28 MiB 0 B 8.9 MiB 2.0 TiB 0.00 0.74 - host us-east-1c-data-0lxvf6
4 us-east-1c 2.00000 1.00000 2 TiB 37 MiB 28 MiB 0 B 8.9 MiB 2.0 TiB 0.00 0.74 58 up osd.4
TOTAL 12 TiB 302 MiB 199 MiB 0 B 103 MiB 12 TiB 0.00
MIN/MAX VAR: 0.58/1.42 STDDEV: 0
```
```
sh-5.1$ ceph osd pool ls detail
pool 1 'ocs-storagecluster-cephblockpool' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 74 lfor 0/0/35 flags hashpspool,selfmanaged_snaps stripe_width 0 target_size_ratio 0.49 application rbd
pool 2 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 19 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr
pool 3 'ocs-storagecluster-cephfilesystem-metadata' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 75 lfor 0/0/35 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
pool 4 'ocs-storagecluster-cephfilesystem-data0' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 76 lfor 0/0/37 flags hashpspool stripe_width 0 target_size_ratio 0.49 application cephfs
pool 5 'ocs-storagecluster-cephblockpool-us-east-1a' replicated size 1 min_size 1 crush_rule 8 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 115 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 6 'ocs-storagecluster-cephblockpool-us-east-1b' replicated size 1 min_size 1 crush_rule 10 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 101 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 7 'ocs-storagecluster-cephblockpool-us-east-1c' replicated size 1 min_size 1 crush_rule 12 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 110 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
```
## Testing the non-resilient storageclass topology aware provisioning
#### Prepare a test Namespace
```
~ % oc create ns test
~ % oc project test
~ % oc create sa test
~ % oc adm policy add-scc-to-user -z test privileged
```
#### Create a pvc
```
~ % cat <<EOF | oc create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: non-resilient-rbd-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: ocs-storagecluster-ceph-non-resilient-rbd
EOF
```
The pvc would be in pending state, waiting for a consumer
```
~ % oc get pvc | grep non-resilient-rbd-pvc
non-resilient-rbd-pvc Pending ocs-storagecluster-ceph-non-resilient-rbd 16s
```
#### Create a pod to consume the pvc
```
~ % cat <<EOF | oc create -f -
apiVersion: v1
kind: Pod
metadata:
name: task-pv-pod
spec:
nodeSelector:
#Change according to failure domain
topology.kubernetes.io/zone: us-east-1a
volumes:
- name: task-pv-storage
persistentVolumeClaim:
claimName: non-resilient-rbd-pvc
containers:
- name: task-pv-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: task-pv-storage
securityContext:
privileged: true
EOF
```
```
~ % oc get pods | grep task
task-pv-pod 1/1 Running 0 57s
```
#### Check the pvc status now & the newly created pv also
```
~ % oc get pvc | grep non-resilient-rbd-pvc
non-resilient-rbd-pvc Bound pvc-d1a26969-a15c-4e60-8f76-847d1fdd6041 1Gi RWO ocs-storagecluster-ceph-non-resilient-rbd 3m49s
~ % oc get pv | grep non-resilient-rbd-pvc
pvc-d1a26969-a15c-4e60-8f76-847d1fdd6041 1Gi RWO Delete Bound openshift-storage/non-resilient-rbd-pvc ocs-storagecluster-ceph-non-resilient-rbd 2m18s
```
#### Check the toolbox pod to see rbd image for new pv is in which pool
```
sh-4.4$ rbd ls ocs-storagecluster-cephblockpool-us-east-1a
csi-vol-937ee9e9-2f99-11ed-b2d0-0a580a83001a
```
#### Match this imageName to the imageName in the pv created above
```
~ % oc get pv pvc-d1a26969-a15c-4e60-8f76-847d1fdd6041 -o=jsonpath='{.spec.csi.volumeAttributes}' | jq
{
"clusterID": "openshift-storage",
"imageFeatures": "layering,deep-flatten,exclusive-lock,object-map,fast-diff",
"imageFormat": "2",
"imageName": "csi-vol-937ee9e9-2f99-11ed-b2d0-0a580a83001a",
"journalPool": "ocs-storagecluster-cephblockpool",
"pool": "ocs-storagecluster-cephblockpool-us-east-1a",
"storage.kubernetes.io/csiProvisionerIdentity": "1662652453217-8081-openshift-storage.rbd.csi.ceph.com",
"topologyConstrainedPools": "[\n {\n \"poolName\": \"ocs-storagecluster-cephblockpool-us-east-1a\",\n \"domainSegments\": [\n {\n \"domainLabel\": \"zone\",\n \"value\": \"us-east-1a\"\n }\n ]\n },\n {\n \"poolName\": \"ocs-storagecluster-cephblockpool-us-east-1b\",\n \"domainSegments\": [\n {\n \"domainLabel\": \"zone\",\n \"value\": \"us-east-1b\"\n }\n ]\n },\n {\n \"poolName\": \"ocs-storagecluster-cephblockpool-us-east-1c\",\n \"domainSegments\": [\n {\n \"domainLabel\": \"zone\",\n \"value\": \"us-east-1c\"\n }\n ]\n }\n]"
}
```
## Scaling up the number of non-resilient osds
By default 1 osd for each failure domain is added.
Patch the storagecluster to increase the number of on-resilent osds per failure domain
Replace 2 in the below patch to the desired number
```
~ $ oc patch storagecluster ocs-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/managedResources/cephNonResilientPools/count", "value": 2 }]'
```
Storagecluster will get into progressing state, wait for it to be ready
```
~ $ oc get storagecluster
NAME AGE PHASE EXTERNAL CREATED AT VERSION
ocs-storagecluster 76m Ready 2024-02-19T05:21:07Z 4.15.0
```
### New OSD pods
```
~ $ oc get pods | grep osd
rook-ceph-osd-0-df7f9c69c-dqwqt 2/2 Running 0 70m
rook-ceph-osd-1-cdc97fd77-7xwmm 2/2 Running 0 70m
rook-ceph-osd-2-6d559549b9-hg7nx 2/2 Running 0 70m
rook-ceph-osd-3-786c94d5d4-x4kc5 2/2 Running 0 60m
rook-ceph-osd-4-58d97f6b77-gz8nx 2/2 Running 0 60m
rook-ceph-osd-5-5d8db7597c-6gbxm 2/2 Running 0 60m
rook-ceph-osd-6-7fc87ffdc7-c5qfz 2/2 Running 0 8m42s
rook-ceph-osd-7-868c5f4c68-t7pbn 2/2 Running 0 8m38s
rook-ceph-osd-8-f94985485-grbvx 2/2 Running 0 8m36s
rook-ceph-osd-prepare-6517706ddfc97525f6914e030258209e-rzvzv 0/1 Completed 0 70m
rook-ceph-osd-prepare-c11cfb23602a82cdc79e25b83369cc54-479kf 0/1 Completed 0 70m
rook-ceph-osd-prepare-eb4107c1221a8047a7746be1c341ecc2-rrtrd 0/1 Completed 0 70m
rook-ceph-osd-prepare-us-east-1a-data-0knkjg-gb7bq 0/1 Completed 0 60m
rook-ceph-osd-prepare-us-east-1a-data-1v5fwd-45ls9 0/1 Completed 0 8m57s
rook-ceph-osd-prepare-us-east-1b-data-0jp5x5-gb5qz 0/1 Completed 0 60m
rook-ceph-osd-prepare-us-east-1b-data-1hfftf-x5b4g 0/1 Completed 0 8m56s
rook-ceph-osd-prepare-us-east-1c-data-0lxvf6-qpw6k 0/1 Completed 0 60m
rook-ceph-osd-prepare-us-east-1c-data-1rm7lr-msb6h 0/1 Completed 0 8m56s
```
### ceph details
```
sh-5.1$ ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 18.00000 - 18 TiB 430 MiB 274 MiB 0 B 155 MiB 18 TiB 0.00 1.00 - root default
-5 18.00000 - 18 TiB 430 MiB 274 MiB 0 B 155 MiB 18 TiB 0.00 1.00 - region us-east-1
-14 6.00000 - 6 TiB 149 MiB 91 MiB 0 B 58 MiB 6.0 TiB 0.00 1.04 - zone us-east-1a
-13 2.00000 - 2 TiB 48 MiB 15 MiB 0 B 32 MiB 2.0 TiB 0.00 1.00 - host ocs-deviceset-gp3-csi-2-data-05ggvh
2 ssd 2.00000 1.00000 2 TiB 48 MiB 15 MiB 0 B 32 MiB 2.0 TiB 0.00 1.00 33 up osd.2
-41 2.00000 - 2 TiB 53 MiB 37 MiB 0 B 16 MiB 2.0 TiB 0.00 1.11 - host us-east-1a-data-0knkjg
3 us-east-1a 2.00000 1.00000 2 TiB 53 MiB 37 MiB 0 B 16 MiB 2.0 TiB 0.00 1.11 44 up osd.3
-61 2.00000 - 2 TiB 48 MiB 39 MiB 0 B 9.5 MiB 2.0 TiB 0.00 1.01 - host us-east-1a-data-1v5fwd
7 us-east-1a 2.00000 1.00000 2 TiB 48 MiB 39 MiB 0 B 9.5 MiB 2.0 TiB 0.00 1.01 36 up osd.7
-4 6.00000 - 6 TiB 132 MiB 91 MiB 0 B 41 MiB 6.0 TiB 0.00 0.92 - zone us-east-1b
-3 2.00000 - 2 TiB 65 MiB 43 MiB 0 B 22 MiB 2.0 TiB 0.00 1.36 - host ocs-deviceset-gp3-csi-1-data-0d98jz
0 ssd 2.00000 1.00000 2 TiB 65 MiB 43 MiB 0 B 22 MiB 2.0 TiB 0.00 1.36 35 up osd.0
-51 2.00000 - 2 TiB 35 MiB 24 MiB 0 B 10 MiB 2.0 TiB 0.00 0.73 - host us-east-1b-data-0jp5x5
5 us-east-1b 2.00000 1.00000 2 TiB 35 MiB 24 MiB 0 B 10 MiB 2.0 TiB 0.00 0.73 40 up osd.5
-56 2.00000 - 2 TiB 32 MiB 24 MiB 0 B 8.4 MiB 2.0 TiB 0.00 0.67 - host us-east-1b-data-1hfftf
6 us-east-1b 2.00000 1.00000 2 TiB 32 MiB 24 MiB 0 B 8.4 MiB 2.0 TiB 0.00 0.67 38 up osd.6
-10 6.00000 - 6 TiB 149 MiB 92 MiB 0 B 57 MiB 6.0 TiB 0.00 1.04 - zone us-east-1c
-9 2.00000 - 2 TiB 73 MiB 40 MiB 0 B 32 MiB 2.0 TiB 0.00 1.52 - host ocs-deviceset-gp3-csi-0-data-0mgxqb
1 ssd 2.00000 1.00000 2 TiB 73 MiB 40 MiB 0 B 32 MiB 2.0 TiB 0.00 1.52 38 up osd.1
-46 2.00000 - 2 TiB 33 MiB 18 MiB 0 B 16 MiB 2.0 TiB 0.00 0.70 - host us-east-1c-data-0lxvf6
4 us-east-1c 2.00000 1.00000 2 TiB 33 MiB 18 MiB 0 B 16 MiB 2.0 TiB 0.00 0.70 40 up osd.4
-66 2.00000 - 2 TiB 43 MiB 34 MiB 0 B 8.8 MiB 2.0 TiB 0.00 0.90 - host us-east-1c-data-1rm7lr
8 us-east-1c 2.00000 1.00000 2 TiB 43 MiB 34 MiB 0 B 8.8 MiB 2.0 TiB 0.00 0.90 38 up osd.8
TOTAL 18 TiB 430 MiB 274 MiB 0 B 155 MiB 18 TiB 0.00
MIN/MAX VAR: 0.67/1.52 STDDEV: 0
```
## Scaling up the size of the non-resilient osds
### Setting the volumeclaimtemplate for the osds
Initially replica-1 osds use the volumeclaimtemplate by copying the one from the first storagedeviceset.
to use a diferent volumeclaimtemplate edit the storagecluster to add the spec which looks like below at spec/managedResources/cephNonResilientPools/
```
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Ti
storageClassName: gp3-csi
volumeMode: Block
```
### Restarted OSD pods
```
~ $ oc get pods | grep osd
rook-ceph-osd-0-94d78d794-7nqhs 2/2 Running 0 28m
rook-ceph-osd-1-58c6f87f64-qw95v 2/2 Running 0 28m
rook-ceph-osd-2-864df8c857-gdx6q 2/2 Running 0 28m
rook-ceph-osd-3-768b79d98f-rk4mb 2/2 Running 0 6m11s
rook-ceph-osd-4-868bdd4f4-s8jw2 2/2 Running 0 5m47s
rook-ceph-osd-5-56b698b7f4-466vm 2/2 Running 0 5m22s
rook-ceph-osd-6-cb6b64845-vljxv 2/2 Running 0 4m57s
rook-ceph-osd-7-5dbc69884f-vq99b 2/2 Running 0 4m25s
rook-ceph-osd-8-7ccdb745cd-wpfmw 2/2 Running 0 4m
rook-ceph-osd-prepare-80058f20e40692a56d589ed4c012ca48-p6zfj 0/1 Completed 0 29m
rook-ceph-osd-prepare-9a3c3db3f6bdb7b8d7715be5ebe233e8-jp44m 0/1 Completed 0 29m
rook-ceph-osd-prepare-eb1536040cd973c8fda358d13a762cc0-ftscr 0/1 Completed 0 29m
rook-ceph-osd-prepare-us-east-1a-data-097ncw-zvzl6 0/1 Completed 0 20m
rook-ceph-osd-prepare-us-east-1a-data-15dbxn-6mv2v 0/1 Completed 0 12m
rook-ceph-osd-prepare-us-east-1b-data-0qfsfd-5d65f 0/1 Completed 0 20m
rook-ceph-osd-prepare-us-east-1b-data-1sfwkr-t5x4s 0/1 Completed 0 12m
rook-ceph-osd-prepare-us-east-1c-data-04bdtc-ts5k4 0/1 Completed 0 20m
rook-ceph-osd-prepare-us-east-1c-data-19htf4-bz4m6 0/1 Completed 0 12m
```
### Changed PVC Sizes
```
~ $ oc get pvc | grep data
ocs-deviceset-gp3-csi-0-data-09cpzv Bound pvc-ccfd8303-7d90-4b66-9bd7-dff235e9ffbf 2Ti RWO gp3-csi <unset> 29m
ocs-deviceset-gp3-csi-1-data-06hgxl Bound pvc-9e6f7d23-62fa-447d-ae76-241b77d36610 2Ti RWO gp3-csi <unset> 29m
ocs-deviceset-gp3-csi-2-data-0rdhrb Bound pvc-a2a02cef-e45e-4d16-aa62-e9added889a2 2Ti RWO gp3-csi <unset> 29m
us-east-1a-data-097ncw Bound pvc-d7c49c55-4655-4ecd-b4af-e764cfbbe048 3Ti RWO gp3-csi <unset> 20m
us-east-1a-data-15dbxn Bound pvc-7566049f-34a5-4a10-970f-69e59130d363 3Ti RWO gp3-csi <unset> 12m
us-east-1b-data-0qfsfd Bound pvc-495abdac-20dd-4ec8-b118-bea24c682d7d 3Ti RWO gp3-csi <unset> 20m
us-east-1b-data-1sfwkr Bound pvc-2da42b5e-08c6-4ad6-8b68-031c3b879242 3Ti RWO gp3-csi <unset> 12m
us-east-1c-data-04bdtc Bound pvc-f801545a-f6b3-4b4a-874c-6d0a7a3450e9 3Ti RWO gp3-csi <unset> 20m
us-east-1c-data-19htf4 Bound pvc-31d9af18-7bff-4306-91e5-7e0f8dc53394 3Ti RWO gp3-csi <unset> 12m
```
### Ceph details
```
sh-5.1$ ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 18.00000 - 23 TiB 5.0 TiB 237 MiB 41 KiB 139 MiB 18 TiB 21.74 1.00 - root default
-5 18.00000 - 23 TiB 5.0 TiB 237 MiB 41 KiB 139 MiB 18 TiB 21.74 1.00 - region us-east-1
-14 6.00000 - 8 TiB 2.0 TiB 78 MiB 14 KiB 46 MiB 6.0 TiB 25.00 1.15 - zone us-east-1a
-13 2.00000 - 2 TiB 53 MiB 19 MiB 0 B 34 MiB 2.0 TiB 0.00 0 - host ocs-deviceset-gp3-csi-2-data-0rdhrb
2 ssd 2.00000 1.00000 2 TiB 53 MiB 19 MiB 0 B 34 MiB 2.0 TiB 0.00 0 32 up osd.2
-51 2.00000 - 3 TiB 1.0 TiB 20 MiB 4 KiB 6.2 MiB 2.0 TiB 33.33 1.53 - host us-east-1a-data-097ncw
5 us-east-1a 2.00000 1.00000 3 TiB 1.0 TiB 20 MiB 4 KiB 6.2 MiB 2.0 TiB 33.33 1.53 43 up osd.5
-66 2.00000 - 3 TiB 1.0 TiB 39 MiB 10 KiB 6.1 MiB 2.0 TiB 33.33 1.53 - host us-east-1a-data-15dbxn
8 us-east-1a 2.00000 1.00000 3 TiB 1.0 TiB 39 MiB 10 KiB 6.1 MiB 2.0 TiB 33.33 1.53 38 up osd.8
-4 6.00000 - 8 TiB 2.0 TiB 79 MiB 12 KiB 46 MiB 6.0 TiB 25.00 1.15 - zone us-east-1b
-3 2.00000 - 2 TiB 57 MiB 23 MiB 0 B 34 MiB 2.0 TiB 0.00 0 - host ocs-deviceset-gp3-csi-1-data-06hgxl
1 ssd 2.00000 1.00000 2 TiB 57 MiB 23 MiB 0 B 34 MiB 2.0 TiB 0.00 0 37 up osd.1
-46 2.00000 - 3 TiB 1.0 TiB 26 MiB 4 KiB 6.2 MiB 2.0 TiB 33.33 1.53 - host us-east-1b-data-0qfsfd
4 us-east-1b 2.00000 1.00000 3 TiB 1.0 TiB 26 MiB 4 KiB 6.2 MiB 2.0 TiB 33.33 1.53 37 up osd.4
-61 2.00000 - 3 TiB 1.0 TiB 30 MiB 8 KiB 6.2 MiB 2.0 TiB 33.33 1.53 - host us-east-1b-data-1sfwkr
7 us-east-1b 2.00000 1.00000 3 TiB 1.0 TiB 30 MiB 8 KiB 6.2 MiB 2.0 TiB 33.33 1.53 40 up osd.7
-10 6.00000 - 7 TiB 1.0 TiB 79 MiB 15 KiB 47 MiB 6.0 TiB 14.29 0.66 - zone us-east-1c
-9 2.00000 - 2 TiB 46 MiB 13 MiB 0 B 33 MiB 2.0 TiB 0.00 0 - host ocs-deviceset-gp3-csi-0-data-09cpzv
0 ssd 2.00000 1.00000 2 TiB 46 MiB 13 MiB 0 B 33 MiB 2.0 TiB 0.00 0 35 up osd.0
-41 2.00000 - 2 TiB 47 MiB 40 MiB 11 KiB 6.7 MiB 2.0 TiB 0.00 0 - host us-east-1c-data-04bdtc
3 us-east-1c 2.00000 1.00000 2 TiB 47 MiB 40 MiB 11 KiB 6.7 MiB 2.0 TiB 0.00 0 37 up osd.3
-56 2.00000 - 3 TiB 1.0 TiB 27 MiB 4 KiB 6.5 MiB 2.0 TiB 33.33 1.53 - host us-east-1c-data-19htf4
6 us-east-1c 2.00000 1.00000 3 TiB 1.0 TiB 27 MiB 4 KiB 6.5 MiB 2.0 TiB 33.33 1.53 43 up osd.6
TOTAL 23 TiB 5.0 TiB 237 MiB 44 KiB 139 MiB 18 TiB 21.74
MIN/MAX VAR: 0/1.53 STDDEV: 16.87
```
Sign in with Wallet
Connect another wallet