# In-Transit compression- Dev Preview: ODF External Mode
## Important
***Applying compression prior to encryption has security implications as it reduces the level of security of messages between peers.
Applying compression after encryption is not efficient and the cost reduction would be very minimal.
So In case both encryption and compression are enabled, compression setting will be ignored by ceph and message will not be compressed. This behaviour can be overriden [by referering this.](https://docs.ceph.com/en/quincy/rados/configuration/msgr2/#confval-mscompresssecure)***
## Important
***For in-transit compression to work we need at least ceph v17 quincy***
## Cases
1. nil--->Compression(✅)
Compression is enabled by patching storagecluster
2. Compression--->nil(✅)
Compression is disabled afterwards
3. Compression,Encryption(❌)
Both encryption & Compression are enabled together
❌ - Encryption & Compression don't work together, only encryption will work
❌ - Encryption can't be turned off on the fly for existing volumes using it
---
## Case-1: Compression is enabled by patching storagecluster
### Creation of storagecluster
#### Creating storagesystem without encryption or compression
* While creating storagecluster don't include encryption enabled or compression enabled in network connections Spec.
* Wait for the storagecluster to become ready & all it's conditions to say successful/completed.
```
~ $ oc get storagecluster
NAME AGE PHASE EXTERNAL CREATED AT VERSION
ocs-external-storagecluster 60m Ready true 2023-07-27T07:10:02Z 4.14.0
```
```
~ $ oc get storagecluster ocs-external-storagecluster -o=jsonpath='{.status.conditions}' | jq
[
{
"lastHeartbeatTime": "2023-07-27T07:10:02Z",
"lastTransitionTime": "2023-07-27T07:10:02Z",
"message": "Version check successful",
"reason": "VersionMatched",
"status": "False",
"type": "VersionMismatch"
},
{
"lastHeartbeatTime": "2023-07-27T08:58:29Z",
"lastTransitionTime": "2023-07-27T07:10:03Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "True",
"type": "ReconcileComplete"
},
{
"lastHeartbeatTime": "2023-07-27T08:58:29Z",
"lastTransitionTime": "2023-07-27T07:13:30Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "True",
"type": "Available"
},
{
"lastHeartbeatTime": "2023-07-27T08:58:29Z",
"lastTransitionTime": "2023-07-27T07:13:30Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "False",
"type": "Progressing"
},
{
"lastHeartbeatTime": "2023-07-27T08:58:29Z",
"lastTransitionTime": "2023-07-27T07:10:02Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "False",
"type": "Degraded"
},
{
"lastHeartbeatTime": "2023-07-27T08:58:29Z",
"lastTransitionTime": "2023-07-27T07:13:30Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "True",
"type": "Upgradeable"
}
]
```
---
### Check if csi is configured to use v1 6789 port
```
~ $ oc get cm rook-ceph-csi-config -o=jsonpath='{.data.csi-cluster-config-json}' | jq
[
{
"clusterID": "openshift-storage",
"monitors": [
"10.1.115.118:6789",
"10.1.115.119:6789",
"10.1.115.120:6789"
],
"namespace": "openshift-storage"
}
]
```
---
### Create workload to test later after enabling compression
#### Create a rbd pvc & a pod to use it
```
~ $ cat <<EOF | oc create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: normal-rbd-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: ocs-external-storagecluster-ceph-rbd
EOF
```
```
~ $ oc get pvc | grep normal
normal-rbd-pvc Bound pvc-2001b9a3-0d28-475a-9c4f-56d820c331b5 1Gi RWO ocs-storagecluster-ceph-rbd 30s
```
```
~ % cat <<EOF | oc create -f -
apiVersion: v1
kind: Pod
metadata:
name: normal-rbd-pod
spec:
nodeSelector:
kubernetes.io/hostname: compute-0
volumes:
- name: rbd-storage
persistentVolumeClaim:
claimName: normal-rbd-pvc
containers:
- name: nginx-container
image: quay.io/mparida/nginx:latest
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: rbd-storage
EOF
```
```
~ $ oc get pods | grep normal
normal-rbd-pod 1/1 Running 0 27s
```
#### rsh into the pod & try using the mounted volume
```
~ $ oc rsh normal-rbd-pod
# cd /usr/share/nginx/html
# ls
lost+found
# touch a
# ls
a lost+found
# sync && exit
```
---
#### Create a cephfs pvc & a pod to use it
```
~ $ cat <<EOF | oc create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: normal-cephfs-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: ocs-external-storagecluster-cephfs
EOF
```
```
~ $ oc get pvc | grep normal
normal-cephfs-pvc Bound pvc-5564dfa1-58d7-4679-82e6-f5280157bbe5 1Gi RWO ocs-storagecluster-cephfs 8s
normal-rbd-pvc Bound pvc-2001b9a3-0d28-475a-9c4f-56d820c331b5 1Gi RWO ocs-storagecluster-ceph-rbd 5m15s
```
```
~ % cat <<EOF | oc create -f -
apiVersion: v1
kind: Pod
metadata:
name: normal-cephfs-pod
spec:
nodeSelector:
kubernetes.io/hostname: compute-0
volumes:
- name: cephfs-storage
persistentVolumeClaim:
claimName: normal-cephfs-pvc
containers:
- name: nginx-container
image: quay.io/mparida/nginx:latest
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: cephfs-storage
EOF
```
```
~ $ oc get pods | grep normal
normal-cephfs-pod 1/1 Running 0 31s
normal-rbd-pod 1/1 Running 0 6m47s
```
#### rsh into the pod & try using the mounted volume
```
~ $ oc rsh normal-cephfs-pod
# cd /usr/share/nginx/html
# ls
# touch a
# ls
a
# sync && exit
```
---
### Enabling Compression
#### Patching the storagecluster to enable compression
```
oc patch storagecluster ocs-external-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/network", "value": {"connections": {"compression": {"enabled": true}}} }]'
```
#### Wait for the storagecluster to get ready & all it's conditions to say successful/completed.
```
~ $ oc get storagecluster
NAME AGE PHASE EXTERNAL CREATED AT VERSION
ocs-external-storagecluster 60m Ready true 2023-07-27T07:10:02Z 4.14.0
```
```
~ $ oc get storagecluster ocs-external-storagecluster -o=jsonpath='{.status.conditions}' | jq
[
{
"lastHeartbeatTime": "2023-07-27T07:10:02Z",
"lastTransitionTime": "2023-07-27T07:10:02Z",
"message": "Version check successful",
"reason": "VersionMatched",
"status": "False",
"type": "VersionMismatch"
},
{
"lastHeartbeatTime": "2023-07-27T09:04:50Z",
"lastTransitionTime": "2023-07-27T07:10:03Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "True",
"type": "ReconcileComplete"
},
{
"lastHeartbeatTime": "2023-07-27T09:04:50Z",
"lastTransitionTime": "2023-07-27T07:13:30Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "True",
"type": "Available"
},
{
"lastHeartbeatTime": "2023-07-27T09:04:50Z",
"lastTransitionTime": "2023-07-27T07:13:30Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "False",
"type": "Progressing"
},
{
"lastHeartbeatTime": "2023-07-27T09:04:50Z",
"lastTransitionTime": "2023-07-27T07:10:02Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "False",
"type": "Degraded"
},
{
"lastHeartbeatTime": "2023-07-27T09:04:50Z",
"lastTransitionTime": "2023-07-27T07:13:30Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "True",
"type": "Upgradeable"
}
]
```
---
### Apply settings on RHCS cluster
#### ssh into the RHCS cluster
#### Apply the in-transit compression ceph configurations
```
# ceph config set global ms_osd_compress_mode force
# ceph config set global rbd_default_map_options ms_mode=prefer-crc
```
#### Check settings on the RHCS cluster
```
# ceph config dump | grep ms
global advanced ms_osd_compress_mode force
global advanced rbd_default_map_options ms_mode=prefer-crc *
```
#### Check ceph mon details, if they have both v1 & v2 addresses
```
# ceph mon dump
epoch 4
fsid 15b49e52-5bf5-11ed-a2e7-0050568fd833
last_changed 2023-06-06T09:40:52.566062+0000
created 2022-11-04T03:59:46.486027+0000
min_mon_release 17 (quincy)
election_strategy: 1
0: [v2:10.1.115.118:3300/0,v1:10.1.115.118:6789/0] mon.rhcs-2-node-1
1: [v2:10.1.115.119:3300/0,v1:10.1.115.119:6789/0] mon.rhcs-2-node-2
2: [v2:10.1.115.120:3300/0,v1:10.1.115.120:6789/0] mon.rhcs-2-node-3
dumped monmap epoch 4
```
### On odf cluster check if csi is configured to use v2 3300 port
```
~ $ oc get cm rook-ceph-csi-config -o=jsonpath='{.data.csi-cluster-config-json}' | jq
[
{
"clusterID": "openshift-storage",
"monitors": [
"10.1.115.118:3300",
"10.1.115.119:3300",
"10.1.115.120:3300"
],
"namespace": "openshift-storage"
}
]
```
---
### Test by creating rbd PVC ,Mounting it in a Pod & using the volume
#### Creating & Checking PVC
```
~ $ cat <<EOF | oc create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: compression-rbd-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: ocs-external-storagecluster-ceph-rbd
EOF
```
```
~ $ oc get pvc | grep compression
encryption-rbd-pvc Bound pvc-c271e73b-92a4-4710-9a6b-71a6d799830f 1Gi RWO ocs-storagecluster-ceph-rbd 67s
```
#### Creating & Checking Pod
```
~ % cat <<EOF | oc create -f -
apiVersion: v1
kind: Pod
metadata:
name: compression-rbd-pod
spec:
nodeSelector:
kubernetes.io/hostname: compute-0
volumes:
- name: rbd-storage
persistentVolumeClaim:
claimName: compression-rbd-pvc
containers:
- name: nginx-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: rbd-storage
EOF
```
```
~ $ oc get pods | grep compression
encryption-rbd-pod 1/1 Running 0 2m41s
```
#### rsh into the pod and try using the mounted volume
```
$ oc rsh compression-rbd-pod
# cd /usr/share/nginx/html
# ls
lost+found
# touch a
# ls
a lost+found
# sync && exit
```
---
### Test by creating cephfs PVC, Mounting it in a Pod & checking the volume
#### Creating & Checking PVC
```
~ $ cat <<EOF | oc create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: compression-cephfs-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: ocs-external-storagecluster-cephfs
EOF
```
```
~ $ oc get pvc | grep compression
compression-cephfs-pvc Bound pvc-25e473e2-a013-4707-bf09-14d99e7482f5 1Gi RWO ocs-storagecluster-cephfs 19s
compression-rbd-pvc Bound pvc-c271e73b-92a4-4710-9a6b-71a6d799830f 1Gi RWO ocs-storagecluster-ceph-rbd 67s
```
#### Creating & Checking Pod
```
~ % cat <<EOF | oc create -f -
apiVersion: v1
kind: Pod
metadata:
name: compression-cephfs-pod
spec:
nodeSelector:
kubernetes.io/hostname: compute-0
volumes:
- name: cephfs-storage
persistentVolumeClaim:
claimName: compression-cephfs-pvc
containers:
- name: nginx-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: cephfs-storage
EOF
```
```
~ $ oc get pods | grep compression
compression-cephfs-pod 0/1 ContainerCreating 0 43s
compression-rbd-pod 1/1 Running 0 2m41s
```
#### rsh into the pod & try using the mounted volume
```
$ oc rsh compression-cephfs-pod
# cd /usr/share/nginx/html
# ls
# touch a
# ls
a
# sync && exit
```
---
### Verify if the earlied created volumes are accesible
```
~ $ oc rsh normal-rbd-pod
# cd /usr/share/nginx/html
# ls
a lost+found
# touch b
# ls
a b lost+found
# sync && exit
```
```
~ $ oc rsh normal-cephfs-pod
# cd /usr/share/nginx/html
# ls
a
# touch b
# ls
a b
# sync && exit
```
## Case 2: Compression is disabled afterwards
### Disabling Compression
#### Patching the storagecluster to disable compression
```
oc patch storagecluster ocs-external-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/network", "value": {"connections": {"compression": {"enabled": false}}} }]'
```
#### Wait for the storagecluster to get ready & all it's conditions to say successful/completed.
```
~ $ oc get storagecluster
NAME AGE PHASE EXTERNAL CREATED AT VERSION
ocs-external-storagecluster 60m Ready true 2023-07-27T07:10:02Z 4.14.0
```
```
~ $ oc get storagecluster ocs-external-storagecluster -o=jsonpath='{.status.conditions}' | jq
[
{
"lastHeartbeatTime": "2023-07-27T07:10:02Z",
"lastTransitionTime": "2023-07-27T07:10:02Z",
"message": "Version check successful",
"reason": "VersionMatched",
"status": "False",
"type": "VersionMismatch"
},
{
"lastHeartbeatTime": "2023-07-27T09:04:50Z",
"lastTransitionTime": "2023-07-27T07:10:03Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "True",
"type": "ReconcileComplete"
},
{
"lastHeartbeatTime": "2023-07-27T09:04:50Z",
"lastTransitionTime": "2023-07-27T07:13:30Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "True",
"type": "Available"
},
{
"lastHeartbeatTime": "2023-07-27T09:04:50Z",
"lastTransitionTime": "2023-07-27T07:13:30Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "False",
"type": "Progressing"
},
{
"lastHeartbeatTime": "2023-07-27T09:04:50Z",
"lastTransitionTime": "2023-07-27T07:10:02Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "False",
"type": "Degraded"
},
{
"lastHeartbeatTime": "2023-07-27T09:04:50Z",
"lastTransitionTime": "2023-07-27T07:13:30Z",
"message": "Reconcile completed successfully",
"reason": "ReconcileCompleted",
"status": "True",
"type": "Upgradeable"
}
]
```
---
### Remove settings on RHCS cluster
#### ssh into the RHCS cluster
#### Apply the in-transit compression ceph configurations
```
# ceph config rm global ms_osd_compress_mode
# ceph config rm global rbd_default_map_options
```
#### Check if the settings are removed
```
# ceph config dump | grep ms
```
---
### On odf cluster check if csi is configured back to use v1 6789 port
```
~ $ oc get cm rook-ceph-csi-config -o=jsonpath='{.data.csi-cluster-config-json}' | jq
[
{
"clusterID": "openshift-storage",
"monitors": [
"10.1.115.118:6789",
"10.1.115.119:6789",
"10.1.115.120:6789"
],
"namespace": "openshift-storage"
}
]
```
---
### Create workload to test after disabling compression
#### Create a rbd pvc & a pod to use it
```
~ $ cat <<EOF | oc create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: normal-rbd-pvc-1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: ocs-external-storagecluster-ceph-rbd
EOF
```
```
~ $ oc get pvc | grep normal
normal-cephfs-pvc Bound pvc-dbd3417c-9aa7-435c-b174-a95798e8185d 1Gi RWO ocs-external-storagecluster-cephfs 33m
normal-rbd-pvc Bound pvc-0206e669-1aa2-45f3-94a5-9fafaa9e70f0 1Gi RWO ocs-external-storagecluster-ceph-rbd 34m
normal-rbd-pvc-1 Bound pvc-a8af66e0-ea9b-43dc-b8dc-bd2936db3711 1Gi RWO ocs-external-storagecluster-ceph-rbd 11s
```
```
~ % cat <<EOF | oc create -f -
apiVersion: v1
kind: Pod
metadata:
name: normal-rbd-pod-1
spec:
nodeSelector:
kubernetes.io/hostname: compute-0
volumes:
- name: rbd-storage
persistentVolumeClaim:
claimName: normal-rbd-pvc-1
containers:
- name: nginx-container
image: quay.io/mparida/nginx:latest
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: rbd-storage
EOF
```
```
~ $ oc get pods | grep normal
normal-cephfs-pod 1/1 Running 0 33m
normal-rbd-pod 1/1 Running 0 35m
normal-rbd-pod-1 1/1 Running 0 21s
```
#### rsh into the pod & try using the mounted volume
```
~ $ oc rsh normal-rbd-pod-1
# cd /usr/share/nginx/html
# ls
lost+found
# touch a
# ls
a lost+found
# sync && exit
```
---
### Test by creating cephfs PVC ,Mounting it in a Pod & using the volume
#### Create a cephfs pvc & a pod to use it
```
~ $ cat <<EOF | oc create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: normal-cephfs-pvc-1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: ocs-external-storagecluster-cephfs
EOF
```
```
~ $ oc get pvc | grep normal
normal-cephfs-pvc Bound pvc-dbd3417c-9aa7-435c-b174-a95798e8185d 1Gi RWO ocs-external-storagecluster-cephfs 35m
normal-cephfs-pvc-1 Bound pvc-4b194783-6b81-4b87-917e-a066828264b4 1Gi RWO ocs-external-storagecluster-cephfs 8s
normal-rbd-pvc Bound pvc-0206e669-1aa2-45f3-94a5-9fafaa9e70f0 1Gi RWO ocs-external-storagecluster-ceph-rbd 36m
normal-rbd-pvc-1 Bound pvc-a8af66e0-ea9b-43dc-b8dc-bd2936db3711 1Gi RWO ocs-external-storagecluster-ceph-rbd 2m21s
```
```
~ % cat <<EOF | oc create -f -
apiVersion: v1
kind: Pod
metadata:
name: normal-cephfs-pod-1
spec:
nodeSelector:
kubernetes.io/hostname: compute-0
volumes:
- name: cephfs-storage
persistentVolumeClaim:
claimName: normal-cephfs-pvc-1
containers:
- name: nginx-container
image: quay.io/mparida/nginx:latest
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: cephfs-storage
EOF
```
```
~ $ oc get pods | grep normal
normal-cephfs-pod 1/1 Running 0 35m
normal-cephfs-pod-1 1/1 Running 0 13s
normal-rbd-pod 1/1 Running 0 37m
normal-rbd-pod-1 1/1 Running 0 2m19s
```
#### rsh into the pod & try using the mounted volume
```
~ $ oc rsh normal-cephfs-pod-1
# cd /usr/share/nginx/html
# ls
# touch a
# ls
a
# sync && exit
```
---
## Case 3: Encryption & Compression are enabled together
### Enabling both encryption & compression
#### Patching the storagecluster
```
oc patch storagecluster ocs-external-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/network", "value": {"connections": {"encryption": {"enabled": true},"compression": {"enabled": true}}} }]'
```
#### Wait for the storagecluster to get ready & all it's conditions to say successful/completed.
---
### Apply settings on RHCS cluster
#### ssh into the RHCS cluster
#### Apply the in-transit encryption & compression ceph configurations
```
# ceph config set global ms_client_mode secure
# ceph config set global ms_cluster_mode secure
# ceph config set global ms_service_mode secure
# ceph config set global rbd_default_map_options ms_mode=secure
# ceph config set global ms_osd_compress_mode force
```
#### Check settings on the RHCS cluster
```
# ceph config dump | grep ms
global basic ms_client_mode secure *
global basic ms_cluster_mode secure *
global advanced ms_osd_compress_mode force
global basic ms_service_mode secure *
global advanced rbd_default_map_options ms_mode=secure *
```
---
### On odf cluster check if csi is configured to use v2 3300 port
```
~ $ oc get cm rook-ceph-csi-config -o=jsonpath='{.data.csi-cluster-config-json}' | jq
[
{
"clusterID": "openshift-storage",
"monitors": [
"10.1.115.118:3300",
"10.1.115.119:3300",
"10.1.115.120:3300"
],
"namespace": "openshift-storage"
}
]
```
---
### Check ocs-operator logs for warning message for both encryption & compression
```
{"level":"info","ts":"2023-07-27T10:27:02Z","logger":"controllers.StorageCluster","msg":"Both in-transit encryption & compression are enabled. To protect security of encrypted messages ceph will ignore compression","Request.Namespace":"openshift-storage","Request.Name":"ocs-external-storagecluster"}
```
---
**This is an unsupported configuration and we should not proceed from here onwards**
We have to decide if we want to use encryption or compression.
---
### If we want to continue with encryption only
#### Patch storagecluster
```
oc patch storagecluster ocs-external-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/network", "value": {"connections": {"compression": {"enabled": false}}} }]'
```
#### changes in RHCS cluster
```
# ceph config rm global ms_osd_compress_mode
```
---
### If we want to continue with compression only
#### Patch storagecluster
```
oc patch storagecluster ocs-external-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/network", "value": {"connections": {"encryption": {"enabled": false}}} }]'
```
#### changes in RHCS cluster
```
# ceph config rm global ms_client_mode
# ceph config rm global ms_cluster_mode
# ceph config rm global ms_service_mode
# ceph config set global rbd_default_map_options ms_mode=prefer-crc
```
#### On odf cluster check if csi is configured back to use v1 6789 port
```
~ $ oc get cm rook-ceph-csi-config -o=jsonpath='{.data.csi-cluster-config-json}' | jq
[
{
"clusterID": "openshift-storage",
"monitors": [
"10.1.115.118:6789",
"10.1.115.119:6789",
"10.1.115.120:6789"
],
"namespace": "openshift-storage"
}
]
```
Sign in with Wallet
Connect another wallet