Read Affinity - HackMD

# Read Affinity We can direct reads for a rbd device to the nearest OSD using `--read_from_replica=localize` and `--crush_location` [kernel rbd(krbd) map options](https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options). ### read_from_replica=localize > When issued a read on a replicated pool, pick the most local OSD for serving it (since 5.8). The locality metric is calculated against the location of the client given with crush_location; a match with the lowest-valued bucket type wins. For example, with default bucket types, an OSD in a matching rack is closer than an OSD in a matching data center, which in turn is closer than an OSD in a matching region. This mode is safe for general use only since Octopus (i.e. after “ceph osd require-osd-release octopus”). Otherwise it should be limited to read-only workloads such as images mapped read-only everywhere or snapshots. https://docs.ceph.com/en/latest/rados/operations/crush-map/#crush-location --- ### `crush_location=x` > Specify the location of the client in terms of CRUSH hierarchy (since 5.8). This is a set of key-value pairs separated from each other by ‘|’, with keys separated from values by ‘:’. Note that ‘|’ may need to be quoted or escaped to avoid it being interpreted as a pipe by the shell. The key is the bucket type name (e.g. rack, datacenter or region with default bucket types) and the value is the bucket name. For example, to indicate that the client is local to rack “myrack”, data center “mydc” and region “myregion”: ##### `crush_location=rack:myrack|datacenter:mydc|region:myregion` >Each key-value pair stands on its own: “myrack” doesn’t need to reside in “mydc”, which in turn doesn’t need to reside in “myregion”. The location is not a path to the root of the hierarchy but rather a set of nodes that are matched independently, owning to the fact that bucket names are unique within a CRUSH map. “Multipath” locations are supported, so it is possible to indicate locality for multiple parallel hierarchies: ##### `crush_location=rack:myrack1|rack:myrack2|datacenter:mydc` --- ### Ceph CSI changes - add `--enable-read-affinity=<true(default)|false>` & `--crush-location-labels=<',' separated label values>` cmdline args to rbd nodeplugin - if enabled, rbd nodeplugin pod will read corresponding labels from k8s node where it is running. - cmdline arguement **`crush-location-labels="topology.kubernetes.io"`** combined with node label **`topology.kubernetes.io/zone=zone1`** is passed as **`--read_from_replica=localize --crush_location="zone:zone1"`** while mapping rbd images. ### Rook changes - Rook adds OSDs to the desired crush map level using node labels (refer [osd topology](https://github.com/rook/rook/blob/master/Documentation/CRDs/Cluster/ceph-cluster-crd.md#osd-topology)). - Add CSI configs to enable read affinity and pass crush location labels. - crush location labels will default to all the labels mentioned [here](https://github.com/rook/rook/blob/master/Documentation/CRDs/Cluster/ceph-cluster-crd.md#osd-topology). ### OCS-operator - TBD (maybe enable it by default ?)