## NCCEPH-416

### Overview

This document outlines what is currently seen as the safest procedure for relocating Elasticsearch data to the LMA Ceph pool.

When relocating Elasticsearch data it is vital that the integrity of Elasticsearch cluster metadata be preserved.

Given the importance of Elasticsearch cluster metadata, the procedure outlined in this document leverages the high availability native to Elasticsearch for data relocation. An alternative method is also reviewed.

### General Procedure

Elasticseach data can be migrated to the LMA Ceph Pool by adding and removing pods from the Elasticsearch cluster. This procedure would require no service downtime. There would be some performance degradation due to data relocation. The steps involved would be:

1. Add a new Elasticsearch pod to the cluster with a PVC that maps to the LMA Ceph pool.

2. Relocate data from an existing Elasticsearch Pod with a PVC that maps to the default Ceph pool.

   curl -XPUT $SERVICE_IP:9200/_cluster/settings -d '{
     "transient" : {
       "cluster.routing.allocation.exclude._ip": "$IP_ADDRESS"
     }
   }'

3. Wait for data to complete relocating and verify Elasticsearch cluster health is "green".

   curl $SERVICE_IP:9200/_cluster/health?pretty=true

4. Permanently delete the Elasticsearch pod that is being removed from the cluster.

5. Reset Elasticsearch routing allocation rules

   curl -XPUT $SERVICE_IP:9200/_cluster/settings -d 
   '{
     "transient" :{
         "cluster.routing.allocation.exclude._ip" : ""
      }
    }'

6. Repeat from Step 1 until all pods using default Ceph pool have been removed from the Elasticsearch cluster.

#### Caveats:
* Extra care is needed when removing master-eligible pods from the Elasticsearch cluster. We would need to be sure that minimum_master_nodes is set appropriately and the that the replacement pod is marked master-eligible. Similar steps could also be needed for ingest/ml nodes if employed.

### Alternative

It would be possible to relocate Elasticsearch data to the LMA Ceph pool by employing the following procedure:

1. Disable Elasticsearch auto data relocation (not technically necessary but will ensure metadata integrity)

   curl -XPUT $SERVICE_IP:9200/_cluster/settings -d 
   '{
    "transient" : {
        "cluster.routing.allocation.enable": "none"
     }
   }'

2. Stop an Elasticsearch pod.
3. Use Ceph to snapshot/clone the underlying RBD image to the LMA pool and flatten the image.
4. Create a PV/PVC for the image in the LMA Ceph pool.
5. Start the Elasticsearch pod back up with the new PVC attached.
6. Remove the original RBD image/snapshot and PV/PVC when the flatten operation is complete.
7. Repeat from step 2 until all RBD images moved to LMA pool.
8. Enable Elasticsearch auto data relocation

   curl -XPUT $SERVICE_IP:9200/_cluster/settings -d 
   '{
    "transient" : {
        "cluster.routing.allocation.enable": "all"
     }
   }'

This alternative would require that minimum_master_nodes be set appropriately. There would be some risk of causing a service outage due to auto data relocation being disabled. Elasticsearch cluster metadata integrity would be maintained as data relocation is disabled.

### Summary

Elasticsearch is a distributed application with it's own metadata regarding data location. Given this, using Elasticsearch itself to relocate data to the LMA Ceph Pool via new pods with PV/PVCs that map to the pool is the safest option. No service downtime would be required. Using Ceph to snapshot/clone existing rbd images is an option but would be safest with Elasticsearch auto data relocation disabled. This would introduce the possibilty of service downtime.