---
# System prepended metadata

title: Untitled

---

```
0. Preparation
[x]    0.1. Instances summary sheet https://docs.google.com/spreadsheets/d/1yhwfv3PWD7AlnZperCV0tMqSxf9X7MhnBV23aZibN6k/edit?usp=sharing
[x]    0.2. Ensure accesses to all instances
[x]    0.3. Perform test run on a copy of any clusters.
                Done, using IAD GFS instances' backup AMIs.
[ ]
```

```
1. Pre-actions check Cluster 1 (IAD)
    Common approach is to check cluster for any errors before performing any modifications.
    If issues are found (except for easy tasks like mount/remount volumes), escalate and postpone executions.

[x]     1.1. Check nagios for issues related to gfs*.iad* hosts

[x]     1.2. Ensure instance i-0af638f14f94e3ce0 (gfs1-1.iad-aws.prod.sli.io) is ready for retype.

[x]             1.2.0. Connect to the instance:
                    ssh 10.79.114.41

[x]             1.2.1. Check mount:
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ mount|grep gfs1
                    /dev/mapper/vg01-gfs1.brick1 on /gluster/gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                    /gluster/sec-gfs1/sec-gfs1.brick1 on /gluster/sec-gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                    gfs1-1.iad-aws.prod.sli.io:/sec-gfs1 on /mnt/sec-gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
                    gfs1-1.iad-aws.prod.sli.io:/gfs1 on /mnt/gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

[x]             1.2.2. Check mount is healthy:
                    Following commands should produce no output:
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null

                    If nothing printed out, proceed to the next step.

                    Following is an example of mounted volume, which has issues:
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null
                    ls: cannot access /mnt/gfs1: Transport endpoint is not connected
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                    ls: cannot access /mnt/sec-gfs1: Transport endpoint is not connected

                    Try easy solution first:
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ umount /mnt/gfs1
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ umount /mnt/sec-gfs1
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                    After issuing above commands, retry whole step starting with 1.2.1.

[x]             1.2.3. Check node VIP status:
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ vip-control status
                    VIP-CONTROL OK - IN VIP: write_file,success; read_file,success; alive;

                    If output of the command is as shown above, proceed to the next step.

                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ vip-control status
                    VIP-CONTROL CRITICAL - NOT IN VIP: write_file,failed; check_failed; remove,already_removed;

                    If output of the command shows above failure, try following:
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ vip-control insert
                    Sat Jul 31 00:38:39 NZST 2021,vip-control,insert,successful

                    If above command succeeded as shown, continue to the next step.

                    If fails, stop execution and escalate to BU SaaS.

[x]             1.2.4. Check peers status:
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ gluster peer status
                    Number of Peers: 3

                    Hostname: gfs1-4.iad-aws.prod.sli.io
                    Uuid: 0183b376-1aa0-4d98-945e-6197a8876f27
                    State: Peer in Cluster (Connected)

                    Hostname: gfs1-2.iad-aws.prod.sli.io
                    Uuid: 95e9d91b-fe51-4e0d-86cd-00097890f113
                    State: Peer in Cluster (Connected)

                    Hostname: gfs1-3.iad-aws.prod.sli.io
                    Uuid: 23884f97-8149-4dd9-b4d3-1faccee03d1a
                    State: Peer in Cluster (Connected)

                    If there are issues with nodes above (not Connected), stop execution and escalate to BU SaaS.

[x]             1.2.5. Check volumes status:
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ gluster volume status
                    Status of volume: gfs1
                    Gluster process                             TCP Port  RDMA Port  Online  Pid
                    ------------------------------------------------------------------------------
                    Brick gfs1-1.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       3510
                    Brick gfs1-2.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       31340
                    Brick gfs1-3.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       3316
                    Brick gfs1-4.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       31413
                    Self-heal Daemon on localhost               N/A       N/A        Y       3665
                    Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       20255
                    Self-heal Daemon on gfs1-2.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       77317
                    Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       66457

                    Task Status of Volume gfs1
                    ------------------------------------------------------------------------------
                    There are no active volume tasks

                    Status of volume: sec-gfs1
                    Gluster process                             TCP Port  RDMA Port  Online  Pid
                    ------------------------------------------------------------------------------
                    Brick gfs1-1.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       3521
                    Brick gfs1-2.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       31351
                    Brick gfs1-3.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       3339
                    Brick gfs1-4.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       31424
                    Self-heal Daemon on localhost               N/A       N/A        Y       3665
                    Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       20255
                    Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       66457
                    Self-heal Daemon on gfs1-2.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       77317

                    Task Status of Volume sec-gfs1
                    ------------------------------------------------------------------------------
                    There are no active volume tasks

                    If any of bricks above are not online, stop execution and escalate to BU SaaS.

                    If there are active volume tasks in progress, wait for completion before continue with modifications.
                    [up to 30min, then escalate]
```

```
[x]     1.3. Ensure instance i-08738fdcf35071be8 (gfs1-2.iad-aws.prod.sli.io) is ready for retype.

[x]             1.3.0. Connect to the instance:
                    ssh 10.79.114.45

[x]             1.3.1. Check mount:
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ mount|grep gfs1
                    /dev/mapper/vg01-gfs1.brick1 on /gluster/gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                    /gluster/sec-gfs1/sec-gfs1.brick1 on /gluster/sec-gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                    gfs1-2.iad-aws.prod.sli.io:/sec-gfs1 on /mnt/sec-gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
                    gfs1-2.iad-aws.prod.sli.io:/gfs1 on /mnt/gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

[x]             1.3.2. Check mount is healthy:
                    Following commands should produce no output:
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null

                    If nothing printed out, proceed to the next step.

                    Following is an example of mounted volume, which has issues:
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null
                    ls: cannot access /mnt/gfs1: Transport endpoint is not connected
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                    ls: cannot access /mnt/sec-gfs1: Transport endpoint is not connected

                    Try easy solution first:
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ umount /mnt/gfs1
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ umount /mnt/sec-gfs1
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                    After issuing above commands, retry whole step starting with 1.3.1.

[x]             1.3.3. Check node VIP status:
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ vip-control status
                    VIP-CONTROL OK - IN VIP: write_file,success; read_file,success; alive;

                    If output of the command is as shown above, proceed to the next step.

                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ vip-control status
                    VIP-CONTROL CRITICAL - NOT IN VIP: write_file,failed; check_failed; remove,already_removed;

                    If output of the command shows above failure, try following:
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ vip-control insert
                    Sat Jul 31 00:38:39 NZST 2021,vip-control,insert,successful


                    If above command succeeded as shown, continue to the next step.

                    If fails, stop execution and escalate to BU SaaS.

[x]             1.3.4. Check peers status:
                    [root@gfs1-2.iad-aws.prod.sli.io ~]$ gluster peer status
                    Number of Peers: 3

                    Hostname: gfs1-1.iad.prod.sli.io
                    Uuid: c9381ca4-99ac-4e69-8823-b9b8e208f873
                    State: Peer in Cluster (Connected)

                    Hostname: gfs1-3.iad-aws.prod.sli.io
                    Uuid: 23884f97-8149-4dd9-b4d3-1faccee03d1a
                    State: Peer in Cluster (Connected)

                    Hostname: gfs1-4.iad-aws.prod.sli.io
                    Uuid: 0183b376-1aa0-4d98-945e-6197a8876f27
                    State: Peer in Cluster (Connected)


                    If there are issues with nodes above (not Connected), stop execution and escalate to BU SaaS.

[x]             1.3.5. Check volumes status:
                    [root@gfs1-1.iad-aws.prod.sli.io ~]$ gluster volume status
                    Status of volume: gfs1
                    Gluster process                             TCP Port  RDMA Port  Online  Pid
                    ------------------------------------------------------------------------------
                    Brick gfs1-1.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       3510
                    Brick gfs1-2.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       31340
                    Brick gfs1-3.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       3316
                    Brick gfs1-4.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       31413
                    Self-heal Daemon on localhost               N/A       N/A        Y       77317
                    Self-heal Daemon on gfs1-1.iad.prod.sli.io  N/A       N/A        Y       3665
                    Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       20255
                    Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       66457

                    Task Status of Volume gfs1
                    ------------------------------------------------------------------------------
                    There are no active volume tasks

                    Status of volume: sec-gfs1
                    Gluster process                             TCP Port  RDMA Port  Online  Pid
                    ------------------------------------------------------------------------------
                    Brick gfs1-1.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       3521
                    Brick gfs1-2.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       31351
                    Brick gfs1-3.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       3339
                    Brick gfs1-4.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       31424
                    Self-heal Daemon on localhost               N/A       N/A        Y       77317
                    Self-heal Daemon on gfs1-1.iad.prod.sli.io  N/A       N/A        Y       3665
                    Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       20255
                    Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       66457

                    Task Status of Volume sec-gfs1
                    ------------------------------------------------------------------------------
                    There are no active volume tasks


                    If any of bricks above are not online, stop execution and escalate to BU SaaS.

                    If there are active volume tasks in progress, wait for completion before continue with modifications.
                    [up to 30min, then escalate]
```

```
[x]     1.4. Ensure instance i-0a0c2ad3bba430706 (gfs1-3.iad-aws.prod.sli.io) is ready for retype.

[x]             1.4.0. Connect to the instance:
                    ssh 10.79.114.199

[x]             1.4.1. Check mount:
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ mount|grep gfs1
                    /dev/mapper/vg01-gfs1.brick1 on /gluster/gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                    /gluster/sec-gfs1/sec-gfs1.brick1 on /gluster/sec-gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                    gfs1-3.iad-aws.prod.sli.io:/sec-gfs1 on /mnt/sec-gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
                    gfs1-3.iad-aws.prod.sli.io:/gfs1 on /mnt/gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

[x]             1.4.2. Check mount is healthy:
                    Following commands should produce no output:
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null

                    If nothing printed out, proceed to the next step.

                    Following is an example of mounted volume, which has issues:
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null
                    ls: cannot access /mnt/gfs1: Transport endpoint is not connected
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                    ls: cannot access /mnt/sec-gfs1: Transport endpoint is not connected

                    Try easy solution first:
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ umount /mnt/gfs1
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ umount /mnt/sec-gfs1
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                    After issuing above commands, retry whole step starting with 1.4.1.

[x]             1.4.3. Check node VIP status:
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ vip-control status
                    VIP-CONTROL OK - IN VIP: write_file,success; read_file,success; alive;

                    If output of the command is as shown above, proceed to the next step.

                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ vip-control status
                    VIP-CONTROL CRITICAL - NOT IN VIP: write_file,failed; check_failed; remove,already_removed;

                    If output of the command shows above failure, try following:
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ vip-control insert
                    Sat Jul 31 00:38:39 NZST 2021,vip-control,insert,successful


                    If above command succeeded as shown, continue to the next step.

                    If fails, stop execution and escalate to BU SaaS.

[x]             1.4.4. Check peers status:
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ gluster peer status
                    Number of Peers: 3

                    Hostname: gfs1-4.iad-aws.prod.sli.io
                    Uuid: 0183b376-1aa0-4d98-945e-6197a8876f27
                    State: Peer in Cluster (Connected)

                    Hostname: gfs1-2.iad-aws.prod.sli.io
                    Uuid: 95e9d91b-fe51-4e0d-86cd-00097890f113
                    State: Peer in Cluster (Connected)

                    Hostname: gfs1-1.iad.prod.sli.io
                    Uuid: c9381ca4-99ac-4e69-8823-b9b8e208f873
                    State: Peer in Cluster (Connected)


                    If there are issues with nodes above (not Connected), stop execution and escalate to BU SaaS.

[x]             1.4.5. Check volumes status:
                    [root@gfs1-3.iad-aws.prod.sli.io ~]$ gluster volume status
                    Status of volume: gfs1
                    Gluster process                             TCP Port  RDMA Port  Online  Pid
                    ------------------------------------------------------------------------------
                    Brick gfs1-1.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       3510
                    Brick gfs1-2.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       31340
                    Brick gfs1-3.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       3316
                    Brick gfs1-4.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       31413
                    Self-heal Daemon on localhost               N/A       N/A        Y       77317
                    Self-heal Daemon on gfs1-1.iad.prod.sli.io  N/A       N/A        Y       3665
                    Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       20255
                    Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       66457

                    Task Status of Volume gfs1
                    ------------------------------------------------------------------------------
                    There are no active volume tasks

                    Status of volume: sec-gfs1
                    Gluster process                             TCP Port  RDMA Port  Online  Pid
                    ------------------------------------------------------------------------------
                    Brick gfs1-1.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       3521
                    Brick gfs1-2.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       31351
                    Brick gfs1-3.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       3339
                    Brick gfs1-4.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       31424
                    Self-heal Daemon on localhost               N/A       N/A        Y       77317
                    Self-heal Daemon on gfs1-1.iad.prod.sli.io  N/A       N/A        Y       3665
                    Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       20255
                    Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       66457

                    Task Status of Volume sec-gfs1
                    ------------------------------------------------------------------------------
                    There are no active volume tasks


                    If any of bricks above are not online, stop execution and escalate to BU SaaS.

                    If there are active volume tasks in progress, wait for completion before continue with modifications.
                    [up to 30min, then escalate]
```

```
[x]     1.5. Ensure instance i-001707ebdd3539c48 (gfs1-4.iad-aws.prod.sli.io) is ready for retype.

[x]             1.5.0. Connect to the instance:
                    ssh 10.79.114.67

[x]             1.5.1. Check mount:
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ mount|grep gfs1
                    /dev/mapper/vg01-gfs1.brick1 on /gluster/gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                    /gluster/sec-gfs1/sec-gfs1.brick1 on /gluster/sec-gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                    gfs1-4.iad-aws.prod.sli.io:/sec-gfs1 on /mnt/sec-gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
                    gfs1-4.iad-aws.prod.sli.io:/gfs1 on /mnt/gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

[x]             1.5.2. Check mount is healthy:
                    Following commands should produce no output:
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null

                    If nothing printed out, proceed to the next step.

                    Following is an example of mounted volume, which has issues:
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null
                    ls: cannot access /mnt/gfs1: Transport endpoint is not connected
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                    ls: cannot access /mnt/sec-gfs1: Transport endpoint is not connected

                    Try easy solution first:
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ umount /mnt/gfs1
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ umount /mnt/sec-gfs1
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                    After issuing above commands, retry whole step starting with 1.5.1.

[x]             1.5.3. Check node VIP status:
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ vip-control status
                    VIP-CONTROL OK - IN VIP: write_file,success; read_file,success; alive;

                    If output of the command is as shown above, proceed to the next step.

                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ vip-control status
                    VIP-CONTROL CRITICAL - NOT IN VIP: write_file,failed; check_failed; remove,already_removed;

                    If output of the command shows above failure, try following:
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ vip-control insert
                    Sat Jul 31 00:38:39 NZST 2021,vip-control,insert,successful


                    If above command succeeded as shown, continue to the next step.

                    If fails, stop execution and escalate to BU SaaS.

[x]             1.5.4. Check peers status:
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ gluster peer status
                    Number of Peers: 3

                    Hostname: gfs1-3.iad-aws.prod.sli.io
                    Uuid: 23884f97-8149-4dd9-b4d3-1faccee03d1a
                    State: Peer in Cluster (Connected)

                    Hostname: gfs1-2.iad-aws.prod.sli.io
                    Uuid: 95e9d91b-fe51-4e0d-86cd-00097890f113
                    State: Peer in Cluster (Connected)

                    Hostname: gfs1-1.iad.prod.sli.io
                    Uuid: c9381ca4-99ac-4e69-8823-b9b8e208f873
                    State: Peer in Cluster (Connected)


                    If there are issues with nodes above (not Connected), stop execution and escalate to BU SaaS.

[x]             1.5.5. Check volumes status:
                    [root@gfs1-4.iad-aws.prod.sli.io ~]$ gluster volume status
                    Status of volume: gfs1
                    Gluster process                             TCP Port  RDMA Port  Online  Pid
                    ------------------------------------------------------------------------------
                    Brick gfs1-1.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       3510
                    Brick gfs1-2.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       31340
                    Brick gfs1-3.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       3316
                    Brick gfs1-4.iad-aws.prod.sli.io:/gluster/g
                    fs1/brick1                                  49152     0          Y       31413
                    Self-heal Daemon on localhost               N/A       N/A        Y       77317
                    Self-heal Daemon on gfs1-1.iad.prod.sli.io  N/A       N/A        Y       3665
                    Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       20255
                    Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       66457

                    Task Status of Volume gfs1
                    ------------------------------------------------------------------------------
                    There are no active volume tasks

                    Status of volume: sec-gfs1
                    Gluster process                             TCP Port  RDMA Port  Online  Pid
                    ------------------------------------------------------------------------------
                    Brick gfs1-1.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       3521
                    Brick gfs1-2.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       31351
                    Brick gfs1-3.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       3339
                    Brick gfs1-4.iad-aws.prod.sli.io:/gluster/s
                    ec-gfs1/brick1                              49153     0          Y       31424
                    Self-heal Daemon on localhost               N/A       N/A        Y       77317
                    Self-heal Daemon on gfs1-1.iad.prod.sli.io  N/A       N/A        Y       3665
                    Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       20255
                    Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                    .io                                         N/A       N/A        Y       66457

                    Task Status of Volume sec-gfs1
                    ------------------------------------------------------------------------------
                    There are no active volume tasks


                    If any of bricks above are not online, stop execution and escalate to BU SaaS.

                    If there are active volume tasks in progress, wait for completion before continue with modifications.
                    [up to 30min, then escalate]
```

```
2. Processing Cluser 1 (IAD)

[x]     2.0. Remove Cluster 1 (IAD) out of rotation:
            [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do ssh ${host} 'runuser -l gbrain bash -c "webcontrol vip remove"'; done
            (allow a minute before performing next step)

[x]     2.1. Ensure Cluster 1 (IAD) is not serving to the clients
            2.1.1. Check vip status on searches:
                [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do ssh ${host} 'runuser -l gbrain bash -c "webcontrol vip status"'; done

            2.1.2. Request healthcheck document from the search hosts:
                [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do echo -n "${host}: "; curl -A "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0" http://${host}/shared/vip-status.txt; done
                expecting all hosts to return down

        2.2. Perform retype of all cluster instances

[x]         2.2.1. Switch off CRON on all GFS instances
                [local]% for host in 10.79.114.41 10.79.114.45 10.79.114.199 10.79.114.67; do ssh ${host} "systemctl disable crond; systemctl stop crond; pkill -f cron -9; pkill -f CRON -9; ps ax|grep -i cron"; done

[x]         2.2.2. Remove all GFS instances out of serving requests
                    [local]% for host in 10.79.114.41 10.79.114.45 10.79.114.199 10.79.114.67; do ssh ${host} "vip-control remove"; done

            2.2.3. Proceed with instance retype
[x]             2.2.3.1. gfs1-1.iad-aws.prod.sli.io
                    [local]% ./automation/bin/run.sh arn:aws:ssm:us-east-1:862508395508:document/co-wu-resize-instance "InstanceId=i-0af638f14f94e3ce0,NewInstanceType=m5.large" 610092376560 us-east-1

[x]             2.2.3.1. gfs1-2.iad-aws.prod.sli.io
                    [local]% ./automation/bin/run.sh arn:aws:ssm:us-east-1:862508395508:document/co-wu-resize-instance "InstanceId=i-08738fdcf35071be8,NewInstanceType=m5.large" 610092376560 us-east-1

[x]             2.2.3.1. gfs1-3.iad-aws.prod.sli.io
                    [local]% ./automation/bin/run.sh arn:aws:ssm:us-east-1:862508395508:document/co-wu-resize-instance "InstanceId=i-0a0c2ad3bba430706,NewInstanceType=m5.large" 610092376560 us-east-1

[x]             2.2.3.1. gfs1-4.iad-aws.prod.sli.io
                    [local]% ./automation/bin/run.sh arn:aws:ssm:us-east-1:862508395508:document/co-wu-resize-instance "InstanceId=i-001707ebdd3539c48,NewInstanceType=m5.large" 610092376560 us-east-1

[x]         2.2.4. Return all GFS instances to serve requests
                [local]% for host in 10.79.114.41 10.79.114.45 10.79.114.199 10.79.114.67; do echo -n "${host}: "; ssh ${host} "vip-control insert"; done
                all hosts expected to return successful result
                    Sat Jul 31 00:38:39 NZST 2021,vip-control,insert,successful
                if any host will return failure, investigate individual host


[x]     2.3. Perform GFS healthcheck for instance i-0af638f14f94e3ce0 (gfs1-1.iad-aws.prod.sli.io).

[x]         2.3.0. Connect to the instance:
                ssh 10.79.114.41

[x]         2.3.1. Check gluster daemon is running:
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ systemctl status glusterd

                If not started, (re)start glusterd
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ systemctl restart glusterd

                If still fail to start all daemons (glusterd, glusterfsd, glusterfs), investigate logs in /var/log/glusterfs/
                [If no solution found in 10 minutes, escalate to BU SaaS]

[x]         2.3.2. Check mount:
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ mount|grep gfs1
                /dev/mapper/vg01-gfs1.brick1 on /gluster/gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                /gluster/sec-gfs1/sec-gfs1.brick1 on /gluster/sec-gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                gfs1-1.iad-aws.prod.sli.io:/sec-gfs1 on /mnt/sec-gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
                gfs1-1.iad-aws.prod.sli.io:/gfs1 on /mnt/gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

                If mounts are missing, try to mount volumes:
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                If mount fails, investigate logs in /var/log/glusterfs/, /var/log/messages, dmesg
                [If no solution found in 10 minutes, escalate to BU SaaS]


[x]         2.3.3. Check mount is healthy:
                Following commands should produce no output:
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null

                If nothing printed out, proceed to the next step.

                Following is an example of mounted volume, which has issues:
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null
                ls: cannot access /mnt/gfs1: Transport endpoint is not connected
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                ls: cannot access /mnt/sec-gfs1: Transport endpoint is not connected

                Try easy solution first:
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ umount /mnt/gfs1
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ umount /mnt/sec-gfs1
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                After issuing above commands, retry whole step starting with 2.3.2.

                If fails for the second time, investigate logs in /var/log/glusterfs/, /var/log/messages, dmesg
                [If no solution found in 10 minutes, escalate to BU SaaS]

[x]         2.3.4. Check volumes status:
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ gluster volume status
                Status of volume: gfs1
                Gluster process                             TCP Port  RDMA Port  Online  Pid
                ------------------------------------------------------------------------------
                Brick gfs1-1.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       3510
                Brick gfs1-2.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       31340
                Brick gfs1-3.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       3316
                Brick gfs1-4.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       31413
                Self-heal Daemon on localhost               N/A       N/A        Y       3665
                Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       20255
                Self-heal Daemon on gfs1-2.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       77317
                Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       66457

                Task Status of Volume gfs1
                ------------------------------------------------------------------------------
                There are no active volume tasks

                Status of volume: sec-gfs1
                Gluster process                             TCP Port  RDMA Port  Online  Pid
                ------------------------------------------------------------------------------
                Brick gfs1-1.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       3521
                Brick gfs1-2.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       31351
                Brick gfs1-3.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       3339
                Brick gfs1-4.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       31424
                Self-heal Daemon on localhost               N/A       N/A        Y       3665
                Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       20255
                Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       66457
                Self-heal Daemon on gfs1-2.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       77317

                Task Status of Volume sec-gfs1
                ------------------------------------------------------------------------------
                There are no active volume tasks

                If any of bricks above are offline, perform following on failed node:
                    [local]% ssh <IP address of the node>
                    [root@gfs1-<node ID>.iad-aws.prod.sli.io ~]$ gluster volume start <volume name> force

                If above command's output will state that there are active volume tasks, continue, but doublecheck in 1h and notify BU SaaS.


[x]         2.3.5. Check peers' status:
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ gluster peer status
                Number of Peers: 3

                Hostname: gfs1-4.iad-aws.prod.sli.io
                Uuid: 0183b376-1aa0-4d98-945e-6197a8876f27
                State: Peer in Cluster (Connected)

                Hostname: gfs1-2.iad-aws.prod.sli.io
                Uuid: 95e9d91b-fe51-4e0d-86cd-00097890f113
                State: Peer in Cluster (Connected)

                Hostname: gfs1-3.iad-aws.prod.sli.io
                Uuid: 23884f97-8149-4dd9-b4d3-1faccee03d1a
                State: Peer in Cluster (Connected)

                If there are issues in above output, something unexpected happened. As if all bricks are online on previous step, but some of peers are disconnected, this is totally wrong. So escalate to BU SaaS and continue.


[x]         2.3.6. Switch on CRON
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ systemctl enable crond
                [root@gfs1-1.iad-aws.prod.sli.io ~]$ systemctl start crond


[x]     2.4. Perform GFS healthcheck for instance i-08738fdcf35071be8 (gfs1-2.iad-aws.prod.sli.io).

[x]         2.4.0. Connect to the instance:
                ssh 10.79.114.45

[x]         2.4.1. Check gluster daemon is running:
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ systemctl status glusterd

                If not started, (re)start glusterd
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ systemctl restart glusterd

                If still fail to start all daemons (glusterd, glusterfsd, glusterfs), investigate logs in /var/log/glusterfs/
                [If no solution found in 10 minutes, escalate to BU SaaS]

[x]         2.4.2. Check mount:
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ mount|grep gfs1
                /dev/mapper/vg01-gfs1.brick1 on /gluster/gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                /gluster/sec-gfs1/sec-gfs1.brick1 on /gluster/sec-gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                gfs1-2.iad-aws.prod.sli.io:/sec-gfs1 on /mnt/sec-gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
                gfs1-2.iad-aws.prod.sli.io:/gfs1 on /mnt/gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

                If mounts are missing, try to mount volumes:
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                If mount fails, investigate logs in /var/log/glusterfs/, /var/log/messages, dmesg
                [If no solution found in 10 minutes, escalate to BU SaaS]


[x]         2.4.3. Check mount is healthy:
                Following commands should produce no output:
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null

                If nothing printed out, proceed to the next step.

                Following is an example of mounted volume, which has issues:
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null
                ls: cannot access /mnt/gfs1: Transport endpoint is not connected
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                ls: cannot access /mnt/sec-gfs1: Transport endpoint is not connected

                Try easy solution first:
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ umount /mnt/gfs1
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ umount /mnt/sec-gfs1
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                After issuing above commands, retry whole step starting with 2.4.2.

                If fails for the second time, investigate logs in /var/log/glusterfs/, /var/log/messages, dmesg
                [If no solution found in 10 minutes, escalate to BU SaaS]

[x]         2.4.4. Check volumes status:
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ gluster volume status
                Status of volume: gfs1
                Gluster process                             TCP Port  RDMA Port  Online  Pid
                ------------------------------------------------------------------------------
                Brick gfs1-1.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       3510
                Brick gfs1-2.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       31340
                Brick gfs1-3.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       3316
                Brick gfs1-4.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       31413
                Self-heal Daemon on localhost               N/A       N/A        Y       3665
                Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       20255
                Self-heal Daemon on gfs1-2.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       77317
                Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       66457

                Task Status of Volume gfs1
                ------------------------------------------------------------------------------
                There are no active volume tasks

                Status of volume: sec-gfs1
                Gluster process                             TCP Port  RDMA Port  Online  Pid
                ------------------------------------------------------------------------------
                Brick gfs1-1.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       3521
                Brick gfs1-2.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       31351
                Brick gfs1-3.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       3339
                Brick gfs1-4.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       31424
                Self-heal Daemon on localhost               N/A       N/A        Y       3665
                Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       20255
                Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       66457
                Self-heal Daemon on gfs1-2.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       77317

                Task Status of Volume sec-gfs1
                ------------------------------------------------------------------------------
                There are no active volume tasks

                If any of bricks above are offline, perform following on failed node:
                    [local]% ssh <IP address of the node>
                    [root@gfs1-<node ID>.iad-aws.prod.sli.io ~]$ gluster volume start <volume name> force

                If above command's output will state that there are active volume tasks, continue, but doublecheck in 1h and notify BU SaaS.


[x]         2.4.5. Check peers' status:
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ gluster peer status
                Number of Peers: 3

                Hostname: gfs1-1.iad.prod.sli.io
                Uuid: c9381ca4-99ac-4e69-8823-b9b8e208f873
                State: Peer in Cluster (Connected)

                Hostname: gfs1-3.iad-aws.prod.sli.io
                Uuid: 23884f97-8149-4dd9-b4d3-1faccee03d1a
                State: Peer in Cluster (Connected)

                Hostname: gfs1-4.iad-aws.prod.sli.io
                Uuid: 0183b376-1aa0-4d98-945e-6197a8876f27
                State: Peer in Cluster (Connected)

                If there are issues in above output, something unexpected happened. As if all bricks are online on previous step, but some of peers are disconnected, this is totally wrong. So escalate to BU SaaS and continue.


[x]         2.4.6. Switch on CRON
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ systemctl enable crond
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ systemctl start crond


[x]     2.5. Perform GFS healthcheck for instance i-0a0c2ad3bba430706 (gfs1-3.iad-aws.prod.sli.io).

[x]         2.5.0. Connect to the instance:
                ssh 10.79.114.199

[x]         2.5.1. Check gluster daemon is running:
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ systemctl status glusterd

                If not started, (re)start glusterd
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ systemctl restart glusterd

                If still fail to start all daemons (glusterd, glusterfsd, glusterfs), investigate logs in /var/log/glusterfs/
                [If no solution found in 10 minutes, escalate to BU SaaS]

[x]         2.5.2. Check mount:
                [root@gfs1-2.iad-aws.prod.sli.io ~]$ mount|grep gfs1
                /dev/mapper/vg01-gfs1.brick1 on /gluster/gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                /gluster/sec-gfs1/sec-gfs1.brick1 on /gluster/sec-gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                gfs1-3.iad-aws.prod.sli.io:/sec-gfs1 on /mnt/sec-gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
                gfs1-3.iad-aws.prod.sli.io:/gfs1 on /mnt/gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

                If mounts are missing, try to mount volumes:
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                If mount fails, investigate logs in /var/log/glusterfs/, /var/log/messages, dmesg
                [If no solution found in 10 minutes, escalate to BU SaaS]


[x]         2.5.3. Check mount is healthy:
                Following commands should produce no output:
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null

                If nothing printed out, proceed to the next step.

                Following is an example of mounted volume, which has issues:
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null
                ls: cannot access /mnt/gfs1: Transport endpoint is not connected
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                ls: cannot access /mnt/sec-gfs1: Transport endpoint is not connected

                Try easy solution first:
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ umount /mnt/gfs1
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ umount /mnt/sec-gfs1
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                After issuing above commands, retry whole step starting with 2.5.2.

                If fails for the second time, investigate logs in /var/log/glusterfs/, /var/log/messages, dmesg
                [If no solution found in 10 minutes, escalate to BU SaaS]

[x]         2.5.4. Check volumes status:
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ gluster volume status
                Status of volume: gfs1
                Gluster process                             TCP Port  RDMA Port  Online  Pid
                ------------------------------------------------------------------------------
                Brick gfs1-1.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       3510
                Brick gfs1-2.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       31340
                Brick gfs1-3.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       3316
                Brick gfs1-4.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       31413
                Self-heal Daemon on localhost               N/A       N/A        Y       3665
                Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       20255
                Self-heal Daemon on gfs1-2.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       77317
                Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       66457

                Task Status of Volume gfs1
                ------------------------------------------------------------------------------
                There are no active volume tasks

                Status of volume: sec-gfs1
                Gluster process                             TCP Port  RDMA Port  Online  Pid
                ------------------------------------------------------------------------------
                Brick gfs1-1.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       3521
                Brick gfs1-2.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       31351
                Brick gfs1-3.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       3339
                Brick gfs1-4.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       31424
                Self-heal Daemon on localhost               N/A       N/A        Y       3665
                Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       20255
                Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       66457
                Self-heal Daemon on gfs1-2.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       77317

                Task Status of Volume sec-gfs1
                ------------------------------------------------------------------------------
                There are no active volume tasks

                If any of bricks above are offline, perform following on failed node:
                    [local]% ssh <IP address of the node>
                    [root@gfs1-<node ID>.iad-aws.prod.sli.io ~]$ gluster volume start <volume name> force

                If above command's output will state that there are active volume tasks, continue, but doublecheck in 1h and notify BU SaaS.


[x]         2.5.5. Check peers' status:
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ gluster peer status
                Number of Peers: 3

                Hostname: gfs1-4.iad-aws.prod.sli.io
                Uuid: 0183b376-1aa0-4d98-945e-6197a8876f27
                State: Peer in Cluster (Connected)

                Hostname: gfs1-2.iad-aws.prod.sli.io
                Uuid: 95e9d91b-fe51-4e0d-86cd-00097890f113
                State: Peer in Cluster (Connected)

                Hostname: gfs1-1.iad.prod.sli.io
                Uuid: c9381ca4-99ac-4e69-8823-b9b8e208f873
                State: Peer in Cluster (Connected)

                If there are issues in above output, something unexpected happened. As if all bricks are online on previous step, but some of peers are disconnected, this is totally wrong. So escalate to BU SaaS and continue.

[x]         2.5.6. Switch on CRON
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ systemctl enable crond
                [root@gfs1-3.iad-aws.prod.sli.io ~]$ systemctl start crond


[x]     2.6. Perform GFS healthcheck for instance i-001707ebdd3539c48 (gfs1-4.iad-aws.prod.sli.io).

[x]         2.6.0. Connect to the instance:
                ssh 10.79.114.67

[x]         2.6.1. Check gluster daemon is running:
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ systemctl status glusterd

                If not started, (re)start glusterd
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ systemctl restart glusterd

                If still fail to start all daemons (glusterd, glusterfsd, glusterfs), investigate logs in /var/log/glusterfs/
                [If no solution found in 10 minutes, escalate to BU SaaS]

[x]         2.6.2. Check mount:
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ mount|grep gfs1
                /dev/mapper/vg01-gfs1.brick1 on /gluster/gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                /gluster/sec-gfs1/sec-gfs1.brick1 on /gluster/sec-gfs1/brick1 type xfs (rw,relatime,attr2,inode64,noquota)
                gfs1-4.iad-aws.prod.sli.io:/sec-gfs1 on /mnt/sec-gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
                gfs1-4.iad-aws.prod.sli.io:/gfs1 on /mnt/gfs1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

                If mounts are missing, try to mount volumes:
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                If mount fails, investigate logs in /var/log/glusterfs/, /var/log/messages, dmesg
                [If no solution found in 10 minutes, escalate to BU SaaS]


[x]         2.6.3. Check mount is healthy:
                Following commands should produce no output:
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null

                If nothing printed out, proceed to the next step.

                Following is an example of mounted volume, which has issues:
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ ls /mnt/gfs1 1>/dev/null
                ls: cannot access /mnt/gfs1: Transport endpoint is not connected
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ ls /mnt/sec-gfs1 1>/dev/null
                ls: cannot access /mnt/sec-gfs1: Transport endpoint is not connected

                Try easy solution first:
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ umount /mnt/gfs1
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ umount /mnt/sec-gfs1
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ mount /mnt/gfs1
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ mount /mnt/sec-gfs1

                After issuing above commands, retry whole step starting with 2.6.2.

                If fails for the second time, investigate logs in /var/log/glusterfs/, /var/log/messages, dmesg
                [If no solution found in 10 minutes, escalate to BU SaaS]

[x]         2.6.4. Check volumes status:
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ gluster volume status
                Status of volume: gfs1
                Gluster process                             TCP Port  RDMA Port  Online  Pid
                ------------------------------------------------------------------------------
                Brick gfs1-1.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       3510
                Brick gfs1-2.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       31340
                Brick gfs1-3.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       3316
                Brick gfs1-4.iad-aws.prod.sli.io:/gluster/g
                fs1/brick1                                  49152     0          Y       31413
                Self-heal Daemon on localhost               N/A       N/A        Y       3665
                Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       20255
                Self-heal Daemon on gfs1-2.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       77317
                Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       66457

                Task Status of Volume gfs1
                ------------------------------------------------------------------------------
                There are no active volume tasks

                Status of volume: sec-gfs1
                Gluster process                             TCP Port  RDMA Port  Online  Pid
                ------------------------------------------------------------------------------
                Brick gfs1-1.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       3521
                Brick gfs1-2.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       31351
                Brick gfs1-3.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       3339
                Brick gfs1-4.iad-aws.prod.sli.io:/gluster/s
                ec-gfs1/brick1                              49153     0          Y       31424
                Self-heal Daemon on localhost               N/A       N/A        Y       3665
                Self-heal Daemon on gfs1-3.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       20255
                Self-heal Daemon on gfs1-4.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       66457
                Self-heal Daemon on gfs1-2.iad-aws.prod.sli
                .io                                         N/A       N/A        Y       77317

                Task Status of Volume sec-gfs1
                ------------------------------------------------------------------------------
                There are no active volume tasks

                If any of bricks above are offline, perform following on failed node:
                    [local]% ssh <IP address of the node>
                    [root@gfs1-<node ID>.iad-aws.prod.sli.io ~]$ gluster volume start <volume name> force

                If above command's output will state that there are active volume tasks, continue, but doublecheck in 1h and notify BU SaaS.


[x]         2.6.5. Check peers' status:
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ gluster peer status
                Number of Peers: 3

                Hostname: gfs1-3.iad-aws.prod.sli.io
                Uuid: 23884f97-8149-4dd9-b4d3-1faccee03d1a
                State: Peer in Cluster (Connected)

                Hostname: gfs1-2.iad-aws.prod.sli.io
                Uuid: 95e9d91b-fe51-4e0d-86cd-00097890f113
                State: Peer in Cluster (Connected)

                Hostname: gfs1-1.iad.prod.sli.io
                Uuid: c9381ca4-99ac-4e69-8823-b9b8e208f873
                State: Peer in Cluster (Connected)

                If there are issues in above output, something unexpected happened. As if all bricks are online on previous step, but some of peers are disconnected, this is totally wrong. So escalate to BU SaaS and continue.

[x]         2.6.6. Switch on CRON
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ systemctl enable crond
                [root@gfs1-4.iad-aws.prod.sli.io ~]$ systemctl start crond
```

```
        2.7. Checking remain cluster components' health
            2.7.1. Search servers
[x]             2.7.1.1. Ensure mounts
                    [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do echo ${host}; ssh ${host} "mount | grep gfs1"; done

                    all hosts should report both gfs1 and sec-gfs1 are mounted
                        systemd-1 on /mnt/sec-gfs1 type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10485)
                        systemd-1 on /mnt/gfs1 type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10489)
                        gfs1.iad-aws.prod.sli.io:/sec-gfs1 on /mnt/sec-gfs1 type fuse.glusterfs (ro,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
                        gfs1.iad-aws.prod.sli.io:/gfs1 on /mnt/gfs1 type fuse.glusterfs (ro,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

                    If any host will not return all expected mounts as an output, mount volumes manually:
                         [local]% ssh <IP address of the host> "mount /mnt/gfs1; mount /mnt/sec-gfs1"

                    If mounting unsuccessful, escalate to BU SaaS.

[x]             2.7.1.2. Ensure mounts are healthy
                    [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do echo ${host}; ssh ${host} "ls /mnt/sec-gfs1 1>/dev/null"; done
                    [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do echo ${host}; ssh ${host} "ls /mnt/gfs1 1>/dev/null"; done

                    All hosts should produce no output.
                    If nothing printed out, proceed to the next step.

                    If any host will produce output like following:
                    ls: cannot access /mnt/gfs1: Transport endpoint is not connected
                    or
                    ls: cannot access /mnt/sec-gfs1: Transport endpoint is not connected

                    Remount volumes on a specific hosts:
                    [local]% ssh <IP address of the host> "umount /mnt/gfs1; umount /mnt/sec-gfs1; sync; sleep 2; mount /mnt/gfs1; mount /mnt/sec-gfs1"

                    Recheck mounts starting from 2.7.1.1.
                    [if there are issues after second run, escalate to BU SaaS]

[x]             2.7.1.3. Ensure Apache is running
                    [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do echo ${host}; ssh ${host} "systemctl status httpd -l"; done

                    on any problem, restart apache on a respective host:
                        [local]% ssh <IP address of the host> "systemctl restart httpd"

[x]             2.7.1.3. Ensure Localbrain is running
                    [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do echo ${host}; ssh ${host} "systemctl status braind -l"; done

                    on any problem, restart apache on a respective host:
                        [local]% ssh <IP address of the host> "systemctl stop braind; sleep 30; systemctl start braind"

            2.7.2. Dory servers
[x]             2.7.2.1. Ensure mounts
                    [local]% for host in 10.79.114.251 10.79.114.88; do echo ${host}; ssh ${host} "mount | grep gfs1"; done

                    all hosts should report gfs1 is mounted
                        gfs1.iad-aws.prod.sli.io:/gfs1 on /mnt/gfs1 type fuse.glusterfs (ro,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

                    If any host will not return all expected mounts as an output, mount volumes manually:
                         [local]% ssh <IP address of the host> "mount /mnt/gfs1"

                    If mounting unsuccessful, escalate to BU SaaS.

[x]             2.7.2.2. Ensure mounts are healthy
                    [local]% for host in 10.79.114.251 10.79.114.88; do echo ${host}; ssh ${host} "ls /mnt/gfs1 1>/dev/null"; done

                    All hosts should produce no output.
                    If nothing printed out, proceed to the next step.

                    If any host will produce output like following:
                    ls: cannot access /mnt/gfs1: Transport endpoint is not connected

                    Remount volumes on a specific hosts:
                    [local]% ssh <IP address of the host> "umount /mnt/gfs1; sync; sleep 2; mount /mnt/gfs1"

                    Recheck mounts starting from 2.7.2.1.
                    [if there are issues after second run, escalate to BU SaaS]

[x]             2.7.2.3. Ensure Apache is running
                    [local]% for host in 10.79.114.251 10.79.114.88; do echo ${host}; ssh ${host} "systemctl status httpd -l"; done

                    on any problem, restart apache on a respective host:
                        [local]% ssh <IP address of the host> "systemctl restart httpd"

[x]             2.7.2.3. Ensure Dory is running
                    [local]% for host in 10.79.114.251 10.79.114.88; do echo ${host}; ssh ${host} "systemctl status dory -l"; done

                    on any problem, restart apache on a respective host:
                        [local]% ssh <IP address of the host> "systemctl stop dory; sleep 30; systemctl start dory"

[ ]         2.7.3. Ensure servers return valid information instead of errors.
                Use Nagios (https://nagios-master.sli.io/nagios/) monitoring to check for errors. If nagios reports issues like following (errors 400 or 404) for any IAD cluster servers, escalate to BU SaaS

                    search1-3.iad.prod asahi braindead
                    Notifications for this service have been disabled
                    CRITICAL	02-08-2021 22:45:42	5d 21h 24m 45s	3/3	asahi/search1-3.iad-aws.prod.sli.io returned 400
```

```
If any of the following steps fail, escalate to BU SaaS.

[ ]     2.8. Returning Cluster 1 (IAD) back to rotation:
            [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do ssh ${host} 'runuser -l gbrain bash -c "webcontrol vip insert"'; done
            (allow a minute before performing next step)

        2.9. Ensure Cluster 1 (IAD) is serving to the clients
[ ]         2.9.1. Check vip status on searches:
                [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do ssh ${host} 'runuser -l gbrain bash -c "webcontrol vip status"'; done

[ ]         2.9.2. Request healthcheck document from the search hosts:
                [local]% for host in 10.79.114.86 10.79.114.150 10.79.114.92 10.79.114.15 10.79.114.94 10.79.114.65; do echo -n "${host}: "; curl -A "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0" http://${host}/shared/vip-status.txt; done
                expecting all hosts to return UP
```


```
3. Processing Cluser 2 (DFW)
4. Processing Cluser 3 (SEA)
```
