## I. How to create and configure a RabbitMQ cluster

### 0. Prerequisites
- 3 VMs (rabbit-1, rabbit-2, rabbit-3) running Ubuntu 20.04/22.04 with hostname properly configured
- SSH access to each VM

### 1. Initial setup
- Update and upgrade each VM: `sudo apt update && sudo apt upgrade -y`
- Install RabbitMQ on all nodes: `sudo apt install rabbitmq-server`
- Enable RabbitMQ management plugin: `sudo rabbitmq-plugins enable rabbitmq_management`

### 2. Cluster configuration
- On the first node **(rabbit-1)**, create the cluster by running:
- `sudo rabbitmqctl stop_app`
- `sudo rabbitmqctl reset`
- `sudo rabbitmqctl start_app`
- `sudo cat /var/lib/rabbitmq/.erlang.cookie` -> `SJ2Et8HOmHJRBwOPOxs1kHzT4vGt7nUM17sBByW/MWP3fcxUNyISiYmi`
Copy the cookie to other nodes
- On the second and third nodes **(rabbit-2, rabbit-3)**, join the cluster by running:
- `sudo systemctl stop rabbitmq-server`
- `echo 'SJ2Et8HOmHJRBwOPOxs1kHzT4vGt7nUM17sBByW/MWP3fcxUNyISiYmi' > /var/lib/rabbitmq/.erlang.cookie`
- `sudo systemctl start rabbitmq-server`
- `sudo rabbitmqctl stop_app`
- `sudo rabbitmqctl reset`
- `sudo rabbitmqctl join_cluster --ram rabbit@rabbit-1`
- `sudo rabbitmqctl start_app`

### 3. Verification
- Access the RabbitMQ web management interface at http://rabbit-1:15672/ with the username and password set up during cluster configuration
- Check that all nodes are listed as part of the cluster in the "Cluster Nodes" section of the management interface

## II. RabbitMQ durable queue

### `Vhost`, `Exchanges`, `Queues`, and `Bindings`

Config environment

export VHOST_NAME=app-1
export USER_NAME=app-1-user
export USER_PASSWORD=adminpassword
export EXCHANGE_NAME=app-1-ex
export QUEUE_NAME=app-1-q
export ROUTING_KEY=app-1-rk

### **0. Create a Vhost and user**

sudo rabbitmqctl add_vhost $VHOST_NAME
sudo rabbitmqctl add_user $USER_NAME $USER_PASSWORD
sudo rabbitmqctl set_user_tags $USER_NAME administrator
sudo rabbitmqctl set_permissions $USER_NAME --vhost $VHOST_NAME ".*" ".*" ".*"
sudo rabbitmqctl list_vhosts
sudo rabbitmqctl list_user_permissions $USER_NAME

### **1. Creating an Exchange**

To create an exchange in RabbitMQ, run the following command:

sudo rabbitmqadmin -u $USER_NAME -p $USER_PASSWORD -V $VHOST_NAME \
     declare exchange \
     name=$EXCHANGE_NAME type=direct

The type of the exchange available `direct`, `topic`, or `fanout`.

### **2. Creating a durable Queue**

To create a queue in RabbitMQ, run the following command:

sudo rabbitmqadmin -u $USER_NAME -p $USER_PASSWORD -V $VHOST_NAME \
     declare queue \
     name=$QUEUE_NAME durable=true queue_type=quorum

### **3. Binding an Exchange to a Queue**

To bind an exchange to a queue in RabbitMQ, run the following command:

sudo rabbitmqadmin -u $USER_NAME -p $USER_PASSWORD -V $VHOST_NAME \
     declare binding \
     source=$EXCHANGE_NAME destination_type=queue destination=$QUEUE_NAME routing_key=$ROUTING_KEY

### **4. Publishing a Message to a Quorum Queue**

To publish a message to a quorum queue in RabbitMQ, run the following command on node **rabbit-1**:

sudo rabbitmqadmin -u $USER_NAME -p $USER_PASSWORD -V $VHOST_NAME \
     publish payload="Hello rabbit `date`" \
     exchange=$EXCHANGE_NAME routing_key=$ROUTING_KEY 

### **5. Testing durability**

To verify the message was published, run the following command:

sudo sudo rabbitmqadmin -u $USER_NAME -p $USER_PASSWORD -V $VHOST_NAME \
     get queue=$QUEUE_NAME

Now poweroff the **rabbit-1** and run get message on **rabbit-2** and **rabbit-3**

ubuntu@rabiit-1:~$ sudo poweroff  

ubuntu@rabiit-2:~$ export VHOST_NAME=app-1
export USER_NAME=app-1-user
export USER_PASSWORD=adminpassword
export EXCHANGE_NAME=app-1-ex
export QUEUE_NAME=app-1-q
export ROUTING_KEY=app-1-rk
ubuntu@rabiit-2:~$ sudo rabbitmqadmin -u $USER_NAME -p $USER_PASSWORD -V $VHOST_NAME \
     get queue=$QUEUE_NAME
     
ubuntu@rabiit-3:~$ export VHOST_NAME=app-1
export USER_NAME=app-1-user
export USER_PASSWORD=adminpassword
export EXCHANGE_NAME=app-1-ex
export QUEUE_NAME=app-1-q
export ROUTING_KEY=app-1-rk
ubuntu@rabiit-2:~$ sudo rabbitmqadmin -u $USER_NAME -p $USER_PASSWORD -V $VHOST_NAME \
     get queue=$QUEUE_NAME

## III. Config a single access point and failover with keepalived

To set up Keepalived with RabbitMQ cluster, follow these steps:

### **1. Install Keepalived on all nodes in the RabbitMQ cluster:**

   $ sudo apt-get install keepalived

### **2. Edit the Keepalived configuration file**

`/etc/keepalived/keepalived.conf` on each node to include the virtual IP address and the RabbitMQ cluster nodes:

   global_defs {
       router_id rabbit1
   }
   vrrp_script chk_rabbitmq {
       script "/usr/local/bin/rabbitmq-check"
       interval 2
   }
   vrrp_instance VI_1 {
       state MASTER
       interface ens3
       virtual_router_id 51
       priority 101
       authentication {
           auth_type PASS
           auth_pass rabbitmq
       }
       virtual_ipaddress {
           10.248.20.100
       }
       track_script {
           chk_rabbitmq
       }
   }

Replace `rabbit1` with a unique router ID on each node. Modify `interface` and `virtual_ipaddress` values as appropriate for your environment.

**Important**: If all three servers in a Keepalived configuration have the same priority number, Keepalived will not be able to determine which server should be the master. This can result in conflicts and instability in the cluster --> **Increase the proirity for each node**

## rabbit1: priority 101
## rabbit2: priority 102
## rabbit3: priority 103

## In this sequence, the master ip will likely fallback from rabbit-3 --> rabbit-2 --> rabbit-1

### **3. Create the `rabbitmq-check` script on each node with the following content:**

#!/bin/bash

# RabbitMQ check script for Keepalived
#
# Check if the RabbitMQ service is running and reachable

# Define the RabbitMQ host and port
RABBITMQ_HOST="localhost"
RABBITMQ_PORT="5672"

# Check RabbitMQ status
sudo rabbitmqctl status 2>&1 >/dev/null

# Check the exit status of command
if [ $? -eq 0 ]; then
    exit 0
else
    exit 1
fi

> **Future** Need to modify this script to check with a real rabbitmq client

Make the script executable:

   $ sudo chmod +x /usr/local/bin/rabbitmq-check

### **4. Restart the Keepalived service on all nodes:**

   $ sudo service keepalived restart

Now, you should have Keepalived running with a virtual IP address that will move between the nodes in the RabbitMQ cluster if one of them goes down.

### **5. Tesing keepalive IP**

Use the command below to test the connectivity with the keepalive IP while stopping `rabbitmq-server` in each node

while true; do sudo rabbitmqadmin --host 10.248.20.100 -u $USER_NAME -p $USER_PASSWORD -V $VHOST_NAME      get queue=$QUEUE_NAME; done

Now we can use the IP 10.248.20.100 as a signle access point for our RabbitMQ

## IV. Load balance with a single access point and failover with keepalived

To maximize resource utilization in a highly available RabbitMQ cluster, you can employ a load-balancing setup with Keepalived. This design ensures that traffic is distributed across multiple servers, allowing for efficient utilization of resources.

   git clone https://github.com/bloomberg/amqpprox.git
   cd amqpprox/
   sudo add-apt-repository ppa:ubuntu-toolchain-r/test
   sudo apt-get update
   sudo apt install gcc-10 gcc-10-base gcc-10-doc g++-10 -y
   sudo apt install libstdc++-10-dev libstdc++-10-doc  -y
   sudo apt-get  install make cmake -y
   sudo apt install python3-pip
   pip install conan==1.59
   conan profile detect
   make setup 
   make init
   make