---
# System prepended metadata

title: RabbitMQ TWG Notes

---

# RabbitMQ TWG Notes

## Pinned notes

https://confluence.diamond.ac.uk/display/SSCC/RabbitMQ

### Tasks

BW: Looking into RabbitMQ server configuration, DH offers to assist
JH: FITSM/ITSM
~~RG: Added RMQ version of queue monitor [(dlstbx/#58)](https://github.com/DiamondLightSource/python-dlstbx/pull/58)~~
SciComp: [HTTPS to admin proxy (SC-2991)](https://jira.diamond.ac.uk/browse/SC-2991)
SciComp: [Host aliases for RabbitMQ servers (SC-2969)](https://jira.diamond.ac.uk/browse/SC-2969)
SciComp: [Set up systemd script for RabbitMQ (SC-2967)](https://jira.diamond.ac.uk/browse/SC-2967)
RG: To chase up above tickets so they can be resolved before project week
DH: 2-way bridge
RG: did set this up monitoring, this is running
??: Alerting

### Upcoming project week
2021-08-13 (Friday) to 2021-08-19 (Thursday)

* Document and link monitoring on Confluence; Alerting
  * https://gitlab.diamond.ac.uk/scisoft/zocalo/-/tree/master/monitoring
* Setting up logging RabbitMQ -> Graylog
  * Erlang 23 / RabbitMQ 3.8: Logging via Lager. That can probably be configured/connected to Graylog (GELF) in some way 
  * ~~Erlang 24 offers a new logging framework (name?) which may make this easier. We can deploy RabbitMQ 3.9+Erlang 24 via the Jenkins job. May or may not suffice to make use of the new logging framework.~~
  * Try to use lager
    * https://github.com/esl/lager_graylog last release 2018, but last activity 2020
    * https://github.com/silviucpp/graylog_lager last activity June, was fix for Erlang 24
* Identify a service for 2-way bridge and set this up
  * Need configuration/support to start a service on RabbitMQ
    * all --live/--test command line arguments, replaced by -e/--environment
  * Service suggestions:
    * Images services, fairly high number of messages, but lost messages don't really matter, leaf node
    * Dispatcher? Could update Synchweb to submit to RMQ
    * Note: liveness-messages go on the same server, so if the Controller runs on ActiveMQ then it won't see RabbitMQ services. We could spin up 2 controllers, but then they'd need separate configuration files. Probably best start with a non-critical service that only runs on RMQ (run on toolservers, or from a cluster array job) and figure out the rest from there.
* FitSM
  * James: Thursday
* Monitoring: See if we can get individual queue monitoring / queue selection on Grafana
* First test scenario
  * ActiveMQ: Run SampleTxnProducer - writes to transient.transaction
  * Bridge: transient.transaction ActiveMQ -> RabbitMQ
  * RabbitMQ: Run SampleTxn - reads from t.t, writes to transient.destination
  * Bridge: transient.destination RabbitMQ -> ActiveMQ
  * ActiveMQ: Run SampleConsumer - reads from t.d
* Actions needed to make this happen:
  * ~~MG: make workflows/zocalo releases, reinstall dials/now~~
  * ~~MG: create second configuration file for RabbitMQ and dials+rabbitmq modules that point to that second config file~~
  * ~~DH: add transient.transaction to the RabbitMQ config~~
  * rewrite pikatransport
  * make `transient.status` a topic

## 2021-03-08
* PMG has worked on the Workflows transport class
* VM 3 misbehaving (DLS-33639)
* Script to start a server:
```bash
module load rabbitmq
/dls_sw/apps/rabbitmq/configuration/start-cluster
```
* What user will the servers run as?
  * Functional account, MG will contact MC and set this up together
* How will the servers be started? (systemd, other, ..?)
  * Will be systemd

## 2021-03-01
* MC has joined us from Scientific Computing
* Machines to be racked at the beginning of Run 2, possibly in Shutdown 1 if MC can find the time
* Basic monitoring with Nagios via Scientific Computing
* Further monitoring to be discussed between PMG and MC
* JH to discuss service catalogue with Anton.

## 2021-02-22

* PMG has made progress on Workflows+Pika
* AH is working on a configuration generator and the first bits have been merged into the zocalo configuration repository
* JH will meet up with NS and AL to discuss how we can approach FitSM for the RabbitMQ service
* There has been no update from Scientific Computing on delivery schedules, at this point it looks unlikely that we will make the next run
* MG to send an update to the project board

## 2021-02-17

JH is the project lead for the FitSM component.


## 2021-02-01

##### `module load rabbitmq` limitations
* Plugins are disabled by default
* PMG has found a way to get this working
* Permissions by default do not allow sending from hosts other than localhost
* Default configuration does not have a user
* Information is on Confluence
* https://confluence.diamond.ac.uk/display/~atd44888/RabbitMQ+Setup

##### Queue setup
* On the dashboard you can import a policies file, that sets up queues and exchanges
* https://www.rabbitmq.com/parameters.html
* Exclusive queues are tied to the lifetime of the connection:
https://www.rabbitmq.com/ha.html#exclusive-queues-are-not-mirrored
https://www.rabbitmq.com/reliability.html#clustering

##### Monitoring
* Scientific computing would like to have rabbitmq in kubernetes because it would make monitoring easier.
* That's not going to happen.

##### Action points:
* [x] PMG to add link to confluence information above
* [ ] MG and PMG to discuss monitoring in more detail
* [ ] PMG and MG to have a look at the configuration/policies


## 2021-01-27

##### [Quorum queues vs. mirrored queues](https://www.rabbitmq.com/quorum-queues.html)
Former are better for data integrity and should be preferred. They do not provide the same features. Notable differences are:
* no exclusivity. We rely on exclusivity
  * in the X-ray centring service: only one instance is allowed to collect results, so that this one instance can aggregate everything
  * in the controller service: to determine which instance is allowed to shut down and spin up and new services.
  * mimas backlog: same reason, only one instance to drip feed jobs into the cluster
  * Further: cluster statistics service, dropfile pickup, `dc_sim_verify`
* no message TTL.
  * We use this in ActiveMQ for development and testing (`transient.` queues), but only because ActiveMQ requires messages to expire before queues can be expired, and what we really care about is the queues expiring.
  * So this is not an issue.

As we need exclusivity we know that not all queues can be quorum queues. We will definitely need some sort of `declare_queue()` function in workflows.

##### Queue configuration
In ActiveMQ queues could be created on the fly. A service could start listen for a name without that name being predefined. Similarly, another service could send messages to a queue that did not already exist.

In RabbitMQ we will need to predefine exchanges/queues in some way:
* services can set up their environment
  * easy to add new services
  * declaration next to code using it
  * obvious problem when eg. PIA service starts before the X-ray centring service, as the destination queue isn't set up
  * Q: Does the configuration translate across the RabbitMQ cluster?
  * Q: What happens if a brand new RabbitMQ cluster node appears? Does the configuration get transferred instantaneously, or will there be a period of potential data loss or failure because destinations are not set up?
* queues can be declared in a central configuration repository
  * more rigid configuration, less flexible
  * single place to go to if you want to know who is responsible for a queue (think DLQ messages)
  * Q: How will this work together with system self tests, where we need to generate hundreds of unique temporary queues?

##### Action points:
* [x] PMG to investigate clustering, monitoring and alerting ([via kubernetes](https://confluence.diamond.ac.uk/display/SC/Kubernetes+User+Guide#KubernetesUserGuide-Monitoring))
* [ ] AH and MG to read through [pika tutorial](https://www.rabbitmq.com/tutorials/tutorial-one-python.html)
* [x] Investigate how other projects do queue configuration


## 2021-01-25

PMG presented her evaluation of 5 client libraries.

Discussion:
* 2 discarded due to development model (we don't really do async/coroutines)
* reference implementation ([pika](https://pypi.org/project/pika/)) has best documentation
* alternative implementation ([amqp](https://github.com/celery/py-amqp)) can be twice as fast, but documentation is out of date
* both implementations are close in terms of the development model

**Result: we will use [pika](https://pypi.org/project/pika/)**, and if performance becomes an issue we can consider switching in the future.

