RabbitMQ

Introduction

RabbitMQ is a message broker: it accepts and forwards messages. You can think about it as a post office: when you put the mail that you want posting in a post box, you can be sure that the letter carrier will eventually deliver the mail to your recipient. In this analogy, RabbitMQ is a post box, a post office, and a letter carrier.

it accepts, stores, and forwards binary blobs of data ‒ messages.

A program that sends messages is a producer.

A queue is the name for the post box in RabbitMQ. Messages can only be stored inside a queue. Many producers can send messages that go to one queue, and many consumers can try to receive data from one queue.

A consumer is a program that mostly waits to receive messages.

Note that the producer, consumer, and broker do not have to reside on the same host; indeed in most applications they don't. An application can be both a producer and consumer, too.

Example

Producer sends messages to the "hello" queue. The consumer receives messages from that queue.

Pika is the Python client recommended by the RabbitMQ team.

python -m pip install pika --upgrade

Sending

send.py will send a single message to the queue:

#!/usr/bin/env python
import pika

# establish a connection with RabbitMQ server.
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

Let's create a hello queue to which the message will be delivered:

channel.queue_declare(queue='hello')

In RabbitMQ a message can never be sent directly to the queue, it always needs to go through an exchange.
Use a default exchange identified by an empty string. This exchange is special ‒ it allows us to specify exactly to which queue the message should go. The queue name needs to be specified in the routing_key parameter:

channel.basic_publish(exchange='',
                      routing_key='hello',
                      body='Hello World!')
print(" [x] Sent 'Hello World!'")

# make sure the network buffers were flushed and our message was actually delivered
connection.close()

Receiving

Our second program receive.py will receive messages from the queue and print them on the screen.
It's a good practice to repeat declaring the queue in both programs.

#!/usr/bin/env python
import pika

# establish a connection with RabbitMQ server.
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

Receiving messages from the queue is more complex. It works by subscribing a callback function to a queue. Whenever we receive a message, this callback function is called by the Pika library.

def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)

# We need to tell RabbitMQ that this particular callback function should receive messages from our hello queue:
channel.basic_consume(queue='hello',
                      auto_ack=True,
                      on_message_callback=callback)

Run example

Run the Docker image:

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.11-management

Install pika.
Open a terminal and execute:

python receive.py
# => [*] Waiting for messages. To exit press CTRL+C

Open another terminal and execute:

python send.py
# => [x] Sent 'Hello World!'

Part 2

The main idea behind Work Queues (aka: Task Queues) is to avoid doing a resource-intensive task immediately and having to wait for it to complete. Instead we schedule the task to be done later. We encapsulate a task as a message and send it to the queue. A worker process running in the background will pop the tasks and eventually execute the job. When you run many workers the tasks will be shared between them.

We will slightly modify the send.py code from our previous example, to allow arbitrary messages to be sent from the command line. This program will schedule tasks to our work queue, so let's name it new_task.py: