## OakNorth Growposium :seedling:

Code and Knowledge Cultivation
with Bulbasaur Insights


---

# SNS and SQS

---

## What are SQS and SNS?

---

<img src="https://i.imgflip.com/88kjwn.jpg" title="made at imgflip.com"/>

---

Spot the difference :female-detective:


---

**Both are**:

- parts of Amazon Web Services (AWS)
- message related services :incoming_envelope:
- Exist to help event-driven and cloud-based systems :cloud:
- 'Simple ... Service'
- Often used together :handshake:

---

Simple NOTIFICATION Service

A 'push' technology

<!-- services are configured to 'pull' from SQS -->

<img src="https://hackmd.io/_uploads/ByOY-O18a.png" height="300px"/>

---

Simple QUEUE Service

A 'pull' technology

(services are configured to 'pull' from SQS)

<img src="https://hackmd.io/_uploads/SkZTzO186.png" height="300px"/>

---

## SNS

Simple Notification Service

---

## Before SNS... :book:

<iframe src="https://giphy.com/embed/xT5LMu7NN4BwQBhhcc" width="480" height="362" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>

---

## Before SNS... :book:

Communication via API calls...

...that needed to be polled for updates synchronously...

...one-to-one server/client relationship

...hard to scale to multiple services

...complex error handling

---

SNS acts as a single **message bus** that can send messages to a variety of devices and platforms :bus:

---

A message bus is:

> A communication system that transfers data between services,
> **without each service needing to know about each other**

:bus:

---

This benefit is known as "fire and forget" :fire:

---

<!-- I imagine some parents sending their kids to school without needing to know anything about where the school is. The school also doesn't need to know where the children came from -->

The endpoints don't need to know about each other

<iframe src="https://giphy.com/embed/3orif3E4D4YucjKuUU" width="480" height="366" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>

---

Each service sends their message to the bus
:envelope: :arrow_right: :bus:

The bus delivers the message to the appropriate endpoints
:bus: :arrow_right: :postbox:

---

<!-- basically SNS - a bus driver that knows where it is going, where each passenger needs to get off, what order they need to be dropped off in, whether they need multiple stops -->
<iframe src="https://giphy.com/embed/xT5LMNZyLRrzHez8pG" width="480" height="366" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>

---

### Bus Features

Uni-directional traffic :arrow_right:
<!-- always drives the same route in the same direction, no return journey! -->

---

### Terminology

- **Senders**
(also known as publishers, producers) :package:

- **Receivers**
(also known as processors, consumers or subscribers) :nerd_face:

---

### How to use SNS

---

1. Create a topic

<!-- This is the bus, or message channel -->


---


---

FIFO = no duplicates, order matters, message delivered exactly once :point_up:

Standard = messages delivered at least once, there may be duplicates or messages out of order :repeat:

---

SNS's core functionality:

When you publish a message to a topic, it fans out the message to all subscribed endpoints

<!-- Put a message on the bus, and it can duplicate that message to all the endpoints that are interested in it (I'm imagining zombie A* children at all the middleclass schools in the area) or zombie buses -->

<iframe src="https://giphy.com/embed/l4hLVfpZQf1Ca0bhm" width="480" height="270" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>

---

SNS endpoints (subscribers) could include:
- email
- SMS (texts)
- Lambda functions
- HTTP endpoints
- SQS queues :eyes:

<iframe src="https://giphy.com/embed/efxrDXet2TJILuGRwO" width="480" height="270" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>

---

<!-- One common design pattern is called “fanout.” In this pattern, a message published to an SNS topic is distributed to a number of SQS queues in parallel. By using this pattern, you can build applications that take advantage parallel, asynchronous processing. -->

<!-- Publish a message to a topic every time a new image is uploaded

Separate SQS queues for:
- generate thumbnails
- image recognition
- store metadata about the image
-->

---

<!-- how this looks in terraform coming up -->


---

How this looks in terraform:

module "invited_applicant_details_submitted" {
  source              = "app.terraform.io/oaknorth/aws_sns_topic/aws"
  version             = "1.4.2"
  sns_topic_name      = "onb-authorisation__invited_applicant_details_submitted"
  subscribed_accounts = [module.terraform_remote_state_aws_organization.outputs.all_accounts["AWS Customer Due Diligence"]]
}

---


---

<!-- to include the settings -->


---

<img src="https://hackmd.io/_uploads/ry4fsRzOT.png" height='710px'>

---

Create a subscription to a topic


<!-- Will talk about this in a bit, but the SQS queue is our endpoint -->

---

resource "aws_sns_topic_subscription" "sns_cdd_business_knowledge_final_approved_subscription_sqs" {
  topic_arn = module.terraform_remote_state_cdd_business_knowledge.outputs.final_approved_sns_topic.arn
  protocol  = "sqs"
  endpoint  = module.sqs_cdd_business_knowledge_final_approved.arn
}

---

<!-- Possible settings - Filter policy: A subscription accepts a message only if the message contains attributes that match those specified in the subscription's filter policy. -->


---

<!-- and in terraform? -->

---

<!-- Redrive policy = what to do with messages that haven't been delivered successfully, e.g. max retry count, interaction between DLQ and main queue

A dead letter queue is a special queue that temporarily stores messages that cannot be processed, to prevent source queue from overflowing with unprocessed messages. -->

---


---

## SQS

Simple Queue Service

---

### What is SQS?

A potential subscriber of SNS,

i.e. a message endpoint

<iframe src="https://giphy.com/embed/l2Je9YgJWAx1zF8EE" width="480" height="366" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>

---

SQS provides a hosted message queue (also known as a mailbox)

---

### Before SQS... :book:

The 'producer-consumer problem'

---

### Scenario:

**Buffer**: the shared resource that holds data produced by producer and consumed by consumer (i.e. the queue) :inbox_tray:

**Producer**: generate data, put it into the buffer, repeat :repeat:

**Consumer**: consume data (remove from the buffer), one piece at a time. :knife_fork_plate:

---

The challenge :game_die:

- Coordinating the producers and consumers efficiently so neither has to wait too long
- Avoid problems like data inconsistency or race conditions.

---

Rules :spiral_note_pad:
- Producer must not insert data when buffer is full
- Consumer must not remove data when the buffer is empty
- Producers and consumers must not access the buffer simulataneously

---

### The Solution

Lock the queue

:lock:
<!-- This ensures that only one of the consumer and producer can access the queue at once, and they can't interfere with each others operations -->

---

In computer science, the lock is called a `mutex`

> Before the producer inserts data into the buffer, it acquires the mutex. After inserting the data, it releases the mutex.
>
> Similarly, before the consumer removes data from the buffer, it acquires the mutex. After removing the data, it releases the mutex.

---

### But...

Using mutexes is quite complex :weary:

---

### Mutex complexity

- Avoiding deadlocks :skull_and_crossbones:
- when each thread is waiting for the other

- Starvation :knife_fork_plate:
- When one thread is perpetually denied access to a resource due to a high-priority thread being greedy

---

### Mutex complexity

- Performance and slow processes :snail:
- each thread has to wait for the mutex before it can execute, which can slow down the program

- Error handling
- If a thread holding a mutex crashes, it can leave the system in an undefined state :ghost:

---

Because of this complexity, lots of higher-level abstractions have been developed to handle a lot of these issues for us
:muscle:

---

:sparkles: SQS :sparkles:
<iframe src="https://giphy.com/embed/hjRde3gO0zkqNY6Bqc" width="480" height="270" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>

---

In SQS, when a producer sends a message to a queue, it doesn't matter how many consumers are reading from the queue.

Each message is processed independently and in isolation.

---

Back to the school bus analogy...

<!-- the bus takes the kids to school (sometimes several schools at once), and each school is a strict military school where the kids queue up a lot and are dealt with one by one (not necessarily in order, and sometimes twice, but definitely each one is processed!) -->

<img src="https://hackmd.io/_uploads/ByTnibX_a.png" height="600px">

---


---

<img src="https://hackmd.io/_uploads/BkbgZQ4Op.png" height="700">

---

### Creating an SQS Queue


<!-- there is fifo and standard in SQS too -->
<!-- FIFO/standard are options for queues and topics. The topics do the message processing, and the queues do the message sending -->

---

Amazon SQS **standard** queues guarantees **at-least-once delivery**:

- storing messages on multiple servers for redundancy
- Resending messages if server is not available (this might mean duplicate messages are sent)

---

<!-- There is some config that queues need to have -->
<img src="https://hackmd.io/_uploads/rJ-KrGEda.png" height="690">

---

<!-- permissions , i.e an access policy is required -->
<img src="https://hackmd.io/_uploads/HJEW8zEup.png" height="690">

---

<!-- in terraform -->

<!-- the custom policy document is the permissions -->

---


<!-- allows the sourceArn (topic) to send messages to the queue -->

---

When a message is delivered, a receipt handle is generated for that delivery and sent to the recipient


---


<!-- The receipt handle is needed for deleting the message or changing its visibility -->

<!--you can't put a message on a queue and then recall it -->

---

### SQS Message

<!-- When you subscribe an Amazon SQS queue to an Amazon SNS topic, you can publish a message to the topic and Amazon SNS sends an Amazon SQS message to the subscribed queue.

The Amazon SQS message contains the subject and message that were published to the topic along with metadata about the message in a JSON document. The Amazon SQS message will look similar to the following JSON document. -->

{
   "Type" : "Notification",
   "MessageId" : "63a3f6b6-d533-4a47-aef9-fcf5cf758c76",
   "TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
   "Subject" : "Testing publish to subscribed queues",
   "Message" : "Hello world! I could be JSON",
   "Timestamp" : "2012-03-29T05:12:16.901Z",
   "SignatureVersion" : "1",
   "Signature" : "EXAMPLEnTrFPa3...",
   "SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",
   "UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:123456789012:MyTopic:c7fe3a54-ab0e-4ec2-88e0-db410a0f2bee"
}

---

### What if message delivery fails? :thinking_face:

If Amazon SNS cannot access a subscribed endpoint due to a client-side or server-side error:
- message delivery will fail

:negative_squared_cross_mark:

---

:skull:

If a dead-letter queue is attached to the subscription:
- failed deliveries held in DLQ for further analysis or reprocessing. :face_with_monocle:

(this would break the order in a FIFO queue)

---

If no dead-letter queue is attached:
- (standard queue) the message is retried indefinitely until the retention period is over
- then SNS discards the message :wastebasket:

---

### SQS/SNS make a system...

<!-- Fault tolerance: SNS and SQS provide fault tolerance by ensuring that messages are not lost if a service is down. They automatically handle retries and failover, making the system more resilient -->
- more resilient
<!-- - SNS and SQS help in decoupling microservices and distributed systems. Instead of having services communicate directly with each other, they can communicate through SNS or SQS. This helps in reducing the complexity of the system and makes it easier to manage and scale -->
- less complex
- easier to scale
<!-- Asynchronous processing: SNS and SQS allow for asynchronous processing of messages. This means that a service can send a message and continue with other tasks without waiting for the message to be processed. -->
- more performant

---

### Used together... :handshake:

- fanout capability of SNS
- SNS pushes messages to SQS
- messages persist in SQS queues

---

:heavy_check_mark: SQS: decouple sending and receiving components, without needing concurrency

:heavy_check_mark:SNS: sends messages to multiple subscribers without needing polling

---


(the end)