HackMDMessage Broker with Nats Jetstream
Overview
Modern applications are becoming more intelligent but also increasingly complex. Engineers often encounter communication issues related to software components built in different programming languages while processing big data, resolving security risks, and simplifying complex infrastructures. Fortunately, some solutions help mitigate these issues.
Message Broker one of them.
Message brokers are a central component of any communication system that contains multiple actors, each with their own way of communicating. Message brokers, also known as message queue (MQ) software, are commonly used in systems like web services when they need to communicate between different components in an application. Message brokers can act as a distributed communications layer, connecting services on various platforms.
Why use a message broker?
Message brokers promote async communications among services so that the sending app doesn’t have to wait for the receiving app's response. This improves fault tolerance and resiliency in the infrastructure. Using message brokers also makes it easier to scale systems.
Using message broker allows for increased control over inter-service communications, ensuring data is sent securely, reliably, and efficiently between app components. Message brokers can play a similar role in integrating multi-cloud environments, enabling communication between workloads and runtimes on different platforms. They also work well with serverless computing, in which individual cloud-hosted services run on an as-needed basis.
How does a message broker work?
In synchronous processing, a source app submits a message to the target app, waits for it to finish its task, and accepts the response before continuing. This is known as a blocking request because the source app can’t take any further action until a response is delivered.
Message brokers rely on asynchronous messaging for inter-application communication. They prevent the loss of essential data and allow systems to continue operating even with intermittent connectivity or lag on public networks. Asynchronous messaging ensures that messages are sent only once (and only once) in the correct order.
Here are some fundamental components of a message broker:
- A producer is an app responsible for transmitting messages. It's linked to the message broker, also known as publishers in the publish/subscribe model.
- A consumer is a service that receives messages waiting in the message broker. They’re known as subscribers in the publish-subscribe pattern.
- A queue or topic is a folder in a system. Message brokers use them to store messages.
- An exchanger is a logical structure or an object that resides on queues and directs the message broker to form a group where consumers or producers can create or listen to send or receive messages.
Let's do it
One of the Message Broker is Nats Jetstream.
What is Nats Jetstream
blablabalablaa
Let's running Jetstream
We will spin up our NATS container, but we will include the -js flag to instruct NATS to start JetStream.
docker run --network host -p 4222:4222 -p 8222:8222 nats -js -m 8222
The output should appear similar to the following:
We can verify that our JetStream instance is running by invoking the JetStream monitoring endpoint: http://localhost:8222/jsz.
curl http://localhost:8222/jsz
Create stream
Before we can publish or consume messages from a stream, we must first create it. The creation step informs NATS that we want the messages to be persisted and instructs NATS how to configure the stream. First, let's look at what we need to create our stream.
Stream Naming Conventions
Unsurprisingly, every stream must have a unique name. JetStream's documentation includes the following naming recommendations:
- Alphanumeric values are recommended.
- Spaces, tabs, period (.), greater than (>), or asterisk (*) are prohibited.
- Limit name length. The JetStream storage directories will include the account, stream name, and consumer name, so a generally safe approach would be to keep names under 32 characters.
- Do not use reserved file names like NUL, LPT1, etc.
- Be aware that some file systems are case insensitive, so do not use stream or account names that would collide in a file system. For example, Foo and foo would collide on a Windows or Mac OSx System.
Persistence
When configuring a stream, we must provide the Storage Type used to persist the stream's messages. There are currently two options: Memory and File. JetStream defaults to File mode if not explicitly declared. This mode will persist stream messages to the filesystem and is required if the stream must persist between server restarts. If maximum throughput is needed, Memory storage can improve performance at the cost of losing all memory streams after a server restart (unless replication is configured in a clustered configuration).
Subject
Streams can have one or more subjects associated with them. Messages in the stream must adhere to the policies defined by the stream.
Push vs Pull Consumers
When creating a stream consumer subscription, we must first decide how messages will be delivered. For example, do we want the stream to push messages to our consumers, or do we want our consumers to pull messages from the stream?
Ephemeral and Durable Consumers
There are two types of stream consumers: Ephemeral and Durable.
Ephemeral Consumers
Ephemeral consumers are not long-lived. By default, JetStream consumers are ephemeral. Each connection appears to the server as a new consumer and will only track the consumer while active. Ephemeral consumers can only be push subscribers.
Durable Consumers
If the consumer is expected to be long-lived, we can create a durable consumer. With a durable consumer, the server will keep track of the consumer's stream position. If the consumer restarts, the server will continue at the last stream position. Pull subscriptions require a durable consumer. Its also important to understand that once a durable consumer has been created, some options cannot be changed.
Using the CLI to Create a Stream
With our JetStream instance running, let's spin up the first (of several) nats-box containers. In a separate terminal instance, run the following command:
docker run --network host --rm -it synadia/nats-box
You should be greeted with the following:
As mentioned in the previous article, nats-box is a containerized NATS toolbox that provides several tools (nats, nats-top, stan-pub, stan-sub, stan-bench) we can use to interact with the NATS server. In this article, we will be using the nats CLI.
Using the nats CLI, we can create our first stream with the following command:
nats str add NOTIFICATIONS --subjects "NOTIFICATIONS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1
Your output should be similar to the following:
We see that the stream NOTIFICATIONS.* was created. Note that we include the (*) wildcard character. We can use the nats CLI to list the current streams
nats str ls
The output should include the NOTIFICATIONS stream:
We can also query the streams configuration using the following command:
nats str info NOTIFICATIONS
The configuration should look similar to the following:
In a new nats-box terminal, we can publish messages to our stream with the following command:
nats pub NOTIFICATIONS.inbound 'new message'
With our first message published, let's now create a Push subscriber. In a separate nats-box terminal, issue the following command:
nats con add NOTIFICATIONS MONITOR --ack none --target monitor.NOTIFICATIONS --deliver last --replay instant --filter '' --heartbeat=10s --flow-control
Your output should appear similar to:
With our stream consumer created, we can subscribe to it using:
nats sub monitor.NOTIFICATIONS
We should start receiving messages in the terminal:
From the highlighted console output, we can see that we received the original message ( new message) and that the idle heartbeat is active, letting us know that we are still subscribed.
Switching back to the terminal where we published the first message, let's publish two more messages:
nats pub NOTIFICATIONS.inbound 'another message'
nats pub NOTIFICATIONS.inbound 'yet another message'
nats str info NOTIFICATIONS
Here we see the primary difference with using message streaming. In the output, we see that the stream currently has three messages. While those messages were delivered to our first consumer, the stream still contains the published messages.
Let's create another consumer with the following command:
nats con add NOTIFICATIONS MONITOR2 --ack none --target monitor2.NOTIFICATIONS --deliver all --replay instant --filter '' --heartbeat=10s --flow-control
Using Golang
git clone git@github.com:sofianhw/go-jetstream-order.git
cd go-jetstream-order
docker build . -t gotainer
docker run --name goexec -v $(pwd):/app -d -i -t gotainer /bin/sh
docker exec -it goexec go run monitor/main.go
docker exec -it goexec go run push-order/main.go
docker exec -it goexec go run update-status/main.go
