# RabbitMQ VS Kafka

## RabbitMQ
<!-- (https://crates.io/crates/lapin) -->

- RabbitMQ is a solid, mature, general-purpose message broker that supports several protocols such as AMQP, MQTT, STOMP and more.
- We can use any of the protocol for any desired scenario
- It can handle High throughput.

- Messages are stored until the consumer ACK the message. Once the message `ACKED` the message is **removed** from the queue.

- It has multiple routing option
- Routing allows messages to reach specific queue

1. Direct
- Routes messages to all queues with an exact match for a `routing key`.
2. Fanout Exchange
- Broadcast a message to every queue that is bound to the exchange.
3. Topic
- Similar to Direct but we can use wildcard matching using `*`


- It supports message priority and we can send message with priority and depending on the message priority the message is placed on the appropriate queue.
- RabbitMQ can consider a message delivered once it’s sent out, or wait for the consumer to manually acknowledgement when it has been received.
- RabbitMq is designed for vertical scaling inorder to scale the rabbitmq broker we need to add more consumer so that the message won't stored inside the broker storage.

## Kafka
<!-- (https://crates.io/crates/rdkafka) -->
- Pub Sub Model.
- Kafka has a persistent storage ( data persisted based on age or size limit
- So the `message is not removed once it is comsumed and it can be replayed`
- Here we need keep small book keeping for processed **offset** in each microservice.

- Kafka uses custom its own protocol so we cannot choose any other protocol.
- It uses simple routing approach
- It supports grouping so grouped consumer can pull from same offset instead start from the begining.
- It does not support the message priority all message are stored and delivered in the order in which they received.

- Kafka producer can ACK once the message stored in the broker storage
- Kafka maintains an offset for each message in a partition. The committed position is the last offset that has been saved. Should the process fail and restart, is this the offset that it will recover to. A consumer in Kafka can either automatically commit offsets periodically, or it can choose to control this committed position manually.
- Kafka retains large amounts of data with very little overhead
- Kafka can scale by adding more nodes to the cluster or by adding more partition to the topics.
- Kafka can scale greater than the rabbitmq
- Uses Last Write Wins strategy and deletes the older version of a message with same key
- Kafka has index but it is not reliable as rdbms index.

- Broker
- handles commit log (how messages are stored in the disk),
- Topic
- Logical name where message are stored in the broker.
- It consists of unit called partition
- One or more partition can make up a single partition
- Single partition replica only exists on only one broker and cannot be split b/w brokers

- ZooKeper
- Helps maintaint consunses in the cluster.
- It is a distributed store that provides discovery, config, sync service in a highly available way.

- Kafka uses os page efficiently to handle million of message and avoids `JVM Heap`

- Kafka Stream
- It exposes sigle interface to all the application

- Kafka Connect
- It is to make integration with other system
- `Source Connector` is used to import data from source into kafka.
- MY_SQL -> KAFKA
- `Sink Connector` is used to export from kafka into different system.


- Kafka Rust c binding library's Bench mark