RabbitMQ VS Kafka

# RabbitMQ VS Kafka ## RabbitMQ  - RabbitMQ is a solid, mature, general-purpose message broker that supports several protocols such as AMQP, MQTT, STOMP and more. - We can use any of the protocol for any desired scenario - It can handle High throughput. - Messages are stored until the consumer ACK the message. Once the message `ACKED` the message is **removed** from the queue. - It has multiple routing option - Routing allows messages to reach specific queue 1. Direct - Routes messages to all queues with an exact match for a `routing key`. 2. Fanout Exchange - Broadcast a message to every queue that is bound to the exchange. 3. Topic - Similar to Direct but we can use wildcard matching using `*` ![](https://i.imgur.com/dgAwrl7.png) - It supports message priority and we can send message with priority and depending on the message priority the message is placed on the appropriate queue. - RabbitMQ can consider a message delivered once it’s sent out, or wait for the consumer to manually acknowledgement when it has been received. - RabbitMq is designed for vertical scaling inorder to scale the rabbitmq broker we need to add more consumer so that the message won't stored inside the broker storage. ## Kafka  - Pub Sub Model. - Kafka has a persistent storage ( data persisted based on age or size limit - So the `message is not removed once it is comsumed and it can be replayed` - Here we need keep small book keeping for processed **offset** in each microservice. - Kafka uses custom its own protocol so we cannot choose any other protocol. - It uses simple routing approach - It supports grouping so grouped consumer can pull from same offset instead start from the begining. - It does not support the message priority all message are stored and delivered in the order in which they received. - Kafka producer can ACK once the message stored in the broker storage - Kafka maintains an offset for each message in a partition. The committed position is the last offset that has been saved. Should the process fail and restart, is this the offset that it will recover to. A consumer in Kafka can either automatically commit offsets periodically, or it can choose to control this committed position manually. - Kafka retains large amounts of data with very little overhead - Kafka can scale by adding more nodes to the cluster or by adding more partition to the topics. - Kafka can scale greater than the rabbitmq - Uses Last Write Wins strategy and deletes the older version of a message with same key - Kafka has index but it is not reliable as rdbms index. - Broker - handles commit log (how messages are stored in the disk), - Topic - Logical name where message are stored in the broker. - It consists of unit called partition - One or more partition can make up a single partition - Single partition replica only exists on only one broker and cannot be split b/w brokers - ZooKeper - Helps maintaint consunses in the cluster. - It is a distributed store that provides discovery, config, sync service in a highly available way. - Kafka uses os page efficiently to handle million of message and avoids `JVM Heap` ![](https://i.imgur.com/PcAXuxs.png) - Kafka Stream - It exposes sigle interface to all the application ![](https://i.imgur.com/rdpJCwN.png) - Kafka Connect - It is to make integration with other system - `Source Connector` is used to import data from source into kafka. - MY_SQL -> KAFKA - `Sink Connector` is used to export from kafka into different system. ![](https://i.imgur.com/5epDVbS.png) - Kafka Rust c binding library's Bench mark ![](https://i.imgur.com/OMNkZsc.png)