---
# System prepended metadata

title: Kafka
tags: [API, Event Driven, GNU, Kafka, Message Queue, Linux, System Design, Distributed System, Software, Server]

---

# Kafka
###### tags: `Kafka` `GNU` `Linux` `Message Queue` `Event Driven` `Distributed System` `Server` `System Design` `Software` `API`

A data pipeline system that is designed for "distributed" & "scalible" producer/consumer models, leveraging sequential reads, effective partitioning, batching, compression, & zero-copy operations.

## Resources

[What is Event Driven Architecture (EDA)? - IBM Technology](https://www.youtube.com/watch?v=o2HJCGcYwoU)
[What is Kafka? - IBM Technology](https://www.youtube.com/watch?v=aj9CDZm0Glc)
[Apache Kafka in 6 minutes - James Cutajar](https://www.youtube.com/watch?v=Ch5VhJzaoaI)
[System Design: Why is Kafka fast? - ByteByteGo](https://www.youtube.com/watch?v=UNUz1-msbOM)

## Event-Driven Message System

~ scalable / distributed / resillient

### Example

microservices / event log storage backup, analysis <-> event backbone <-> user-ends sub to pub
![](https://i.imgur.com/I30XVvO.png =500x)

:::info
**feasible functions such as:**
- event logging & analysis
- streaming processing
- comunication with pub-sub instead of connection to EVERY microservices / end-points
![](https://i.imgur.com/yNgkwja.png =350x)
:::

### Message Queue

![](https://i.imgur.com/baNWW8H.png =500x)

[為什麼需要 Message Queue?](https://medium.com/starbugs/讓任務排隊吧-message-queue-1-de949e274c43)

### Streaming Sequentially

producers -> brokers in partitions -> comsumers
![](https://i.imgur.com/Glo3ksX.png =250x)

comsumers follow their record in order to tracing & blocks expired in time
![](https://i.imgur.com/HVHYY7E.png =700x)






