Apache Kafka
What is it?
Kafka is a distributed event-streaming platform that lets systems publish and read continuous streams of records reliably and at huge scale.
Explain like I'm 5
Why was it created?
It was built to handle massive, continuous streams of events (like activity logs) that traditional message queues struggled to move and store reliably.
Where is it used?
- Real-time analytics pipelines
- Activity and log aggregation
- Event-driven microservices
- Stream processing
Why should developers care?
Kafka is the backbone of many real-time data pipelines. Backend, data, and platform engineers frequently work with it.
How does it work?
Producers append records to topics, which are split into partitions and stored on disk. Consumers read at their own pace by tracking an offset, so many consumers can replay the same stream independently.
Real-world example
A ride-share app publishes every location ping to Kafka; pricing, mapping, and fraud systems each read the same stream for their own purposes.
Common use cases
- Event streaming between services
- Log and metrics pipelines
- Decoupling producers from consumers
- Replaying past events
Advantages
- Extremely high throughput
- Durable, replayable event log
- Decouples systems
- Scales horizontally
Disadvantages
- Operationally complex to run well
- Overkill for simple task queues
- Steeper learning curve than basic queues
When should you use it?
When you need a durable, high-volume stream that multiple systems consume independently.
When should you avoid it?
For simple background-job queues where a lightweight broker is enough.
Alternatives
Related terms
Interview questions
Beginner
- What is a topic in Kafka?
- How is Kafka different from a traditional queue?
Intermediate
- What is a partition and why does it matter?
- What is a consumer offset?
Senior
- How do you guarantee ordering and exactly-once processing?
- How would you size partitions for throughput?
Common misconceptions
- "Kafka is just a message queue" — it's a durable, replayable log; consumers don't delete messages by reading them.
- "Reading removes the message" — messages stay until they age out by retention policy.
Fun facts
- It was named after the writer Franz Kafka by its creators at LinkedIn.
- A single cluster can handle trillions of messages per day at large companies.
Timeline
- 2011 — Open-sourced by LinkedIn
- 2012 — Became an Apache project
Learning resources
Quick summary
Kafka is a durable, high-throughput event-streaming log that lets many systems read the same data independently and in order.
Cheat sheet
- Distributed event-streaming log
- Topics split into partitions
- Consumers track their own offset
- Durable and replayable