Lesson · ⏱ 7 min read
🌊 Streaming data: Kafka and real-time pipelines
EnglishHinglish
🏠 Real-life example
Batch processing is like reading the morning newspaper: yesterday's news, all at once. Streaming is like a live cricket score app: every ball updates instantly. Fraud detection, live delivery tracking and real-time recommendations need the live-score approach.
💡 In simple words
In streaming, events (a click, a payment, a GPS ping) flow continuously. Apache Kafka is the standard backbone: producers write events to topics, Kafka stores them durably in order, and consumers read them at their own pace. Stream processors (Spark Structured Streaming, Flink) compute things on the fly, such as rolling counts and features for fraud models. Streaming adds complexity: late or duplicate events, ordering and exactly-once guarantees. Start with batch, and add streaming only where freshness truly matters.
🪜 Step by step
- Producers send events (a payment, a click) to a Kafka topic as they happen.
- Kafka stores them in order, in partitions, for a set time, so several consumers can read independently.
- Consumers (Flink, Spark Streaming or a simple service) process events: filter, enrich, compute running counts.
- Results go to a database, a feature store for real time ML features, or an alert.
- Handle duplicates and late events: design consumers to be idempotent and use event time windows.
🔑 Key terms
- Event
- A single fact that happened at a time, e.g. 'payment of ₹499 at 10:02'.
- Topic
- A named stream of events in Kafka.
- Consumer
- A service that reads and processes events from a topic.
👀 See it in action
UPI app ──payment event──▶ Kafka topic 'payments'
│
┌───────────────────────┼───────────────────────┐
fraud scorer live dashboard warehouse loader
(reacts in < 1 s) (updates per second) (hourly batch)One stream, many independent consumers, each at its own speed.
⚠️ Common mistakes to avoid
- Choosing streaming when hourly batch meets the need. Streaming is harder to build and run.
- Assuming each event arrives exactly once and in order. Plan for duplicates and delays.
✅ Quick check
Which use case most needs streaming instead of a nightly batch?
- Yearly HR summary
- Blocking a fraudulent payment as it happens
- Monthly finance report
Show answer
B. Blocking a fraudulent payment as it happens: Fraud must be stopped in seconds. A report tomorrow is too late.
🙋 Questions learners ask
Is Kafka a database?
Not really. It's a durable event log for moving data between systems. You still store results in databases.
Where does streaming help AI?
Real time features for fraud and recommendations, and keeping search or vector indexes up to date within seconds.
📌 Remember
Batch = periodic, simpler. Streaming = real-time, more complex. Use it when seconds matter.
📖 Words from this lesson (AI dictionary)
Learn this as part of a full course
Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:
Continue in the course