SD - Message Processing

Imagine a busy restaurant kitchen where food orders come in, get processed by the chefs, and are eventually delivered to customers. Different messaging systems work like different ways the restaurant organizes and manages orders.

1. Kafka: The Smart Order Board

• There are multiple order boards (Kafka Topics) where incoming orders are written.
• Each order is time-stamped and stays on the board for a long time.
• Multiple chefs (consumers) poll frequently or look at the board to process order as soon as they arrive.
• If a chef misses an order, they can go back and check the history.
• Orders are organized into lanes (partitions) so that multiple chefs can work in parallel.
• Chefs can form consumer groups, where a team of chefs collectively works on the same order board, ensuring load is balanced among them.

Different Kafka/Kitchen Setups

1. Real-Time Processing Setup:

• The order board updates continuously with new orders.
• Orders stay visible for a configured time period (retention period).
• Chefs pick up orders as they appear.
• Configuration in Kafka:
  • Short retention period (e.g., a few hours or a day) to avoid storing unnecessary history.
  • Consumers poll frequently to process messages as soon as they arrive.
• Great for: Live dashboards, instant notifications, real-time tracking.

2. Complete History Setup:

• The order board maintains infinite retention of orders.
• New chefs can access the complete history of past orders.
• Configuration in Kafka:
  • Long-term storage enabled
  • Full retention policy activated
• Perfect for: Audit trails, analytics, debugging issues.

3. Current State Setup:

• Only the latest version of each order stays on the board.
• If a customer changes their order, the old one is replaced.
• Focuses on current status rather than history.
• Unlike the Complete History Setup, which retains all past orders, this setup only keeps the latest state of each order, making it more lightweight and optimized for real-time state tracking rather than event retention.
• Ideal for: Inventory levels, current menu prices, user preferences.

4. Multi-Team Setup:

• Different kitchen teams work from the same order board.
• Each team tracks their own progress independently.
• Teams don't interfere with each other's work.
• Key Contrast: Unlike Real-Time Processing Setup, which processes orders as they come in, this setup ensures that multiple independent groups can process the same data without conflicting with each other.
• Configuration in Kafka:
  • Multiple consumer groups ensure different teams process the same data independently.
• Best for: Parallel processing, multiple department coordination.

2. ActiveMQ/SQS: Head Chef Assigns In Order

• A head chef receives incoming orders and assigns each one to the next available chef.
• Once assigned, no other chef can take that order.
• Orders wait in line if all chefs are busy (queueing mechanism).
• After preparation, the order ticket is removed (message is deleted).
• Push vs Pull: In ActiveMQ/SQS, the head chef assigns (pushes) orders to available chefs. In Kafka, chefs check the order board and pick up the next order when they are ready to work on it.
• No History: Once an order is taken, it disappears.

3. Redis Pub/Sub: Order Announcement System

• The kitchen has a loudspeaker system where new orders are called out.
• All available chefs hear the announcement at the same time.
• If a chef hears the order and decides to work on it, they start preparing the meal immediately.
• If a chef isn't present or paying attention, they miss the announcement forever.
• No records of past announcements are kept.
• Push vs Pull: In Redis Pub/Sub, orders are announced over a loudspeaker, meaning all chefs receive the message (push) at the same time without having to check for new orders (pull). In Kafka, chefs must check the order board and pick up new orders themselves (pull-based system).
• Ephemeral: No history—if you're not listening, you miss the order.

4. Redis Streams: Last 100 Live Orders

• Orders are displayed in the order they arrive with timestamps.
• Multiple chefs can take orders and prepare them at different speeds.
• The system keeps only the last 100 orders visible at any time.
• Chefs can review past orders within retention limits before they expire.
• Limited Replayability:
  • Redis Streams does not store an indefinite history of orders. Instead, it keeps only the most recent ones (e.g., last 100 orders).
  • If a chef wants to review older orders, they must do so before they expire or get deleted.
  • In contrast, Kafka retains a full log for a configurable duration (days/weeks) and allows replaying from any past point
• Processing Flexibility: Unlike a queue (e.g., Redis List), where once a chef picks an order, it disappears for everyone.

Summary

System	Kitchen Role	Best For	Limitation
Kafka	Smart Order Board	Real-time & historical event processing, analytics, parallel consumer groups	Requires more storage, consumers must poll
ActiveMQ/SQS	Order Distribution System	Task queues, one-time message delivery, load balancing	No message history, strict first-come-first-serve
Redis Pub/Sub	Kitchen Announcement System	Live notifications, real-time updates, chat messaging	No replay or history, must be listening to receive
Redis Streams	Order Tracking System	Recent event replay, flexible processing speeds, lightweight history	Limited retention, older messages expire