Kafka

Kafka Architecture

Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, durable messaging.

Core Components

Kafka Cluster:

┌─────────────────────────────────────────────────┐
│                  Kafka Cluster                   │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐         │
│  │ Broker 1│  │ Broker 2│  │ Broker 3│         │
│  └─────────┘  └─────────┘  └─────────┘         │
│       ↑            ↑            ↑               │
│  ┌─────────────────────────────────────┐        │
│  │         ZooKeeper / KRaft          │        │
│  └─────────────────────────────────────┘        │
└─────────────────────────────────────────────────┘
         ↑                        ↓
    Producers               Consumers

Key Concepts

Component	Description
Broker	Kafka server storing data
Topic	Category/feed of messages
Partition	Topic subdivision for parallelism
Offset	Unique message ID within partition
Producer	Publishes messages to topics
Consumer	Reads messages from topics
Consumer Group	Group of consumers load-balancing reads

Data Flow

Producer → Topic → Partition → Consumer Group → Consumer

1. Producer sends message to topic
2. Topic routes to partition (based on key hash)
3. Message appended to partition log
4. Consumer Group assigns partitions to consumers
5. Consumer reads messages in order

Topics and Partitions

Topic Structure

Topic: user-events

Partition 0: [msg0, msg3, msg6, msg9, ...]
Partition 1: [msg1, msg4, msg7, msg10, ...]
Partition 2: [msg2, msg5, msg8, msg11, ...]

Each partition is an ordered, immutable sequence of messages

Partitioning Strategy

// Key-based partitioning (default)
ProducerRecord<String, String> record = new ProducerRecord<>(
    "user-events",    // topic
    "user-123",       // key (used for partitioning)
    eventJson         // value
);
// Partition = hash(key) % num_partitions

// Round-robin partitioning (no key)
producer.send(new ProducerRecord<>("logs", logMessage));

Partition Configuration

# server.properties
num.partitions=6

# Topic-specific
kafka-topics.sh --create --topic orders \
  --partitions 12 \
  --replication-factor 3

Partitioning Best Practices

Consideration	Recommendation
Number of partitions	2-3x target throughput
Key selection	Use related entity ID
Ordering	Guaranteed within partition
Scalability	Can add partitions (not reduce)

Partition Limits

Partition constraints:
- Each partition on single broker
- Replication across brokers
- More partitions = more file handles
- More partitions = longer recovery time

Rule of thumb: < 4000 partitions per broker

Consumer Groups

How Consumer Groups Work

Consumer Group: order-processor

Topic: orders (3 partitions)

Partition 0 → Consumer 1
Partition 1 → Consumer 2
Partition 2 → Consumer 3

Each partition consumed by exactly ONE consumer in group

Scaling Consumers

3 Partitions, 3 Consumers (optimal):

P0 → C1
P1 → C2
P2 → C3

3 Partitions, 6 Consumers (over-provisioned):

P0 → C1
P1 → C2
P2 → C3
C4 → idle
C5 → idle
C6 → idle

Max parallelism = number of partitions

Consumer Group Example

Properties props = new Properties();
props.put("group.id", "analytics-group");
props.put("bootstrap.servers", "kafka:9092");
props.put("enable.auto.commit", "false");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("user-events"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        processEvent(record.value());
    }
    consumer.commitSync();  // Commit offsets
}

Offset Management

Offset Tracking:

Partition 0: [msg0, msg1, msg2, msg3, msg4]
                     ↑
              Committed offset = 2
              (msg0, msg1 processed)
              (msg2, msg3, msg4 pending)

Commit strategies:
- Auto-commit: Periodic background commit
- Sync commit: Blocking commit after processing
- Async commit: Non-blocking commit with callback