Partitions

Partition Strategy

Partition Assignment Methods

1. Key-Based Partitioning:
   partition = hash(key) % num_partitions
   Same key → Same partition
   Guarantees ordering per key

2. Round-Robin:
   partition = counter % num_partitions
   Even distribution
   No ordering guarantee

3. Custom Partitioner:
   Business logic determines partition
   Flexible but complex

Implementation

// Custom partitioner
public class OrderPartitioner implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                        Object value, byte[] valueBytes, Cluster cluster) {
        int numPartitions = cluster.partitionCountForTopic(topic);
        
        // High-priority orders go to dedicated partition
        Order order = (Order) value;
        if (order.isHighPriority()) {
            return 0;  // Dedicated partition
        }
        
        // Regular orders distributed by customer
        return Math.abs(key.hashCode()) % numPartitions;
    }
}

// Configuration
props.put("partitioner.class", "com.example.OrderPartitioner");

Partitioning Strategies

Strategy	Description	Use Case
Hash-based	hash(key) % partitions	General purpose
Range-based	Key ranges to partitions	Time-series data
Custom	Business logic	Priority routing
Sticky	Affinity to partition	Session data

Ordering Guarantees

Within Partition

Partition 0:
[msg0] → [msg1] → [msg2] → [msg3]
  ↓        ↓        ↓        ↓
Offset 0  Offset 1  Offset 2  Offset 3

Ordering guaranteed within partition!

Across Partitions

Partition 0: [msg0] → [msg2] → [msg4]
Partition 1: [msg1] → [msg3] → [msg5]

NO ordering guarantee across partitions!
msg1 may arrive before msg0

Achieving Global Order

Option 1: Single partition (no parallelism)
Option 2: Use key for entity ordering
Option 3: Sequence numbers in messages

Ordering Examples

# Example 1: User events (ordered by user)
{
    "user_id": "123",  # Key - ensures ordering per user
    "event": "profile_updated",
    "timestamp": "2024-01-15T10:00:00Z"
}

# Example 2: Order events (ordered by order)
{
    "order_id": "456",  # Key - ensures ordering per order
    "status": "shipped",
    "timestamp": "2024-01-15T10:05:00Z"
}

# Example 3: Global events (no ordering guarantee)
{
    "event_type": "system_alert",
    "data": {...}
    # No key - round-robin to partitions
}

Ordering Trade-offs

Requirement	Solution	Trade-off
Entity ordering	Key by entity	Limited parallelism
Global ordering	Single partition	No parallelism
No ordering needed	Random partitioning	Maximum parallelism

Partition Rebalancing

What is Rebalancing?

Trigger: Consumer joins or leaves group

Before:
P0 → C1
P1 → C2
P2 → C3

C4 joins:
P0 → C1
P1 → C2
P2 → C3
P3 → C4  (new partition assigned)

Or:
P0 → C1
P1 → C2
P2 → C3
P3 → C4

Rebalance Strategies

1. Range Assignment:
   - Consecutive partitions to consumers
   - C0: P0, P1
   - C1: P2, P3

2. Round-Robin:
   - Distribute evenly
   - C0: P0, P2
   - C1: P1, P3

3. Sticky:
   - Minimize partition movement
   - Keep existing assignments
   - Only move what's necessary

Rebalance Impact

During Rebalance:
- All consumers pause
- Partitions reassigned
- Offsets may be lost
- Latency spike

Duration: 100ms - few seconds

Reducing Rebalance Impact

// Incremental cooperative rebalancing (Kafka 2.4+)
props.put("partition.assignment.strategy",
    "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

// Benefits:
// - Only affected partitions revoked
// - Non-stop processing continues
// - Faster rebalance

Rebalance Monitoring

def monitor_rebalance(consumer_group):
    # Track rebalance events
    rebalance_count = get_metric('consumer.rebalance.count')
    rebalance_latency = get_metric('consumer.rebalance.latency')
    
    # Alert on frequent rebalances
    if rebalance_count > threshold:
        alert('Frequent consumer rebalances')
    
    # Alert on long rebalances
    if rebalance_latency > 5000:  # 5 seconds
        alert('Long consumer rebalance')

Best Practices

1. Use cooperative rebalancing (Kafka 2.4+)
2. Keep heartbeat interval low for fast detection
3. Set session timeout appropriately
4. Monitor rebalance frequency
5. Avoid unnecessary consumer restarts