Topics

Topic Design

Naming Conventions

Topic Naming Patterns:

1. Entity-based:
   orders
   users
   products

2. Event-based:
   order.created
   order.updated
   user.registered

3. Domain-based:
   ecommerce.orders
   ecommerce.payments
   analytics.events

4. Purpose-based:
   orders-input
   orders-processed
   orders-dlq

Design Principles

Principle	Description
Single Responsibility	One topic per event type
Descriptive Names	Clear purpose from name
Consistent Format	Uniform naming convention
Separation of Concerns	Different topics for different domains
Future-proof	Consider evolution

Topic Structure

# Well-designed topics
topics = {
    # Core business events
    'orders.created': 'New orders',
    'orders.updated': 'Order status changes',
    'orders.cancelled': 'Cancelled orders',
    
    # User events
    'users.registered': 'New user signups',
    'users.updated': 'Profile changes',
    
    # System events
    'system.audit': 'Audit logs',
    'system.metrics': 'Application metrics'
}

Topic vs Queue

Aspect	Topic (Pub-Sub)	Queue (Point-to-Point)
Consumers	Multiple get copy	One gets message
Retention	Messages retained	Messages deleted
Replay	Yes	No
Use Case	Event broadcasting	Task distribution

Topic Partitioning

Partitioning Strategy

Topic: orders (6 partitions)

Key-based:
order_id=1 → hash(1) % 6 = partition 3
order_id=2 → hash(2) % 6 = partition 1
order_id=3 → hash(3) % 6 = partition 4

Same key always → same partition (ordering guarantee)

Key Selection

# Good key choices
def get_partition_key(message):
    # Option 1: Entity ID (maintains entity ordering)
    return message['order_id']
    
    # Option 2: User ID (user-level ordering)
    return message['user_id']
    
    # Option 3: Null (round-robin, no ordering)
    return None

# Partition assignment
partition = hash(key) % num_partitions

Partition Count

Partition Count Guidelines:

- Start with: 2-3x target throughput
- Can increase: Yes (but not decrease)
- Consider: Consumer count, broker count
- Max per broker: ~4000 (file handles)

Example:
Target: 100K messages/sec
Broker: 10K messages/sec capacity
Need: 10 brokers minimum
Partitions: 20-30 (2-3x brokers)

Hot Partition Problem

Problem: One partition gets disproportionate traffic

Solution 1: Better key distribution
Solution 2: Shard hot key across partitions
Solution 3: Increase partitions and rebalance

Partition Reassignment

# Reassign partitions
kafka-reassign-partitions.sh \
  --reassignment-json-file reassignment.json \
  --execute

# JSON format
{
  "partitions": [
    {"topic": "orders", "partition": 0, "replicas": [1, 2, 3]}
  ]
}

Topic Management

Topic Lifecycle

1. Creation:
   kafka-topics.sh --create \
     --topic orders \
     --partitions 6 \
     --replication-factor 3

2. Configuration:
   kafka-topics.sh --alter \
     --topic orders \
     --config retention.ms=604800000

3. Deletion:
   kafka-topics.sh --delete --topic orders

Topic Configuration

# Key configurations
retention.ms=604800000      # 7 days retention
cleanup.policy=delete       # delete or compact
max.message.bytes=1048576   # 1MB max message
min.insync.replicas=2       # Durability requirement

Topic Monitoring

# Monitor topic health
def monitor_topic(topic, expected_partitions=6):
    # Check partition count
    partitions = kafka.partitions(topic)
    if len(partitions) != expected_partitions:
        alert(f"Partition count mismatch: {topic}")
    
    # Check consumer lag
    for partition in partitions:
        lag = get_consumer_lag(topic, partition)
        if lag > 10000:
            alert(f"High lag on {topic}:{partition}")
    
    # Check replication
    for partition in partitions:
        replicas = get_replica_count(topic, partition)
        if replicas < 2:
            alert(f"Low replication: {topic}:{partition}")

Topic Cleanup Policies

Policy	Description	Use Case
Delete	Remove old data	Standard topics
Compact	Keep latest per key	State topics
Compact+Delete	Both	State with TTL

Best Practices

1. Use descriptive names with domain prefix
2. Set retention based on needs
3. Monitor lag and health
4. Plan partition count carefully
5. Document topic purposes