Delivery Semantics

At-Most-Once Delivery

At-most-once guarantees a message is delivered zero or one time - never more.

How it Works

At-Most-Once Flow:

1. Producer sends message
2. Message stored in queue
3. Consumer receives message
4. Consumer processes message
5. Consumer acknowledges (BEFORE processing)

If consumer crashes after ack:
- Message lost!
- Never reprocessed
- But never duplicated

Implementation

# Kafka auto-commit (at-most-once)
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "5000");

# Messages committed before processing
# Crash after commit = message lost

Trade-offs

Advantage	Disadvantage
Fastest delivery	May lose messages
Simplest implementation	No retry guarantee
Lowest latency	Data loss risk

When to Use

• Log collection (losing some logs OK)
• Metrics collection (approximate is fine)
• Non-critical events
• High throughput, low importance

Example

# At-most-once for metrics
def process_metric(metric):
    # Ack first (message may be lost if crash)
    queue.acknowledge(message_id)
    
    # Then process
    store_metric(metric)

At-Least-Once Delivery

At-least-once guarantees a message is delivered one or more times - never zero.

How it Works

At-Least-Once Flow:

1. Producer sends message
2. Consumer receives message
3. Consumer processes message
4. Consumer acknowledges (AFTER processing)

If consumer crashes before ack:
- Message redelivered
- May be processed twice
- But never lost

Implementation

# Kafka manual commit (at-least-once)
props.put("enable.auto.commit", "false");

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        processMessage(record.value());  // Process first
    }
    consumer.commitSync();  // Then commit
}
# Crash before commit = message redelivered

Deduplication Required

class AtLeastOnceConsumer:
    def __init__(self, db):
        self.db = db
    
    def process(self, message):
        # Check if already processed
        if self.db.is_processed(message['id']):
            return  # Skip duplicate
        
        # Process and mark as processed
        with self.db.transaction():
            self.handle(message)
            self.db.mark_processed(message['id'])

Trade-offs

Advantage	Disadvantage
No message loss	May process duplicates
Reliable delivery	Needs idempotency
Common pattern	Slightly more complex

When to Use

• Order processing
• Payment processing
• Any critical business logic
• When duplicates can be handled

Exactly-Once Delivery

Exactly-once guarantees a message is delivered exactly one time - no more, no less.

How it Works

Exactly-Once Flow:

1. Producer sends message with unique ID
2. Broker deduplicates by ID
3. Consumer processes message
4. Consumer commits with transaction
5. Consumer marks as processed atomically

Combines:
- Idempotent producer (no duplicates from retries)
- Transactional consumer (atomic processing + commit)

Kafka Implementation

// Idempotent producer
props.put("enable.idempotence", "true");
props.put("acks", "all");
props.put("retries", Integer.MAX_VALUE);

// Transactional consumer
props.put("isolation.level", "read_committed");

// Transaction example
producer.initTransactions();
producer.beginTransaction();
try {
    // Process message
    processMessage(record.value());
    
    // Produce output
    producer.send(new ProducerRecord<>("output-topic", result));
    
    // Commit offset atomically
    producer.sendOffsetsToTransaction(offsets, consumerGroupId);
    
    producer.commitTransaction();
} catch (Exception e) {
    producer.abortTransaction();
}

Components

1. Idempotent Producer:
   - Each message has unique sequence ID
   - Broker deduplicates
   - Prevents producer retries causing duplicates

2. Transactional Consumer:
   - Read committed isolation level
   - Only sees committed messages
   - Atomic processing + offset commit

3. Consumer Offset Transactions:
   - Process message + commit offset atomically
   - Prevents: processed but not committed

Trade-offs

Advantage	Disadvantage
No duplicates, no loss	Highest complexity
Strongest guarantee	Performance overhead
True exactly-once	Limited support

When to Use

• Financial transactions
• Critical business operations
• When duplicates are unacceptable
• When at-least-once + idempotency isn't sufficient