Idempotency

What is Idempotency

Idempotency means an operation produces the same result whether executed once or multiple times.

Definition

f(f(x)) = f(x)

Applying operation twice = same result as applying once

Examples

Idempotent:
- HTTP GET (read-only)
- HTTP PUT (replace)
- HTTP DELETE
- SET key = value

Non-Idempotent:
- HTTP POST (create)
- INCR counter
- APPEND to list

Why Idempotency Matters

Problem without idempotency:
1. Client sends POST /orders
2. Server processes, creates order
3. Network fails before response
4. Client retries
5. Server creates ANOTHER order (duplicate!)

With idempotency:
1. Client sends POST /orders with idempotency key
2. Server processes, creates order
3. Network fails before response
4. Client retries with same key
5. Server detects key, returns existing order

Idempotency in Distributed Systems

Scenario	Need Idempotency
Retries	Yes
Message processing	Yes
Payment processing	Critical
Read operations	Natural
Write operations	Must implement

Implementing Idempotency

Idempotency Key Pattern

class IdempotentAPI:
    def __init__(self, db):
        self.db = db
    
    def create_order(self, order_data, idempotency_key):
        # Check if already processed
        existing = self.db.get('idempotency', idempotency_key)
        if existing:
            return existing['result']  # Return cached result
        
        # Process order
        result = self.process_order(order_data)
        
        # Store result atomically
        self.db.store('idempotency', idempotency_key, {
            'result': result,
            'timestamp': time.time(),
            'ttl': 86400  # 24 hours
        })
        
        return result

Database-Level Idempotency

-- Unique constraint
create table orders (
    idempotency_key uuid primary key,
    order_data jsonb,
    result jsonb,
    created_at timestamp
);

-- Insert with ignore
INSERT INTO orders (idempotency_key, order_data, result)
VALUES ($1, $2, $3)
ON CONFLICT (idempotency_key) DO NOTHING;

Conditional Updates

def update_inventory(item_id, quantity, expected_version):
    current = db.get('inventory', item_id)
    
    if current['version'] != expected_version:
        return {'error': 'Version conflict'}
    
    db.update('inventory', item_id, {
        'quantity': current['quantity'] - quantity,
        'version': current['version'] + 1
    })
    return {'success': True}

Natural Idempotency

PUT /users/123 {"name": "John"}
- Always sets name to John
- Multiple calls = same result
- Natural idempotency

DELETE /users/123
- Deletes user if exists
- Multiple calls = same result
- Natural idempotency

Idempotency Keys

Key Generation

import uuid

# Method 1: UUID
key = str(uuid.uuid4())

# Method 2: Deterministic from request
import hashlib
key = hashlib.sha256(
    json.dumps(request_data, sort_keys=True).encode()
).hexdigest()

# Method 3: Client-provided
key = request.headers.get('Idempotency-Key')

Key Storage

class IdempotencyStore:
    def __init__(self, redis_client):
        self.redis = redis_client
    
    def store_result(self, key, result, ttl=86400):
        """Store idempotency result with TTL"""
        self.redis.setex(
            f"idempotency:{key}",
            ttl,
            json.dumps(result)
        )
    
    def get_result(self, key):
        """Get cached result"""
        result = self.redis.get(f"idempotency:{key}")
        if result:
            return json.loads(result)
        return None
    
    def exists(self, key):
        """Check if key exists"""
        return self.redis.exists(f"idempotency:{key}") > 0

API Design

# API with idempotency support
@app.route('/api/orders', methods=['POST'])
def create_order():
    # Get or generate idempotency key
    idempotency_key = request.headers.get('Idempotency-Key')
    if not idempotency_key:
        idempotency_key = str(uuid.uuid4())
    
    # Check idempotency
    existing = idempotency_store.get_result(idempotency_key)
    if existing:
        return jsonify(existing), 200  # Return existing result
    
    # Process new request
    result = process_order(request.json)
    
    # Store result
    idempotency_store.store_result(idempotency_key, result)
    
    return jsonify(result), 201

Key Best Practices

1. Client generates key or server generates and returns
2. Store with TTL to prevent unbounded growth
3. Atomic check-and-store to prevent race conditions
4. Document idempotency behavior
5. Return cached result for duplicate requests

Key Expiry

# Different TTLs for different operations
ttl_config = {
    'payment': 86400 * 7,  # 7 days
    'order': 86400 * 24,   # 24 hours
    'notification': 3600,  # 1 hour
    'analytics': 300       # 5 minutes
}