Functional Requirements

Requirement	Description
Submit Jobs	Producers can enqueue jobs with payload, priority, and optional scheduling constraints
Retry Failed Jobs	Automatically retry transient failures with configurable backoff and max attempts
Priority Queues	Support multiple priority levels (critical, high, normal, low) with weighted scheduling
Job Status Tracking	Producers can query job state (pending, processing, completed, failed) in real time
Scheduled Jobs	Support cron-like scheduling for recurring tasks and delayed one-time jobs
Job Results	Store and retrieve results or errors for completed/failed jobs
Cancel Jobs	Allow cancellation of pending or scheduled jobs before execution

Non-Functional Requirements

Requirement	Target	Rationale
At-least-once Delivery	Guaranteed	Avoiding lost jobs is critical for business workflows
Fault Tolerance	99.99% availability	System must survive node failures without losing jobs
Horizontal Scaling	Linear throughput growth	Must handle 10K+ jobs/sec with adding workers
Low Latency Submission	< 10ms p99	Job submission should not be a bottleneck
Monitoring	Real-time dashboards	Operators need visibility into queue health and job flow
Ordering	Per-priority FIFO	Jobs within same priority should execute in submission order

Core Architecture

┌─────────────┐    ┌──────────────┐    ┌─────────────────┐    ┌──────────────┐
│  Producer A  │───▶│              │    │   Worker Pool   │    │              │
└─────────────┘    │              │    │  ┌───┐ ┌───┐   │    │  Result      │
┌─────────────┐    │  Job Queue   │───▶│  │ W1│ │ W2│...│───▶│  Store       │
│  Producer B  │───▶│  (Redis/     │    │  └───┘ └───┘   │    │  (DB/S3)     │
└─────────────┘    │   SQS)       │    └─────────────────┘    └──────────────┘
                   │              │                                │
┌─────────────┐    │  Priority    │    ┌─────────────────┐        │
│  Scheduler   │───▶│  Partitions  │    │  Dead Letter    │◀───────┘
│  (Cron)      │    │              │    │  Queue (DLQ)    │
└─────────────┘    └──────────────┘    └─────────────────┘
                            │
                   ┌─────────────────┐
                   │  Monitoring &   │
                   │  Metrics (CW)   │
                   └─────────────────┘

Component Responsibilities:

• Producers: Submit jobs via REST API or SDK. Each job gets a unique ID, payload, priority, and optional schedule.
• Job Queue: durable message broker holding pending jobs. Supports priority partitions, delayed delivery, and visibility timeouts.
• Workers: Pull jobs from queue, execute them, report results. Stateless and horizontally scalable.
• Result Store: Persists job outcomes (success results or error details) for producer retrieval.
• Scheduler: Manages cron-like and delayed jobs, moving them to the active queue at the correct time.
• Dead Letter Queue: Holds jobs that exceeded max retries, requiring manual inspection.
• Monitoring: Publishes metrics to CloudWatch/StatsD for dashboards and alerts.

Technology Choices

Component	Option A	Option B	Option C
Queue Broker	Amazon SQS	Redis (Bull/Redis Streams)	RabbitMQ
Result Store	DynamoDB	PostgreSQL	S3
Worker Runtime	Python/Celery	Node.js/BullMQ	Go
Monitoring	CloudWatch	Prometheus+Grafana	Datadog
Scheduler	AWS EventBridge	Custom cron service	Redis delayed sets

Key Design Decisions:

1. SQS vs Redis: SQS is fully managed and scales automatically, but Redis offers lower latency and richer data structures (sorted sets for priorities). For interviews, Redis is often preferred for discussion depth.
2. At-least-once vs Exactly-once: At-least-once is practical; exactly-once requires distributed transactions and is rarely worth the complexity. Workers must be idempotent.
3. Pull vs Push: Pull model (workers poll queue) gives workers natural backpressure. Push model (broker pushes to workers) reduces latency but requires rate limiting.
4. Stateless Workers: Workers hold no state, making horizontal scaling trivial. Any worker can process any job.

State Machine

                    ┌──────────┐
                    │ SUBMITTED│
                    └────┬─────┘
                         │ (enqueue)
                         ▼
                    ┌──────────┐
            ┌──────▶│  PENDING │◀──────┐
            │       └────┬─────┘       │
            │            │ (dequeue)    │
            │            ▼              │
            │       ┌──────────┐       │
            │       │PROCESSING│       │ (retry)
            │       └──┬───┬───┘       │
            │          │   │           │
            │ (success)│   │(failure)   │
            │          ▼   ▼           │
            │   ┌─────┐ ┌──────┐      │
            │   │DONE │ │FAILED│──────┘
            │   └─────┘ └──┬───┘
            │              │ (max retries exceeded)
            │              ▼
            │       ┌──────────┐
            │       │  DEAD    │
            └───────│  LETTER  │
  (cancel)          └──────────┘

State Definitions

State	Description	Stored In	TTL
SUBMITTED	Job just created, not yet in queue	Job Table	N/A
PENDING	In queue, waiting for a worker to claim	Queue Broker	24 hours
PROCESSING	Worker has claimed the job and is executing	Queue Broker + Worker	15 min
COMPLETED	Job finished successfully	Result Store	30 days
FAILED	Job failed but may be retried	Job Table	7 days
DEAD_LETTER	Exhausted all retries, needs manual review	DLQ	90 days
CANCELLED	Producer explicitly cancelled before execution	Job Table	7 days

State Transitions & Side Effects

From	To	Trigger	Side Effects
SUBMITTED	PENDING	Enqueue succeeds	Publish job.queued event
PENDING	PROCESSING	Worker dequeues (visibility timeout set)	Publish job.started event, start timeout timer
PROCESSING	COMPLETED	Worker returns success	Store result, publish job.completed event, remove from queue
PROCESSING	FAILED	Worker returns error	Publish job.failed event, schedule retry if retries remain
FAILED	PENDING	Retry scheduler fires	Increment attempt counter, re-enqueue with delay
FAILED	DEAD_LETTER	Max retries exceeded	Publish job.dead_letter event, alert operator
PENDING	CANCELLED	Producer calls cancel API	Remove from queue, publish job.cancelled event
PROCESSING	PENDING	Visibility timeout expires	Worker didn't complete in time, re-queue for another worker

Visibility Timeout (In-Flight Protection)

When a worker dequeues a job, the broker marks it as in-flight with a visibility timeout (e.g., 30 seconds). If the worker doesn't acknowledge completion within this window, the job reappears in the queue for another worker.

Timeline:
  0s: Worker A dequeues Job X (visibility timeout = 30s)
  5s: Worker A starts processing
  25s: Worker A completes → sends ACK → visibility timeout cleared
  
  Failure scenario:
  0s: Worker B dequeues Job Y (visibility timeout = 30s)
  5s: Worker B crashes
  30s: Visibility timeout expires → Job Y reappears in queue
  31s: Worker C dequeues Job Y → processes successfully

Visibility timeout should be 2x the expected job execution time to avoid duplicate processing while still detecting failures quickly.

Idempotency Design

Since at-least-once delivery means jobs may be processed multiple times, every worker must be idempotent.

Strategy	Implementation	Use Case
Unique Job ID Check	Before processing, check Result Store if job ID already has a result	Universal (required for all jobs)
Deduplication Token	Producer includes a unique token per logical operation	API calls, payments
Conditional Database Writes	UPDATE ... WHERE job_id = X AND status != 'completed'	Database mutations
Idempotency Keys	Use business-level keys (e.g., order ID) to prevent duplicate effects	Financial transactions

# Example: Idempotent worker
async def process_job(job):
    # 1. Check if already processed
    existing = await result_store.get(job.id)
    if existing:
        return existing  # Already done, return cached result
    
    # 2. Process with conditional write
    result = await do_work(job.payload)
    
    # 3. Store result atomically (only if not already stored)
    success = await result_store.put_if_absent(job.id, result)
    if not success:
        return await result_store.get(job.id)  # Another worker beat us
    
    return result

Scheduled & Recurring Jobs

Type	Implementation	Example
Delayed Job	Store in Redis sorted set with score = execute_at timestamp	Send email in 24 hours
Cron Job	Scheduler service polls cron table, creates job instances at scheduled times	Daily report generation
Recurring Job	Cron job template that spawns new job instances on schedule	Hourly data sync

Redis Sorted Set for Delayed Jobs:

  ZADD delayed_jobs <timestamp> <job_json>

Scheduler loop (every second):
  now = current_timestamp()
  jobs = ZRANGEBYSCORE delayed_jobs 0 now
  for job in jobs:
      ZREM delayed_jobs job  # Remove from delayed set
      LPUSH queue:high job   # Push to active queue

Data Model

-- Jobs table (primary metadata)
CREATE TABLE jobs (
    job_id      UUID PRIMARY KEY,
    producer_id VARCHAR(64),
    job_type    VARCHAR(128),       -- e.g., 'send_email', 'generate_report'
    payload     JSONB,
    priority    INTEGER DEFAULT 2,  -- 0=critical, 1=high, 2=normal, 3=low
    status      VARCHAR(20),        -- pending, processing, completed, failed, dead_letter, cancelled
    attempts    INTEGER DEFAULT 0,
    max_retries INTEGER DEFAULT 3,
    created_at  TIMESTAMP,
    updated_at  TIMESTAMP,
    scheduled_at TIMESTAMP,         -- NULL = immediate, otherwise = run at this time
    idempotency_key VARCHAR(256) UNIQUE
);

-- Job results table
CREATE TABLE job_results (
    job_id    UUID PRIMARY KEY REFERENCES jobs(job_id),
    result    JSONB,
    error     TEXT,
    duration  INTEGER,  -- milliseconds
    worker_id VARCHAR(64),
    completed_at TIMESTAMP
);

-- Worker stats table (for monitoring)
CREATE TABLE worker_stats (
    worker_id   VARCHAR(64) PRIMARY KEY,
    hostname    VARCHAR(128),
    status      VARCHAR(20),
    jobs_processed INTEGER DEFAULT 0,
    last_heartbeat TIMESTAMP,
    started_at  TIMESTAMP
);

Worker Pool Architecture

┌──────────────────────────────────────────────────────┐
│                   Load Balancer (ALB)                 │
└──────────────┬──────────────┬──────────────┬────────┘
               │              │              │
       ┌───────▼──────┐ ┌────▼───────┐ ┌───▼────────┐
       │  Worker Pod 1 │ │Worker Pod 2│ │Worker Pod N│
       │  ┌─────────┐  │ │ ┌────────┐ │ │ ┌────────┐│
       │  │Consumer │  │ │ │Consumer│ │ │ │Consumer││
       │  │Group-A  │  │ │ │Group-A │ │ │ │Group-B ││
       │  └─────────┘  │ │ └────────┘ │ │ └────────┘│
       │  ┌─────────┐  │ │ ┌────────┐ │ │           │
       │  │Consumer │  │ │ │Consumer│ │ │           │
       │  │Group-B  │  │ │ │Group-B │ │ │           │
       │  └─────────┘  │ │ └────────┘ │ │           │
       └──────────────┘ └────────────┘ └───────────┘

Consumer Groups ensure each job is delivered to exactly
one worker within a group, enabling parallel processing
without duplication.

Consumer Groups & Partition Assignment

Using Redis Streams as the queue:

Stream: jobs_stream
├── Partitions: 0, 1, 2, ..., N-1
│
Consumer Group A (priority: high)
├── Consumer A1 → reads partitions 0, 1
├── Consumer A2 → reads partitions 2, 3
└── Consumer A3 → reads partitions 4, 5

Consumer Group B (priority: normal)
├── Consumer B1 → reads partitions 0, 1, 2
└── Consumer B2 → reads partitions 3, 4, 5

Partition assignment algorithm:

1. Each worker joins a consumer group and announces itself.
2. The group coordinator collects all active workers.
3. Partitions are distributed evenly: worker[i].partitions = all_partitions[i::num_workers].
4. On worker failure (missed heartbeat), partitions are reassigned to surviving workers.

Auto-Scaling Strategy

Metrics → Scaling Decision → Action

1. Queue Depth (primary signal)
   depth_per_worker = queue_depth / num_active_workers
   if depth_per_worker > THRESHOLD_HIGH (100): scale UP
   if depth_per_worker < THRESHOLD_LOW (10):   scale DOWN

2. Processing Latency (secondary signal)
   if avg_processing_time > MAX_LATENCY:     scale UP

3. Worker Utilization
   utilization = jobs_processed / (time_period * max_concurrent)
   if utilization > 0.9: scale UP
   if utilization < 0.2:  scale DOWN (with cooldown)

Scaling Cooldown:
  - Scale UP:   30 seconds cooldown
  - Scale DOWN: 5 minutes cooldown (avoid flapping)

Scaling Bounds:
  - Min workers: 2 (for high availability)
  - Max workers: 100 (cost/safety limit)
  - Step size:   add/remove 2 workers at a time

Backpressure Mechanism

When workers are overwhelmed and the queue grows too large:

Mechanism	Description	Implementation
Queue Depth Limit	Stop accepting new jobs when queue exceeds threshold	Return HTTP 503 from producer API
Rate Limiting	Throttle producers sending jobs too fast	Token bucket at API gateway
Worker Throttling	Workers slow down consumption rate	Increase poll interval
Load Shedding	Drop low-priority jobs when under extreme load	Skip re-enqueue for lowest priority

if queue_depth > HARD_LIMIT (100,000):
    reject_new_jobs()
    alert_operator("Queue depth critical: {queue_depth}")

elif queue_depth > SOFT_LIMIT (50,000):
    enable_rate_limiting(producer_api, rate=1000/sec)
    scale_up_workers(step=4)

Retry & Exponential Backoff

RETRY_CONFIG = {
    'max_attempts': 3,
    'base_delay_ms': 1000,      # 1 second
    'max_delay_ms': 30000,       # 30 seconds
    'backoff_multiplier': 2,
    'jitter': True               # Add randomness to prevent thundering herd
}

# Backoff formula:
# delay = min(base_delay * (backoff_multiplier ^ attempt), max_delay)
# delay += random(0, delay * 0.1)  # 10% jitter

# Attempt 1: 1s + jitter
# Attempt 2: 2s + jitter  
# Attempt 3: 4s + jitter
# Attempt 4: → DEAD_LETTER (max retries exceeded)

Retry Classification:

Error Type	Retryable?	Example
Network timeout	Yes	API call timed out, transient
Rate limit exceeded	Yes	Third-party API throttle
Service unavailable	Yes	Upstream service temporarily down
Invalid payload	No	Malformed JSON, missing required fields
Authentication failure	No	Invalid credentials, permission denied
Business logic error	No	Order already cancelled, duplicate entry

Dead Letter Queue & Recovery

Dead Letter Queue (DLQ)
├── Job metadata (original payload, attempts, errors)
├── All error logs from each attempt
├── Timestamp of death (when moved to DLQ)
└── Producer who submitted the job

Recovery Options:
1. Manual retry via admin API: POST /admin/jobs/{id}/retry
2. Bulk retry: POST /admin/dlq/retry-all?filter=job_type:send_email
3. Inspection dashboard: View job details, errors, and decide
4. Auto-expiry: DLQ entries expire after 90 days

Monitoring & Observability

Key Metrics

Metric	Type	Alert Threshold
jobs.submitted	Counter (per min)	N/A (baseline)
jobs.processed	Counter (per min)	< 50% of submitted
jobs.failed	Counter (per min)	> 5% of processed
jobs.in_dlq	Gauge	> 100
queue.depth	Gauge	> 50,000
queue.age_seconds	Gauge	> 300 (oldest unprocessed job)
worker.processing_time_ms	Histogram (p99)	> 30,000
worker.utilization	Gauge	> 90%
worker.count	Gauge	< 2 (availability risk)

Dashboard Layout

┌─────────────────────────────────────────────────┐
│  Job Queue Health Dashboard                      │
├─────────────────┬───────────────────────────────┤
│  Queue Depth    │  Jobs/sec (submitted vs done) │
│  ████████ 45K  │  ▁▂▃▅▆▇█▇▆▅  (sparkline)    │
├─────────────────┼───────────────────────────────┤
│  Failure Rate   │  Worker Count                 │
│  ████░░░ 2.1%  │  ██████████ 12 active         │
├─────────────────┼───────────────────────────────┤
│  DLQ Count      │  Avg Processing Time          │
│  ██░░░░░░ 47   │  ████░░░░░░ 1.2s p99          │
└─────────────────┴───────────────────────────────┘

Alerting Rules

alerts:
  - name: HighFailureRate
    condition: jobs.failed / jobs.processed > 0.05 for 5m
    action: page-oncall
    
  - name: QueueBacklog
    condition: queue.depth > 100000 for 10m
    action: page-oncall, auto-scale
    
  - name: StuckJobs
    condition: queue.age_seconds > 600
    action: page-oncall
    
  - name: DLQBreach
    condition: jobs.in_dlq > 500
    action: slack-alert, create-ticket
    
  - name: WorkerDown
    condition: worker.count < 2 for 2m
    action: auto-replace, page-oncall

Complete System Flow

1. Producer POST /jobs { type: "send_email", payload: {...}, priority: 1 }
   │
   ▼
2. API validates payload, generates job_id, writes to Jobs Table (SUBMITTED)
   │
   ▼
3. API enqueues job to Redis Stream: XADD jobs_stream <payload>
   │
   ▼
4. Job state updated to PENDING in Jobs Table
   │
   ▼
5. Worker XREADGROUP from jobs_stream (claims partition)
   │
   ▼
6. Job state updated to PROCESSING, visibility timeout starts
   │
   ▼
7. Worker checks idempotency: SELECT FROM job_results WHERE job_id = ?
   │
   ├── Already exists → return cached result (idempotent hit)
   │
   └── Not found → execute job logic
       │
       ├── Success → XACK stream, write result to job_results, state=COMPLETED
       │              Publish job.completed event
       │
       └── Failure → XACK stream, increment attempts
                     if attempts < max_retries:
                         schedule retry with exponential backoff
                         state → FAILED → PENDING
                     else:
                         move to DLQ, state → DEAD_LETTER
                         alert operator

This design ensures reliable, scalable, and observable job processing suitable for Amazon-scale production workloads.