API Gateway (HLD)

Problem Statement

Design an API Gateway that serves as the single entry point for all client traffic to a microservices architecture. The gateway must handle routing, authentication, rate limiting, request/response transformation, caching, and load balancing -- all while adding minimal latency.

An API Gateway is analogous to a front desk in a large building: every visitor checks in there, gets verified, is directed to the right floor, and follows the building's rules. Without it, each microservice would need to implement its own auth, rate limiting, and protocol handling -- leading to massive duplication and inconsistency.

Functional Requirements

Feature	Description
Request Routing	Route incoming requests to appropriate backend services based on path, headers, method, or query parameters
Authentication and Authorization	Validate JWT tokens, OAuth 2.0 access tokens, API keys, and support mTLS for service-to-service communication
Rate Limiting	Enforce per-client, per-endpoint, and global rate limits using distributed counters
Request Transformation	Manipulate headers, rewrite paths, transform request bodies, and translate between protocols (HTTP to gRPC)
Response Caching	Cache GET responses with configurable TTLs and cache invalidation mechanisms
Load Balancing	Distribute traffic across backend instances using round-robin, least connections, or weighted algorithms
Circuit Breaking	Detect failing backend services and trip circuit breakers to prevent cascading failures
Logging and Monitoring	Capture request/response logs, latency metrics, error rates, and throughput data

Non-Functional Requirements

Requirement	Target	Rationale
Latency	<10ms overhead per request	Gateway adds a hop; must remain negligible
Throughput	100,000+ requests/second	Must handle peak traffic for a large-scale service
Availability	99.99% uptime	Single point of failure for all backend services
Scalability	Horizontal scaling to 50+ nodes	Handle traffic spikes and geographic distribution
Security	OWASP Top 10 compliance	Gateway is the internet-facing edge
Observability	End-to-end request tracing	Debugging across microservices requires correlation

Capacity Estimation

Target: 100,000 RPS peak
Average request size: 2 KB (in) + 5 KB (out) = 7 KB per request
Bandwidth: 100K * 7 KB = 700 MB/s = 5.6 Gbps

Memory for rate limiting (Redis):
- 100K active clients * 1 KB per client = 100 MB
- Token bucket counters: negligible

Memory for response caching:
- 10% of requests cacheable (GET)
- Cache 10K unique responses * 10 KB avg = 100 MB
- Total cache memory: ~500 MB with overhead

Disk for logging (if local): Not recommended -- stream to centralized store

Key Design Decisions

Monolithic vs. Sidecar Gateway

• Monolithic gateway: Single deployable unit, simpler to manage, but couples all cross-cutting concerns
• Sidecar (Envoy pattern): Each service gets its own proxy, more flexible, used by Istio/Linkerd service meshes
• For SDE-1 interview: Recommend monolithic gateway for simplicity, mention sidecar as evolution path

Managed vs. Custom

• AWS API Gateway: Fully managed, auto-scales, integrates with Lambda/EC2, but vendor lock-in and cost at scale
• Kong / APISIX: Open-source, plugin-based, runs on your infra, more control
• Custom (Envoy + filters): Maximum flexibility, higher operational cost
• Recommendation: Start with managed, plan migration path to self-hosted as scale demands

End-to-End Request Flow

The gateway processes each request through an ordered pipeline of filters. Each filter is independent, testable, and can be enabled/disabled per route.

+---------------------------------------------------------------------+
|                        CLIENT (Mobile/Web/IoT)                     |
+------------------------------+--------------------------------------+
                               | HTTPS/TLS
                               v
+---------------------------------------------------------------------+
|                    LOAD BALANCER (L4/L7)                           |
|              Distributes across gateway nodes                      |
|              Health checks, SSL termination                        |
+------------------------------+--------------------------------------+
                               |
                               v
+---------------------------------------------------------------------+
|                       API GATEWAY NODE                             |
|                                                                     |
|  +----------+  +----------+  +----------+  +----------+            |
|  |  TLS     |-> |  Route   |-> |  Auth    |-> |  Rate    |            |
|  |  Terminate|  |  Resolve |  |  Filter |  |  Limit   |            |
|  +----------+  +----------+  +----------+  +----------+            |
|       |                                                       |
|       v                                                       v
|  +----------+  +----------+  +----------+  +----------+            |
|  | Transform|-> |  Cache   |-> | Circuit  |-> |  LB to   |---> BACKEND
|  |  Request |  |  Lookup  |  |  Breaker |  | Upstream |     SERVICES
|  +----------+  +----------+  +----------+  +----------+            |
|                                                                     |
|  Response path: Reverse through filters + cache store              |
+---------------------------------------------------------------------+

Filter Chain Implementation

Each filter implements a common interface. The gateway chains them in order for every request.

// Core filter interface
type Filter interface {
    Name() string
    Execute(ctx *RequestContext) FilterResult
    OnResponse(ctx *RequestContext, resp *Response) Response
}

type FilterResult struct {
    Continue   bool
    StatusCode int
    Body       string
}

func ExecuteFilterChain(filters []Filter, ctx *RequestContext) FilterResult {
    for _, filter := range filters {
        result := filter.Execute(ctx)
        log.Printf("Filter=%s Duration=%v Continue=%v",
            filter.Name(), ctx.FilterLatency(filter.Name()), result.Continue)
        if !result.Continue {
            return result
        }
    }
    return FilterResult{Continue: true}
}

Route Resolution

Routes are defined as a mapping from request attributes to backend services.

routes:
  - path: /api/v1/users/{id}
    method: GET
    upstream: user-service
    filters:
      - jwt-auth
      - rate-limit: { rps: 1000, per: client }
      - cache: { ttl: 30s, vary: [Authorization] }

  - path: /api/v1/orders/**
    method: POST
    upstream: order-service
    filters:
      - jwt-auth
      - rate-limit: { rps: 100, per: client }
      - transform-request: { add-header: { X-Request-Source: gateway } }

  # Weighted routing for canary deployments
  - path: /api/v1/checkout
    upstream: checkout-service
    weight: 90
  - path: /api/v1/checkout
    upstream: checkout-service-canary
    weight: 10

Route Matching Algorithm

Routes are matched using a trie-based structure for O(path-length) lookup, not O(N) linear scan.

Route Trie Structure:

/ --- api/ --- v1/ --- users/ --- {id}    -> user-service
                        |--- orders/ --- {id} -> order-service
              |--- v2/ --- users/ --- {id}    -> user-service-v2

Header-based routing:
  X-API-Version: 2.x  -> user-service-v2
  X-API-Version: 1.x  -> user-service-v1

Method-based:
  POST /users  -> user-service (create)
  GET  /users  -> user-service (read)

func ResolveRoute(req *http.Request) (*Route, map[string]string) {
    pathParts := strings.Split(req.URL.Path, "/")
    node := routeTrie.Root
    params := make(map[string]string)

    for _, part := range pathParts {
        if child, ok := node.Children[part]; ok {
            node = child
        } else if paramChild, ok := node.ParamChild; ok {
            params[paramChild.Name] = part
            node = paramChild
        } else if wildcard, ok := node.Wildcard; ok {
            params["*"] = strings.Join(pathParts[1:], "/")
            node = wildcard
            break
        } else {
            return nil, nil
        }
    }

    route := node.GetRoute(req.Method)
    if len(route.Alternatives) > 0 {
        route = weightedSelect(route, route.Alternatives)
    }
    return route, params
}

Request Context

The gateway maintains a per-request context object that flows through the entire pipeline.

type RequestContext struct {
    RequestID       string
    ClientIP        string
    Route           *Route
    Params          map[string]string
    Headers         http.Header
    AuthInfo        *AuthInfo
    RateLimitInfo   *RateLimitInfo
    CacheKey        string
    StartTime       time.Time
    UpstreamLatency time.Duration
    metadata        map[string]interface{}
}

Authentication Pipeline

The gateway supports multiple authentication methods. Each route specifies which method(s) are required.

+--------------------------------------------------+
|                   AUTH FILTER PIPELINE            |
|                                                  |
|  Request arrives                                 |
|       |                                          |
|       v                                          |
|  +-------------+                                 |
|  | Has JWT?    |--Yes--> Validate JWT signature   |
|  +------+------+         + expiry               |
|         No               |                       |
|         |                v                       |
|         v         +--------------+               |
|  +-------------+  | Extract      |               |
|  | Has API Key?|--Yes--> claims   |               |
|  +------+------+  | Check scopes |               |
|         No        | Validate key |               |
|         |         +--------------+               |
|         v                                        |
|  +-------------+                                 |
|  | Has mTLS    |--Yes--> Verify client cert       |
|  +------+------+         against CA chain        |
|         No               + revocation check      |
|         v                                        |
|    401 Unauthorized                              |
+--------------------------------------------------+

JWT Validation Implementation

type JWTAuthFilter struct {
    secretKey []byte
    publicKey *rsa.PublicKey
    issuer    string
    audience  string
    clockSkew time.Duration
}

func (f *JWTAuthFilter) Execute(ctx *RequestContext) FilterResult {
    token := extractBearerToken(ctx.Headers)
    if token == "" {
        return FilterResult{Continue: false, StatusCode: 401,
            Body: `{"error": "missing authorization token"}`}
    }

    claims := &JWTClaims{}
    tokenObj, err := jwt.ParseWithClaims(token, claims, func(t *jwt.Token) (interface{}, error) {
        if _, ok := t.Method.(*jwt.SigningMethodRSA); !ok {
            return nil, fmt.Errorf("unexpected signing method: %v", t.Header["alg"])
        }
        return f.publicKey, nil
    })

    if err != nil || !tokenObj.Valid {
        return FilterResult{Continue: false, StatusCode: 401,
            Body: `{"error": "invalid token"}`}
    }

    if claims.Issuer != f.issuer {
        return FilterResult{Continue: false, StatusCode: 401,
            Body: `{"error": "invalid issuer"}`}
    }

    now := time.Now()
    if now.Before(claims.NotBefore.Add(-f.clockSkew)) ||
       now.After(claims.Expiration.Add(f.clockSkew)) {
        return FilterResult{Continue: false, StatusCode: 401,
            Body: `{"error": "token expired"}`}
    }

    ctx.AuthInfo = &AuthInfo{
        UserID:    claims.Subject,
        Scopes:    claims.Scopes,
        ClientID:  claims.ClientID,
        ExpiresAt: claims.Expiration,
    }
    return FilterResult{Continue: true}
}

Rate Limiting with Token Bucket

The token bucket algorithm allows burst traffic while enforcing average rate limits.

Token Bucket Visualization:

Bucket Capacity: 10 tokens
Refill Rate: 2 tokens/second

Time 0:  [##########] 10/10 tokens (full)
Time 0:  Request 3   -> [#######   ] 7/10 tokens
Time 0:  Request 5   -> [##        ] 2/10 tokens
Time 0:  Request 1   -> [#         ] 1/10 tokens
Time 0:  Request 1   -> [          ] 0/10 tokens -> REJECT (429)
Time 1:  (refill +2)  -> [##        ] 2/10 tokens
Time 2:  (refill +2)  -> [####      ] 4/10 tokens

Distributed Rate Limiting with Redis

Using Redis allows rate limiting across multiple gateway nodes with atomic operations.

type DistributedRateLimiter struct {
    redis *redis.Client
}

func (r *DistributedRateLimiter) Allow(
    key string,
    maxTokens int64,
    refillRate float64,
) (bool, RateLimitInfo) {
    // Lua script for atomic token bucket operations
    const script = `
        local key = KEYS[1]
        local max_tokens = tonumber(ARGV[1])
        local refill_rate = tonumber(ARGV[2])
        local now = tonumber(ARGV[3])
        local requested = tonumber(ARGV[4])

        local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
        local tokens = tonumber(bucket[1]) or max_tokens
        local last_refill = tonumber(bucket[2]) or now

        local elapsed = now - last_refill
        local new_tokens = math.min(max_tokens, tokens + elapsed * refill_rate)

        if new_tokens >= requested then
            new_tokens = new_tokens - requested
            redis.call('HMSET', key, 'tokens', new_tokens, 'last_refill', now)
            redis.call('EXPIRE', key, math.ceil(max_tokens / refill_rate) * 2)
            return {1, new_tokens}
        else
            redis.call('HMSET', key, 'tokens', new_tokens, 'last_refill', now)
            redis.call('EXPIRE', key, math.ceil(max_tokens / refill_rate) * 2)
            return {0, 0}
        end
    `
    now := float64(time.Now().UnixMicro()) / 1_000_000
    result, err := r.redis.Eval(script, []string{key},
        maxTokens, refillRate, now, 1).Result()

    if err != nil {
        // Fail open: allow request if Redis is down
        return true, RateLimitInfo{Remaining: maxTokens}
    }

    results := result.([]interface{})
    allowed := results[0].(int64) == 1
    remaining := results[1].(int64)

    return allowed, RateLimitInfo{
        Remaining: remaining,
        Limit:     maxTokens,
        ResetAt:   time.Now().Add(time.Duration(float64(time.Second) / refillRate)),
    }
}

Rate Limit Response Headers

HTTP/1.1 200 OK
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 847
X-RateLimit-Reset: 1692345678
X-RateLimit-Policy: 1000;w=60

Multi-Dimensional Rate Limiting

Rate Limit Hierarchy (checked in order):

1. Global limit:      1,000,000 RPS across all clients
2. Per-client limit:  Client A -> 10,000 RPS
3. Per-endpoint:      /api/v1/users -> 5,000 RPS (all clients)
4. Per-client+endpoint: Client A -> /api/v1/users -> 1,000 RPS

Example: Client A sends 2,000 RPS to /api/v1/users
  -> Global:          PASS (1M limit)
  -> Per-client:      PASS (10K limit)
  -> Per-endpoint:    PASS (5K limit)
  -> Per-client+endpoint: FAIL (1K limit) -> 429

API Key Management

api_keys:
  - key: "ak_live_abc123..."
    client: "partner-acme"
    scopes: ["read", "write"]
    rate_limit: 5000
    allowed_ips: ["203.0.113.0/24"]
    expires_at: "2025-12-31T23:59:59Z"

  - key: "ak_test_xyz789..."
    client: "partner-beta"
    scopes: ["read"]
    rate_limit: 1000
    allowed_ips: ["*"]
    expires_at: "2024-06-30T23:59:59Z"

mTLS for Service-to-Service

+----------+     +--------------+     +--------------+
|  Client  |---->| API Gateway  |---->|   Backend    |
|          |     |              |     |   Service    |
| presents |     | validates    |     | validates    |
| cert     |     | client cert  |     | gateway cert |
|          |     | against CA   |     | against CA   |
+----------+     +--------------+     +--------------+

Certificate chain:
  Root CA -> Intermediate CA -> Client/Gateway Certificate

Validation steps:
  1. Verify certificate is signed by trusted CA
  2. Check certificate has not expired
  3. Verify certificate is not revoked (CRL/OCSP)
  4. Match CN/SAN against expected service identity

Request/Response Transformation

The gateway can modify requests and responses at the edge without touching backend code.

Transformation Types:

1. Header Manipulation:
   +---------------------+     +---------------------+
   | GET /users/123      |     | GET /users/123      |
   | Host: api.client.com| --> | Host: user-svc.int  |
   | X-Request-ID: abc   |     | X-Request-ID: abc   |
   |                     |     | X-Gateway-Node: gw-3|
   +---------------------+     +---------------------+

2. Path Rewriting:
   /api/v2/users/123  ->  /internal/v1/users/123
   /public/products   ->  /products

3. Body Transformation:
   Request:  { "userName": "john" }  ->  { "name": "john", "source": "api" }
   Response: { "internal_id": 42 }  ->  { "id": 42 }

4. Protocol Translation:
   HTTP/JSON request  ->  gRPC call to backend
   REST response      ->  GraphQL response shaping

type TransformFilter struct {
    rules []TransformRule
}

type TransformRule struct {
    Match       RouteMatch
    RequestOps  []RequestTransform
    ResponseOps []ResponseTransform
}

type RequestTransform struct {
    AddHeaders       map[string]string
    RemoveHeaders    []string
    RewritePath      string
    AddBodyFields    map[string]interface{}
    RemoveBodyFields []string
}

func (t *TransformFilter) Execute(ctx *RequestContext) FilterResult {
    for _, rule := range t.rules {
        if rule.Match.Matches(ctx) {
            for _, op := range rule.RequestOps {
                if op.RewritePath != "" {
                    ctx.Route.Path = applyRegexRewrite(ctx.Route.Path, op.RewritePath)
                }
                for k, v := range op.AddHeaders {
                    ctx.Headers.Set(k, interpolateVars(v, ctx))
                }
                for _, h := range op.RemoveHeaders {
                    ctx.Headers.Del(h)
                }
            }
        }
    }
    return FilterResult{Continue: true}
}

Response Caching

Cache Decision Flow:

Request arrives
    |
    v
Is method GET or HEAD? --No--> Skip cache
    |
    Yes
    v
Is response cacheable?  --No--> Skip cache
(check Cache-Control,      |
 Expires, no-store)       Yes
    |                      |
    v                      v
Generate cache key    Cache HIT? --Yes--> Return cached response
(path + query +          |
 vary headers)          No
    |                    |
    |                    v
    |              Forward to backend
    |                    |
    |                    v
    |              Store response in cache
    |              (async, do not block response)
    |              Set TTL from Cache-Control
    +---------------------+

Cache Implementation

type ResponseCache struct {
    store  CacheStore
    config CacheConfig
}

type CacheStore interface {
    Get(key string) (*CachedResponse, bool)
    Set(key string, resp *CachedResponse, ttl time.Duration)
    Delete(key string)
    InvalidatePattern(pattern string)
}

type CachedResponse struct {
    StatusCode int
    Headers    http.Header
    Body       []byte
    CachedAt   time.Time
    ExpiresAt  time.Time
}

func (c *ResponseCache) GenerateKey(ctx *RequestContext) string {
    hasher := sha256.New()
    hasher.Write([]byte(ctx.Route.Method))
    hasher.Write([]byte(ctx.Route.Path))
    hasher.Write([]byte(sortedQueryString(ctx.Route.Query)))
    for _, header := range ctx.Route.CacheVary {
        hasher.Write([]byte(header + ":" + ctx.Headers.Get(header)))
    }
    return hex.EncodeToString(hasher.Sum(nil))
}

func (c *ResponseCache) OnResponse(ctx *RequestContext, resp *Response) Response {
    if ctx.Route.Method != "GET" || resp.StatusCode != 200 {
        return *resp
    }
    if !isCacheable(resp.Headers) {
        return *resp
    }
    go func() {
        cached := &CachedResponse{
            StatusCode: resp.StatusCode,
            Headers:    resp.Headers.Clone(),
            Body:       resp.Body,
            CachedAt:   time.Now(),
            ExpiresAt:  time.Now().Add(c.config.DefaultTTL),
        }
        c.store.Set(ctx.CacheKey, cached, c.config.DefaultTTL)
    }()
    resp.Headers.Set("X-Cache", "MISS")
    return *resp
}

Cache Invalidation Strategies

1. TTL-Based (Passive):
   - Set TTL on each cached response
   - Simple, eventual consistency
   - Good for: product catalogs, search results

2. Event-Driven (Active):
   - Backend publishes invalidation events to message queue
   - Gateway subscribes and invalidates affected keys
   - Good for: user profile updates, order status changes

   Backend --publish--> Kafka/SQS --consume--> Gateway Invalidation Worker

3. Pattern Invalidation:
   - Invalidate all keys matching a pattern
   - Example: /api/v1/users/* when user updates profile
   - Implementation: Redis SCAN + DEL or prefix-based namespaces

4. Versioned Keys:
   - Cache key includes version: /users/123?v=3
   - Increment version to invalidate all cached versions

Circuit Breaker Pattern

Circuit Breaker States:

          Success
    +------------------+
    |                  |
    v                  |
+---------+    +---------+    +---------+
| CLOSED  |--->|  OPEN   |--->|HALF-OPEN|
| (normal)|    |(failing)|    |(testing)|
+---------+    +---------+    +---------+
    ^                  |            |
    |                  |            |
    +------------------+            |
         Failure threshold          |
         exceeded                   |
                                   v
                            Test request
                            succeeds
                                   |
                            +------+
                            |
                      Back to CLOSED

Closed State:
  - All requests pass through
  - Track success/failure count
  - If failure rate > threshold -> trip to OPEN

Open State:
  - All requests fail immediately (fallback response)
  - No requests reach backend
  - After timeout -> transition to HALF-OPEN

Half-Open State:
  - Allow limited test requests through
  - If they succeed -> CLOSED
  - If they fail -> OPEN (reset timeout)

type CircuitBreaker struct {
    state            BreakerState
    failureCount     int
    successCount     int
    lastFailure      time.Time
    failureThreshold int
    successThreshold int
    timeout          time.Duration
    mu               sync.RWMutex
}

func (cb *CircuitBreaker) Execute(fn func() error) error {
    cb.mu.RLock()
    state := cb.state
    cb.mu.RUnlock()

    switch state {
    case StateOpen:
        if time.Since(cb.lastFailure) > cb.timeout {
            cb.mu.Lock()
            cb.state = StateHalfOpen
            cb.mu.Unlock()
        } else {
            return &CircuitOpenError{Message: "circuit breaker is open"}
        }
    case StateHalfOpen:
        if cb.successCount > 0 {
            return &CircuitOpenError{Message: "circuit breaker half-open"}
        }
    }

    err := fn()

    cb.mu.Lock()
    defer cb.mu.Unlock()

    if err != nil {
        cb.failureCount++
        cb.lastFailure = time.Now()
        if cb.failureCount >= cb.failureThreshold {
            cb.state = StateOpen
        }
        return err
    }

    cb.successCount++
    cb.failureCount = 0
    if cb.state == StateHalfOpen && cb.successCount >= cb.successThreshold {
        cb.state = StateClosed
    }
    return nil
}

Monitoring and Observability

Metrics Collection Pipeline:

+----------+     +----------+     +------------+     +----------+
| Gateway  |---->| Metrics  |---->| Time Series|---->|Dashboard |
|  Node    |     | Exporter |     |  Store     |     | (Grafana)|
+----------+     +----------+     +------------+     +----------+
     |
     +------> Structured Logs --> Log Aggregator (ELK/Datadog)
     +------> Traces ----------> Trace Collector (Jaeger/X-Ray)

Key Metrics:

Metric	Description	Alert Threshold
gateway_requests_total	Total requests by route, status, method	Sudden drop
gateway_request_duration_seconds	Request latency histogram	p99 > 100ms
gateway_active_connections	Current open connections	> 80% of max
gateway_rate_limit_rejections	Requests rejected by rate limiter	Spike
gateway_circuit_breaker_trips	Circuit breaker state changes	Any trip
gateway_cache_hit_ratio	Cache hits / (hits + misses)	< 30%
gateway_upstream_latency_seconds	Backend response time by service	p99 > 200ms
gateway_error_rate	5xx responses / total requests	> 1%

Structured Logging:

{
  "request_id": "req-abc-123",
  "timestamp": "2025-01-15T10:30:00Z",
  "client_ip": "203.0.113.42",
  "method": "GET",
  "path": "/api/v1/users/123",
  "status": 200,
  "latency_ms": 45,
  "upstream": "user-service",
  "upstream_latency_ms": 38,
  "filters": {
    "auth": { "status": "pass", "user_id": "u-456" },
    "rate_limit": { "status": "pass", "remaining": 847 },
    "cache": { "status": "miss" },
    "circuit_breaker": { "status": "closed" }
  }
}

Horizontal Scaling

Scaling Architecture:

+---------------------------------------------+
|              DNS / Anycast                   |
|         (route to nearest region)            |
+-------------------+-------------------------+
                    |
                    v
+---------------------------------------------+
|         Regional Load Balancer               |
|     (distribute across gateway nodes)        |
+-------+---------------+---------------+-----+
        |               |               |
        v               v               v
   +---------+    +---------+    +---------+
   | Gateway |    | Gateway |    | Gateway |
   | Node 1  |    | Node 2  |    | Node 3  |
   +----+----+    +----+----+    +----+----+
        |               |               |
        +---------------+---------------+
                        |
                        v
              +------------------+
              |   Redis Cluster  |
              | (rate limit,     |
              |  cache state)    |
              +------------------+

Stateless Design Principles:

1. No local session state: All per-request state stored in Redis or passed in request context
2. Connection pooling: Reuse connections to backend services, pool per upstream
3. Shared configuration: Route configs loaded from central store (etcd/Consul), hot-reloaded
4. Graceful shutdown: Drain in-flight requests before terminating, remove from LB first

Connection Pool Configuration:

upstream_pools:
  user-service:
    max_connections: 200
    max_idle_time: 30s
    connect_timeout: 5s
    read_timeout: 30s
    write_timeout: 30s
    retry_policy:
      max_retries: 3
      retry_on: ["5xx", "connection_error", "timeout"]
      backoff: exponential  # 100ms, 200ms, 400ms
    health_check:
      interval: 10s
      path: /health
      healthy_threshold: 3
      unhealthy_threshold: 2

Data Model

CREATE TABLE routes (
    id              UUID PRIMARY KEY,
    path_pattern    VARCHAR(512) NOT NULL,
    method          VARCHAR(10) NOT NULL,
    upstream_id     UUID REFERENCES upstreams(id),
    weight          INT DEFAULT 100,
    priority        INT DEFAULT 0,
    enabled         BOOLEAN DEFAULT true,
    created_at      TIMESTAMP DEFAULT NOW(),
    updated_at      TIMESTAMP DEFAULT NOW(),
    UNIQUE(path_pattern, method)
);

CREATE TABLE upstreams (
    id                  UUID PRIMARY KEY,
    name                VARCHAR(255) NOT NULL,
    service_type        VARCHAR(50) NOT NULL,
    base_url            VARCHAR(1024) NOT NULL,
    health_check_path   VARCHAR(255) DEFAULT '/health',
    circuit_breaker_config JSONB,
    connection_pool     JSONB,
    load_balance        VARCHAR(50) DEFAULT 'round_robin',
    created_at          TIMESTAMP DEFAULT NOW()
);

CREATE TABLE rate_limit_rules (
    id          UUID PRIMARY KEY,
    route_id    UUID REFERENCES routes(id),
    client_scope VARCHAR(50) NOT NULL,
    max_tokens  BIGINT NOT NULL,
    refill_rate DECIMAL(10,2) NOT NULL,
    burst       BIGINT,
    enabled     BOOLEAN DEFAULT true
);

CREATE TABLE api_keys (
    id          UUID PRIMARY KEY,
    key_hash    VARCHAR(255) NOT NULL UNIQUE,
    client_name VARCHAR(255) NOT NULL,
    scopes      TEXT[] NOT NULL,
    rate_limit_id UUID REFERENCES rate_limit_rules(id),
    allowed_ips INET[],
    expires_at  TIMESTAMP NOT NULL,
    created_at  TIMESTAMP DEFAULT NOW(),
    revoked_at  TIMESTAMP
);

CREATE TABLE cache_configs (
    id                UUID PRIMARY KEY,
    route_id          UUID REFERENCES routes(id),
    ttl_seconds       INT NOT NULL,
    cache_key_template VARCHAR(512),
    vary_headers      TEXT[],
    invalidate_on     TEXT[],
    enabled           BOOLEAN DEFAULT true
);

CREATE TABLE transform_rules (
    id          UUID PRIMARY KEY,
    route_id    UUID REFERENCES routes(id),
    phase       VARCHAR(20) NOT NULL,
    operations  JSONB NOT NULL,
    priority    INT DEFAULT 0
);

Deployment Options Comparison

Aspect	AWS API Gateway	Kong (Self-Hosted)	Custom (Envoy)
Setup	Fully managed, minutes	Docker/K8s, hours	Custom code, weeks
Latency	10-30ms overhead	2-5ms overhead	<1ms overhead
Scaling	Auto-scales	Horizontal scaling	Full control
Cost	$3.50/M requests + $0.09/hr	Infra cost only	Infra + dev cost
Customization	Limited plugins	Plugin ecosystem	Unlimited
Maintenance	Zero	Moderate	High
Best for	Startups, rapid prototyping	Mid-scale, need flexibility	Large scale, specific needs