Traffic Management

Canary and Progressive Delivery

Kubernetes rolling updates move replicas, not traffic percentages. To send 5% of requests to a new version you need a mesh (or an ingress controller) that splits at L7:

http:
- route:
  - destination: { host: api, subset: stable }
    weight: 95
  - destination: { host: api, subset: canary }
    weight: 5

Progressive delivery wraps the split in gates: shift 5%, watch error rate and p99 against an SLO budget, advance to 50% then 100%, and snap weights back to 0 on breach (Flagger automates this loop against Prometheus). Blue/green is the simpler cousin: full cutover by flipping the Service selector, at double capacity cost.

Retries, Timeouts, Circuit Breaking

A DestinationRule carries the resilience policy:

trafficPolicy:
  connectionPool:
    http: { http1MaxPendingRequests: 100, maxRequests: 10 }
  outlierDetection:
    consecutive5xxErrors: 5
    interval: 10s
    baseEjectionTime: 30s

VirtualService controls retry behavior:

http:
- retries:
    attempts: 3
    perTryTimeout: 2s
    retryOn: connect-failure,refused-stream,503
  timeout: 6s

Outlier detection ejects repeatedly failing hosts so the pool keeps serving from healthy ones; circuit breaking caps pending work so one slow dependency cannot exhaust client resources. Fault injection (abort with 503s, or fixed delays) rehearses client handling of exactly those failures before production sees them.

Pitfalls and Interview Angles

Retry storms. Retrying an overloaded service multiplies load by the attempt count — always bound attempts with a per-try timeout shorter than the overall deadline, and retry only idempotent calls.

Mesh + app double retries. If your code retries AND the proxy retries, attempts compound; pick one layer as the retry owner.

Ejection too aggressive. With 2 replicas, aggressive outlier detection can eject everyone and cause total blackout — size pools or relax thresholds.

Weights without gates. Static 90/10 forever is not a canary; without automated metric gates you will ship a bad release confidently.

Interview angle: lay out a canary plan with explicit percentages, the metrics you watch, the rollback trigger, and where retries live in your stack.