Monitoring and Security

The Five Golden Signals

Google's SRE book defines five golden signals that apply to any service. Amazon interviewers expect you to know these.

Signal	Definition	How to Measure	Example
Latency	Time to service a request	Histogram of response times (p50, p95, p99)	p99 latency = 150ms
Traffic	Demand on the system	Requests per second (QPS)	10,000 QPS at peak
Errors	Rate of failed requests	Error count / total requests	0.1% error rate
Saturation	How full the resource is	CPU, memory, disk, network utilization	CPU at 75%
Saturation	Pending work	Queue depth, thread pool usage	SQS queue depth = 5,000

Latency Percentiles

Always discuss latency in terms of percentiles, not averages. Averages hide tail latency problems.

Percentile	What It Tells You	Why It Matters
p50 (median)	Typical experience for most users	Baseline performance
p95	What 1 in 20 users experience	Good SLA target
p99	Worst-case for most users	SLA compliance, capacity planning
p99.9	Extreme outliers	Identifies systemic issues

Example: "Our API has an average latency of 50ms, which looks good. But our p99 is 2 seconds, meaning 1% of users (100 out of 10,000) experience a 2-second wait. We need to investigate the slow queries causing this tail latency."

The RED Method (For Request-Driven Services)

RED stands for Rate, Errors, Duration. Apply this to every microservice.

• Rate: Requests per second. "How much traffic is this service getting?"

  • Metric: http_requests_total (counter)
  • Dashboard: Requests/sec graph

• Errors: Failed requests. "Is the service healthy?"

  • Metric: http_requests_failed_total (counter)
  • Dashboard: Error rate as percentage of total requests

• Duration: Request latency. "Is the service fast enough?"

  • Metric: http_request_duration_seconds (histogram)
  • Dashboard: p50, p95, p99 latency graphs

RED Application Example:
For a Product Search API:

• Rate: 5,000 QPS during peak, 500 QPS off-peak
• Errors: < 0.05% (target SLA: 99.95% availability)
• Duration: p50 = 30ms, p95 = 80ms, p99 = 150ms

The USE Method (For Resources)

USE stands for Utilization, Saturation, Errors. Apply this to infrastructure components.

Resource	Utilization	Saturation	Errors
CPU	% time busy	Run queue length, context switches	Hardware errors
Memory	% used	Swap usage, OOM kills	ECC errors
Disk I/O	% busy	I/O wait, queue depth	Media errors
Network	% bandwidth used	TCP retransmits, dropped packets	Interface errors

USE Application Example:
For a database server:

• CPU Utilization: 60% average, 85% peak (need to monitor for saturation)
• Memory Saturation: 2GB swap used (indicates memory pressure)
• Disk I/O Errors: 0 (healthy)

Amazon-Specific Monitoring Expectations

Amazon interviewers expect you to proactively mention monitoring as part of your design. This signals operational maturity.

When presenting your design, include:

• "We will instrument every API endpoint with RED metrics and export to CloudWatch."
• "We will set up dashboards for the five golden signals on each microservice."
• "We will collect application logs to CloudWatch Logs with structured JSON format for querying."
• "We will use X-Ray for distributed tracing across service boundaries."

Metrics vs Logs vs Traces

Type	What It Captures	Use Case	Example
Metrics	Numeric aggregations over time	Dashboards, alerting	CPU usage, request count, error rate
Logs	Discrete events with context	Debugging, audit	Request/response details, error stack traces
Traces	End-to-end request flow	Distributed debugging	Full request path through 5 microservices

All three are needed for complete observability. Metrics tell you something is wrong, logs tell you why, traces tell you where.

Alert Design Principles

Bad alerts cause alert fatigue. Good alerts drive action. Follow these principles:

1. Every alert must be actionable — If no one can do anything about it, do not alert on it
2. Alert on symptoms, not causes — Alert on high latency, not on high CPU (CPU might be high but the system is fine)
3. Use multiple severity levels — Not everything is critical
4. Include runbooks — Every alert should link to a document explaining what to do

Severity Levels

Level	Name	Response Time	Examples	Notification
P0	Critical	5 minutes	Full outage, data loss, security breach	PagerDuty + phone call
P1	High	15 minutes	Degraded performance, partial outage	PagerDuty
P2	Medium	1 hour	Non-critical feature down, elevated errors	Slack + email
P3	Low	Next business day	Warning thresholds, capacity planning	Email only

Threshold Design

Static Thresholds: Fixed values that trigger alerts.

• "Alert if error rate > 1% for 5 minutes"
• "Alert if p99 latency > 500ms for 3 minutes"
• Good for: Known baselines, SLA violations

Dynamic Thresholds: Adapt based on historical patterns.

• "Alert if latency is 2x the rolling 24-hour average"
• "Alert if traffic deviates 3 standard deviations from the norm"
• Good for: Seasonal patterns, growth trends

Composite Alerts: Combine multiple conditions.

• "Alert only if error rate > 1% AND latency > 300ms for 5 minutes"
• Reduces false positives from transient spikes

Alerting Rules Example

# High Latency Alert
- alert: HighLatency
  expr: histogram_quantile(0.99, http_request_duration_seconds) > 0.5
  for: 3m
  labels:
    severity: P1
  annotations:
    summary: "p99 latency exceeds 500ms"
    runbook: "https://wiki/runbooks/high-latency"

# High Error Rate Alert
- alert: HighErrorRate
  expr: rate(http_requests_failed_total[5m]) / rate(http_requests_total[5m]) > 0.01
  for: 5m
  labels:
    severity: P1
  annotations:
    summary: "Error rate exceeds 1%"
    runbook: "https://wiki/runbooks/high-error-rate"

# Capacity Alert
- alert: HighCPU
  expr: avg(cpu_utilization) > 0.85
  for: 15m
  labels:
    severity: P2
  annotations:
    summary: "CPU utilization above 85% for 15 minutes"
    runbook: "https://wiki/runbooks/high-cpu"

Escalation and On-Call

On-Call Rotation:

• Primary on-call engineer responds to P0/P1 alerts
• Secondary on-call is backup if primary does not acknowledge within 5 minutes
• Weekly rotation to prevent burnout

Escalation Path:

1. Alert triggers → PagerDuty notification to on-call engineer
2. No acknowledgment in 5 minutes → Escalate to secondary
3. No acknowledgment in 15 minutes → Escalate to engineering manager
4. No resolution in 30 minutes → Escalate to VP of Engineering

Incident Response Workflow:

1. Acknowledge the alert
2. Assess impact (how many users affected?)
3. Mitigate first (rollback, feature flag, scale up)
4. Investigate root cause
5. Communicate status updates every 15 minutes
6. Post-incident review within 48 hours

Amazon Leadership Principle: Insist on the Highest Standards

Show that you have a complete monitoring and response strategy. Mention:

• "We will have dashboards for every service showing the RED metrics."
• "Every alert will have a runbook linked in the alert description."
• "We will conduct chaos engineering tests monthly to verify our monitoring catches real failures."
• "We will review alert effectiveness quarterly and tune thresholds to reduce noise."

Security in System Design Interviews

Amazon interviewers expect you to proactively address security. Bring it up without being asked. This shows production thinking.

Authentication (Who Are You?)

JWT (JSON Web Tokens):

• Stateless tokens containing user claims
• Signed with HMAC or RSA
• No server-side session storage needed
• Token typically expires in 15-60 minutes
• Refresh tokens for long-lived sessions

OAuth 2.0:

• Third-party authentication ("Sign in with Google")
• Authorization code flow for web apps
• Client credentials flow for service-to-service
• PKCE (Proof Key for Code Exchange) for mobile apps

When to Use What:

Scenario	Recommended
Single-page app with API	JWT with refresh tokens
Third-party login	OAuth 2.0
Service-to-service	mTLS or IAM roles
Internal tools	SSO (SAML/OIDC)

Authorization (What Can You Do?)

RBAC (Role-Based Access Control):

• Users are assigned roles (admin, editor, viewer)
• Roles define permissions
• Simple, easy to audit
• Example: "Admin can delete users, Editor can edit posts, Viewer can only read"

**ABAC (Attribute-Based Access Control):

• Permissions based on attributes (department, clearance level, time of day)
• More granular than RBAC
• Example: "Managers can approve expenses up to $10,000; directors up to $50,000"

Amazon-Specific: Use IAM roles and policies for service-level authorization. Every AWS resource should have least-privilege IAM policies.

Encryption

At Rest:

• Database encryption: AES-256 (RDS, DynamoDB, S3)
• File encryption: KMS-managed keys
• Disk encryption: EBS encryption enabled by default
• Key management: AWS KMS for key rotation and auditing

In Transit:

• TLS 1.2+ for all API communications
• Certificate management with AWS Certificate Manager
• Internal service communication: mTLS (mutual TLS)
• API Gateway: enforce HTTPS, reject HTTP

API Security

Rate Limiting:

• Per-user rate limits: 100 requests/minute per user
• Per-IP rate limits: 1000 requests/minute per IP
• Global rate limits: protect backend from total overload
• Implementation: Token bucket algorithm, Redis-based sliding window

Input Validation:

• Validate all inputs server-side (never trust client)
• Use schema validation (JSON Schema, OpenAPI)
• Sanitize inputs to prevent injection attacks
• Content-Type validation: reject unexpected content types

CORS (Cross-Origin Resource Sharing):

• Whitelist specific origins (never use * in production)
• Limit allowed methods and headers
• Set appropriate max-age for preflight caching

CSP (Content Security Policy):

• Restrict which resources the browser can load
• Prevent XSS attacks by whitelisting script sources
• Report violations for monitoring

Data Security

PII (Personally Identifiable Information) Handling:

• Minimize PII collection (only what you need)
• Encrypt PII at rest and in transit
• Mask PII in logs (never log full credit cards, SSNs)
• Implement data retention policies (delete after X days)
• Right to deletion (GDPR compliance)

Data Masking:

• Log masking: Credit card: ****-****-****-1234
• API responses: return only necessary fields
• Database: column-level encryption for sensitive fields

Audit Logging:

• Log all access to sensitive data (who, what, when)
• Immutable audit logs (write-once, append-only)
• Retain audit logs for compliance (7 years for financial data)
• Integration with CloudTrail for AWS API audit

Amazon Security Expectations

Amazon interviewers expect you to mention:

• "All API calls go through API Gateway with OAuth 2.0 token validation."
• "PII fields are encrypted with KMS-managed keys and never logged."
• "We use VPC with security groups to restrict network access between services."
• "We rotate encryption keys quarterly using KMS key rotation."
• "We conduct annual penetration testing and address findings within 30 days."

Amazon Leadership Principle: Trust and Transparency

Security is about protecting customer data. Connect your security decisions to customer trust:

• "We encrypt all customer PII because protecting their data is foundational to maintaining trust."
• "We implement audit logging so we can prove to customers that their data is only accessed by authorized personnel."
• "We follow the principle of least privilege—each service only has access to the data it absolutely needs."