Pastebin

Requirements & Scope

Functional Requirements

1. Create Paste: Users can create a paste with text content, syntax highlighting language, title, and optional expiration.
2. Read Paste: Users can view a paste via a unique URL (e.g., pastebin.com/abc123).
3. Raw View: Users can view the raw text content (no HTML rendering) via /raw/{pasteKey}.
4. Syntax Highlighting: Support 100+ programming languages for code highlighting.
5. Auto-Expire: Pastes can expire after a configurable duration (10 min, 1 hour, 1 day, 1 week, 1 month, never).
6. Delete Paste: Users can manually delete their pastes.
7. Search/Search: Users can search pastes by title, content, or language (public pastes only).
8. User Dashboard: Logged-in users can see all their pastes with management options.

Non-Functional Requirements

Requirement	Target
Read Latency	< 100ms for paste retrieval
Availability	99.9% uptime
Throughput	10M pastes created per day, 100M reads per day
Storage	Average paste size 10 KB, max 10 MB
Read:Write Ratio	~10:1 (reads significantly outnumber writes)
Data Durability	Pastes must not be lost until they expire or are deleted
Cost	Minimize storage cost for billions of pastes

Capacity Estimation

Writes: 10M pastes/day ≈ 115 pastes/sec
Reads:  100M reads/day ≈ 1,150 reads/sec
Storage per year: 10M × 365 × 10 KB = 36.5 TB/year
Storage (5 years): ~182.5 TB
Cache (hot 10% of reads): 10M reads/day × 10 KB = 100 GB in Redis

Key Clarifying Questions

• What is the maximum paste size? → 10 MB for free tier, 100 MB for premium.
• Are pastes public or private? → Both: public (searchable), unlisted (link-only), private (account only).
• Do we need versioning? → No, pastes are immutable once created.
• Do we need user accounts? → Yes, for managing pastes. Anonymous pastes allowed.
• What languages for syntax highlighting? → 100+ via Highlight.js or Prism.js.

Storage Design

Storage Architecture

┌──────────────────────────────────────────────────────────┐
│                    Storage Layer                          │
│                                                          │
│  ┌─────────────────┐         ┌─────────────────────┐    │
│  │   Amazon S3     │         │   MySQL / DynamoDB  │    │
│  │                 │         │                     │    │
│  │  Paste Content  │         │   Paste Metadata    │    │
│  │  (raw text,     │         │   - paste_key       │    │
│  │   up to 10MB)   │         │   - user_id         │    │
│  │                 │         │   - title           │    │
│  │  Bucket:        │         │   - language        │    │
│  │  pastebin-      │         │   - visibility      │    │
│  │  content-{env}  │         │   - expires_at      │    │
│  │                 │         │   - created_at      │    │
│  └─────────────────┘         │   - s3_key          │    │
│                              └─────────────────────┘    │
│                                                          │
│  ┌──────────────────────────────────────────────────┐   │
│  │              CDN (CloudFront)                     │   │
│  │  - Cache raw paste content (TTL: 24h)             │   │
│  │  - Cache static assets (JS, CSS, fonts)           │   │
│  └──────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────┘

Why Separate Content from Metadata?

Aspect	Content (S3)	Metadata (SQL/DynamoDB)
Size	Large (up to 10 MB)	Small (fixed schema, ~500 bytes)
Access Pattern	Read-heavy, rarely updated	Frequent reads, occasional deletes
Cost	S3: $0.023/GB/month (very cheap)	DB: higher per-GB cost
Caching	CDN-cacheable, immutable	Application-cacheable
Search	Not searchable	Searchable via DB indexes

Metadata Schema (SQL)

CREATE TABLE pastes (
    id            BIGINT PRIMARY KEY AUTO_INCREMENT,
    paste_key     VARCHAR(10) UNIQUE NOT NULL,   -- base62-encoded id
    user_id       BIGINT,
    title         VARCHAR(200),
    language      VARCHAR(50) DEFAULT 'text',
    visibility    ENUM('public', 'unlisted', 'private') DEFAULT 'public',
    expires_at    TIMESTAMP NULL,                -- NULL = never expires
    created_at    TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    s3_key        VARCHAR(500) NOT NULL,         -- path in S3 bucket
    size_bytes    INT DEFAULT 0
);

CREATE INDEX idx_paste_key ON pastes(paste_key);
CREATE INDEX idx_user_id ON pastes(user_id, created_at DESC);
CREATE INDEX idx_expires ON pastes(expires_at);
CREATE INDEX idx_visibility ON pastes(visibility, created_at DESC);

Metadata Schema (DynamoDB)

Table: pastes
Partition Key: paste_key (String)

Attributes:
  user_id: Number
  title: String
  language: String
  visibility: String
  expires_at: Number (epoch, 0 = never)
  created_at: Number (epoch)
  s3_key: String
  size_bytes: Number

GSI: user_id-index (for user dashboard queries)
GSI: visibility-created_at-index (for public paste browsing)

S3 Key Structure

s3://pastebin-content-{env}/{year}/{month}/{day}/{paste_key}.txt

Example:
s3://pastebin-content-prod/2026/08/16/abc123.txt

Why partition by date?
  - S3 listing is faster with date partitions
  - Makes lifecycle policies easier (delete old data)
  - Better S3 performance (distributes across prefix partitions)

Key Generation

import string

ALPHABET = string.ascii_letters + string.digits  # 62 chars
BASE = len(ALPHABET)

def base62_encode(num: int) -> str:
    if num == 0:
        return ALPHABET[0]
    chars = []
    while num > 0:
        chars.append(ALPHABET[num % BASE])
        num //= BASE
    return ''.join(reversed(chars))

# Auto-increment ID → paste key
# ID 1 → "b"
# ID 2 → "c"
# ID 62 → "ba"
# ID 3844 → "baa"
# ID 1000000 → "4c92" (4 chars)

Why auto-increment ID + base62? It guarantees no collisions, the key length grows logarithmically with ID, and IDs are globally unique across shards.

API & Architecture

REST API Design

Create Paste

POST /api/v1/pastes

Headers:
  Authorization: Bearer <token>  (optional for anonymous)

Request Body:
{
  "title": "My Python Script",
  "content": "def hello():\n    print('Hello, World!')",
  "language": "python",
  "visibility": "public",        // public | unlisted | private
  "expiration": "10m"            // 10m | 1h | 1d | 1w | 1m | never
}

Response (201 Created):
{
  "pasteKey": "abc123",
  "url": "https://pastebin.com/abc123",
  "rawUrl": "https://pastebin.com/raw/abc123",
  "createdAt": "2026-08-16T10:00:00Z",
  "expiresAt": "2026-08-16T10:10:00Z",
  "sizeBytes": 42
}

Read Paste

GET /api/v1/pastes/{pasteKey}

Response (200 OK):
{
  "pasteKey": "abc123",
  "title": "My Python Script",
  "content": "def hello():\n    print('Hello, World!')",
  "language": "python",
  "visibility": "public",
  "createdAt": "2026-08-16T10:00:00Z",
  "expiresAt": "2026-08-16T10:10:00Z",
  "sizeBytes": 42,
  "user": { "username": "developer1" }
}

Raw Paste Content

GET /api/v1/pastes/{pasteKey}/raw

Response: text/plain

def hello():
    print('Hello, World!')

Delete Paste

DELETE /api/v1/pastes/{pasteKey}

Headers:
  Authorization: Bearer <token>

Response (204 No Content)

List User's Pastes

GET /api/v1/users/{userId}/pastes?page=1&limit=20

Response (200 OK):
{
  "pastes": [...],
  "pagination": { "page": 1, "totalPages": 5, "totalItems": 92 }
}

High-Level Architecture

┌──────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────┐
│  Client   │────▶│ API Gateway  │────▶│ Paste Service│────▶│    S3    │
│  (Web/    │     │  (Rate Limit,│     │ (CRUD,       │     │ (Content │
│  Mobile)  │     │   Auth, CDN) │     │  Validation) │     │  Store)  │
└──────────┘     └──────────────┘     └──────┬───────┘     └──────────┘
                                             │
                                             │
                                             ▼
                                      ┌──────────────┐
                                      │    Redis     │
                                      │   Cache      │
                                      └──────────────┘
                                             │
                                             ▼
                                      ┌──────────────┐
                                      │    MySQL /   │
                                      │   DynamoDB   │
                                      │  (Metadata)  │
                                      └──────────────┘

Request Flow — Create Paste

1. Client sends POST /api/v1/pastes with content
2. API Gateway: rate limit check, authentication (optional)
3. Paste Service:
   a. Validate input (title, content size, language)
   b. Generate unique paste_key (base62 of auto-increment ID)
   c. Upload content to S3 at path /{year}/{month}/{day}/{pasteKey}.txt
   d. Insert metadata record into DB
   e. Return paste_key and URLs to client

Request Flow — Read Paste

1. Client sends GET /api/v1/pastes/{pasteKey}
2. API Gateway: rate limit check
3. Paste Service:
   a. Check Redis cache for metadata (key: paste:{pasteKey})
      - HIT → use cached metadata
      - MISS → query DB, populate cache (TTL 1h)
   b. Check if paste is expired (expires_at < NOW())
      - Expired → return 404, trigger async delete
   c. Check visibility (private pastes need owner auth)
   d. Fetch content from S3 (or CDN for raw view)
      - For rendered view: return metadata + content
      - For raw view: proxy S3 response as text/plain
   e. Return response to client

Request Flow — Raw Paste (CDN Optimized)

GET /raw/{pasteKey}

1. Request hits CloudFront CDN
   - Cache key: /raw/{pasteKey}
   - TTL: 24 hours (pastes are immutable)
   - If cached → return directly from edge (fastest path)

2. Cache miss → origin server:
   a. Look up s3_key from metadata (DB or cache)
   b. Proxy request to S3
   c. Return response with CDN cache headers

Benefit: Popular raw pastes are served from CDN edge,
         reducing origin server load by 90%+.

Syntax Highlighting

Server-side (for API responses):
  - Return raw content + language identifier
  - Client applies Highlight.js / Prism.js rendering

Alternative (pre-rendered):
  - Generate highlighted HTML on paste creation
  - Store in S3 alongside raw content: {pasteKey}_highlighted.html
  - Serve pre-rendered HTML (no client-side processing)

Rate Limiting

Limits:
  - Anonymous: 10 pastes/hour per IP
  - Authenticated: 100 pastes/hour per user
  - Read: 1000 requests/minute per IP

Implementation:
  - Sliding window counter in Redis
  - Key: rate_limit:{user_id}:{window}
  - For anonymous: rate_limit:ip:{ip_address}:{window}

Scaling & Optimization

Scaling Strategy

                     ┌──────────────────────────┐
                     │     CloudFront CDN        │
                     │  (Static assets + raw     │
                     │   paste content)           │
                     └────────────┬─────────────┘
                                  │
                     ┌────────────▼─────────────┐
                     │      Load Balancer        │
                     │   (Application Load       │
                     │    Balancer - ALB)         │
                     └────────────┬─────────────┘
                                  │
            ┌─────────────────────┼─────────────────────┐
            │                     │                      │
      ┌─────▼─────┐      ┌───────▼───────┐      ┌───────▼───────┐
      │Paste Svc 1│      │ Paste Svc 2   │      │ Paste Svc N   │
      │(Stateless)│      │ (Stateless)   │      │ (Stateless)   │
      └─────┬─────┘      └───────┬───────┘      └───────┬───────┘
            │                     │                      │
            └─────────────────────┼──────────────────────┘
                                  │
            ┌─────────────────────┼──────────────────────┐
            │                     │                      │
      ┌─────▼─────┐      ┌───────▼───────┐      ┌───────▼───────┐
      │   Redis    │      │   Redis       │      │   Redis       │
      │  Cluster   │      │  Cluster      │      │  Cluster      │
      │ (3 master, │      │ (replica)     │      │ (replica)     │
      │  3 replica)│      │               │      │               │
      └─────┬─────┘      └───────┬───────┘      └───────┬───────┘
            │                     │                      │
            └─────────────────────┼──────────────────────┘
                                  │
            ┌─────────────────────┼──────────────────────┐
            │                     │                      │
      ┌─────▼─────┐      ┌───────▼───────┐      ┌───────▼───────┐
      │  MySQL     │      │  MySQL        │      │  MySQL        │
      │  Primary   │      │  Read Replica │      │  Read Replica │
      └───────────┘      └───────────────┘      └───────────────┘
            │
            ▼
      ┌───────────┐
      │ Amazon S3 │
      │ (Content) │
      └───────────┘

CDN Caching Strategy

CloudFront Distribution:

1. Static Assets (JS, CSS, images):
   - Path: /static/*, /assets/*
   - TTL: 30 days (versioned filenames)
   - Compression: gzip, brotli

2. Raw Paste Content:
   - Path: /raw/*
   - TTL: 24 hours (pastes are immutable after creation)
   - Cache Key: /raw/{pasteKey}

3. Rendered Paste Pages:
   - Path: /{pasteKey}
   - TTL: 1 hour (for public pastes)
   - Vary: Accept-Encoding
   - Bypass cache for private/unlisted pastes

Cache Invalidation:
  - On paste delete: invalidate /raw/{pasteKey} and /{pasteKey}
  - On paste update (if allowed): invalidate affected paths

TTL-Based Expiration

Expiration Options:
  10 minutes  → 600 seconds
  1 hour      → 3600 seconds
  1 day       → 86400 seconds
  1 week      → 604800 seconds
  1 month     → 2592000 seconds
  never       → NULL

Two-Phase Expiration:

Phase 1 — Soft Delete (Lazy):
  - On read: check if expires_at < NOW()
  - If expired: return 404 to client
  - Publish "paste.expired" event to Kafka

Phase 2 — Hard Delete (Background):
  - Worker consumes "paste.expired" events
  - Deletes S3 object: s3.deleteObject(s3_key)
  - Deletes DB record: DELETE FROM pastes WHERE paste_key = ?
  - Invalidates CDN cache

Garbage Collection

Background Job: paste-gc

Schedule: Every hour

Process:
  1. SELECT paste_key, s3_key FROM pastes
     WHERE expires_at IS NOT NULL
     AND expires_at < NOW()
     LIMIT 1000

  2. For each expired paste:
     a. Delete S3 object
     b. Delete DB record
     c. Invalidate CDN cache

  3. Log metrics:
     - pastes_deleted_total
     - s3_objects_deleted_total
     - gc_duration_seconds

Scaling GC:
  - Partition work by paste_key hash
  - Multiple GC workers process different partitions
  - Monitor GC lag (oldest unprocessed expired paste)

Database Sharding

Strategy: Hash-based sharding on paste_key

Shard Assignment:
  Shard 0: hash(paste_key) % 4 == 0
  Shard 1: hash(paste_key) % 4 == 1
  Shard 2: hash(paste_key) % 4 == 2
  Shard 3: hash(paste_key) % 4 == 3

Each Shard:
  - 1 primary (writes)
  - 2 read replicas (reads)
  - Separate connection pool per shard

Shard Router:
  - Application-level routing
  - Consistent hashing for minimal reshuffling
  - Shard map cached in Redis, refreshed every 5 min

Read-Heavy Optimization

Technique 1: Application-Level Cache (Caffeine/Guava)
  - In-memory LRU cache on each app server
  - TTL: 60 seconds
  - Stores hot paste metadata (top 1% = 50% of reads)
  - Reduces Redis load by 10-50x

Technique 2: Cache Warming
  - On app server startup, pre-load top 1000 pastes into local cache
  - Periodically refresh based on access frequency

Technique 3: Connection Pooling
  - HikariCP for MySQL (pool size: 20 per shard)
  - Jedis/Lettuce for Redis (pool size: 50)
  - Avoid connection creation overhead on each request

Monitoring & Observability

Key Metrics:
  - Paste creation rate (pastes/sec)
  - Read latency (p50, p95, p99)
  - Cache hit rate (Redis + CDN)
  - S3 request rate and latency
  - DB connection pool utilization
  - GC lag (oldest unprocessed expired paste)
  - Storage growth rate (GB/day)

Alerts:
  - Read p99 > 200ms → investigate CDN/cache/DB
  - Cache hit rate < 70% → possible hot key issue
  - GC lag > 1 hour → scale GC workers
  - S3 5xx errors > 0.1% → check S3 health
  - Storage growth anomaly → possible abuse

Security Considerations

1. Content Moderation:
   - Scan public pastes for malware/phishing links
   - Integrate with VirusTotal API on creation
   - Flag and quarantine suspicious content

2. Abuse Prevention:
   - Rate limiting per IP and per user
   - CAPTCHA for anonymous paste creation after 5 pastes/hour
   - Block paste creation from known bot IPs

3. Access Control:
   - Private pastes: verify user_id matches owner on read
   - Unlisted pastes: accessible via link only, not searchable
   - Public pastes: indexed and searchable

4. Content Security:
   - Sanitize paste content before rendering (prevent XSS)
   - CSP headers on rendered paste pages
   - Content-Type enforcement on raw endpoint

Cost Estimation

Per 10M pastes/day (10 KB average):

  S3 Storage: 36.5 TB/year × $0.023/GB = $840/year ≈ $70/month
  S3 Requests: 10M PUT + 100M GET × $0.0004/1K = $44/month
  CloudFront: 100M reads × $0.085/10K = $850/month
  MySQL (RDS): 4 shards × r5.xlarge = $2,400/month
  Redis (ElastiCache): 3 nodes × r5.large = $900/month
  Compute (EC2): 10 servers × m5.xlarge = $1,400/month
  Total: ~$5,664/month

Interview Tips

1. Start with storage design — The S3 + metadata DB split is the key insight. Explain why large content shouldn't go in a database.
2. Explain base62 key generation — Show you understand the tradeoff between key length and space.
3. CDN for raw pastes — This is a major optimization. Pastes are immutable, so CDN caching is very effective.
4. Two-phase expiration — Lazy delete on read + background GC is more robust than either approach alone.
5. Cost optimization — S3 is extremely cheap for storage; focus cost discussion on compute and CDN.
6. Privacy model — Explain public vs unlisted vs private visibility levels clearly.