YouTube

Requirements & Scope

Functional Requirements

Core Features

Upload Video — Creators upload videos with title, description, tags, thumbnails; support for large files (up to 12 hours / 256 GB)
Watch Video — Viewers stream video with adaptive quality, controls (play/pause/seek), captions, and speed adjustment
Search — Full-text search across titles, descriptions, tags, channel names with filters (duration, upload date, type)
Like/Dislike — Engagement signals on videos, visible counts
Comment — Threaded comments with likes, replies, and moderation
Subscribe — Follow channels, receive notifications for new uploads
Playlists — Create, edit, share playlists; auto-play next in playlist
Channel Pages — Creator channels with videos, playlists, about section, subscriber count
Shorts — Vertical short-form video (up to 60 seconds)
Live Streaming — Real-time video broadcast with chat

Non-Functional Requirements

Requirement	Target
Availability	99.99% uptime
Upload throughput	500+ hours of video uploaded per minute globally
View throughput	1B+ hours of video watched per day
Latency (streaming)	Video starts < 2 seconds
Latency (live)	< 5 seconds end-to-end for live streams
Scale	2B+ monthly active users
Content variety	800M+ videos in catalog
Device support	Web, mobile (iOS/Android), Smart TVs, gaming consoles

Key Constraints

Videos range from 3-second Shorts to 12-hour long-form content
Must handle viral spikes (video goes from 0 to 100M views in hours)
Content moderation required at scale (millions of uploads daily)
Copyright detection (Content ID) must run on every upload
Global audience: content must be accessible worldwide with low latency

Capacity Estimation

Assumptions:
- 2B monthly active users, 800M daily active
- 500 hours uploaded per minute = 8.33 hours/second
- Average video: 7 minutes, 720p
- Average viewing session: 30 minutes/day

Storage per video (7 min, 720p H.264, 2.5 Mbps):
  Size = 2.5 Mbps * 420s = 131 MB per resolution
  With 5 resolutions: ~655 MB per video
  + 1 original quality: ~655 MB additional
  Total per video: ~1.3 GB (multiple resolutions)

Daily upload storage:
  500 hrs/min * 60 min/hr * 24 hrs = 720,000 hours/day
  720,000 * 1.3 GB = ~936 TB new storage per day

Viewing bandwidth:
  800M users * 30 min/day = 24B minutes/day
  At 2.5 Mbps avg: 24B * 60s * 2.5 Mbps = 3.6 EB/day

Cache hit target: 90%+ from CDN edge

High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                        Clients                             │
│   Web Browser  │  Mobile App  │  Smart TV  │  API Clients  │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│                    API Gateway / Load Balancer              │
│            (Rate limiting, Auth, Routing, CDN)             │
└────┬──────────┬──────────┬──────────┬──────────┬────────────┘
     │          │          │          │          │
     ▼          ▼          ▼          ▼          ▼
┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
│ Upload │ │Streaming│ │Search  │ │User    │ │Social  │
│Service │ │Service │ │Service │ │Service │ │Service │
└───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘
    │          │          │          │          │
    ▼          ▼          ▼          ▼          ▼
┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
│Transcode│ │CDN     │ │Elastic-│ │MySQL   │ │Redis   │
│Pipeline │ │Edge    │ │search  │ │/Cassan-│ │Cache   │
│(K8s)   │ │Servers │ │Cluster │ │dra     │ │        │
└────────┘ └────────┘ └────────┘ └────────┘ └────────┘

Video Upload Pipeline

Upload Architecture

YouTube's upload pipeline is one of the most complex distributed systems in the world, handling 500+ hours of video per minute.

Upload Flow

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│  Creator  │───▶│  Upload  │───▶│  Chunk   │───▶│   S3     │
│  Client   │    │  Service │    │  Manager │    │  Origin  │
└──────────┘    └──────────┘    └──────────┘    └────┬─────┘
                                                     │
                          ┌───────────────────────────┤
                          │                           │
                          ▼                           ▼
                    ┌──────────┐              ┌──────────┐
                    │ Metadata │              │Transcode │
                    │ Service  │              │ Pipeline │
                    └──────────┘              └────┬─────┘
                          │                        │
                          ▼                        ▼
                    ┌──────────┐              ┌──────────┐
                    │ MySQL /  │              │  CDN     │
                    │Cassandra │              │  Push    │
                    └──────────┘              └──────────┘

Chunked Upload Protocol

Large video files are split into chunks for reliable upload:

Video File (2 GB)
┌────────────────────────────────────────────────────┐
│  Chunk 1  │  Chunk 2  │  Chunk 3  │ ... │ Chunk N │
│  (5 MB)   │  (5 MB)   │  (5 MB)   │     │ (last)  │
└─────┬─────┴─────┬─────┴─────┬─────┴─────┴────┬────┘
      │           │           │                │
      ▼           ▼           ▼                ▼
   Upload 1   Upload 2   Upload 3          Upload N
   (parallel) (parallel) (parallel)       (sequential)

Each chunk uploaded independently with:
- Chunk index
- Resume token (for interrupted uploads)
- Integrity checksum (MD5)

Benefits of chunked upload:

Resumability: If upload fails, resume from last successful chunk
Parallelism: Upload multiple chunks simultaneously
Progress tracking: Show upload progress to user
Error isolation: Failed chunk doesn't require full restart
Bandwidth optimization: Adapt chunk size to connection speed

Video Processing Pipeline

After upload to S3, the video enters a multi-stage processing pipeline:

Stage 1: Validation & Virus Scan
    │
    ▼
Stage 2: Content ID (Copyright Detection)
    │  - Audio fingerprinting
    │  - Video fingerprinting
    │  - Match against rights holder database
    │  - Flag for monetization or takedown
    │
    ▼
Stage 3: Content Moderation
    │  - ML model: NSFW detection
    │  - ML model: Violence detection
    │  - ML model: Hate speech (audio/text)
    │  - Queue for human review if flagged
    │
    ▼
Stage 4: Transcoding (Parallel)
    │  - Decode source → Encode to multiple formats/resolutions
    │  - Run on Kubernetes cluster with GPU workers
    │
    ▼
Stage 5: Thumbnail Generation
    │  - Extract 3 auto-generated thumbnails
    │  - Allow custom thumbnail upload
    │
    ▼
Stage 6: Metadata Indexing
    │  - Index in Elasticsearch for search
    │  - Update recommendation features
    │  - Notify subscribers
    │
    ▼
Stage 7: CDN Distribution
       - Push to CDN edge servers
       - Pre-position based on channel's audience geography

Transcoding in Detail

YouTube transcodes every uploaded video into multiple formats and resolutions:

Output Format	Resolution	Codec	Bitrate	Use Case
VP9 2160p	3840×2160	VP9	20 Mbps	4K displays
VP9 1080p	1920×1080	VP9	8 Mbps	HD displays
VP9 720p	1280×720	VP9	4 Mbps	Tablets
H.264 1080p	1920×1080	H.264	5 Mbps	Broad compatibility
H.264 720p	1280×720	H.264	2.5 Mbps	Mobile
H.264 480p	854×480	H.264	1.2 Mbps	Slow connections
H.264 360p	640×360	H.264	700 kbps	Ultra-low bandwidth
Audio only	-	AAC	128 kbps	Audio streaming
HLS	Multiple	H.264	Adaptive	iOS/Safari
DASH	Multiple	VP9/H.264	Adaptive	Android/Web

Parallel Transcoding Architecture

┌──────────────────────────────────────────────────────┐
│              Transcoding Orchestrator                 │
│  ┌────────────────────────────────────────────────┐  │
│  │ 1. Analyze source video (duration, codec, res) │  │
│  │ 2. Determine target formats based on bitrate   │  │
│  │    ladder and source quality                    │  │
│  │ 3. Split video into segments for parallel work │  │
│  └─────────────────────┬──────────────────────────┘  │
│                        │                             │
│  ┌─────────────────────┼─────────────────────────┐   │
│  │  Worker Pool (Kubernetes)                     │   │
│  │  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │   │
│  │  │Worker 1│ │Worker 2│ │Worker 3│ │Worker N│ │   │
│  │  │1080p   │ │720p    │ │480p    │ │360p    │ │   │
│  │  │VP9     │ │H.264   │ │H.264   │ │H.264   │ │   │
│  │  └────────┘ └────────┘ └────────┘ └────────┘ │   │
│  └───────────────────────────────────────────────┘   │
│                        │                             │
│  ┌─────────────────────┼─────────────────────────┐   │
│  │  Completion Monitor                           │   │
│  │  - Track all worker progress                  │   │
│  │  - Detect failures, retry with backoff        │   │
│  │  - When all complete → push to CDN            │   │
│  └───────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────┘

FFmpeg Example Command

# Transcode to 720p H.264 with AAC audio
ffmpeg -i input.mp4 \
  -c:v libx264 -preset medium -crf 23 \
  -vf scale=1280:720 \
  -c:a aac -b:a 128k \
  -movflags +faststart \
  output_720p.mp4

# Generate HLS manifest with multiple quality levels
ffmpeg -i input.mp4 \
  -filter_complex "[0:v]split=3[v1][v2][v3];\n  [v1]scale=1920:1080[v1out];\n  [v2]scale=1280:720[v2out];\n  [v3]scale=854:480[v3out]" \
  -map "[v1out]" -c:v:0 libx264 -b:v:0 5M \
  -map "[v2out]" -c:v:1 libx264 -b:v:1 2.5M \
  -map "[v3out]" -c:v:2 libx264 -b:v:2 1M \
  -map 0:a -c:a aac \
  -f hls -hls_time 6 -hls_playlist_type vod \
  -master_pl_name master.m3u8 \
  -var_stream_map "v:0,a:0 v:1,a:0 v:2,a:0" \
  stream_%v/playlist.m3u8

Upload Service Design

POST /api/v1/upload/init
  Request: { title, description, tags, categoryId }
  Response: { uploadId, chunkSize, uploadUrls[] }

PUT /api/v1/upload/{uploadId}/chunk/{chunkIndex}
  Request: Binary chunk data
  Response: { status: "uploaded", checksum: "..." }

POST /api/v1/upload/{uploadId}/complete
  Request: { thumbnailUrl?, captions? }
  Response: { videoId, status: "processing" }

GET /api/v1/upload/{uploadId}/status
  Response: { status: "processing", progress: 65%, currentStage: "transcoding" }

Streaming & CDN

Video Streaming Architecture

Adaptive Bitrate Streaming (ABR)

YouTube uses the same ABR approach as Netflix — chunked video with client-side quality switching.

┌────────────────────────────────────────────────────┐
│                Client Request Flow                 │
│                                                   │
│  1. Client requests manifest (MPD or M3U8)        │
│     GET /videos/{id}/manifest.mpd                 │
│                                                   │
│  2. Manifest lists all available qualities:        │
│     - 1080p VP9 @ 8 Mbps                          │
│     - 720p H.264 @ 2.5 Mbps                       │
│     - 480p H.264 @ 1.2 Mbps                       │
│     - 360p H.264 @ 700 kbps                       │
│                                                   │
│  3. Client starts with initial quality             │
│     (usually 360p-480p for fast start)             │
│                                                   │
│  4. Client monitors:                               │
│     - Download speed of recent chunks              │
│     - Buffer occupancy level                       │
│     - Network type (WiFi/4G/3G)                    │
│                                                   │
│  5. Client switches quality dynamically            │
│     - Good conditions → request higher quality     │
│     - Poor conditions → drop to lower quality      │
└────────────────────────────────────────────────────┘

YouTube's Bitrate Ladder

Quality	Resolution	Video Codec	Video Bitrate	Audio Codec	Audio Bitrate
2160p60	3840×2160	VP9	20 Mbps	Opus	128 kbps
2160p	3840×2160	VP9	16 Mbps	Opus	128 kbps
1440p60	2560×1440	VP9	12 Mbps	Opus	128 kbps
1440p	2560×1440	VP9	10 Mbps	Opus	128 kbps
1080p60	1920×1080	VP9	8 Mbps	Opus	128 kbps
1080p	1920×1080	VP9/H.264	5 Mbps	AAC	128 kbps
720p60	1280×720	VP9/H.264	3.5 Mbps	AAC	128 kbps
720p	1280×720	H.264	2.5 Mbps	AAC	128 kbps
480p	854×480	H.264	1.2 Mbps	AAC	128 kbps
360p	640×360	H.264	700 kbps	AAC	96 kbps
240p	426×240	H.264	400 kbps	AAC	64 kbps
144p	256×144	H.264	200 kbps	AAC	48 kbps

CDN Architecture

YouTube uses Google's global CDN infrastructure (Google Global Cache - GGC):

┌─────────────────────────────────────────────────────────────┐
│                    Google Origin Servers                    │
│               (Video files, all resolutions)               │
└────────────────────────┬────────────────────────────────────┘
                         │
         ┌───────────────┼───────────────┐
         ▼               ▼               ▼
   ┌──────────┐   ┌──────────┐    ┌──────────┐
   │ Regional │   │ Regional │    │ Regional │
   │ PoP #1   │   │ PoP #2   │    │ PoP #N   │
   │(US-East) │   │(EU-West) │    │(APAC)    │
   └────┬─────┘   └────┬─────┘    └────┬─────┘
        │              │               │
   ┌────┴─────┐   ┌────┴─────┐    ┌────┴─────┐
   │  Google  │   │  Google  │    │  Google  │
   │  Global  │   │  Global  │    │  Global  │
   │  Cache   │   │  Cache   │    │  Cache   │
   │  (GGC)   │   │  (GGC)   │    │  (GGC)   │
   │ in ISPs  │   │ in ISPs  │    │ in ISPs  │
   └────┬─────┘   └────┬─────┘    └────┬─────┘
        │              │               │
   ┌────┴─────┐   ┌────┴─────┐    ┌────┴─────┐
   │  End     │   │  End     │    │  End     │
   │  Users   │   │  Users   │    │  Users   │
   └──────────┘   └──────────┘    └──────────┘

Google Global Cache (GGC)

Google deploys GGC appliances inside ISP networks (similar to Netflix Open Connect):

Feature	Detail
Deployment	Inside ISP data centers globally
Storage	100s of TB per appliance
Content	Most popular videos for that ISP
Refresh	Updated based on popularity trends
Scale	Deployed in 1000s of ISPs worldwide
Benefit	Eliminates cross-ISP traffic, reduces latency

Content Caching Strategy

Content Popularity Distribution (Zipf's Law):

Views
  │
  │█
  │█
  │██
  │███
  │█████
  │████████
  │██████████████
  │████████████████████████
  │████████████████████████████████████████
  └──────────────────────────────────────── Videos (ranked by popularity)

- Top 1% of videos: 80% of views
- Top 10% of videos: 95% of views
- Long tail: 90% of videos get very few views

Caching tiers:
- Tier 1 (GGC in ISP): Top 1000 videos per region
- Tier 2 (Regional PoP): Top 10,000 videos per region
- Tier 3 (Origin): All 800M+ videos

Request Routing

When a user requests a video:

DNS resolution → Directs to nearest Google PoP based on geography and network topology
GGC check → Is the content cached on the local GGC appliance?
- If YES → Serve directly from GGC (lowest latency)
- If NO → Fetch from Regional PoP, cache locally, then serve
ABR manifest → Client receives manifest and begins requesting chunks
Chunk serving → Each chunk served from optimal location

Live Streaming Architecture

YouTube Live uses low-latency protocols:

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│  Creator  │───▶│  Ingest  │───▶│ Transcode│───▶│   CDN    │
│  (OBS/    │    │  Server  │    │ (Real-   │    │  Edge    │
│  Encoder) │    │          │    │  time)   │    │          │
└──────────┘    └──────────┘    └──────────┘    └────┬─────┘
                                                     │
                                              ┌──────┴──────┐
                                              │  Viewers    │
                                              │  (1-15s     │
                                              │  latency)   │
                                              └─────────────┘

Protocols:
- Ingest: RTMP (Real-Time Messaging Protocol)
- Streaming: DASH (primary), HLS (fallback)
- Low-latency: LL-HLS, WebRTC (for <5s latency)

Video Chunk Size Comparison

Platform	Typical Chunk Size	Use Case
YouTube	2-5 seconds	On-demand streaming
Netflix	5 seconds	On-demand streaming
Twitch	2-4 seconds	Live streaming
YouTube Live	1-2 seconds	Live streaming

Smaller chunks = lower latency but more HTTP requests and manifest complexity

Recommendations & Scaling

YouTube Recommendation System

YouTube's recommendation engine drives 70% of total watch time on the platform.

Recommendation Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Candidate Generation                     │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ 1. Collaborative Filtering (user similarity)        │   │
│  │ 2. Content-Based (video features, metadata)         │   │
│  │ 3. Trending / Popular in region                     │   │
│  │ 4. Subscription feed                                 │   │
│  │ 5. Search history                                    │   │
│  │                                                      │   │
│  │ Input: User features + Context (time, device)        │   │
│  │ Output: ~1000 candidate videos                       │   │
│  └─────────────────────────┬───────────────────────────┘   │
│                            │                                │
│                            ▼                                │
│  ┌─────────────────────────────────────────────────────┐   │
│  │              Ranking Model (Deep Neural Network)    │   │
│  │                                                      │   │
│  │  Features:                                           │   │
│  │  - User watch history (last 100 videos)              │   │
│  │  - User search history                               │   │
│  │  - User demographics                                 │   │
│  │  - Video features (title, description, tags, etc.)   │   │
│  │  - Engagement signals (likes, comments, shares)      │   │
│  │  - Time since upload                                 │   │
│  │  - User's device and time of day                     │   │
│  │                                                      │   │
│  │  Output: Predicted:                                  │   │
│  │  - P(click) - probability user clicks thumbnail      │   │
│  │  - P(watch > 50%) - probability of significant watch │   │
│  │  - P(like) - probability of positive engagement      │   │
│  │  - Expected watch time (regression)                  │   │
│  └─────────────────────────┬───────────────────────────┘   │
│                            │                                │
│                            ▼                                │
│  ┌─────────────────────────────────────────────────────┐   │
│  │              Re-ranking & Filtering                  │   │
│  │  - Remove already watched                           │   │
│  │  - Apply diversity rules (no 5 of same channel)     │   │
│  │  - Apply freshness boost for new uploads             │   │
│  │  - Filter blocked/sensitive content                  │   │
│  │  - Balance exploration vs exploitation               │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

Deep Learning Ranking Model

YouTube uses a two-tower neural network:

┌──────────────────┐     ┌──────────────────┐
│   User Tower     │     │  Video Tower      │
│                  │     │                   │
│  Watch history   │     │  Video ID         │
│  (sequence)      │     │  Title (text)     │
│       │          │     │  Description      │
│       ▼          │     │  Tags             │
│  ┌─────────┐     │     │  Category         │
│  │  LSTM / │     │     │  Duration         │
│  │  Trans- │     │     │  Engagement       │
│  │  former │     │     │       │           │
│  └────┬────┘     │     │       ▼           │
│       │          │     │  ┌─────────┐      │
│       ▼          │     │  │  DNN    │      │
│  User Embedding  │     │  └────┬────┘      │
│  (128-d vector)  │     │       │           │
└────────┬─────────┘     │  Video Embedding  │
         │               │  (128-d vector)   │
         │               └────────┬──────────┘
         │                        │
         └───────────┬────────────┘
                     │
              ┌──────▼──────┐
              │  Cosine     │
              │  Similarity │
              │  Score      │
              └──────┬──────┘
                     │
              ┌──────▼──────┐
              │  Combine    │
              │  with other │
              │  features   │
              └──────┬──────┘
                     │
              ┌──────▼──────┐
              │  Final      │
              │  Prediction │
              └─────────────┘

Engagement Signals Used

Signal	Weight	Description
Watch time	High	Total seconds watched (most important)
Click-through rate	High	Thumbnail impressions → clicks
Completion rate	Medium	% of video watched
Likes	Medium	Positive engagement
Comments	Medium	Discussion/engagement
Shares	Medium	Virality signal
Subscribe after watch	High	Strong positive signal
Not interested	High (negative)	User explicitly rejected
Swipe away	High (negative)	Quick dismissal

Content Moderation Pipeline

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│  Upload  │───▶│  ML      │───▶│  Human   │───▶│  Decision│
│          │    │  Screen  │    │  Review  │    │          │
└──────────┘    └────┬─────┘    └────┬─────┘    └────┬─────┘
                     │               │               │
              ┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐
              │ Auto-approve│ │ Queue for   │ │ Approve /   │
              │ (95% of     │ │ human review│ │ Reject /    │
              │  uploads)   │ │ (5% of      │ │ Age-restrict│
              │             │ │  uploads)   │ │             │
              └─────────────┘ └─────────────┘ └─────────────┘

ML Models for Moderation:
- NSFW detection (nudity, explicit content)
- Violence/gore detection
- Hate speech detection (audio transcription + text)
- Spam/scam detection
- Copyright detection (Content ID)
- Misinformation flags (for certain topics)

Scaling Strategies

Database Sharding:

User Data (2B+ users):
- Shard by user_id % N
- N = number of shards (start with 1024, grow as needed)
- Each shard: ~2M users
- Hot shards: popular users get more connections
- Solution: consistent hashing + virtual nodes

Video Metadata (800M+ videos):
- Shard by video_id % N
- Popular videos: cache in Redis/Memcached
- Long-tail videos: served from database directly

Horizontal Scaling:

Stateless Services:
- Upload Service: Scale horizontally behind load balancer
- Search Service: Stateless, scale with Elasticsearch cluster
- User Service: Stateless, connect to sharded database
- Streaming Service: Stateless, serve from CDN

Stateful Services:
- Transcoding: Kubernetes with auto-scaling based on queue depth
- Database: Sharding + read replicas
- Cache: Redis cluster with consistent hashing

CDN Scaling:

YouTube uses Google's infrastructure:
- 200+ edge locations worldwide
- Google Global Cache (GGC) in ISPs
- Dynamic origin selection based on load
- Automatic failover between PoPs
- Capacity scales with Google's infrastructure

Search Architecture

┌──────────┐    ┌──────────┐    ┌──────────┐
│  User    │───▶│  API     │───▶│Elastic-  │
│  Query   │    │  Gateway │    │search    │
└──────────┘    └──────────┘    └────┬─────┘
                                     │
                              ┌──────┴──────┐
                              │  Index      │
                              │  Structure  │
                              │             │
                              │ - Title     │
                              │ - Desc      │
                              │ - Tags      │
                              │ - Channel   │
                              │ - Captions  │
                              │ - Views     │
                              │ - Upload    │
                              │   date      │
                              └─────────────┘

Search Features:
- Autocomplete (prefix matching on popular queries)
- Typo tolerance (fuzzy matching)
- Query expansion (synonyms)
- Personalization (based on watch history)
- Filters (duration, date, type, features)

Storage Breakdown

Data Type	Storage System	Reason
Video files	Google Cloud Storage (S3-like)	Large binary objects, high throughput
Video metadata	MySQL (sharded)	Structured data, ACID transactions
User data	MySQL (sharded)	Structured, relational
Watch history	Bigtable / Cassandra	High write throughput, time-series
Search index	Elasticsearch	Full-text search, autocomplete
Recommendations cache	Redis / Memcached	Low-latency reads, hot data
Thumbnails	CDN + GCS	Served frequently, cacheable
Subtitles/Captions	Cloud Storage	Text files, infrequent access

Practice Problems

0/3solved

advancedDesign YouTube (Design YouTube) System

Design a scalable YouTube (Design YouTube) system. Cover high-level architecture, data model, and API design.

Solution

// Complete system design:
// - Functional + Non-functional requirements
// - Capacity estimation
// - Data model (SQL/NoSQL choice)
// - API endpoints
// - Component architecture
// - Scaling strategy
// - Monitoring & reliability

advancedYouTube (Design YouTube) Scaling

How would you scale YouTube (Design YouTube) to handle 10x the current load? Identify bottlenecks and solutions.

Solution

// Scaling approach:
// 1. Load balancing
// 2. Database sharding/replication
// 3. Cache layer (Redis)
// 4. CDN for static assets
// 5. Async processing (queues)
// 6. Microservices decomposition

advancedYouTube (Design YouTube) Failure Modes

Analyze potential failure modes for YouTube (Design YouTube) and design mitigation strategies.

Solution

// Failure mitigation:
// 1. Redundancy (multi-AZ)
// 2. Circuit breakers
// 3. Retry with backoff
// 4. Dead letter queues
// 5. Health checks
// 6. Graceful degradation

Flashcards

Question

How does YouTube's chunked upload work?

Click to reveal answer

Answer

Video files are split into 5MB chunks uploaded in parallel. Each chunk has an index and checksum. If upload fails, it resumes from the last successful chunk using a resume token, avoiding full re-upload.

Question

What are the stages of YouTube's video processing pipeline?

Click to reveal answer

Answer

1) Validation/Virus Scan → 2) Content ID (Copyright) → 3) Content Moderation (ML + Human) → 4) Parallel Transcoding (multiple resolutions/codecs) → 5) Thumbnail Generation → 6) Metadata Indexing → 7) CDN Distribution

Question

What is Content ID on YouTube?

Click to reveal answer

Answer

Automated copyright detection system that runs on every uploaded video. Uses audio and video fingerprinting to match against a database of rights holders. Can auto-monetize (ads for owner), block, or track the video.

Question

How does YouTube's recommendation system rank videos?

Click to reveal answer

Answer

Two-stage process: 1) Candidate Generation filters 800M videos down to ~1000 candidates using collaborative filtering, content-based, and trending signals. 2) Deep neural network ranks candidates using features like watch history, engagement, video metadata, and user context.

Question

What is Google Global Cache (GGC)?

Click to reveal answer

Answer

Custom servers placed inside ISP data centers worldwide that cache the most popular YouTube videos. Similar to Netflix Open Connect. Eliminates cross-ISP traffic, reduces latency, and handles 90%+ of requests from edge cache.

Question

How does YouTube handle database scaling for 2B+ users?

Click to reveal answer

Answer

MySQL sharding by user_id, with consistent hashing and virtual nodes. Each shard handles ~2M users. Hot data cached in Redis. Video metadata also sharded by video_id. Elasticsearch handles search queries independently.

Question

What protocols does YouTube use for live streaming?

Click to reveal answer

Answer

Ingest: RTMP from creator's encoder. Streaming: DASH (primary) + HLS (fallback) for viewers. Low-latency options: LL-HLS and WebRTC for sub-5-second latency. End-to-end latency: 1-15 seconds depending on protocol.

Revision Notes

Key Takeaways

1.YouTube uploads use chunked protocol (5MB chunks, parallel, resumable) for reliability at scale
2.Video processing pipeline has 7 stages: validation → Content ID → moderation → transcoding → thumbnails → indexing → CDN
3.Transcoding runs in parallel on Kubernetes workers, producing 12+ output formats per video
4.Google Global Cache (GGC) places custom servers inside ISPs to serve popular videos locally
5.Recommendation system drives 70% of watch time using two-stage architecture: candidate generation + DNN ranking
6.Content moderation combines ML auto-screening (95% auto-approve) with human review (5% flagged)
7.YouTube handles 500+ hours of video uploaded per minute through distributed processing pipelines
8.Database scaling uses MySQL sharding by user_id with consistent hashing

Interview Tips

•Start with capacity estimation: 500 hours/min upload, 1B hours/day viewing, 2B users
•Draw the upload pipeline first — it's the most complex and unique part of YouTube vs Netflix
•Explain chunked upload with resumability — this is a key interview talking point
•Discuss parallel transcoding architecture with Kubernetes and FFmpeg examples
•Compare YouTube's CDN (Google GGC) with Netflix's Open Connect — both place servers in ISPs
•For recommendations, explain the two-stage architecture (candidate generation → ranking)
•Cover content moderation as a real-world scaling challenge (ML + human review)
•Address live streaming if time permits — RTMP ingest, LL-HLS/WebRTC for low latency
•Mention database sharding strategy and how to handle hot partitions (popular channels/videos)

Cheat Sheet

YouTube System Design - Cheat Sheet

Architecture Overview

API Gateway → Microservices (Upload, Streaming, Search, User, Social) → Data Stores
CDN: Google Global Cache (GGC) inside ISPs + Google regional PoPs
Storage: GCS (video origin), MySQL (metadata, sharded), Cassandra (watch history), Elasticsearch (search), Redis (cache)

Upload Pipeline

Chunked upload (5MB chunks, parallel, resumable)
Processing: Validation → Content ID → Moderation → Transcoding → Thumbnails → Indexing → CDN
Transcoding: Parallel on Kubernetes, 12+ output formats (VP9 4K, H.264 360p-1080p, HLS, DASH)
FFmpeg-based, runs 10+ workers per video simultaneously

Streaming

Adaptive Bitrate (ABR): Client monitors buffer, switches quality per chunk
Bitrate ladder: 144p to 4K, VP9 for HD, H.264 for compatibility
Chunk size: 2-5 seconds for on-demand, 1-2 seconds for live
Manifest: DASH MPD (primary) + HLS M3U8 (fallback)

CDN (Google Global Cache)

Custom appliances inside ISPs (like Netflix Open Connect)
Caches top 1000 videos per ISP region
200+ Google edge locations + GGC in 1000s of ISPs
90%+ requests served from edge cache

Recommendations

Drives 70% of watch time
Two-stage: Candidate Generation (~1000 videos) → Deep Neural Network Ranking
Signals: watch time, CTR, completion rate, likes, comments, shares, subscriptions
Features: user watch history (LSTM/Transformer), video embeddings, context

Content Moderation

ML models screen all uploads (NSFW, violence, hate speech)
95% auto-approved, 5% queued for human review
Content ID: audio/video fingerprinting for copyright

Scaling Strategies

Databases: Sharded MySQL (user_id % N), consistent hashing
Services: Stateless, horizontal scaling behind load balancers
Transcoding: Kubernetes with auto-scaling based on queue depth
CDN: Google's global infrastructure scales automatically

Key Metrics

500+ hours uploaded per minute
2B+ monthly users
70% watch time from recommendations
800M+ videos in catalog
1B+ hours watched per day

Design Tips

Start with the upload pipeline (most complex part)
Discuss parallel transcoding with FFmpeg examples
Explain GGC and how it differs from traditional CDNs
Cover the two-stage recommendation architecture
Mention content moderation as a scaling challenge

Requirements & Scope

Functional Requirements

Core Features

Non-Functional Requirements

Key Constraints

Capacity Estimation

High-Level Architecture

Video Upload Pipeline

Upload Architecture

Upload Flow

Chunked Upload Protocol

Video Processing Pipeline

Transcoding in Detail

Parallel Transcoding Architecture

FFmpeg Example Command

Upload Service Design

Streaming & CDN

Video Streaming Architecture

Adaptive Bitrate Streaming (ABR)

YouTube's Bitrate Ladder

CDN Architecture

Google Global Cache (GGC)

Content Caching Strategy

Request Routing

Live Streaming Architecture

Video Chunk Size Comparison

Recommendations & Scaling

YouTube Recommendation System

Recommendation Architecture

Deep Learning Ranking Model

Engagement Signals Used

Content Moderation Pipeline

Scaling Strategies

Search Architecture

Storage Breakdown

Practice Problems

Quiz

Flashcards

Revision Notes

Key Takeaways

Interview Tips

Cheat Sheet

YouTube System Design - Cheat Sheet

Architecture Overview

Upload Pipeline

Streaming

CDN (Google Global Cache)

Recommendations

Content Moderation

Scaling Strategies

Key Metrics

Design Tips