Instagram Feed

Functional Requirements

Non-Functional Requirements

Capacity Estimation

Traffic per day:
  DAU: ~500 million
  Feed requests: 500M × 10 feeds/day = 5B feed requests/day
  Uploads: 100M photos/day = ~1,160 uploads/sec
  Storage: 100M photos × 2MB avg = 200TB/day
  CDN traffic: 5B feed views × 2MB = 10 EB/day (served from CDN)

Storage (5 years):
  200TB/day × 365 × 5 = 365 PB raw
  With replicas (3x): ~1.1 EB

Database sizing:
  User table: 2B rows × 1KB = 2 TB
  Post metadata: 365B posts × 0.5KB = 182 TB
  Fanout table: massive — needs Cassandra or similar

Upload Flow

┌─────────┐     ┌──────────────┐     ┌─────────────┐     ┌──────────────────┐     ┌─────────┐
│ Client  │────▶│ Upload       │────▶│ S3 (Raw)    │────▶│ Image            │────▶│ S3      │
│         │     │ Service      │     │             │     │ Processor        │     │(Resized)│
└─────────┘     └──────────────┘     └─────────────┘     └──────────────────┘     └─────────┘
      │                                    │                      │                      │
      │  Signed URL                        │                      │  Multiple sizes      │
      │◀───────────────────────────────────│                      │  ── Thumbnail (150x) │
      │                                    │                      │  ── Medium (640x)    │
      │                                    │                      │  ── Full (1080x)     │
      │                                    │                      │  ── Original         │
      │                                    │                      │                      │
      │                                    ▼                      ▼                      ▼
      │                              ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
      │                              │ Metadata     │     │ CDN (Cloud-  │     │ CDN Edge     │
      │                              │ Service      │     │ front)       │     │ Servers      │
      │                              └──────────────┘     └──────────────┘     └──────────────┘
      │                                    │
      │                                    ▼
      │                              ┌──────────────┐
      │                              │ PostgreSQL   │
      │                              │ (User, Post  │
      │                              │  Metadata)   │
      │                              └──────────────┘
      │
      │  Post ID
      ▼

Detailed Upload Pipeline

Step 1: Client Request

# Client requests a pre-signed S3 URL
POST /api/v1/upload/request
Body: { "content_type": "image/jpeg", "file_size": 2048000 }

# Response
{
    "upload_url": "https://s3.amazonaws.com/bucket/prefix?X-Amz-Signature=...",
    "upload_id": "uuid-v4",
    "expires_in": 300
}

Step 2: Direct S3 Upload

PUT https://s3.amazonaws.com/bucket/prefix?X-Amz-Signature=...
Content-Type: image/jpeg

<binary data>

# S3 returns 200 OK, triggers SQS event

Step 3: Async Image Processing (SQS + Workers)

# Worker picks up message from SQS
import boto3
from PIL import Image
import io

def process_image(s3_bucket, s3_key, upload_id):
    # Download original from S3
    s3 = boto3.client('s3')
    original = s3.get_object(Bucket=s3_bucket, Key=s3_key)['Body'].read()

    # Generate multiple resolutions
    sizes = {
        'thumbnail': (150, 150),
        'medium': (640, 640),
        'full': (1080, 1080),
    }

    for size_name, dimensions in sizes.items():
        img = Image.open(io.BytesIO(original))
        img.thumbnail(dimensions, Image.LANCZOS)

        buffer = io.BytesIO()
        img.save(buffer, format='JPEG', quality=85, optimize=True)
        buffer.seek(0)

        output_key = f"{upload_id}/{size_name}.jpg"
        s3.put_object(
            Bucket=s3_bucket,
            Key=output_key,
            Body=buffer,
            ContentType='image/jpeg',
            CacheControl='max-age=31536000',  # 1 year
            Metadata={'upload_id': upload_id}
        )

    # Store metadata in database
    store_metadata(upload_id, {
        'thumbnail_url': f"https://cdn.example.com/{upload_id}/thumbnail.jpg",
        'medium_url': f"https://cdn.example.com/{upload_id}/medium.jpg",
        'full_url': f"https://cdn.example.com/{upload_id}/full.jpg",
        'original_url': f"https://s3.amazonaws.com/{s3_bucket}/{s3_key}",
        'width': img.width,
        'height': img.height,
        'processed_at': datetime.utcnow()
    })

Image Storage Strategy

Video Processing

Videos require a more complex pipeline:

Video Upload → S3 → SQS → MediaConvert (AWS) or FFmpeg Workers
    ├── Transcode to H.264 (MP4)
    ├── Generate HLS manifest (adaptive bitrate)
    ├── Extract thumbnail at 0s, 25%, 50%, 75%
    ├── Generate preview GIF (5s loop)
    ├── Upload to S3 + CloudFront
    └── Store metadata in DB

CDN Configuration

# CloudFront Distribution
distribution:
  origins:
    - domain: s3-bucket.s3.amazonaws.com
      path: /media
      cache_behavior:
        default_ttl: 86400        # 24 hours
        max_ttl: 31536000         # 1 year
        min_ttl: 0
        allowed_methods: [GET, HEAD]
        cached_methods: [GET, HEAD]
        compress: true
  behaviors:
    - path_pattern: "/media/*/thumbnail.*"
      default_ttl: 604800         # 7 days (thumbnails rarely change)
    - path_pattern: "/media/*/full.*"
      default_ttl: 86400          # 24 hours
    - path_pattern: "/media/original/*"
      default_ttl: 0              # No cache (direct S3)

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│                          FEED GENERATION SYSTEM                        │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ┌──────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────┐  │
│  │ User     │───▶│ Fanout       │───▶│ Feed Cache   │───▶│ Feed     │  │
│  │ Posts    │    │ Service      │    │ (Redis)      │    │ Service  │  │
│  └──────────┘    └──────────────┘    └──────────────┘    └──────────┘  │
│       │                │                                       │       │
│       │                ▼                                       ▼       │
│       │         ┌──────────────┐                         ┌──────────┐  │
│       │         │ Fanout       │                         │ User     │  │
│       │         │ Workers      │                         │ Timeline │  │
│       │         └──────────────┘                         │ (Sorted │  │
│       │                │                                  │  Sets)   │  │
│       │                ▼                                  └──────────┘  │
│       │         ┌──────────────┐                                      │
│       └────────▶│ Cassandra    │  (Feed data — partitioned by user_id) │
│                 └──────────────┘                                      │
└─────────────────────────────────────────────────────────────────────────┘

Fanout-on-Write Strategy

When a user posts, we immediately push that post to all their followers' feeds.

class FeedFanoutService:
    def __init__(self):
        self.redis = RedisCluster()
        self.cassandra = CassandraClient()
        self.sqs = boto3.client('sqs')

    def fanout_post(self, post: Post):
        """Push post to all followers' feed caches."""
        author_id = post.author_id
        followers = self.get_followers(author_id)

        # Celebrity threshold: accounts with > 10K followers
        if len(followers) > 10_000:
            # For celebrities, use fanout-on-read instead
            self.mark_celebrity_post(post)
            return

        # Normal users: fanout to all followers
        batch_size = 100
        for i in range(0, len(followers), batch_size):
            batch = followers[i:i + batch_size]
            self.sqs.send_message(
                QueueUrl=self.fanout_queue_url,
                MessageBody=json.dumps({
                    'post_id': post.id,
                    'follower_ids': batch,
                    'timestamp': post.created_at
                })
            )

    def process_fanout(self, message):
        """Worker processes fanout message."""
        data = json.loads(message.body)
        post_id = data['post_id']
        timestamp = data['timestamp']

        for follower_id in data['follower_ids']:
            # Add to Redis sorted set (feed cache)
            pipe = self.redis.pipeline()
            pipe.zadd(
                f"feed:{follower_id}",
                {post_id: timestamp}
            )
            pipe.zremrangebyrank(f"feed:{follower_id}", 0, -2001)  # Keep last 2000
            pipe.expire(f"feed:{follower_id}", 86400 * 7)  # 7 day TTL
            pipe.execute()

            # Also persist to Cassandra for durability
            self.cassandra.execute("""
                INSERT INTO user_feed (user_id, post_id, post_time)
                VALUES (%s, %s, %s)
            """, (follower_id, post_id, timestamp))

Fanout-on-Read Strategy (Celebrity Fallback)

class FeedService:
    def get_feed(self, user_id: str, limit: int = 20) -> List[Post]:
        """Get feed for a user. Hybrid: cached + on-read."""
        # Step 1: Get cached feed (fanout-on-write posts)
        cached_post_ids = self.redis.zrevrange(
            f"feed:{user_id}", 0, limit * 2
        )

        # Step 2: Fetch posts from followed celebrities (fanout-on-read)
        celebrity_posts = self.get_celebrity_posts(user_id, limit=50)

        # Step 3: Merge and deduplicate
        all_post_ids = set(cached_post_ids)
        all_post_ids.update([p.id for p in celebrity_posts])

        # Step 4: Rank the merged feed
        posts = self.get_posts_by_ids(list(all_post_ids))
        ranked = self.rank_feed(posts, user_id)

        return ranked[:limit]

    def get_celebrity_posts(self, user_id: str, limit: int) -> List[Post]:
        """Get recent posts from celebrities this user follows."""
        celebrities = self.get_followed_celebrities(user_id)
        posts = []

        for celeb_id in celebrities:
            # Get recent posts directly from DB
            recent = self.post_repository.get_recent(
                author_id=celeb_id,
                limit=5
            )
            posts.extend(recent)

        return sorted(posts, key=lambda p: p.created_at, reverse=True)[:limit]

Feed Ranking Algorithm

class FeedRanker:
    """Hybrid ranking: chronological + engagement signals."""

    def rank_feed(self, posts: List[Post], user_id: str) -> List[Post]:
        user_preferences = self.get_user_preferences(user_id)

        scored = []
        for post in posts:
            score = self.calculate_score(post, user_id, user_preferences)
            scored.append((post, score))

        # Sort by score descending
        scored.sort(key=lambda x: x[1], reverse=True)
        return [post for post, score in scored]

    def calculate_score(self, post, user_id, prefs):
        # Time decay: posts lose relevance over time
        age_hours = (datetime.utcnow() - post.created_at).total_seconds() / 3600
        time_score = 1.0 / (1.0 + age_hours * 0.1)  # Half-life ~10 hours

        # Engagement signals
        engagement = (
            post.likes_count * 1.0 +
            post.comments_count * 2.0 +
            post.shares_count * 3.0
        )
        engagement_score = math.log1p(engagement) / 10.0

        # Affinity: how close is this user to the author?
        affinity = self.get_affinity_score(user_id, post.author_id)

        # Content type preference
        type_prefs = {
            'photo': prefs.get('photo_weight', 1.0),
            'video': prefs.get('video_weight', 1.2),
            'carousel': prefs.get('carousel_weight', 1.1),
        }
        type_score = type_prefs.get(post.content_type, 1.0)

        # Final weighted score
        return (
            0.3 * time_score +
            0.3 * engagement_score +
            0.25 * affinity +
            0.15 * type_score
        )

    def get_affinity_score(self, user_id, author_id):
        """Score based on past interactions with this author."""
        interactions = self.get_interactions(user_id, author_id)
        return min(1.0, interactions / 100.0)  # Normalize to [0, 1]

Database Schema (Cassandra)

-- Feed table: partition by user_id, cluster by post_time DESC
CREATE TABLE user_feed (
    user_id UUID,
    post_id UUID,
    post_time TIMESTAMP,
    author_id UUID,
    PRIMARY KEY (user_id, post_time)
) WITH CLUSTERING ORDER BY (post_time DESC)
  AND default_time_to_live = 604800;  -- 7 days TTL

-- Post table: partition by post_id
CREATE TABLE posts (
    post_id UUID,
    author_id UUID,
    content_type TEXT,
    caption TEXT,
    media_urls MAP<TEXT, TEXT>,  -- {'thumbnail': '...', 'full': '...'}
    likes_count COUNTER,
    comments_count COUNTER,
    created_at TIMESTAMP,
    PRIMARY KEY (post_id)
);

-- Fanout table: track which posts were fanned out to whom
CREATE TABLE feed_fanout (
    user_id UUID,
    post_id UUID,
    fanned_at TIMESTAMP,
    PRIMARY KEY (user_id, post_id)
) WITH default_time_to_live = 604800;

Multi-Layer Caching Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        CACHING HIERARCHY                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Layer 1: Client Cache                                              │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  iOS/Android: NSUserDefaults / SharedPreferences            │   │
│  │  - Last 50 feed items                                       │   │
│  │  - User profile (24h TTL)                                    │   │
│  │  - Image thumbnails (in-memory LRU, 50MB limit)             │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                      │
│                              ▼                                      │
│  Layer 2: CDN Cache (CloudFront)                                    │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  - All image/video assets                                    │   │
│  │  - 24h default TTL, 1 year max TTL                          │   │
│  │  - Edge locations: 200+ PoPs worldwide                       │   │
│  │  - Cache hit ratio target: > 95%                             │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                      │
│                              ▼                                      │
│  Layer 3: Application Cache (Redis Cluster)                         │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  - Feed cache: sorted sets (user_id → post_ids by time)     │   │
│  │  - User profiles: hash maps                                  │   │
│  │  - Session data: auth tokens                                 │   │
│  │  - Rate limiting counters                                    │   │
│  │  - Sharded across 100+ nodes, each node: 256GB RAM           │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                      │
│                              ▼                                      │
│  Layer 4: Database (Cassandra / PostgreSQL)                          │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  - Source of truth for all data                              │   │
│  │  - Read replicas for heavy read queries                      │   │
│  │  - Cassandra: feed data, posts, fanout                       │   │
│  │  - PostgreSQL: user accounts, settings, relationships        │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Redis Feed Cache Implementation

class FeedCache:
    """Redis-based feed cache with smart eviction."""

    def __init__(self):
        self.redis = RedisCluster()
        self.FEED_TTL = 86400 * 7  # 7 days
        self.MAX_FEED_SIZE = 2000  # Max posts per user feed

    def add_to_feed(self, user_id: str, post_id: str, score: float):
        """Add post to user's feed cache."""
        key = f"feed:{user_id}"
        pipe = self.redis.pipeline()

        # Add to sorted set (score = timestamp for chronological)
        pipe.zadd(key, {post_id: score})

        # Trim to keep only latest N posts
        pipe.zremrangebyrank(key, 0, -(self.MAX_FEED_SIZE + 1))

        # Set TTL
        pipe.expire(key, self.FEED_TTL)

        pipe.execute()

    def get_feed(self, user_id: str, offset: int = 0, limit: int = 20) -> List[str]:
        """Get paginated feed from cache."""
        key = f"feed:{user_id}"

        # Get post IDs in reverse chronological order
        post_ids = self.redis.zrevrange(key, offset, offset + limit - 1)

        if not post_ids:
            return []  # Cache miss — need to rebuild from Cassandra

        return post_ids

    def rebuild_feed(self, user_id: str):
        """Rebuild feed cache from Cassandra (on cache miss)."""
        # Get recent posts from fanout table
        rows = cassandra.execute("""
            SELECT post_id, post_time FROM user_feed
            WHERE user_id = %s
            LIMIT %s
        """, (user_id, self.MAX_FEED_SIZE))

        pipe = self.redis.pipeline()
        key = f"feed:{user_id}"

        for row in rows:
            pipe.zadd(key, {str(row.post_id): row.post_time.timestamp()})

        pipe.expire(key, self.FEED_TTL)
        pipe.execute()

Cache Invalidation Strategy

CDN Caching Headers

# When uploading media to S3, set proper cache headers
def upload_to_s3_with_caching(upload_id, image_data, size_type):
    s3.put_object(
        Bucket=BUCKET,
        Key=f"media/{upload_id}/{size_type}.jpg",
        Body=image_data,
        ContentType='image/jpeg',
        CacheControl='public, max-age=31536000, immutable',
        # immutable = browser won't revalidate (content-addressable URL)
        Metadata={
            'Content-Hash': hashlib.sha256(image_data).hexdigest()
        }
    )

# CDN invalidation for emergency cases (costs $0.005/invalidation)
def invalidate_cdn(paths: List[str]):
    cloudfront.create_invalidation(
        DistributionId=DISTRIBUTION_ID,
        Items=[{'Path': p, 'Quantity': len(paths)}],
        InvalidationBatch={
            'Paths': {'Items': paths, 'Quantity': len(paths)},
            'CallerReference': str(time.time())
        }
    )

Stories Implementation

class StoriesService:
    """Ephemeral stories with 24-hour TTL."""

    def __init__(self):
        self.redis = RedisCluster()
        self.STORY_TTL = 86400  # 24 hours in seconds

    def post_story(self, user_id: str, media_url: str) -> str:
        story_id = str(uuid.uuid4())
        story_data = {
            'id': story_id,
            'user_id': user_id,
            'media_url': media_url,
            'created_at': time.time(),
            'viewers': []  # List of user_ids who viewed
        }

        # Store in Redis with TTL
        pipe = self.redis.pipeline()
        pipe.setex(
            f"story:{story_id}",
            self.STORY_TTL,
            json.dumps(story_data)
        )

        # Add to user's story ring
        pipe.zadd(
            f"stories:{user_id}",
            {story_id: time.time()}
        )
        pipe.expire(f"stories:{user_id}", self.STORY_TTL)

        pipe.execute()
        return story_id

    def get_stories(self, user_id: str) -> List[dict]:
        """Get stories from followed users."""
        following = self.get_following(user_id)
        stories = []

        for followee_id in following:
            story_ids = self.redis.zrevrange(
                f"stories:{followee_id}", 0, 9  # Latest 10
            )
            for sid in story_ids:
                data = self.redis.get(f"story:{sid}")
                if data:  # Still exists (not expired)
                    story = json.loads(data)
                    if user_id not in story['viewers']:
                        stories.append(story)

        # Sort by recency
        stories.sort(key=lambda s: s['created_at'], reverse=True)
        return stories

Performance Metrics

System Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                            INSTAGRAM ARCHITECTURE                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────┐                                                              │
│  │  Mobile  │──┐                                                           │
│  │  Client  │  │                                                           │
│  └──────────┘  │    ┌──────────────┐    ┌──────────────┐                  │
│                ├───▶│  API Gateway │───▶│  Load        │                  │
│  ┌──────────┐  │    │  (Kong/AWS)  │    │  Balancer    │                  │
│  │  Web     │──┘    └──────────────┘    └──────┬───────┘                  │
│  │  Client  │                                  │                           │
│  └──────────┘                    ┌─────────────┼─────────────┐            │
│                                  │             │             │            │
│                                  ▼             ▼             ▼            │
│                           ┌───────────┐ ┌───────────┐ ┌───────────┐     │
│                           │  Upload   │ │   Feed    │ │  Stories  │     │
│                           │  Service  │ │  Service  │ │  Service  │     │
│                           └─────┬─────┘ └─────┬─────┘ └─────┬─────┘     │
│                                 │             │             │             │
│                           ┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐     │
│                           │    SQS    │ │   Redis   │ │   Redis   │     │
│                           └─────┬─────┘ └─────┬─────┘ └─────┬─────┘     │
│                                 │             │             │             │
│                           ┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐     │
│                           │  Image    │ │Cassandra  │ │ Cassandra │     │
│                           │ Processor │ │           │ │           │     │
│                           └─────┬─────┘ └───────────┘ └───────────┘     │
│                                 │                                        │
│                           ┌─────▼─────┐                                  │
│                           │    S3     │                                  │
│                           │  (Media)  │                                  │
│                           └─────┬─────┘                                  │
│                                 │                                        │
│                           ┌─────▼─────┐                                  │
│                           │ CloudFront│                                  │
│                           │   (CDN)   │                                  │
│                           └───────────┘                                  │
└─────────────────────────────────────────────────────────────────────────────┘