News Feed

Functional Requirements

Core features of a Facebook/LinkedIn-style news feed:

• Create Posts: Users publish text, images, videos, links, or status updates
• View Feed: Users see a feed of posts from connections/friends, sorted chronologically or by relevance
• Like/Comment: Users engage with posts through reactions and threaded comments
• Share/Repost: Users share posts to their own network
• Follow Connections: Users connect with others to see their posts in feed
• Notifications: Users receive alerts for interactions on their posts

Non-Functional Requirements

Requirement	Target	Rationale
Feed Generation Latency	< 500ms	Users expect instant feed load
Availability	99.99%	Social platforms must always be accessible
Consistency	Eventual consistency OK	A few seconds delay for new posts is acceptable
Throughput	100K posts/sec write, 500K feed reads/sec	LinkedIn-scale numbers
Durability	All posts must be persisted permanently	Users expect data persistence
Media Support	Images, videos, links with previews	Rich content is essential
Scalability	1B+ registered users, 500M+ DAU	Must handle massive scale

Core Entities

User:      { user_id, name, email, profile_pic, created_at }
Post:      { post_id, user_id, content, media_urls, post_type, visibility, created_at }
Like:      { like_id, user_id, post_id, reaction_type, created_at }
Comment:    { comment_id, user_id, post_id, parent_id, content, created_at }
Follow:    { follower_id, followee_id, created_at }
Feed:      { user_id, post_ids[], last_updated }
Share:     { share_id, user_id, original_post_id, commentary, created_at }

Capacity Estimation

Write load: 100M posts/day = ~1,160 posts/sec avg, ~3,000 posts/sec peak
Read load: Each user views feed ~15 times/day = 7.5B reads/day = ~87K reads/sec avg
Storage: 100M posts/day * 2KB avg (including metadata) = 200GB/day = ~73TB/year
Media: 100M posts * 30% have media * 500KB avg = 15TB/day
Comments: ~500M comments/day = ~5.8K comments/sec

Key Design Decisions

1. Feed ordering: Chronological (simple) vs algorithmic/ranked (engagement-optimized)
2. Fanout strategy: Push vs pull vs hybrid for feed generation
3. Media handling: CDN for images/videos, async upload processing
4. Comment threading: Flat vs nested comment structure
5. Real-time updates: Long-polling, WebSockets, or periodic polling

How Fanout on Write Works

When a user creates a post, the system immediately pushes the post ID to all followers' pre-computed feed caches.

User creates post
    |
    v
Post Service stores post in DB
    |
    v
Publishes "new_post" event to Kafka
    |
    v
Fanout Workers consume event
    |
    +---> Push post_id to Follower 1's feed cache
    +---> Push post_id to Follower 2's feed cache
    +---> Push post_id to Follower 3's feed cache
    +---> ... (all followers)

Implementation

Fanout Worker Logic

def fanout_post(post, author):
    follower_ids = get_follower_ids(author.id)
    
    pipeline = redis.pipeline()
    for follower_id in follower_ids:
        # Add post to follower's feed sorted set (scored by timestamp)
        pipeline.zadd(
            f"feed:{follower_id}",
            {post.id: post.created_at.timestamp()}
        )
        # Trim to keep only last 1000 posts per user
        pipeline.zremrangebyrank(f"feed:{follower_id}", 0, -1001)
        
        # Send notification (async, separate pipeline)
        if is_active(follower_id):
            notify(follower_id, post)
    
    pipeline.execute()  # Single atomic batch

Feed Read Path

def get_feed(user_id, page=1, count=50):
    offset = (page - 1) * count
    
    # 1. Get post IDs from Redis sorted set
    post_ids = redis.zrevrange(f"feed:{user_id}", offset, offset + count - 1)
    
    # 2. Batch fetch full post details from database
    posts = batch_get_posts(post_ids)
    
    # 3. Hydrate with author info, like counts, comment counts
    hydrated = hydrate_posts(posts)
    
    return hydrated

Pros and Cons

Pros	Cons
Feed reads are extremely fast (O(1) lookup)	Write amplification: 1 post = N writes (N = follower count)
No computation at read time	Wasted work for users who never check their feed
Simple read path, easy to cache	Popular users cause massive fanout bursts
Predictable read latency	Storage overhead: each user's feed is duplicated

When to Use Fanout on Write

• Users typically have a bounded number of connections (< 5,000)
• Read-to-write ratio is high (users read much more than they post)
• Low latency feed generation is critical
• System can tolerate some write amplification

How Fanout on Read Works

When a user opens their feed, the system dynamically fetches recent posts from all their connections and merges them in real-time.

User opens feed
    |
    v
Feed Service receives request
    |
    v
Get list of followees (who user follows)
    |
    v
Query recent posts from each followee (parallel)
    |
    +---> User A: latest 20 posts
    +---> User B: latest 20 posts
    +---> User C: latest 20 posts
    +---> ... (all followees)
    |
    v
Merge all posts, sort by timestamp
    |
    v
Apply ranking model (optional)
    |
    v
Return top N posts to client

Implementation

Feed Read Logic

def get_feed(user_id, count=50):
    # 1. Get followee list
    followee_ids = get_followees(user_id)  # From graph DB or cache
    
    # 2. Fetch recent posts from each followee (parallel queries)
    all_posts = []
    with ThreadPool(max_workers=20) as pool:
        futures = [
            pool.submit(get_recent_posts, fid, limit=20)
            for fid in followee_ids
        ]
        for future in futures:
            all_posts.extend(future.result())
    
    # 3. Merge and sort by timestamp
    all_posts.sort(key=lambda p: p.created_at, reverse=True)
    
    # 4. Apply ranking (optional)
    ranked_posts = rank_posts(all_posts, user_id)
    
    return ranked_posts[:count]

Database Queries

-- Get recent posts from a specific user
SELECT post_id, content, media_urls, created_at
FROM posts
WHERE user_id = ? AND visibility = 'public'
ORDER BY created_at DESC
LIMIT 20;

-- Get followees of a user
SELECT followee_id
FROM follows
WHERE follower_id = ?;

Pros and Cons

Pros	Cons
No write amplification	Slow reads: must query all followees
Always shows freshest content	Latency scales with number of followees
No wasted computation	Computationally expensive at read time
Storage efficient (no duplicate feeds)	Hard to meet <500ms SLA for users with many connections

Caching Optimizations

1. Cache followee lists in Redis to avoid graph DB lookups on every read
2. Cache recent posts per user in Redis to reduce DB queries
3. Use read replicas for post queries to distribute read load
4. Pre-warm feeds for high-traffic users during off-peak hours

The Hybrid Strategy

Combine fanout-on-write for normal users with fanout-on-read for high-follower accounts.

User creates post
    |
    v
Post Service stores post
    |
    v
Check author's follower count:
    |
    +-- < 10,000 followers --> Fanout Service pushes to followers' feeds
    |
    +-- >= 10,000 followers --> No fanout; pull at read time

Feed Read with Hybrid

def get_feed_hybrid(user_id, count=50):
    # 1. Get pre-computed feed from Redis (push-based posts)
    pre_computed_ids = redis.zrevrange(f"feed:{user_id}", 0, count * 2)
    pre_computed_posts = batch_get_posts(pre_computed_ids)
    
    # 2. Get followed high-follower accounts
    high_follower_ids = get_followed_high_follower_accounts(user_id)
    
    # 3. Pull recent posts from high-follower accounts
    pull_posts = []
    for hf_id in high_follower_ids:
        posts = get_recent_posts(hf_id, limit=5)
        pull_posts.extend(posts)
    
    # 4. Merge, deduplicate, and sort
    all_posts = merge_and_dedup(pre_computed_posts, pull_posts)
    all_posts.sort(key=lambda p: p.created_at, reverse=True)
    
    # 5. Apply ranking
    ranked = apply_ranking_model(all_posts, user_id)
    
    return ranked[:count]

Threshold Design

Follower Count	Strategy	Fanout Cost	Read Cost
< 1,000	Push only	Minimal	Very fast
1,000 - 10,000	Push only	Moderate	Fast
10,000 - 100,000	Hybrid (pull this user's posts)	N/A for this user	Slight delay
> 100,000	Pull only	Zero	Must pull

Migration Logic

When a user crosses the threshold (e.g., from 9,999 to 10,000 followers):

1. Stop pushing new posts to followers' feeds
2. Existing pushed posts remain in followers' caches until they expire
3. Mark user as "high-follower" in metadata
4. Followers' read path automatically starts pulling from this user

Benefits of Hybrid

• 95% of users have <1,000 followers — fanout-on-write works perfectly
• Top 5% of users (influencers, brands) use pull — avoids massive fanout
• Feed latency remains <500ms for all users
• Write amplification stays bounded
• This is what Facebook, LinkedIn, and Instagram actually use

Feed Ranking Models

Chronological (Simple)

Sort posts by creation timestamp, newest first. Simple but may show low-quality or irrelevant content.

Engagement-Based Ranking

Predict the probability a user will engage (like, comment, share, click) with each post.

score(post, user) = w1 * recency(post)
                  + w2 * affinity(user, post.author)
                  + w3 * content_quality(post)
                  + w4 * engagement_rate(post)
                  + w5 * post_type_weight(post.type)

Ranking Signal Breakdown

Signal	Description	Weight
Recency	How recent is the post (exponential decay)	0.3
Affinity	How often user interacts with author	0.25
Engagement Rate	Likes/comments/shares per impression	0.2
Content Quality	Text quality, image resolution, spam score	0.15
Post Type	Video > Photo > Link > Text (engagement varies)	0.1

ML Ranking Pipeline

1. Candidate Generation: Fetch ~500 candidate posts (pull + push feeds)
2. Feature Extraction: Extract user, post, and interaction features
3. Model Scoring: Run through ranking model (e.g., XGBoost, neural net)
4. Filtering: Remove blocked/muted users, already-seen posts
5. Diversification: Ensure mix of content types, authors
6. Return top 50 posts to client

Optimization Strategies

CDN for Media

Client --> CDN Edge (cached media)
         |
         +--> Cache HIT: serve from edge (< 50ms)
         +--> Cache MISS: fetch from S3 origin, cache at edge

• Images served from nearest CDN edge location
• Videos use adaptive bitrate streaming (HLS/DASH)
• Thumbnails pre-generated and cached

Background Pre-fetching

On app open (background thread):
1. Fetch next page of feed while user reads current page
2. Pre-load images for visible posts
3. Pre-fetch comments for posts near viewport

Async Processing

• Like/comment counts updated asynchronously via Kafka events
• Notification delivery is async (separate service)
• Media processing (thumbnails, transcoding) is async
• Feed ranking model runs in background, updates cached feed scores

Pagination & Infinite Scroll

GET /api/feed?cursor=<timestamp>&limit=50

Response:
{
  "posts": [...],
  "next_cursor": "2024-01-15T10:30:00Z",
  "has_more": true
}

Use cursor-based pagination (not offset) for consistent results during concurrent writes.

Handling Viral Content

When a post gets millions of views/engagements in minutes:

1. Rate limit comments to prevent database overload
2. Cache hot post at CDN edge and application cache
3. Queue comment writes through Kafka to smooth out spikes
4. Shard post data across multiple partitions if needed
5. Degrade gracefully: show cached like counts, delay real-time updates