Requirements & Scope
Functional Requirements
- Upload/Download Files: Users can upload files of any type (up to 5GB) and download them to any device
- Sync Across Devices: Changes made on one device automatically sync to all other devices within seconds
- Share Files: Users can share files/folders with specific users or generate shareable links
- Folder Hierarchy: Support nested folder structures like a file system
- Version History: Maintain full version history; users can restore previous versions
- Offline Support: Users can work offline; changes sync when connectivity resumes
Non-Functional Requirements
- Strong Consistency: File metadata must be strongly consistent (read-after-write guarantee)
- Conflict Resolution: Handle concurrent edits gracefully with user notification
- Offline Support: Local queue of changes; sync when back online
- Data Durability: 99.999999999% (11 nines) durability via replication
- High Availability: 99.99% uptime; system must handle node failures gracefully
- Latency: File sync within 1-2 seconds for small files (<10MB)
- Scalability: Support 100M+ users with petabytes of storage
Capacity Estimation
Assumptions:
- 100M users, 50M daily active
- Average 200 files/user, 500KB average file size
- 5% of files change daily: 100M * 200 * 0.05 = 1B file changes/day
- Write QPS: 1B / 86400 ≈ 11,500 QPS
- Storage: 100M * 200 * 500KB = 10PB total
- Read QPS: 3x writes ≈ 35,000 QPS
File Sync Protocol
Block-Level Sync (Delta Sync)
Instead of uploading entire files, split files into fixed-size blocks and only transfer changed blocks.
┌─────────────────────────────────────────────────┐
│ BLOCK-LEVEL SYNC │
├─────────────────────────────────────────────────┤
│ │
│ Original File (12MB) │
│ ┌──────┬──────┬──────┐ │
│ │ Block│ Block│ Block│ 3 x 4MB blocks │
│ │ A │ B │ C │ │
│ └──────┴──────┴──────┘ │
│ │
│ Modified File (12MB) │
│ ┌──────┬──────┬──────┐ │
│ │ Block│ Block│ Block│ │
│ │ A' │ B │ C' │ Only A and C changed │
│ └──────┴──────┴──────┘ │
│ │
│ Transfer: Only blocks A' and C' (8MB vs 12MB) │
└─────────────────────────────────────────────────┘
Sync Protocol Flow
┌──────────┐ ┌──────────┐
│ Client │ │ Server │
│ (Device) │ │ │
└─────┬─────┘ └────┬─────┘
│ │
│ 1. Connect + Send Local DB │
│ (file hashes, versions) │
│──────────────────────────────>│
│ │
│ 2. Server compares with │
│ metadata DB │
│ │
│ 3. Send list of changes │
│ (new/modified/deleted files) │
│<──────────────────────────────│
│ │
│ 4. Client sends changed │
│ blocks for upload │
│──────────────────────────────>│
│ │
│ 5. Server stores blocks │
│ in S3 │
│ │
│ 6. Send confirmation + │
│ updated metadata │
│<──────────────────────────────│
│ │
│ 7. Client receives changes │
│ from other devices │
│<──────────────────────────────│
│ │
│ 8. Client downloads new │
│ blocks │
│<──────────────────────────────│
│ │
Client-Side Metadata Database
Each client maintains a local SQLite database tracking:
CREATE TABLE files (
file_id TEXT PRIMARY KEY,
file_path TEXT NOT NULL,
file_hash TEXT NOT NULL,
block_hashes JSON, -- Array of block hashes
version INTEGER NOT NULL,
last_modified TIMESTAMP,
sync_status TEXT, -- 'synced', 'pending', 'conflict'
device_id TEXT
);
CREATE TABLE blocks (
block_hash TEXT PRIMARY KEY,
file_id TEXT,
block_index INTEGER,
local_path TEXT, -- Path to local block file
uploaded BOOLEAN DEFAULT FALSE
);
Hashing Strategy
- File-level: SHA-256 hash of entire file content (stored in metadata)
- Block-level: SHA-256 hash of each 4MB block (used for deduplication)
- Rolling hash: Content-defined chunking for variable-size blocks (advanced)
- Merkle tree: Hash tree for efficient change detection
Storage Architecture
Two-Tier Storage System
┌─────────────────────────────────────────────────────────────┐
│ CLOUD DRIVE ARCHITECTURE │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ ┌──────────────┐ ┌─────────────────┐ │
│ │ Client │───>│ Upload Service│───>│ Block Storage │ │
│ │ App │ │ (Worker) │ │ (S3) │ │
│ └────┬─────┘ └──────┬───────┘ └─────────────────┘ │
│ │ │ │
│ │ │ ┌─────────────────┐ │
│ │ ├───>│ Metadata Service │ │
│ │ │ │ (API Server) │ │
│ │ │ └────────┬────────┘ │
│ │ │ │ │
│ │ │ ┌────────▼────────┐ │
│ │ │ │ Database │ │
│ │ │ │ (MySQL/PG) │ │
│ │ │ └─────────────────┘ │
│ │ │ │
│ │ │ ┌─────────────────┐ │
│ └─────────────────┼───>│ Notification │ │
│ │ │ Service (WS) │ │
│ │ └─────────────────┘ │
│ │ │
│ │ ┌─────────────────┐ │
│ ├───>│ Deduplication │ │
│ │ │ Service │ │
│ │ └─────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Metadata Service
Stores file/folder hierarchy, versions, permissions:
Metadata Tables:
users
├── user_id (PK)
├── email
├── name
├── storage_quota
└── storage_used
files
├── file_id (PK)
├── parent_folder_id (FK)
├── file_name
├── file_type
├── file_size
├── latest_version_id (FK)
├── created_by (FK users)
└── created_at
versions
├── version_id (PK)
├── file_id (FK files)
├── version_number
├── file_hash (SHA-256)
├── block_count
├── created_by (FK users)
└── created_at
blocks
├── block_id (PK)
├── version_id (FK versions)
├── block_hash (SHA-256)
├── block_index
├── block_size
└── storage_path (S3 key)
permissions
├── permission_id (PK)
├── resource_id (FK)
├── resource_type (file/folder)
├── grantee_id (FK users)
├── permission_level (view/edit/owner)
└── granted_at
Block Storage (S3)
S3 Bucket Structure:
s3://cloud-drive-blocks/
├── blocks/
│ ├── {block_hash_1}.block
│ ├── {block_hash_2}.block
│ └── ...
├── tmp/
│ └── {upload_id}.part # Incomplete multipart uploads
└── metadata/
└── {file_id}/
└── version_{n}.json
Block Storage Properties:
- Storage Class: S3 Standard (frequent access)
- Versioning: Enabled (for safety)
- Lifecycle: Move to Glacier after 90 days of no access
- Encryption: AES-256 at rest, TLS in transit
Deduplication Strategy
Content-Addressable Storage (CAS) - blocks identified by content hash:
import hashlib
def deduplicate_block(block_data: bytes) -> str:
"""Store block only if it doesn't already exist."""
block_hash = hashlib.sha256(block_data).hexdigest()
# Check if block already exists in S3
if not s3.head_object(Bucket='blocks', Key=block_hash):
s3.put_object(
Bucket='blocks',
Key=block_hash,
Body=block_data
)
return block_hash
def calculate_dedup_ratio():
"""Typical dedup ratio: 2-3x for user files."""
# Same file uploaded by multiple users: 1 block stored
# Common files (README, etc.): shared across all users
pass
Durability & Replication
- S3: 11 nines durability (99.999999999%)
- Cross-Region Replication: Replicate to 2nd region
- Versioning: S3 object versioning for safety
- Checksums: SHA-256 verified on upload/download
- Multipart Upload: Large files split into parts, retried on failure
Sharing & Collaboration
Permission Model
┌─────────────────────────────────────────────────────┐
│ PERMISSION HIERARCHY │
├─────────────────────────────────────────────────────┤
│ │
│ Organization (Owner) │
│ │ │
│ ├── Team Folder (Edit) │
│ │ ├── Project A (View) │
│ │ │ ├── doc1.pdf │
│ │ │ └── doc2.docx │
│ │ └── Project B (Edit) │
│ │ └── shared.xlsx │
│ │ │
│ └── Personal Folder (Owner) │
│ ├── private.txt (No access) │
│ └── shared.md (Edit) │
│ │
└─────────────────────────────────────────────────────┘
Permission Levels:
- Owner: Full control, delete, share, change permissions
- Editor: Upload, edit, delete, share (no permission changes)
- Viewer: Download, preview only
- Commenter: View + add comments
Sharing Flow
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Sharer │ │ Permission│ │ Recipient│
│ (Client) │ │ Service │ │ (Client) │
└─────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
│ 1. Share file │ │
│ (email, perm) │ │
│───────────────>│ │
│ │ │
│ 2. Validate │ │
│ permissions │ │
│ │ │
│ 3. Create ACL │ │
│ entry │ │
│ │ │
│ 4. Send invite │ │
│ notification │ │
│─────────────────────────────── >│
│ │ │
│ │ 5. Accept │
│ │<───────────────│
│ │ │
│ │ 6. Grant │
│ │ access │
│ │───────────────>│
│ │ │
Link Sharing
def create_share_link(file_id: str, user_id: str, expires_in_days: int = 7) -> dict:
"""Generate a shareable link with optional expiry."""
# Generate unique token
token = secrets.token_urlsafe(32)
# Store link metadata
db.execute("""
INSERT INTO share_links
(token, file_id, created_by, expires_at)
VALUES (%s, %s, %s, NOW() + INTERVAL '%s days')
""", [token, file_id, user_id, expires_in_days])
return {
"link": f"https://drive.example.com/s/{token}",
"expires_at": datetime.now() + timedelta(days=expires_in_days)
}
Real-Time Sync Updates
WebSocket for live sync notifications:
class SyncWebSocket:
def on_connect(self, user_id: str):
"""Subscribe to user's file changes."""
self.subscribe(f"user:{user_id}:files")
def on_file_changed(self, file_id: str, version: int):
"""Notify all devices of file change."""
devices = self.get_user_devices(user_id)
for device in devices:
self.send(device.websocket, {
"event": "file_changed",
"file_id": file_id,
"new_version": version
})
def on_conflict(self, file_id: str, conflicting_versions: list):
"""Notify user of conflict requiring manual resolution."""
self.send(user_websocket, {
"event": "conflict_detected",
"file_id": file_id,
"versions": conflicting_versions
})
Conflict Resolution Strategies
- Last-Write-Wins (LWW): Simple, but may lose data
- Operational Transform (OT): For text files (Google Docs style)
- Version Vectors: Track causality, detect concurrent edits
- Manual Resolution: Show both versions, let user decide
Practice Problems
0/3solved
Design Cloud Drive (Design Google Drive/Dropbox) System
Design a scalable Cloud Drive (Design Google Drive/Dropbox) system. Cover high-level architecture, data model, and API design.
Solution
// Complete system design:
// - Functional + Non-functional requirements
// - Capacity estimation
// - Data model (SQL/NoSQL choice)
// - API endpoints
// - Component architecture
// - Scaling strategy
// - Monitoring & reliabilityCloud Drive (Design Google Drive/Dropbox) Scaling
How would you scale Cloud Drive (Design Google Drive/Dropbox) to handle 10x the current load? Identify bottlenecks and solutions.
Solution
// Scaling approach:
// 1. Load balancing
// 2. Database sharding/replication
// 3. Cache layer (Redis)
// 4. CDN for static assets
// 5. Async processing (queues)
// 6. Microservices decompositionCloud Drive (Design Google Drive/Dropbox) Failure Modes
Analyze potential failure modes for Cloud Drive (Design Google Drive/Dropbox) and design mitigation strategies.
Solution
// Failure mitigation:
// 1. Redundancy (multi-AZ)
// 2. Circuit breakers
// 3. Retry with backoff
// 4. Dead letter queues
// 5. Health checks
// 6. Graceful degradationQuiz
1. Why use block-level sync instead of full file upload?
2. What is the primary purpose of content-addressable storage (CAS) in cloud drive?
3. Which database is best for storing file metadata and hierarchy?
4. How does the sync protocol handle offline changes?
5. What durability guarantee does S3 provide for block storage?
Flashcards
Question
What is block-level sync (delta sync)?
Click to reveal answer
Answer
Splitting files into fixed-size blocks (typically 4MB) and only transferring blocks that have changed, rather than uploading entire files.
Question
What is content-addressable storage (CAS)?
Click to reveal answer
Answer
Storing data blocks by their content hash (e.g., SHA-256) rather than traditional file paths. Enables automatic deduplication.
Question
Why use MySQL/PostgreSQL for metadata instead of NoSQL?
Click to reveal answer
Answer
File hierarchies involve relational queries (parent-child, permissions) and require strong consistency, which relational databases handle better than NoSQL.
Question
How does the client-side metadata DB work?
Click to reveal answer
Answer
Each device maintains a local SQLite database tracking file hashes, versions, and sync status. On connect, client sends its DB state to server for comparison.
Question
What is the typical deduplication ratio for cloud storage?
Click to reveal answer
Answer
2-3x storage savings through content-addressable storage. Same files uploaded by multiple users share one physical copy.
Revision Notes
Key Takeaways
- 1.Block-level sync (delta sync) is essential for efficient file transfers
- 2.Content-addressable storage enables automatic deduplication
- 3.Two-tier storage: metadata in relational DB, blocks in S3
- 4.Client-side database enables offline work and efficient sync
- 5.WebSocket/SSE for real-time sync notifications across devices
- 6.ACL-based permissions for sharing with fine-grained access control
- 7.Version history with conflict resolution strategies
Interview Tips
- •Start with requirements: clarify sync model (real-time vs periodic), file size limits, offline needs
- •Explain block-level sync with a diagram showing 4MB blocks and only changed blocks transferred
- •Discuss tradeoffs: block size (4MB balances overhead vs granularity), hashing algorithm (SHA-256 for security vs faster hashes)
- •Address durability: S3 provides 11 nines, but discuss replication strategy
- •Explain conflict resolution: LWW for simplicity, OT for collaborative editing
- •Discuss scalability: how to handle 100M+ users, storage partitioning, CDN for hot files
Cheat Sheet
Cloud Drive Cheat Sheet
Architecture Components
- Upload Service: Handles file uploads, splits into blocks
- Metadata Service: File hierarchy, versions, permissions
- Block Storage: S3 for block data (CAS)
- Notification Service: WebSocket/SSE for real-time sync
- Dedup Service: Content-hash-based deduplication
Key Design Decisions
- Block size: 4MB (balances overhead vs. granularity)
- Hashing: SHA-256 for blocks and files
- Database: MySQL/PostgreSQL for metadata (relational)
- Object storage: S3 for blocks (durability, scalability)
- Sync protocol: Client compares local DB with server
Sync Flow
- Client connects, sends local DB state
- Server compares, returns list of changes
- Client uploads changed blocks
- Server stores blocks, updates metadata
- Server broadcasts changes to other devices
Conflict Resolution
- Last-write-wins: Simple, may lose data
- Operational Transform: For text files (Google Docs)
- Version vectors: Track causality
- Manual: User chooses
Durability
- S3: 11 nines (99.999999999%)
- Cross-region replication for safety
- Versioning and checksums