Video Streaming Platform

Upload, transcode, CDN delivery and adaptive bitrate playback for YouTube-scale video.

AdvancedProduct17 min read

Read these first

Web Crawler Design

Design StreamHub VOD and live streaming: creators upload or go live; viewers watch with adaptive bitrate streaming at YouTube scale — 500M hours watched/month, global CDN delivery, under 2 seconds startup latency.

Requirements

Functional requirements

Upload video files (up to 4K, 12 hours max for VOD).
Live streaming with under 5 seconds end-to-end latency.
Adaptive bitrate playback (360p to 4K) based on client bandwidth.
Thumbnail and preview generation.
View count analytics and creator dashboard.
Resume playback from last watched position.

Non-functional requirements

Scale: 500M hours watched/month; 1M concurrent live streams at peak.
Upload: Support 10 GB files via multipart upload.
Playback latency: Under 2 seconds startup for VOD; under 5 seconds for live.
Availability: 99.99% for playback path.
Storage: 10M new hours uploaded/month × 5 GB/hour (multi-bitrate) = 50 PB/month raw.

Out of scope: Live chat (separate system), content moderation ML, DRM for premium content.

Estimation

500M hours/month × 3600 sec = 1.8T seconds of video/month.
At 2 Mbps average playback: 1.8T × 2 Mbps ≈ 450 EB/month egress — CDN is mandatory.
1M concurrent live streams × 3 Mbps ingest ≈ 3 Tbps ingest at peak — regional ingest clusters.
Transcode: 1 hour video → ~30 min wall time on 1 GPU → 10M hours/month needs ~200 GPU workers sustained.

Video pipeline (AWS)

Presigned S3 upload → MediaConvert → CloudFront HLS delivery.

API design

Java

# Upload (multipart)
POST /v1/videos/upload/init:
  body: { "title": "...", "size_bytes": 5368709120 }
  response: { "upload_id": "up_123", "presigned_urls": [...] }

POST /v1/videos/upload/complete:
  body: { "upload_id": "up_123", "parts": [{ "part_number": 1, "etag": "..." }] }

# Playback
GET /v1/videos/{id}/manifest.m3u8          # HLS master playlist
GET /v1/videos/{id}/segments/{quality}/{segment}.ts

# Live
POST /v1/streams/live/start
POST /v1/streams/live/{id}/ingest          # RTMP/WebRTC ingest endpoint
GET  /v1/streams/live/{id}/manifest.m3u8

Java

CREATE TABLE videos (
    id              BIGINT PRIMARY KEY,
    creator_id      BIGINT NOT NULL,
    title           VARCHAR(256),
    status          VARCHAR(16),     -- uploading, transcoding, ready, failed
    duration_sec    INT,
    s3_raw_key      VARCHAR(512),
    created_at      TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE video_renditions (
    video_id     BIGINT NOT NULL,
    quality      VARCHAR(8) NOT NULL,   -- 360p, 720p, 1080p, 4k
    s3_prefix    VARCHAR(512) NOT NULL,
    bitrate_kbps INT,
    PRIMARY KEY (video_id, quality)
);

High-level architecture

CloudFront edge delivery

Video segments cached at PoPs; S3 origin fetch only on cache miss.

Pipeline stages

Upload service: Issues presigned S3 URLs for multipart upload; tracks upload state.
Transcode fleet: GPU workers pull jobs from queue; FFmpeg produces HLS segments at each bitrate.
Origin storage: S3 stores raw upload + transcoded segments organised by /{video_id}/{quality}/segment_N.ts.
CDN (CloudFront): Caches segments at edge PoPs; 95%+ cache hit ratio for popular content.
Manifest service: Generates master .m3u8 playlist listing available bitrates.
Live ingest: RTMP/WebRTC receiver → real-time transcode → HLS segmenter → CDN (5 s segments).

Deep dive: adaptive bitrate streaming

Java

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
https://cdn.streamhub.com/videos/v_9912/360p/manifest.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1280x720
https://cdn.streamhub.com/videos/v_9912/720p/manifest.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=8000000,RESOLUTION=1920x1080
https://cdn.streamhub.com/videos/v_9912/1080p/manifest.m3u8

Quality	Resolution	Bitrate	Segment size (2s)
360p	640×360	800 Kbps	~200 KB
720p	1280×720	2.8 Mbps	~700 KB
1080p	1920×1080	8 Mbps	~2 MB
4K	3840×2160	20 Mbps	~5 MB

360p
Resolution640×360
Bitrate800 Kbps
Segment size (2s)~200 KB
720p
Resolution1280×720
Bitrate2.8 Mbps
Segment size (2s)~700 KB
1080p
Resolution1920×1080
Bitrate8 Mbps
Segment size (2s)~2 MB
4K
Resolution3840×2160
Bitrate20 Mbps
Segment size (2s)~5 MB

Deep dive: live streaming path

Live vs VOD differences

Ingest: Creator streams via RTMP to nearest ingest server (GeoDNS routed).
Real-time transcode: GPU encodes 2-second HLS segments on the fly — no waiting for full upload.
Low-latency HLS: Chunk size reduced to 1 s; CDN cache TTL near zero for live edge.
DVR window: Keep last 2 hours of live segments in S3 for instant replay.
Viewer sync: All viewers within 5–10 seconds of each other (trade latency for stability).

Quick recall

Everything you need if you only revisit this box.

Upload: presigned multipart S3 → transcode queue → GPU FFmpeg → HLS segments per bitrate.
Playback: CDN serves segments; player uses ABR to switch quality based on bandwidth.
Live: RTMP ingest → real-time transcode → 1–2 s HLS segments → CDN with low TTL.
500M hours/month requires CDN for 95%+ egress; never serve video from origin directly.
Store raw + all renditions in S3; manifest service generates master .m3u8 playlist.
10M hours uploaded/month × 5 GB/hour multi-bitrate ≈ 50 PB/month storage planning.

Test yourself

Answer these before moving on — recall is what makes it stick.