System Design: Data Storage at Scale
Master blob storage, distributed file systems, storage paradigm comparisons, Data Lakes vs Warehouses, temperature tiering, and media storage pipelines.
1. Object Storage vs Block Storage vs File Storage
Modern cloud infrastructure provides three distinct storage paradigms tailored for different read/write access patterns and file structure requirements.
Object Storage vs Block Storage vs File Storage Architecture Topologies
System DesignComparing flat key-value object namespaces, low-latency VM blocks, and shared POSIX file directories
100%
Loading system design canvas…
[
{
"id": "st-obj-box",
"type": "rectangle",
"x": 30,
"y": 40,
"width": 260,
"height": 280,
"strokeColor": "#2563eb",
"backgroundColor": "#eff6ff",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13000,
"version": 1,
"versionNonce": 13001,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "st-obj-box-text"
}
]
},
{
"id": "st-obj-box-text",
"type": "text",
"x": 40,
"y": 135,
"width": 240,
"height": 90,
"strokeColor": "#1e40af",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13002,
"version": 1,
"versionNonce": 13003,
"isDeleted": false,
"text": "Object Storage (AWS S3)\n\nFlat Namespace\nKey-Value Access\nRich Metadata Tags\nScale: Petabytes+",
"fontSize": 12,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Object Storage (AWS S3)\n\nFlat Namespace\nKey-Value Access\nRich Metadata Tags\nScale: Petabytes+",
"containerId": "st-obj-box"
},
{
"id": "st-blk-box",
"type": "rectangle",
"x": 310,
"y": 40,
"width": 260,
"height": 280,
"strokeColor": "#059669",
"backgroundColor": "#ecfdf5",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13004,
"version": 1,
"versionNonce": 13005,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "st-blk-box-text"
}
]
},
{
"id": "st-blk-box-text",
"type": "text",
"x": 320,
"y": 135,
"width": 240,
"height": 90,
"strokeColor": "#065f46",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13006,
"version": 1,
"versionNonce": 13007,
"isDeleted": false,
"text": "Block Storage (AWS EBS)\n\nRaw Unformatted Blocks\nAttached to Single VM\nUltra-low Latency (NVMe)\nIdeal for DB Disks",
"fontSize": 12,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Block Storage (AWS EBS)\n\nRaw Unformatted Blocks\nAttached to Single VM\nUltra-low Latency (NVMe)\nIdeal for DB Disks",
"containerId": "st-blk-box"
},
{
"id": "st-file-box",
"type": "rectangle",
"x": 590,
"y": 40,
"width": 270,
"height": 280,
"strokeColor": "#8b4dde",
"backgroundColor": "#f3e8ff",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13008,
"version": 1,
"versionNonce": 13009,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "st-file-box-text"
}
]
},
{
"id": "st-file-box-text",
"type": "text",
"x": 600,
"y": 135,
"width": 250,
"height": 90,
"strokeColor": "#6b21a8",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13010,
"version": 1,
"versionNonce": 13011,
"isDeleted": false,
"text": "File Storage (AWS EFS)\n\nHierarchical Directories\nPOSIX File Permissions\nShared NFS / SMB Protocols\nShared across VM Nodes",
"fontSize": 12,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "File Storage (AWS EFS)\n\nHierarchical Directories\nPOSIX File Permissions\nShared NFS / SMB Protocols\nShared across VM Nodes",
"containerId": "st-file-box"
}
]Object Storage (AWS S3, Google Cloud Storage)
Stores data as immutable objects in a flat namespace. Each object contains payload data, a unique key ID, and custom metadata tags. Accessed over HTTP REST APIs.
Block Storage (AWS EBS, SAN)
Exposes raw unformatted block volumes formatted with traditional file systems (ext4, NTFS) and attached directly to Virtual Machines over high-speed NVMe buses. Ideal for databases.
File Storage (AWS EFS, Network Attached Storage NFS/SMB)
Provides shared hierarchical directory structures with POSIX file locking permissions, accessible concurrently by hundreds of application server instances.
2. Blob Storage & Distributed File Systems (HDFS, GFS)
Distributed file systems (HDFS, GFS) split massive petabyte files into fixed-size block chunks (e.g. 128 MB) and distribute them across a cluster of commodity hardware servers with automatic 3x replication factor.
Distributed File System (HDFS NameNode & DataNode Replication)
System DesignTracing NameNode metadata block mappings and DataNode 3x block replication for fault tolerance
100%
Loading system design canvas…
[
{
"id": "hdfs-box",
"type": "rectangle",
"x": 30,
"y": 40,
"width": 830,
"height": 280,
"strokeColor": "#10b981",
"backgroundColor": "#ecfdf5",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13012,
"version": 1,
"versionNonce": 13013,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "hdfs-box-text"
}
]
},
{
"id": "hdfs-box-text",
"type": "text",
"x": 40,
"y": 166,
"width": 810,
"height": 28,
"strokeColor": "#047857",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13014,
"version": 1,
"versionNonce": 13015,
"isDeleted": false,
"text": "Distributed File System (HDFS / GFS Architecture)",
"fontSize": 14,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Distributed File System (HDFS / GFS Architecture)",
"containerId": "hdfs-box"
},
{
"id": "hdfs-master",
"type": "rectangle",
"x": 340,
"y": 85,
"width": 210,
"height": 65,
"strokeColor": "#8b4dde",
"backgroundColor": "#f3e8ff",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13016,
"version": 1,
"versionNonce": 13017,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "hdfs-master-text"
}
]
},
{
"id": "hdfs-master-text",
"type": "text",
"x": 350,
"y": 96.875,
"width": 190,
"height": 41.25,
"strokeColor": "#6b21a8",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13018,
"version": 1,
"versionNonce": 13019,
"isDeleted": false,
"text": "NameNode (Master)\nMetadata Registry & Block Maps\n(File.txt -> [Block1, Block2])",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "NameNode (Master)\nMetadata Registry & Block Maps\n(File.txt -> [Block1, Block2])",
"containerId": "hdfs-master"
},
{
"id": "h-arr1",
"type": "arrow",
"x": 340,
"y": 115,
"width": -170,
"height": 75,
"strokeColor": "#8b4dde",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "dashed",
"roughness": 0,
"opacity": 100,
"points": [
[
0,
0
],
[
-170,
75
]
],
"endArrowhead": "arrow",
"seed": 13020,
"version": 1,
"versionNonce": 13021,
"isDeleted": false
},
{
"id": "h-arr2",
"type": "arrow",
"x": 445,
"y": 150,
"width": 0,
"height": 40,
"strokeColor": "#8b4dde",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "dashed",
"roughness": 0,
"opacity": 100,
"points": [
[
0,
0
],
[
0,
40
]
],
"endArrowhead": "arrow",
"seed": 13022,
"version": 1,
"versionNonce": 13023,
"isDeleted": false
},
{
"id": "h-arr3",
"type": "arrow",
"x": 550,
"y": 115,
"width": 170,
"height": 75,
"strokeColor": "#8b4dde",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "dashed",
"roughness": 0,
"opacity": 100,
"points": [
[
0,
0
],
[
170,
75
]
],
"endArrowhead": "arrow",
"seed": 13024,
"version": 1,
"versionNonce": 13025,
"isDeleted": false
},
{
"id": "hdfs-dn1",
"type": "rectangle",
"x": 60,
"y": 190,
"width": 220,
"height": 95,
"strokeColor": "#10b981",
"backgroundColor": "#d1fae5",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13026,
"version": 1,
"versionNonce": 13027,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "hdfs-dn1-text"
}
]
},
{
"id": "hdfs-dn1-text",
"type": "text",
"x": 70,
"y": 216.875,
"width": 200,
"height": 41.25,
"strokeColor": "#047857",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13028,
"version": 1,
"versionNonce": 13029,
"isDeleted": false,
"text": "DataNode 1 (Rack 1)\nStores Block 1 (Replica A)\nStores Block 2 (Replica B)",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "DataNode 1 (Rack 1)\nStores Block 1 (Replica A)\nStores Block 2 (Replica B)",
"containerId": "hdfs-dn1"
},
{
"id": "hdfs-dn2",
"type": "rectangle",
"x": 335,
"y": 190,
"width": 220,
"height": 95,
"strokeColor": "#10b981",
"backgroundColor": "#d1fae5",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13030,
"version": 1,
"versionNonce": 13031,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "hdfs-dn2-text"
}
]
},
{
"id": "hdfs-dn2-text",
"type": "text",
"x": 345,
"y": 216.875,
"width": 200,
"height": 41.25,
"strokeColor": "#047857",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13032,
"version": 1,
"versionNonce": 13033,
"isDeleted": false,
"text": "DataNode 2 (Rack 1)\nStores Block 1 (Replica B)\nStores Block 2 (Replica A)",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "DataNode 2 (Rack 1)\nStores Block 1 (Replica B)\nStores Block 2 (Replica A)",
"containerId": "hdfs-dn2"
},
{
"id": "hdfs-dn3",
"type": "rectangle",
"x": 610,
"y": 190,
"width": 220,
"height": 95,
"strokeColor": "#10b981",
"backgroundColor": "#d1fae5",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13034,
"version": 1,
"versionNonce": 13035,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "hdfs-dn3-text"
}
]
},
{
"id": "hdfs-dn3-text",
"type": "text",
"x": 620,
"y": 216.875,
"width": 200,
"height": 41.25,
"strokeColor": "#047857",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13036,
"version": 1,
"versionNonce": 13037,
"isDeleted": false,
"text": "DataNode 3 (Rack 2)\nStores Block 1 (Replica C)\nFault Tolerant 3x Sync",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "DataNode 3 (Rack 2)\nStores Block 1 (Replica C)\nFault Tolerant 3x Sync",
"containerId": "hdfs-dn3"
}
]NameNode (Master Node)
Maintains in-memory metadata mappings of file paths to block IDs and tracks which physical DataNode servers host each block replica.
DataNodes (Worker Nodes)
Stores raw block chunks on local disk drives and streams data blocks directly to client applications without passing data payloads through the NameNode.
Review how the HDFS Chunk Simulator below calculates block locations and 3x replica node placement.
HDFS Distributed Chunk Routing & Block Replication Simulator
Splitting file streams into fixed-size block chunks and distributing 3x replicas across cluster DataNodes
Press Run to execute the code and see output here.
3. Data Lakes, Warehouses & Temperature Storage Tiering
Enterprise data architectures balance analytics query performance against long-term storage costs using tiered storage lifecycles.
Data Lakes vs Data Warehouses & Temperature Storage Tiering
System DesignComparing schema-on-read Data Lakes against schema-on-write Warehouses and Hot/Warm/Cold storage costs
100%
Loading system design canvas…
[
{
"id": "dl-box",
"type": "rectangle",
"x": 30,
"y": 40,
"width": 410,
"height": 280,
"strokeColor": "#2563eb",
"backgroundColor": "#eff6ff",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13038,
"version": 1,
"versionNonce": 13039,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "dl-box-text"
}
]
},
{
"id": "dl-box-text",
"type": "text",
"x": 40,
"y": 166,
"width": 390,
"height": 28,
"strokeColor": "#1e40af",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13040,
"version": 1,
"versionNonce": 13041,
"isDeleted": false,
"text": "Data Lake vs Data Warehouse",
"fontSize": 14,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Data Lake vs Data Warehouse",
"containerId": "dl-box"
},
{
"id": "dl-lake",
"type": "rectangle",
"x": 50,
"y": 90,
"width": 170,
"height": 100,
"strokeColor": "#3b82f6",
"backgroundColor": "#dbeafe",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13042,
"version": 1,
"versionNonce": 13043,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "dl-lake-text"
}
]
},
{
"id": "dl-lake-text",
"type": "text",
"x": 60,
"y": 112.5,
"width": 150,
"height": 55,
"strokeColor": "#1e40af",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13044,
"version": 1,
"versionNonce": 13045,
"isDeleted": false,
"text": "Data Lake (S3/Hadoop)\nSchema-on-Read\nRaw JSON, Logs, Images\nMachine Learning ML",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Data Lake (S3/Hadoop)\nSchema-on-Read\nRaw JSON, Logs, Images\nMachine Learning ML",
"containerId": "dl-lake"
},
{
"id": "dl-wh",
"type": "rectangle",
"x": 250,
"y": 90,
"width": 170,
"height": 100,
"strokeColor": "#059669",
"backgroundColor": "#d1fae5",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13046,
"version": 1,
"versionNonce": 13047,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "dl-wh-text"
}
]
},
{
"id": "dl-wh-text",
"type": "text",
"x": 260,
"y": 112.5,
"width": 150,
"height": 55,
"strokeColor": "#065f46",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13048,
"version": 1,
"versionNonce": 13049,
"isDeleted": false,
"text": "Data Warehouse\nSchema-on-Write\nStructured Tables\nSQL BI Analytics",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Data Warehouse\nSchema-on-Write\nStructured Tables\nSQL BI Analytics",
"containerId": "dl-wh"
},
{
"id": "dl-desc",
"type": "rectangle",
"x": 50,
"y": 205,
"width": 370,
"height": 100,
"strokeColor": "#2563eb",
"backgroundColor": "#eff6ff",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13050,
"version": 1,
"versionNonce": 13051,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "dl-desc-text"
}
]
},
{
"id": "dl-desc-text",
"type": "text",
"x": 60,
"y": 241,
"width": 350,
"height": 28,
"strokeColor": "#1e40af",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13052,
"version": 1,
"versionNonce": 13053,
"isDeleted": false,
"text": "Lakehouse Pattern:\nCombines cheap object storage of Data Lakes with ACID transactions and SQL engines of Warehouses.",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Lakehouse Pattern:\nCombines cheap object storage of Data Lakes with ACID transactions and SQL engines of Warehouses.",
"containerId": "dl-desc"
},
{
"id": "tier-box",
"type": "rectangle",
"x": 460,
"y": 40,
"width": 410,
"height": 280,
"strokeColor": "#d97706",
"backgroundColor": "#fffbe6",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13054,
"version": 1,
"versionNonce": 13055,
"isDeleted": false
},
{
"id": "tier-box-text",
"type": "text",
"x": 490,
"y": 176,
"width": 350,
"height": 28,
"strokeColor": "#92400e",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13056,
"version": 1,
"versionNonce": 13057,
"isDeleted": false,
"text": "Storage Temperature Lifecycle Tiering",
"fontSize": 14,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Storage Temperature Lifecycle Tiering",
"containerId": "t-warm"
},
{
"id": "t-hot",
"type": "rectangle",
"x": 480,
"y": 90,
"width": 370,
"height": 50,
"strokeColor": "#ef4444",
"backgroundColor": "#fee2e2",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13058,
"version": 1,
"versionNonce": 13059,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "t-hot-text"
}
]
},
{
"id": "t-hot-text",
"type": "text",
"x": 490,
"y": 101,
"width": 350,
"height": 28,
"strokeColor": "#991b1b",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13060,
"version": 1,
"versionNonce": 13061,
"isDeleted": false,
"text": "Hot Tier (S3 Standard): Millisecond Access | Highest $/GB",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Hot Tier (S3 Standard): Millisecond Access | Highest $/GB",
"containerId": "t-hot"
},
{
"id": "t-a1",
"type": "arrow",
"x": 665,
"y": 140,
"width": 0,
"height": 25,
"strokeColor": "#d97706",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"points": [
[
0,
0
],
[
0,
25
]
],
"endArrowhead": "arrow",
"seed": 13062,
"version": 1,
"versionNonce": 13063,
"isDeleted": false
},
{
"id": "t-warm",
"type": "rectangle",
"x": 480,
"y": 165,
"width": 370,
"height": 50,
"strokeColor": "#f59e0b",
"backgroundColor": "#fef3c7",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13064,
"version": 1,
"versionNonce": 13065,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "tier-box-text"
},
{
"type": "text",
"id": "t-warm-text"
}
]
},
{
"id": "t-warm-text",
"type": "text",
"x": 490,
"y": 176,
"width": 350,
"height": 28,
"strokeColor": "#b45309",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13066,
"version": 1,
"versionNonce": 13067,
"isDeleted": false,
"text": "Warm Tier (S3 IA): 30-day Access | Lower $/GB",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Warm Tier (S3 IA): 30-day Access | Lower $/GB",
"containerId": "t-warm"
},
{
"id": "t-a2",
"type": "arrow",
"x": 665,
"y": 215,
"width": 0,
"height": 25,
"strokeColor": "#d97706",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"points": [
[
0,
0
],
[
0,
25
]
],
"endArrowhead": "arrow",
"seed": 13068,
"version": 1,
"versionNonce": 13069,
"isDeleted": false
},
{
"id": "t-cold",
"type": "rectangle",
"x": 480,
"y": 240,
"width": 370,
"height": 60,
"strokeColor": "#3b82f6",
"backgroundColor": "#dbeafe",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13070,
"version": 1,
"versionNonce": 13071,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "t-cold-text"
}
]
},
{
"id": "t-cold-text",
"type": "text",
"x": 490,
"y": 256,
"width": 350,
"height": 28,
"strokeColor": "#1e40af",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13072,
"version": 1,
"versionNonce": 13073,
"isDeleted": false,
"text": "Cold Tier (Glacier Deep Archive): 12hr Retrieval | Lowest $/GB",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Cold Tier (Glacier Deep Archive): 12hr Retrieval | Lowest $/GB",
"containerId": "t-cold"
}
]Data Warehouses (Snowflake, BigQuery, Redshift)
Optimized for structured SQL analytics queries. Uses a schema-on-write approach where data is transformed (ETL) before ingestion.
Data Lakes (AWS S3 + Glue, Apache Iceberg)
Stores raw structured, semi-structured (JSON, Parquet), and unstructured (images, audio) data in native format using a flexible schema-on-read approach.
Hot vs Warm vs Cold Storage Lifecycle
Hot Tier (Standard S3): Instant millisecond access for active data. Warm Tier (S3 IA): Lower storage cost for data accessed once per month. Cold Tier (Glacier): Ultra-low cost for long-term archives with 1-12 hour retrieval times.
4. Content Storage and Retrieval for Media Systems
High-throughput media platforms (YouTube, Netflix) decouple file upload ingestion from video streaming delivery. Clients request temporary presigned URLs from API servers, enabling direct browser-to-S3 uploads that bypass application servers entirely.
Media Storage, Transcoding Pipeline & CDN Delivery Architecture
System DesignTracing presigned URL uploads, asynchronous HLS transcoding, and global CDN edge distribution
100%
Loading system design canvas…
[
{
"id": "m-client",
"type": "rectangle",
"x": 30,
"y": 120,
"width": 120,
"height": 60,
"strokeColor": "#334155",
"backgroundColor": "#f8fafc",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13074,
"version": 1,
"versionNonce": 13075,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "m-client-text"
}
]
},
{
"id": "m-client-text",
"type": "text",
"x": 40,
"y": 136,
"width": 100,
"height": 28,
"strokeColor": "#0f172a",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13076,
"version": 1,
"versionNonce": 13077,
"isDeleted": false,
"text": "Client Mobile App",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Client Mobile App",
"containerId": "m-client"
},
{
"id": "m-a1",
"type": "arrow",
"x": 150,
"y": 135,
"width": 60,
"height": -50,
"strokeColor": "#2563eb",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"points": [
[
0,
0
],
[
60,
-50
]
],
"endArrowhead": "arrow",
"seed": 13078,
"version": 1,
"versionNonce": 13079,
"isDeleted": false
},
{
"id": "m-api",
"type": "rectangle",
"x": 210,
"y": 60,
"width": 150,
"height": 50,
"strokeColor": "#2563eb",
"backgroundColor": "#eff6ff",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13080,
"version": 1,
"versionNonce": 13081,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "m-api-text"
}
]
},
{
"id": "m-api-text",
"type": "text",
"x": 220,
"y": 71,
"width": 130,
"height": 28,
"strokeColor": "#1e40af",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13082,
"version": 1,
"versionNonce": 13083,
"isDeleted": false,
"text": "API Gateway\n(Generates Presigned URL)",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "API Gateway\n(Generates Presigned URL)",
"containerId": "m-api"
},
{
"id": "m-a2",
"type": "arrow",
"x": 150,
"y": 160,
"width": 240,
"height": 0,
"strokeColor": "#10b981",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"points": [
[
0,
0
],
[
240,
0
]
],
"endArrowhead": "arrow",
"seed": 13084,
"version": 1,
"versionNonce": 13085,
"isDeleted": false
},
{
"id": "m-s3-raw",
"type": "rectangle",
"x": 390,
"y": 130,
"width": 140,
"height": 60,
"strokeColor": "#10b981",
"backgroundColor": "#d1fae5",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13086,
"version": 1,
"versionNonce": 13087,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "m-s3-raw-text"
}
]
},
{
"id": "m-s3-raw-text",
"type": "text",
"x": 400,
"y": 146,
"width": 120,
"height": 28,
"strokeColor": "#047857",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13088,
"version": 1,
"versionNonce": 13089,
"isDeleted": false,
"text": "Raw S3 Bucket\n(Raw Video MP4)",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Raw S3 Bucket\n(Raw Video MP4)",
"containerId": "m-s3-raw"
},
{
"id": "m-a3",
"type": "arrow",
"x": 530,
"y": 160,
"width": 60,
"height": 0,
"strokeColor": "#8b4dde",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"points": [
[
0,
0
],
[
60,
0
]
],
"endArrowhead": "arrow",
"seed": 13090,
"version": 1,
"versionNonce": 13091,
"isDeleted": false
},
{
"id": "m-transcoder",
"type": "rectangle",
"x": 590,
"y": 130,
"width": 140,
"height": 60,
"strokeColor": "#8b4dde",
"backgroundColor": "#f3e8ff",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13092,
"version": 1,
"versionNonce": 13093,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "m-transcoder-text"
}
]
},
{
"id": "m-transcoder-text",
"type": "text",
"x": 600,
"y": 146,
"width": 120,
"height": 28,
"strokeColor": "#6b21a8",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13094,
"version": 1,
"versionNonce": 13095,
"isDeleted": false,
"text": "Media Transcoder\n(HLS 1080p, 720p, 480p)",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "Media Transcoder\n(HLS 1080p, 720p, 480p)",
"containerId": "m-transcoder"
},
{
"id": "m-a4",
"type": "arrow",
"x": 730,
"y": 160,
"width": 60,
"height": 0,
"strokeColor": "#d97706",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"points": [
[
0,
0
],
[
60,
0
]
],
"endArrowhead": "arrow",
"seed": 13096,
"version": 1,
"versionNonce": 13097,
"isDeleted": false
},
{
"id": "m-cdn",
"type": "rectangle",
"x": 790,
"y": 130,
"width": 90,
"height": 60,
"strokeColor": "#f59e0b",
"backgroundColor": "#fef3c7",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"roundness": {
"type": 3
},
"seed": 13098,
"version": 1,
"versionNonce": 13099,
"isDeleted": false,
"boundElements": [
{
"type": "text",
"id": "m-cdn-text"
}
]
},
{
"id": "m-cdn-text",
"type": "text",
"x": 800,
"y": 146,
"width": 70,
"height": 28,
"strokeColor": "#b45309",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"seed": 13100,
"version": 1,
"versionNonce": 13101,
"isDeleted": false,
"text": "CDN Edge\n(Global Stream)",
"fontSize": 11,
"fontFamily": 5,
"textAlign": "center",
"verticalAlign": "middle",
"originalText": "CDN Edge\n(Global Stream)",
"containerId": "m-cdn"
}
]Review the Node.js S3 Presigned URL generator and Multipart Upload manager below to see direct cloud storage uploads in practice.
S3 Presigned URL Upload & Multipart Chunked Upload Manager
Generating secure presigned S3 URLs for direct client uploads and managing multipart chunk completions
Press Run to execute the code and see output here.
Data Storage at Scale Knowledge Verification
1. What defines Object Storage like AWS S3 or Google Cloud Storage?
2. How does HDFS achieve fault tolerance across distributed clusters?
3. What characterizes a Data Lake compared to a Data Warehouse?
4. What is the main trade-off of Cold Storage (Glacier)?
5. Why do media systems generate presigned URLs for object uploads?