System Design: Scalability Concepts
Master the core architectural strategies to scale distributed web services from single-server setups to high-concurrency systems handling millions of requests.
1. Vertical Scaling vs Horizontal Scaling
When user traffic grows, application infrastructure must handle higher request rates. Systems scale through two fundamental approaches: adding hardware capacity to an existing machine or adding more machines to a distributed network.
Vertical vs Horizontal Scaling Architecture
System DesignComparing single-server hardware upgrades against multi-node load balanced pools
Vertical Scaling (Scale-Up)
Upgrading a single server with more processing cores, higher capacity RAM, or faster solid-state drives. Think of swapping a delivery sedan for a cargo truck.
- Pros:Zero application architecture code changes required. Simple operations with no distributed networking overhead.
- Cons:Hard physical hardware ceiling. Exponential cost growth at high tiers. Single point of failure if the host crashes.
Horizontal Scaling (Scale-Out)
Distributing incoming workloads across a pool of commodity server instances running behind a load balancer. Think of deploying a fleet of delivery vans.
- Pros:Virtually unlimited scaling capacity. High fault tolerance and continuous availability during node maintenance.
- Cons:Requires stateless application design, distributed caching, load balancers, and complex deployment pipelines.
2. Stateless vs Stateful Services
State refers to data stored by a service across client HTTP requests, such as active user sessions, shopping cart items, or temporary upload buffers. How a service manages state dictates its ability to scale horizontally.
Stateful Services
Stateful nodes store user session data directly in local server memory or local disk drives. Subsequent requests from that user must hit the exact same server instance, requiring sticky session routing on load balancers.
Stateless Services
Stateless nodes process incoming requests using only the data provided in the request payload or retrieved from external shared datastores (such as Redis or PostgreSQL). Any node can process any user request.
Before inspecting the implementation code below, notice how moving session storage from local server RAM into a shared Redis cluster allows application servers to spin up or terminate without interrupting active user sessions.
Node.js Express: Stateful Anti-Pattern vs Stateless Redis Pattern
Comparing fragile local memory session storage against scalable shared Redis caching
Press Run to execute the code and see output here.
3. Scaling Reads vs Scaling Writes
Most web applications exhibit asymmetric traffic profiles. Social media feeds and content platforms often see a 100:1 read-to-write ratio, whereas logging pipelines and IoT telemetry platforms experience heavy write traffic. Reads and writes require different scaling techniques.
Read-Heavy Scaling Strategies
- Read Replicas:Spreading SELECT queries across multiple database read replicas while pointing write queries to a single primary node.
- Caching:Placing fast in-memory stores (Redis, Memcached) in front of databases to serve hot data instantly.
- CDN Offloading:Caching static assets, API JSON payloads, and media files at global edge locations.
Write-Heavy Scaling Strategies
- Asynchronous Message Queues:Buffering incoming write spikes in Kafka or RabbitMQ before background workers process database writes.
- Database Sharding:Partitioning write operations across distinct physical master database nodes.
- Write-Behind Caching:Writing changes to an in-memory cache first, then asynchronously committing batch updates to storage.
4. Auto-Scaling
Auto-scaling dynamically adjusts the number of active compute resources based on real-time system metrics. It ensures high performance during unexpected traffic bursts while cutting cloud costs during off-peak hours.
Metric Triggers
Monitors metrics like CPU utilization, memory thresholds, HTTP request rate per second, or message queue lag length.
Scale-Out Velocity
Spins up new container instances or virtual machines instantly when load crosses configured high-water mark thresholds.
Cool-Down Windows
Prevents rapid node creation and destruction (flapping) by enforcing stabilization wait windows during scale-in events.
The declarative manifest below illustrates how Kubernetes configures an automated scaling policy based on target CPU and memory metrics.
Kubernetes Horizontal Pod Autoscaler (HPA) Configuration
Defining metric thresholds and min/max replica boundaries for automated container scaling
Press Run to execute the code and see output here.
5. Sharding vs Partitioning
While often used interchangeably, partitioning and sharding represent distinct database division concepts. Partitioning is the broad umbrella term for splitting a dataset into subsets. Sharding specifically describes horizontal partitioning across multiple separate database servers.
Architectural Distinction
- Partitioning:Dividing a database table into smaller pieces located on the SAME physical server machine or database engine.
- Sharding:Dividing a dataset across MULTIPLE independent physical database servers, where each server manages its own subset of rows.
6. Horizontal Partitioning vs Vertical Partitioning
Database tables can be sliced in two dimensions: vertically by column attributes or horizontally by data rows.
Horizontal vs Vertical Partitioning Visual Model
System DesignSplitting database structures by column domain versus row ID distributions
Vertical Partitioning (Column Split)
Splitting a single wide table into separate tables containing subsets of columns. Frequently used to separate lightweight, high-frequency fields (like passwords and emails) from heavy, infrequently accessed fields (like profile bios or image blobs).
Horizontal Partitioning / Sharding (Row Split)
Splitting table rows across multiple shards using a shard key router (such as user ID modulo shard count). Each shard keeps the exact same table column schema but holds a distinct subset of total rows.
7. Hotspot Problem and Mitigation
A hotspot (also known as the celebrity key problem or skewed access pattern) occurs when traffic heavily concentrates on a single shard or database key. For instance, when a global celebrity publishes a post, millions of reads and comments target one single shard while neighboring shards remain idle.
Key Salting
Appending random integer suffixes (e.g. key_0 through key_9) to distribute writes for viral items across 10 sub-shards.
Local Read Caching
Caching hot items in application server memory directly to intercept 99% of read traffic before reaching database shards.
Write Buffer Aggregation
Queueing viral item updates in Redis and flushing aggregated counts to the database once every few seconds.
Review how the shard router below calculates standard hash targets versus salted targets for viral entity keys.
Shard Key Router & Hotspot Mitigation Algorithm
Implementing consistent hash routing with salted key distribution for viral entity hotspots
Press Run to execute the code and see output here.
8. Fan-out Systems
A fan-out system describes how messages or events propagate from a single source to multiple destinations. This pattern is central to social media feeds, notification systems, and message pub-sub brokers.
Fan-Out Architecture: Push Model vs Pull Model
System DesignComparing pre-computed write fan-out against on-demand read fan-out algorithms
Push Model (Fan-Out on Write)
When a user publishes a post, background workers deliver the post directly into every follower's home feed cache.
- Read Performance:O(1) instant feed fetch for readers.
- Write Drawback:Massive write amplification when a user with millions of followers posts.
Pull Model (Fan-Out on Read)
Posts are stored only in the creator's post table. When a follower requests their timeline, the system fetches and merges posts from all followed creators.
- Write Performance:O(1) instant write time for creators.
- Read Drawback:Heavy join and sort computation overhead when rendering timelines.
Production Hybrid Fan-Out Strategy
Leading platforms combine both models. Standard users with under 10,000 followers use Fan-Out on Write for fast timeline delivery. Celebrity accounts with millions of followers switch to Fan-Out on Read, merging celebrity posts into timelines only when active followers open the application.
Scalability Concepts Knowledge Verification
1. What defines vertical scaling (scale-up)?
2. How do stateless application services handle scale?
3. Which technique effectively scales read-heavy workloads?
4. What is horizontal database partitioning (sharding)?
5. What is the main benefit of fan-out on write (push model)?