System Design: Caching Concepts and Architecture
Learn how high-performance caching layers reduce system latency from milliseconds to microseconds, protect databases from heavy reads, and scale distributed architectures.
1. What is Caching and Why It Matters
Caching is the technique of storing copies of data in a fast, temporary storage layer (typically RAM) so future requests for that data can be served faster than fetching it from primary storage (like relational databases or disk drives).
Reading data from main system RAM takes roughly 100 nanoseconds, whereas reading data from an SSD drive takes around 100 microseconds, and executing a complex SQL database query across a network can take anywhere from 5 to 50 milliseconds. Caching leverages this speed differential to deliver massive performance gains.
Latency Reduction
Serves requests in sub-millisecond timeframes by bypassing expensive disk read I/O operations and complex database table joins.
Database Offloading
Intercepts up to 90% of repetitive query volume, freeing primary database CPU cores to process critical write transactions.
Cost Efficiency
Allows systems to handle 10x traffic bursts using smaller, cheaper database nodes paired with lightweight in-memory cache clusters.
2. Caching Layers Across the Stack
Caching operates at every tier of modern web application architecture, from client browsers to edge networks, application servers, and database engines.
Multi-Tier Web Application Caching Layers
System DesignTracing request flows through Client, Edge CDN, Application RAM, and Distributed Redis caches
Client-Side & CDN Caching
- Client-Side:Browsers store static assets and API responses using Cache-Control HTTP headers, ETags, and Service Workers. Bypasses network traffic entirely.
- CDN Caching:Content Delivery Networks (Cloudflare, CloudFront) cache static assets and HTML pages at edge points of presence near users worldwide.
Application & Database Caching
- Application Level:In-memory key-value stores (Redis, Memcached) sit next to backend services to cache serialized objects, session data, and query results.
- Database Level:Database engines use internal buffer pools (like MySQL InnoDB buffer pool) to cache index pages and table data in database RAM.
3. Cache Writing Patterns
Cache writing patterns define how data flows between the application, the cache storage, and the primary persistent database during read and write operations.
Cache-Aside vs Write-Through vs Write-Back Workflows
System DesignComparing read/write sequence paths across lazy loading, synchronous updates, and async batching
Cache-Aside (Lazy Loading)
The application inspects the cache first. If a cache miss occurs, the application fetches data from the database, populates the cache for subsequent reads, and returns the result.
- Best for:Read-heavy workloads with non-critical immediate write consistency needs.
Write-Through Cache
The application writes data to the cache layer, and the cache layer synchronously writes to the database before confirming success to the application caller.
- Best for:Applications requiring high data consistency between cache and storage.
Write-Back (Write-Behind) Cache
The application writes to the cache, which acknowledges completion immediately. Background processes flush cached updates in batch to the database asynchronously.
- Best for:Write-heavy applications (like gaming score counters or view loggers). Risk of data loss if cache crashes before flushing.
Write-Around Cache
Data is written directly to the database, bypassing the cache completely. The cache is populated only when a subsequent read miss occurs.
- Best for:Datasets written once and rarely read immediately (such as archived log records).
Examine how the Node.js implementation below executes the Cache-Aside pattern paired with explicit invalidation on data updates.
Node.js & Redis: Cache-Aside Implementation with Invalidation
Handling cache reads, TTL expiration settings, database fallbacks, and explicit deletion on mutations
Press Run to execute the code and see output here.
4. Cache Eviction Policies
Because main memory (RAM) is finite and more expensive than storage disks, caches eventually reach maximum allocated capacity. Cache eviction policies dictate which existing keys are removed to make space for incoming items.
Least Recently Used (LRU)
Evicts the key that has not been accessed for the longest duration of time. Highly popular default policy for general web workloads.
Least Frequently Used (LFU)
Evicts keys with the lowest total access count. Ideal for workloads where certain hot items remain popular across long spans.
First In First Out (FIFO)
Evicts keys in exact order of creation timestamp, regardless of access frequency or recency.
Random Eviction
Evicts keys completely at random. Offers low memory management computational overhead.
5. Cache Invalidation Strategies
Cache invalidation ensures that stale cached data is replaced or purged when underlying database records change. Managing invalidation effectively prevents users from seeing outdated state.
Time-to-Live (TTL) Expiration
Assigning an explicit lifetime (e.g. 300 seconds) to cache keys. Once the timer expires, the cache engine automatically deletes the key, forcing the next read to fetch fresh data from the database.
Explicit Event-Driven Invalidation
When an API endpoint mutates data (e.g. UPDATE or DELETE query), the application explicitly deletes or overwrites the associated cache key immediately.
Versioned Key Namespacing
Appending version hashes to cache keys (e.g. user:102:v3). Incrementing the version counter instantly renders old keys obsolete without requiring mass key scanning.
6. Distributed Caching: Redis vs Memcached
A distributed cache is a dedicated cluster of memory nodes shared across multiple backend application servers. Redis and Memcached are the industry standards for distributed caching.
Redis Architecture & Features
- Rich Data Structures:Supports Strings, Hashes, Lists, Sets, Sorted Sets, and HyperLogLogs.
- Persistence Options:RDB snapshots and AOF logs allow cache recovery after cluster restarts.
- Features:Built-in Pub/Sub, Lua scripting, replication, geospatial indexes, and Sentinel/Cluster modes.
Memcached Architecture & Features
- Pure Key-Value Store:Stores plain strings and serialized binary blobs without complex structure overhead.
- Multi-Threaded Architecture:Excellent performance scaling on large multi-core server hardware.
- Simplicity:Lightweight, simple memory slab allocation model designed purely for high-speed object caching.
7. Cache Stampede (Thundering Herd Problem)
A Cache Stampede occurs when a high-traffic cache key expires or gets invalidated. Hundreds or thousands of concurrent incoming requests experience a cache miss simultaneously and attempt to recompute the same database query at the exact same instant, overwhelming primary database CPU limits.
Distributed Mutex Lock
Using a lock key (Redlock) so only ONE worker recomputes the missing key while other requests wait briefly.
Probabilistic Refresh
Asynchronously recomputing keys in background before TTL expiration based on access frequency algorithms (XFetch).
Stale-While-Revalidate
Serving slightly stale cached data to clients immediately while triggering an asynchronous background refresh job.
Study how the distributed mutex implementation below guards database capacity during high-concurrency cache misses.
Distributed Mutex Implementation to Prevent Cache Stampede
Using Redis atomic NX locks to serialize database re-computation across application nodes
Press Run to execute the code and see output here.
Caching Concepts Knowledge Verification
1. What is the primary purpose of caching?
2. How does the Cache-Aside (Lazy Loading) pattern work?
3. Which eviction policy removes keys not accessed for the longest time?
4. How does Write-Back caching differ from Write-Through?
5. What resolves the Cache Stampede (Thundering Herd) problem?