System Design: Networking & Communication
Master modern networking patterns, CDN acceleration, API Gateways, Service Meshes, real-time streaming techniques, and asynchronous event communication across distributed software architectures.
Latency, Throughput, and Bandwidth
Every distributed network operation is bound by three foundational physical and operational constraints: Latency, Throughput, and Bandwidth. Misunderstanding these definitions often leads engineers to attempt fixing performance bottlenecks by scaling hardware resources that do not address the underlying bottleneck.
Core Concepts and Mathematical Definitions
- Latency: The time required for a packet of data to travel from its source to its destination and return an acknowledgment (Round-Trip Time, or RTT). Total latency is the sum of Propagation Delay (distance divided by speed of light in fiber), Transmission Delay (data size divided by channel rate), Queueing Delay (buffer wait times in router hardware), and Processing Delay (CPU packet header inspection).
- Bandwidth: The maximum theoretical data capacity of a transmission medium, expressed in bits per second (e.g. 10 Gbps Ethernet). It represents the width of the network pipe.
- Throughput: The actual rate of successful data delivery over a channel per unit time, expressed in bits per second or requests per second (RPS). Throughput is always less than or equal to Bandwidth due to packet headers, retransmissions, protocol handshakes, and network congestion.
The Fluid Network Analogy
Think of a water delivery pipeline system to visualize the distinction:
- Bandwidth:The diameter of the pipe. A wider pipe allows more water molecules to travel side-by-side simultaneously.
- Latency:The velocity at which water flows through the pipe from end to end. A longer physical pipe increases latency regardless of its diameter.
- Throughput:The volume of usable water that actually exits the pipe tap per minute, accounting for friction, turbulence, and valve restrictions.
Bandwidth-Delay Product and Little's Law
The total amount of data in flight inside a network cable at any instant is governed by the Bandwidth-Delay Product (BDP):
For example, a 1 Gbps connection between New York and London with an RTT of 70 ms has a BDP calculated as:
If the TCP window size is configured smaller than 8.75 MB, the sender must pause transmission while waiting for ACKs, reducing active throughput far below the available 1 Gbps bandwidth limit.
System concurrency is further modeled using Little's Law from queueing theory:
Where L is the average number of concurrent requests in the system, lambda is the arrival rate (throughput in RPS), and W is the average service time (latency). If your backend service processes 5,000 requests per second and each request takes 200 ms (0.2 sec), your application must maintain at least L = 5,000 x 0.2 = 1,000 active concurrent connections or threads to avoid queue buildup.
Latency vs Throughput Architectural Trade-offs Matrix
Trade-off MatrixSystem design trade-offs between request batching, payload compression, HTTP/2 multiplexing, and TCP connection pooling.
Content Delivery Network (CDN)
A Content Delivery Network (CDN) is a globally distributed network of edge servers deployed across hundreds of Points of Presence (POPs). The fundamental mission of a CDN is to shorten the physical distance between the client and the data source, reducing TCP/TLS handshake latency and serving cached static and dynamic assets near the user.
CDN Edge POP Routing vs Origin Miss Flow
CDN FlowEdge servers intercept client connections via Anycast DNS, returning cached assets in milliseconds or fetching missing content from the origin server.
Routing Mechanisms: Anycast IP vs Geo-DNS
CDNs route client traffic to the nearest physical Edge POP using one of two primary architectural methods:
Anycast IP vs Geo-DNS Routing
- Anycast IP Routing:Multiple edge servers around the world advertise the exact same BGP IP address. Internet routers automatically select the shortest network hop path using Border Gateway Protocol.
- Geo-DNS Routing:The authoritative DNS server inspects the client recursive resolver IP address and returns a distinct IP address pointing to the geographically closest POP.
Cache Control Headers and Invalidation Strategies
Caching policies are declared at the origin server using standard HTTP headers:
- public: Allows intermediate CDN proxies and client browser caches to store the response payload.
- max-age=3600: Directs browser client caches to consider the object fresh for 1 hour.
- s-maxage=86400: Shared CDN edge proxy caches hold the object fresh for 24 hours (overriding max-age for proxies).
- stale-while-revalidate=60: Edge proxies serve stale cached responses immediately for an extra 60 seconds while asynchronously fetching fresh content from the origin in the background.
Invalidation Paradigms
When data changes on the origin before TTL expiration, cache invalidation can be triggered via three mechanisms:
- Purge by URL: Explicit purge request sent via API to evict a specific file path from all edge POP caches within 1 to 5 seconds.
- Cache Tags / Surrogate Keys: Origin attaches `Cache-Tag: user-981, product-402` HTTP headers. Purging `product-402` instantly evicts thousands of cached pages matching that tag in a single operation.
- Cache Busting (Hashed Filenames): Assets are published with content hashes in filenames (e.g. `main.a8f912c.js`). HTML files maintain short TTLs, while static JS/CSS files set `max-age=31536000, immutable`, bypassing manual purge calls entirely.
Edge Computing
Edge computing moves application execution logic from centralized cloud data centers directly onto edge CDN POPs situated within single-digit milliseconds of end users. This model transforms CDNs from passive file caches into programmable serverless runtime platforms.
Execution Runtimes: V8 Isolates vs Docker Containers
Centralized cloud serverless functions (like AWS Lambda inside VPCs) use virtual machines or micro-containers (Firecracker), which suffer from cold start delays ranging from 100 ms to several seconds. In contrast, modern edge compute platforms (Cloudflare Workers, Fastly Compute@Edge) leverage Google V8 Isolates or WebAssembly (Wasm) runtimes.
V8 Isolates run thousands of separate Javascript execution contexts within a single shared OS process space. Memory isolation is enforced by the C++ engine rather than OS hardware virtualization, reducing startup times to under 5 milliseconds and memory footprints to a few megabytes per isolate.
Cloudflare Worker Edge Script Example
Edge authentication token verification and dynamic geo-routing script running inside V8 Isolates.
Press Run to execute the code and see output here.
Edge Trade-offs and Limitations
Edge Computing Architectural Constraints
- State Storage Limitations:Edge nodes are ephemeral. Writes to Edge Key-Value (KV) stores rely on asynchronous global propagation, introducing eventual consistency windows up to 60 seconds.
- Database Connection Pools:Edge workers cannot maintain long-lived TCP connection pools to traditional SQL relational databases without HTTP database proxy bridges (such as Prisma Data Proxy or Supabase Hyperbeam).
- CPU Execution Limits:Edge routines are limited to strict CPU limits (typically 10ms to 50ms per request) to preserve multi-tenant process stability.
Proxy vs Reverse Proxy
While both forward proxies and reverse proxies act as intermediaries standing between a client network location and a target server location, their architectural positions and operational motives are exact mirror images.
Forward Proxy vs Reverse Proxy Architecture
Proxy ArchitecturesVisual comparison of client network protection (Forward Proxy) versus origin server protection (Reverse Proxy).
Comparative Architectural Breakdown
- Forward Proxy: Sits in front of client devices within a private corporate network. It intercepts outbound client requests to the internet. Use cases include enforcing egress corporate filtering, hiding internal IP addresses, caching external site responses, and enforcing TLS interception monitoring.
- Reverse Proxy: Sits in front of origin servers within an infrastructure data center. It intercepts inbound external client requests from the public internet. Use cases include hiding backend infrastructure topology, TLS termination, TCP load balancing, request compression, rate limiting, and static file caching.
Layer 4 vs Layer 7 Proxying
Proxies operate at different layers of the Open Systems Interconnection (OSI) network stack:
Layer 4 vs Layer 7 Proxies
- Layer 4 Proxy (Transport Layer):Forwards raw TCP/UDP packets based strictly on IP addresses and TCP port numbers (e.g. HAProxy TCP mode). It cannot read HTTP headers or cookies, offering sub-millisecond routing speeds with minimal CPU overhead.
- Layer 7 Proxy (Application Layer):Parses full HTTP/HTTPS headers, URLs, cookies, and JSON payloads (e.g. NGINX, Envoy). It makes routing decisions based on request paths or headers, enables SSL termination, but consumes significantly higher CPU for packet parsing.
NGINX Layer 7 Reverse Proxy Configuration Example
Reverse proxy request routing, upstream binding, and TLS termination directives.
Press Run to execute the code and see output here.
API Gateway
An API Gateway is a specialized Layer 7 reverse proxy tailored explicitly for managing application programming interfaces (APIs) in microservices architectures. Rather than forcing external client applications to make disparate network calls to dozens of discrete microservices, the API Gateway acts as a single centralized entry gate.
Core Gateway Responsibilities
- Request Routing & Path Mapping: Directs inbound `/api/v1/orders` to the Order Microservice cluster and `/api/v1/billing` to the Payment Microservice cluster.
- Authentication & Authorization: Validates OAuth2 access tokens and JWT cryptographic signatures at the edge, rejecting unauthorized requests before they consume internal microservice CPU cycles.
- Rate Limiting & Throttling: Enforces API quota thresholds per client IP or API key using distributed algorithms like Token Bucket or Sliding Window Log in Redis.
- Protocol Translation: Transports external HTTP/2 REST or JSON calls over public networks and converts them into binary gRPC over HTTP/2 requests inside the private cluster network.
- Response Aggregation (BFF Pattern): Accepts a single client query, makes parallel internal calls to three microservices, aggregates the JSON responses, and returns a single payload to mobile clients to conserve cellular radio battery.
API Gateway Failure Modes
Centralizing traffic through an API Gateway introduces explicit engineering risks:
Gateway Bottlenecks and Single Points of Failure
- Single Point of Failure (SPOF):If the API Gateway cluster crashes or suffers misconfiguration, the entire digital application goes offline. High-availability deployments require horizontal auto-scaling behind active-active cloud load balancers.
- Monolithic Gateway Antipattern:Allowing multiple independent engineering teams to embed domain-specific business logic into gateway plugins creates a shared deployment bottleneck. Gateway code should strictly handle cross-cutting infrastructure policy.
Service Mesh (Istio, Linkerd)
As microservices clusters scale to hundreds of individual applications, managing inter-service communication (east-west traffic) directly within application application code becomes unmanageable. A Service Mesh solves this by inserting a dedicated infrastructure layer that handles service discovery, traffic routing, security encryption, and operational telemetry automatically.
Service Mesh Control Plane vs Data Plane (Sidecar Pattern)
Service MeshApplication containers communicate strictly over local loopback (localhost) to Envoy sidecar proxies, which enforce mTLS encryption and retries.
Architecture: Control Plane vs Data Plane
A service mesh is strictly split into two operational layers:
- Data Plane: Composed of high-performance sidecar proxy containers (such as Envoy or Linkerd-proxy) deployed alongside every application instance inside the same Kubernetes Pod. All network ingress and egress traffic is transparently intercepted via Linux `iptables` rules and routed through the local proxy container.
- Control Plane (Istiod, Linkerd Control Plane): Provides centralized configuration management. It translates high-level traffic routing rules, updates service discovery registries, and issues X.509 cryptographic certificates to data plane sidecars for automated mutual TLS (mTLS) handshakes.
Key Operational Capabilities
- Automated mTLS: Encrypts all internal east-west container traffic with zero application code changes, performing automated certificate rotation every 24 hours.
- Traffic Shifting (Canary & Blue-Green): Directs 95% of production traffic to `v1.2` of a service while routing 5% to `v1.3` based on HTTP headers or exact percentage splits.
- Fault Injection & Chaos Testing: Artificially injects 200 ms latency delays or 5% HTTP 503 error rates into specific microservice routes to test system resilience against cascading failures.
- Distributed Tracing Context Propagation: Sidecars inspect and record W3C Trace Context HTTP headers (`traceparent`, `tracestate`), emitting telemetry data to Jaeger or Zipkin collector backends automatically.
Service Mesh Cost and Overhead Trade-offs
Implementing a service mesh introduces distinct resource penalties:
Long Polling, Short Polling, Server-Sent Events
Standard HTTP request-response cycles are inherently client-initiated. When a backend server needs to deliver updates to a browser client immediately as events occur, traditional HTTP requires architectural modifications.
Architectural Comparison of Real-Time Patterns
1. Short Polling Architecture
Short PollingClient repeatedly sends HTTP GET requests at fixed intervals to query for new updates.
2. Long Polling (Comet) Architecture
Long PollingServer holds the HTTP request connection open until new event data becomes available.
3. Server-Sent Events (SSE) Architecture
Server-Sent EventsPersistent single HTTP connection streaming real-time push event data from server to client.
| Protocol / Pattern | Directionality | Transport Protocol | Network Efficiency | Primary Use Case |
|---|---|---|---|---|
| Short Polling | Unidirectional (Client Pull) | HTTP/1.1 or HTTP/2 | Very Low (High HTTP header waste) | Low-frequency status checks (e.g. batch job progress) |
| Long Polling | Unidirectional (Simulated Push) | HTTP/1.1 or HTTP/2 | Medium (Frequent connection resets) | Legacy chat apps, fallbacks for restricted firewalls |
| Server-Sent Events | Unidirectional (Server Push) | HTTP/1.1 or HTTP/2 (`text/event-stream`) | High (Single persistent HTTP stream) | Stock tickers, sports scores, LLM token streaming |
| WebSockets | Full-Duplex (Bi-directional) | TCP Framing (`ws://` / `wss://`) | Very High (Minimal 2 to 14 byte frame headers) | Multiplayer gaming, collaborative text editors |
Node.js Server-Sent Events (SSE) Implementation
Express.js SSE Endpoint Handler Example
Server-Sent Events HTTP streaming handler setting text/event-stream headers and pushing interval updates.
Press Run to execute the code and see output here.
Pub/Sub Model
The Publish-Subscribe (Pub/Sub) model is an asynchronous messaging pattern that strictly decouples message producers from message consumers. Publishers publish events to named logical channels (Topics) without knowledge of which downstream applications will consume them. Subscribers express interest in topics and process events independently.
Pub/Sub Broker Fan-Out Routing Architecture
Pub/Sub BrokerPublisher dispatches events to a Topic, which clones and fans out messages into independent worker queues.
Core Concepts & Messaging Guarantees
- Topics & Fan-Out Routing: A topic acts as a logical event stream. The broker performs fan-out delivery, cloning an incoming `order.created` event into independent queues for separate consumer worker groups.
- At-Least-Once Delivery: The broker guarantees that every message will be delivered to consumers at least once. If an acknowledgment (ACK) is not returned before a timeout, the broker redelivers the message. Consumers must be implemented idempotently to handle duplicate messages safely.
- At-Most-Once Delivery: Messages are delivered once without waiting for acknowledgments. If a consumer crashes mid-processing, the message is lost forever. Useful for non-critical high-frequency telemetry.
- Exactly-Once Processing: Achieved by combining at-least-once broker delivery with idempotent consumer state persistence using unique message deduplication keys (e.g. `event_id`).
Message Brokers: Kafka vs RabbitMQ
Log-Based (Kafka) vs Smart-Broker (RabbitMQ)
- Apache Kafka (Log-Based Broker):Messages are appended to disk as an immutable partition log. Consumers track their own read position using numeric offsets. Messages can be replayed repeatedly.
- RabbitMQ (Smart Broker):Messages are held in memory/disk queues and actively pushed to consumers. Once an ACK is received, the broker deletes the message from the queue immediately.
Webhooks
Webhooks are user-defined HTTP POST callbacks that allow applications to transmit real-time event notifications to external third-party systems automatically as events occur. Unlike polling APIs where clients query servers repeatedly, webhooks allow provider systems to push notifications directly to client endpoint URLs.
Architecture of a Production-Grade Webhook System
- Event Generation: A user completes a payment transaction on a payment platform.
- Asynchronous Task Queue: The platform writes a `payment_intent.succeeded` job payload into an internal task queue (e.g. Redis Celery / RabbitMQ).
- Worker Dispatch: Outbound webhook worker processes pick up the job, pull the target client HTTPS URL, and compute a cryptographic HMAC signature header.
- HTTP Delivery: The worker transmits an HTTP POST request containing the JSON payload to the subscriber endpoint with a strict 5-second connection timeout.
Security Engineering: Signature Verification
Because webhook receiver endpoints are publicly accessible HTTP URLs on the internet, receivers must verify that incoming payloads were sent by the legitimate provider and were not tampered with in transit by a attacker. This is accomplished using Hash-based Message Authentication Codes (HMAC).
Node.js Express Webhook Signature Verification Handler
Verifies HMAC SHA-256 signatures with timing-safe comparison to prevent spoofing and side-channel attacks.
Press Run to execute the code and see output here.
Resilience Strategies for Webhook Delivery
Retries and Exponential Backoff with Dead-Letter Queues
- Exponential Backoff Retries:If a client server returns HTTP 500 or times out, the webhook sender retries delivery using exponential intervals (e.g. retry after 1min, 5min, 30min, 2hours, 24hours).
- Dead-Letter Queue (DLQ):After 10 failed retry attempts over 72 hours, the delivery job is moved to a DLQ, and an email notification is dispatched to the developer to inspect their failing endpoint.
- Idempotency Keys:Webhook payloads must contain a unique `id` (e.g. `evt_91823719`). Receiver code must record processed event IDs in a cache or database to ignore duplicate retry deliveries safely.
Knowledge Check
Knowledge Check
1. What is the precise relationship between Bandwidth and Throughput in networking?
2. How does an API Gateway differ from a standard Reverse Proxy?
3. What primary architectural advantage does a Service Mesh sidecar pattern provide over application-level HTTP client libraries?
4. Which communication model is optimal for streaming unidirectional server updates to web browsers over standard HTTP without full-duplex socket overhead?
5. How should a receiver authenticate an incoming Webhook HTTP POST payload from a third-party service like Stripe or GitHub?