System Design: Monitoring, Logging, and Observability

Master the Three Pillars of Observability (Metrics, Logs, Traces), centralized ELK logging, distributed tracing with OpenTelemetry, Prometheus & Grafana, SRE alerting, health probes, and APM.

1. Metrics, Logs, and Traces (The Three Pillars)

While traditional monitoring informs operators when a system is failing (black-box symptom detection), modern observability allows engineers to infer the internal state of complex distributed microservices based on external outputs (white-box debugging).

The Three Pillars of Observability Architecture Comparison

System Design

Contrasting numerical time-series metrics, structured event logs, and distributed request traces

100%

Loading system design canvas…

[
  {
    "id": "p-metrics",
    "type": "rectangle",
    "x": 30,
    "y": 40,
    "width": 260,
    "height": 260,
    "strokeColor": "#2563eb",
    "backgroundColor": "#eff6ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17000,
    "version": 1,
    "versionNonce": 17001,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "p-metrics-text"
      }
    ]
  },
  {
    "id": "p-metrics-text",
    "type": "text",
    "x": 40,
    "y": 94.375,
    "width": 240,
    "height": 151.25,
    "strokeColor": "#1e40af",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17002,
    "version": 1,
    "versionNonce": 17003,
    "isDeleted": false,
    "text": "1. Metrics (Aggregable)\n\nNumerical time-series values\nKey-value dimension tags\n\n• Counters (req_count)\n• Gauges (cpu_memory)\n• Histograms (p99 latency)\n\nPros: Low storage cost\nUse for: Real-time Alerts",
    "fontSize": 11,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "1. Metrics (Aggregable)\n\nNumerical time-series values\nKey-value dimension tags\n\n• Counters (req_count)\n• Gauges (cpu_memory)\n• Histograms (p99 latency)\n\nPros: Low storage cost\nUse for: Real-time Alerts",
    "containerId": "p-metrics"
  },
  {
    "id": "p-logs",
    "type": "rectangle",
    "x": 315,
    "y": 40,
    "width": 260,
    "height": 260,
    "strokeColor": "#059669",
    "backgroundColor": "#ecfdf5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17004,
    "version": 1,
    "versionNonce": 17005,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "p-logs-text"
      }
    ]
  },
  {
    "id": "p-logs-text",
    "type": "text",
    "x": 325,
    "y": 94.375,
    "width": 240,
    "height": 151.25,
    "strokeColor": "#065f46",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17006,
    "version": 1,
    "versionNonce": 17007,
    "isDeleted": false,
    "text": "2. Logs (Contextual)\n\nStructured JSON events\nTimestamped payload context\n\n• User IDs / IP Addresses\n• Error stack traces\n• Audit compliance logs\n\nPros: Exact detail\nUse for: Root Cause Debug",
    "fontSize": 11,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "2. Logs (Contextual)\n\nStructured JSON events\nTimestamped payload context\n\n• User IDs / IP Addresses\n• Error stack traces\n• Audit compliance logs\n\nPros: Exact detail\nUse for: Root Cause Debug",
    "containerId": "p-logs"
  },
  {
    "id": "p-traces",
    "type": "rectangle",
    "x": 600,
    "y": 40,
    "width": 260,
    "height": 260,
    "strokeColor": "#8b5cf6",
    "backgroundColor": "#f3e8ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17008,
    "version": 1,
    "versionNonce": 17009,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "p-traces-text"
      }
    ]
  },
  {
    "id": "p-traces-text",
    "type": "text",
    "x": 610,
    "y": 94.375,
    "width": 240,
    "height": 151.25,
    "strokeColor": "#5b21b6",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17010,
    "version": 1,
    "versionNonce": 17011,
    "isDeleted": false,
    "text": "3. Traces (Distributed)\n\nEnd-to-end request journeys\nContext propagation across RPCs\n\n• TraceID (Global request)\n• SpanID (Unit of work)\n• Latency bottlenecks\n\nPros: Microservice map\nUse for: Distributed profiling",
    "fontSize": 11,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "3. Traces (Distributed)\n\nEnd-to-end request journeys\nContext propagation across RPCs\n\n• TraceID (Global request)\n• SpanID (Unit of work)\n• Latency bottlenecks\n\nPros: Microservice map\nUse for: Distributed profiling",
    "containerId": "p-traces"
  }
]

`1. Metrics (Time-Series)`

Aggregable numerical data points measured over time intervals. Low storage overhead.

Counters: Monotonically increasing values (total requests)
Gauges: Fluctuating values (CPU %, memory usage)
Histograms: Distribution buckets for calculating p95/p99 latency

`2. Logs (Structured Events)`

Discrete timestamped JSON records emitted during execution. Provides rich context.

Contextual metadata: user_id, order_id, IP address
Error stack traces & debug event trails
High storage cost; requires index tiering & retention

`3. Traces (Request Journeys)`

Tracks the lifecycle of a request as it traverses across multiple microservices.

TraceID: Unique ID representing the end-to-end request
SpanID: Segment representing a specific unit of work (RPC, SQL query)
Context Propagation: Passed via HTTP/gRPC headers

2. Centralized Logging Architecture (ELK Stack & Streaming Pipelines)

In high-scale microservice architectures running hundreds of ephemeral container pods, logging to local disk files is impossible to debug. Centralized logging streams, buffers, transforms, and indexes logs into full-text search clusters.

Enterprise Centralized Logging Pipeline (Fluentbit, Kafka, Logstash & Elasticsearch)

System Design

Buffering peak log spikes with Apache Kafka and indexing JSON inverted search indices for Kibana querying

100%

Rendering diagram…

flowchart LR
    subgraph K8s [Kubernetes Nodes / VM Fleet]
        App1[Service Pod 1<br/>JSON stdout]
        App2[Service Pod 2<br/>JSON stdout]
        Agent[Log Shipper DaemonSet<br/>Fluentbit / Filebeat / Vector]
        App1 --> Agent
        App2 --> Agent
    end

    subgraph Buffer [Streaming Ingestion Buffer]
        Kafka[(Apache Kafka / AWS Kinesis<br/>Topic: raw-logs<br/>Zero Data Loss Buffer)]
    end

    subgraph Processing [Log Transformation Engine]
        Logstash[Logstash / Vector Workers<br/>GeoIP, Grok parsing, PII Masking]
    end

    subgraph Storage [Search & Analytics Engine]
        ES[(Elasticsearch / OpenSearch<br/>Indexed inverted indices<br/>Hot-Warm-Cold Tiering)]
    end

    subgraph Visualization [UI & Query Layer]
        Kibana[Kibana / Grafana UI<br/>Full-text log search & alerts]
    end

    Agent -- High-throughput streaming --> Kafka
    Kafka --> Logstash
    Logstash -- Bulk indexing HTTP --> ES
    ES --> Kibana

    classDef k8sStyle fill:#eff6ff,stroke:#2563eb,stroke-width:2px,color:#1e40af;
    classDef bufStyle fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e;
    classDef esStyle fill:#ecfdf5,stroke:#059669,stroke-width:2px,color:#065f46;
    classDef uiStyle fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px,color:#5b21b6;

    class App1,App2,Agent k8sStyle;
    class Kafka bufStyle;
    class Logstash,ES esStyle;
    class Kibana uiStyle;

`Log Ingestion & Buffering`

Protecting storage engines from volume spikes.

DaemonSet Log Shippers: Lightweight agents (Fluentbit, Vector) ship pod stdout streams
Kafka Buffer: Decouples log producers from slower Elasticsearch indexing writes to prevent log loss during traffic spikes

`Transformation & Index Lifecycle (ILM)`

Managing storage costs across time.

Structured JSON: Always emit structured JSON; avoid raw text strings
Hot Tier (NVMe SSD): Active indexing & search (0-7 days)
Warm/Cold Tier (HDD / S3 Object Storage): Read-only archived logs (8-90 days)

Production Structured JSON Logger with OpenTelemetry Trace ID Correlation

A structured JSON logger that attaches OpenTelemetry trace IDs so logs can be correlated across services.

3. Distributed Tracing (OpenTelemetry, Jaeger & W3C Trace Context)

When an API request hops across 10 microservices and takes 2 seconds, distributed tracing pinpoint exactly which microservice, database query, or downstream RPC caused the latency bottleneck.

Distributed Request Trace Sequence & W3C Context Propagation

System Design

Tracing TraceID and SpanID hierarchy from API Gateway to slow PostgreSQL database queries

100%

Rendering diagram…

sequenceDiagram
    autonumber
    actor Client as User / Mobile App
    participant GW as API Gateway (Root Span)
    participant Auth as Auth Service (Child Span 1)
    participant Order as Order Service (Child Span 2)
    participant DB as Postgres Database (Child Span 3)
    participant Jaeger as Jaeger / OTel Collector

    Note over Client,GW: HTTP GET /orders/123
    GW->>GW: Generate TraceID: 4bf92f... & RootSpanID: a1
    GW->>Auth: HTTP GET /verify (Header: traceparent: 00-4bf92f...-a1-01)
    activate Auth
    Auth->>Auth: SpanID: b2 (Parent: a1)
    Auth-->>GW: 200 OK (Auth Validated - 12ms)
    deactivate Auth
    GW->>Jaeger: Emit Span: Auth Verification (12ms)

    GW->>Order: HTTP POST /get-order (Header: traceparent: 00-4bf92f...-a1-01)
    activate Order
    Order->>Order: SpanID: c3 (Parent: a1)
    Order->>DB: SQL Query SELECT * FROM orders (SpanID: d4, Parent: c3)
    activate DB
    DB-->>Order: Row Returned (140ms - SLOW QUERY!)
    deactivate DB
    Order->>Jaeger: Emit Span: DB Query (140ms, tag: sql.query)
    Order-->>GW: 200 OK (Order Payload - 155ms)
    deactivate Order
    GW->>Jaeger: Emit Span: Order Fulfillment (155ms)
    GW-->>Client: 200 OK (Total Response Time: 172ms)
    GW->>Jaeger: Emit Root Span: End-to-End Request (172ms)

`W3C Trace Context Standard`

Cross-language context header format: traceparent: 00-{traceId}-{spanId}-{flags}.

Trace ID: 16-byte random hex string shared across all child spans in the request
Span ID: 8-byte unique identifier for the specific operation
Baggage: Key-value metadata propagated across process boundaries (e.g. tenant_id)

`Sampling Strategies (Head vs Tail)`

Controlling storage and network overhead.

Head-Based Sampling: API Gateway samples a fixed % of traces (e.g. 5%) at request start
Tail-Based Sampling: Collector buffers all spans in memory and retains 100% of traces that error (5xx) or exceed latency thresholds (>500ms)

4. Monitoring Architecture (Prometheus Pull Scraping & Grafana)

Prometheus has become the cloud-native standard for metrics monitoring. It employs an active pull model where the server periodically scrapes registered HTTP endpoints.

Prometheus Metrics Pull Architecture & Grafana TSDB Topology

System Design

Dynamic Kubernetes service discovery, time-series storage (TSDB), PromQL queries, and Alertmanager routing

100%

Loading system design canvas…

[
  {
    "id": "prom-svc1",
    "type": "rectangle",
    "x": 30,
    "y": 60,
    "width": 160,
    "height": 60,
    "strokeColor": "#3b82f6",
    "backgroundColor": "#eff6ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17012,
    "version": 1,
    "versionNonce": 17013,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "prom-svc1-text"
      }
    ]
  },
  {
    "id": "prom-svc1-text",
    "type": "text",
    "x": 40,
    "y": 76,
    "width": 140,
    "height": 28,
    "strokeColor": "#1e40af",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17014,
    "version": 1,
    "versionNonce": 17015,
    "isDeleted": false,
    "text": "Order Service Pod\n/metrics endpoint",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Order Service Pod\n/metrics endpoint",
    "containerId": "prom-svc1"
  },
  {
    "id": "prom-svc2",
    "type": "rectangle",
    "x": 30,
    "y": 140,
    "width": 160,
    "height": 60,
    "strokeColor": "#3b82f6",
    "backgroundColor": "#eff6ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17016,
    "version": 1,
    "versionNonce": 17017,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "prom-svc2-text"
      }
    ]
  },
  {
    "id": "prom-svc2-text",
    "type": "text",
    "x": 40,
    "y": 156,
    "width": 140,
    "height": 28,
    "strokeColor": "#1e40af",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17018,
    "version": 1,
    "versionNonce": 17019,
    "isDeleted": false,
    "text": "Payment Service Pod\n/metrics endpoint",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Payment Service Pod\n/metrics endpoint",
    "containerId": "prom-svc2"
  },
  {
    "id": "prom-svc3",
    "type": "rectangle",
    "x": 30,
    "y": 220,
    "width": 160,
    "height": 60,
    "strokeColor": "#3b82f6",
    "backgroundColor": "#eff6ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17020,
    "version": 1,
    "versionNonce": 17021,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "prom-svc3-text"
      }
    ]
  },
  {
    "id": "prom-svc3-text",
    "type": "text",
    "x": 40,
    "y": 236,
    "width": 140,
    "height": 28,
    "strokeColor": "#1e40af",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17022,
    "version": 1,
    "versionNonce": 17023,
    "isDeleted": false,
    "text": "Auth Service Pod\n/metrics endpoint",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Auth Service Pod\n/metrics endpoint",
    "containerId": "prom-svc3"
  },
  {
    "id": "prom-a1",
    "type": "arrow",
    "x": 200,
    "y": 90,
    "width": 80,
    "height": 60,
    "strokeColor": "#2563eb",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        80,
        60
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 17024,
    "version": 1,
    "versionNonce": 17025,
    "isDeleted": false
  },
  {
    "id": "prom-a2",
    "type": "arrow",
    "x": 200,
    "y": 170,
    "width": 80,
    "height": 0,
    "strokeColor": "#2563eb",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        80,
        0
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 17026,
    "version": 1,
    "versionNonce": 17027,
    "isDeleted": false
  },
  {
    "id": "prom-a3",
    "type": "arrow",
    "x": 200,
    "y": 250,
    "width": 80,
    "height": -60,
    "strokeColor": "#2563eb",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        80,
        -60
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 17028,
    "version": 1,
    "versionNonce": 17029,
    "isDeleted": false
  },
  {
    "id": "prom-server",
    "type": "rectangle",
    "x": 280,
    "y": 50,
    "width": 280,
    "height": 240,
    "strokeColor": "#d97706",
    "backgroundColor": "#fffbeb",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17030,
    "version": 1,
    "versionNonce": 17031,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "prom-server-text"
      }
    ]
  },
  {
    "id": "prom-server-text",
    "type": "text",
    "x": 290,
    "y": 121.875,
    "width": 260,
    "height": 96.25,
    "strokeColor": "#92400e",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17032,
    "version": 1,
    "versionNonce": 17033,
    "isDeleted": false,
    "text": "Prometheus Server (Pull Engine)\n\n• Service Discovery (K8s API)\n• Scrape Interval (every 15s)\n• Local TSDB (Time Series DB)\n• PromQL Evaluation Engine\n• Alerting Rules Evaluation",
    "fontSize": 11,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Prometheus Server (Pull Engine)\n\n• Service Discovery (K8s API)\n• Scrape Interval (every 15s)\n• Local TSDB (Time Series DB)\n• PromQL Evaluation Engine\n• Alerting Rules Evaluation",
    "containerId": "prom-server"
  },
  {
    "id": "prom-out1",
    "type": "arrow",
    "x": 560,
    "y": 120,
    "width": 80,
    "height": -30,
    "strokeColor": "#059669",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        80,
        -30
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 17034,
    "version": 1,
    "versionNonce": 17035,
    "isDeleted": false
  },
  {
    "id": "prom-out2",
    "type": "arrow",
    "x": 560,
    "y": 220,
    "width": 80,
    "height": 10,
    "strokeColor": "#ef4444",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        80,
        10
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 17036,
    "version": 1,
    "versionNonce": 17037,
    "isDeleted": false
  },
  {
    "id": "prom-grafana",
    "type": "rectangle",
    "x": 640,
    "y": 60,
    "width": 220,
    "height": 75,
    "strokeColor": "#059669",
    "backgroundColor": "#ecfdf5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17038,
    "version": 1,
    "versionNonce": 17039,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "prom-grafana-text"
      }
    ]
  },
  {
    "id": "prom-grafana-text",
    "type": "text",
    "x": 650,
    "y": 78.75,
    "width": 200,
    "height": 37.5,
    "strokeColor": "#065f46",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17040,
    "version": 1,
    "versionNonce": 17041,
    "isDeleted": false,
    "text": "Grafana Dashboards\nReal-time PromQL visual graphs,\np99 latency, and heatmaps",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Grafana Dashboards\nReal-time PromQL visual graphs,\np99 latency, and heatmaps",
    "containerId": "prom-grafana"
  },
  {
    "id": "prom-alertmgr",
    "type": "rectangle",
    "x": 640,
    "y": 195,
    "width": 220,
    "height": 75,
    "strokeColor": "#ef4444",
    "backgroundColor": "#fee2e2",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17042,
    "version": 1,
    "versionNonce": 17043,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "prom-alertmgr-text"
      }
    ]
  },
  {
    "id": "prom-alertmgr-text",
    "type": "text",
    "x": 650,
    "y": 213.75,
    "width": 200,
    "height": 37.5,
    "strokeColor": "#991b1b",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17044,
    "version": 1,
    "versionNonce": 17045,
    "isDeleted": false,
    "text": "Alertmanager Engine\nDeduplication, silences, and\nrouting to PagerDuty / Slack",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Alertmanager Engine\nDeduplication, silences, and\nrouting to PagerDuty / Slack",
    "containerId": "prom-alertmgr"
  }
]

`Prometheus Pull Model Advantages`

Why pull architecture scales well in container clusters.

Automatic Health Detection: If a scrape fails, Prometheus immediately knows the target is down
Prevents Overload: Prometheus controls scrape rates; target services never get overwhelmed pushing telemetry
Service Discovery: Automatically tracks pods as K8s auto-scales replicas up and down

`PromQL (Prometheus Query Language)`

Powerful expressions for calculating percentiles.

Rate calculation: rate(http_requests_total[5m])
p99 Latency: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
Error %: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

5. SRE Alerting Systems & Preventing Alert Fatigue

Poorly tuned alerting causes Alert Fatigue where engineers ignore pages. Effective SRE alerts trigger exclusively on user-impacting symptoms and Service Level Objective (SLO) burn rates.

SRE Alerting & Incident Escalation Workflow

System Design

Symptom-based evaluation, alert deduplication, inhibition trees, and severity tier routing

100%

Rendering diagram…

flowchart TD
    Metrics[Prometheus / APM Metrics] --> Rules{Evaluate Alerting Rules<br/>e.g. 5xx Error Rate > 1% for 3m}
    Rules -- Alert Fires --> AM[Alertmanager / PagerDuty Routing]
    
    subgraph Routing & Deduplication
        AM --> Dedup[Deduplication & Inhibition<br/>Suppress child alerts if Root Switch is down]
        Dedup --> Group[Group related pod alerts into 1 incident]
    end

    Group --> Severity{Alert Severity Tier}

    Severity -- P1 / Critical (Customer Facing SLO breach) --> P1[Page On-Call Engineer<br/>Phone Call + PagerDuty Push Alert]
    Severity -- P2 / High (Elevated latency / Replica down) --> P2[Urgent Slack Channel Notification]
    Severity -- P3 / Warning (Disk at 75% / Non-urgent) --> P3[Create Jira Ticket / Low-priority Slack]

    P1 --> Action[Acknowledge, Triage Runbook & Auto-Remediation]

    classDef metricStyle fill:#eff6ff,stroke:#2563eb,stroke-width:2px,color:#1e40af;
    classDef alertStyle fill:#fee2e2,stroke:#ef4444,stroke-width:2px,color:#991b1b;
    classDef warnStyle fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e;

    class Metrics,Rules,AM,Dedup,Group metricStyle;
    class Severity,P1,Action alertStyle;
    class P2,P3 warnStyle;

`SRE Alerting Principles (Google SRE Book)`

Design alerts that are actionable, urgent, and symptom-oriented.

Alert on Symptoms, Not Causes: Page on high 5xx error rate or elevated user latency, NOT on single CPU blips or memory thresholds
Multi-Window Multi-Burn-Rate Alerts: Detect rapid budget burns (14x burn rate over 1h) and slow budget drains (2x burn rate over 6h)
Alert Inhibition: If an entire Data Center switch goes offline, suppress individual pod unreachable alerts to avoid 500 duplicate pages

6. Health Check Endpoints: Liveness vs Readiness Probes

Container orchestrators like Kubernetes use dedicated probe endpoints to decide whether to restart an unresponsive container or temporarily remove it from load balancer routing.

Liveness vs Readiness Probe Load Balancer & Container Lifecycle Decisions

System Design

Contrasting container restart triggers against traffic routing removal

100%

Loading system design canvas…

[
  {
    "id": "hc-live-box",
    "type": "rectangle",
    "x": 30,
    "y": 40,
    "width": 400,
    "height": 260,
    "strokeColor": "#2563eb",
    "backgroundColor": "#eff6ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17046,
    "version": 1,
    "versionNonce": 17047,
    "isDeleted": false
  },
  {
    "id": "hc-live-box-text",
    "type": "text",
    "x": 40,
    "y": 156,
    "width": 380,
    "height": 28,
    "strokeColor": "#1e40af",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17048,
    "version": 1,
    "versionNonce": 17049,
    "isDeleted": false,
    "text": "Liveness Probe (/health/live)",
    "fontSize": 13,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Liveness Probe (/health/live)"
  },
  {
    "id": "hc-live-target",
    "type": "rectangle",
    "x": 50,
    "y": 90,
    "width": 160,
    "height": 60,
    "strokeColor": "#3b82f6",
    "backgroundColor": "#dbeafe",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17050,
    "version": 1,
    "versionNonce": 17051,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "hc-live-target-text"
      }
    ]
  },
  {
    "id": "hc-live-target-text",
    "type": "text",
    "x": 60,
    "y": 106,
    "width": 140,
    "height": 28,
    "strokeColor": "#1e40af",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17052,
    "version": 1,
    "versionNonce": 17053,
    "isDeleted": false,
    "text": "App Container\nInternal Event Loop",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "App Container\nInternal Event Loop",
    "containerId": "hc-live-target"
  },
  {
    "id": "hc-l-a1",
    "type": "arrow",
    "x": 210,
    "y": 120,
    "width": 60,
    "height": 0,
    "strokeColor": "#2563eb",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        60,
        0
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 17054,
    "version": 1,
    "versionNonce": 17055,
    "isDeleted": false
  },
  {
    "id": "hc-live-resp",
    "type": "rectangle",
    "x": 270,
    "y": 90,
    "width": 140,
    "height": 60,
    "strokeColor": "#10b981",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17056,
    "version": 1,
    "versionNonce": 17057,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "hc-live-resp-text"
      }
    ]
  },
  {
    "id": "hc-live-resp-text",
    "type": "text",
    "x": 280,
    "y": 106,
    "width": 120,
    "height": 28,
    "strokeColor": "#047857",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17058,
    "version": 1,
    "versionNonce": 17059,
    "isDeleted": false,
    "text": "HTTP 200 OK\nProcess Responsive",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "HTTP 200 OK\nProcess Responsive",
    "containerId": "hc-live-resp"
  },
  {
    "id": "hc-live-fail",
    "type": "rectangle",
    "x": 50,
    "y": 170,
    "width": 360,
    "height": 110,
    "strokeColor": "#ef4444",
    "backgroundColor": "#fee2e2",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17060,
    "version": 1,
    "versionNonce": 17061,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "hc-live-fail-text"
      }
    ]
  },
  {
    "id": "hc-live-fail-text",
    "type": "text",
    "x": 60,
    "y": 200,
    "width": 340,
    "height": 50,
    "strokeColor": "#991b1b",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17062,
    "version": 1,
    "versionNonce": 17063,
    "isDeleted": false,
    "text": "If Liveness Fails (Deadlock / OOM):\n• Kubernetes kills the stalled container immediately\n• Initiates pod restart policy (CrashLoopBackOff if chronic)\n• DOES NOT check external database health!",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "If Liveness Fails (Deadlock / OOM):\n• Kubernetes kills the stalled container immediately\n• Initiates pod restart policy (CrashLoopBackOff if chronic)\n• DOES NOT check external database health!",
    "containerId": "hc-live-fail"
  },
  {
    "id": "hc-ready-box",
    "type": "rectangle",
    "x": 460,
    "y": 40,
    "width": 400,
    "height": 260,
    "strokeColor": "#059669",
    "backgroundColor": "#ecfdf5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17064,
    "version": 1,
    "versionNonce": 17065,
    "isDeleted": false
  },
  {
    "id": "hc-ready-box-text",
    "type": "text",
    "x": 470,
    "y": 156,
    "width": 380,
    "height": 28,
    "strokeColor": "#065f46",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17066,
    "version": 1,
    "versionNonce": 17067,
    "isDeleted": false,
    "text": "Readiness Probe (/health/ready)",
    "fontSize": 13,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Readiness Probe (/health/ready)"
  },
  {
    "id": "hc-ready-target",
    "type": "rectangle",
    "x": 480,
    "y": 90,
    "width": 160,
    "height": 60,
    "strokeColor": "#059669",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17068,
    "version": 1,
    "versionNonce": 17069,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "hc-ready-target-text"
      }
    ]
  },
  {
    "id": "hc-ready-target-text",
    "type": "text",
    "x": 490,
    "y": 106,
    "width": 140,
    "height": 28,
    "strokeColor": "#065f46",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17070,
    "version": 1,
    "versionNonce": 17071,
    "isDeleted": false,
    "text": "App + DB / Cache\nDependency Check",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "App + DB / Cache\nDependency Check",
    "containerId": "hc-ready-target"
  },
  {
    "id": "hc-r-a1",
    "type": "arrow",
    "x": 640,
    "y": 120,
    "width": 60,
    "height": 0,
    "strokeColor": "#059669",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        60,
        0
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 17072,
    "version": 1,
    "versionNonce": 17073,
    "isDeleted": false
  },
  {
    "id": "hc-ready-resp",
    "type": "rectangle",
    "x": 700,
    "y": 90,
    "width": 140,
    "height": 60,
    "strokeColor": "#10b981",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17074,
    "version": 1,
    "versionNonce": 17075,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "hc-ready-resp-text"
      }
    ]
  },
  {
    "id": "hc-ready-resp-text",
    "type": "text",
    "x": 710,
    "y": 106,
    "width": 120,
    "height": 28,
    "strokeColor": "#047857",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17076,
    "version": 1,
    "versionNonce": 17077,
    "isDeleted": false,
    "text": "HTTP 200 OK\nReady for Traffic",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "HTTP 200 OK\nReady for Traffic",
    "containerId": "hc-ready-resp"
  },
  {
    "id": "hc-ready-fail",
    "type": "rectangle",
    "x": 480,
    "y": 170,
    "width": 360,
    "height": 110,
    "strokeColor": "#f59e0b",
    "backgroundColor": "#fef3c7",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 17078,
    "version": 1,
    "versionNonce": 17079,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "hc-ready-fail-text"
      }
    ]
  },
  {
    "id": "hc-ready-fail-text",
    "type": "text",
    "x": 490,
    "y": 200,
    "width": 340,
    "height": 50,
    "strokeColor": "#92400e",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 17080,
    "version": 1,
    "versionNonce": 17081,
    "isDeleted": false,
    "text": "If Readiness Fails (DB Overloaded / Cache Warming):\n• Load Balancer removes pod from endpoint routing\n• Zero traffic sent to this instance\n• POD IS NOT KILLED! Allows graceful background recovery",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "If Readiness Fails (DB Overloaded / Cache Warming):\n• Load Balancer removes pod from endpoint routing\n• Zero traffic sent to this instance\n• POD IS NOT KILLED! Allows graceful background recovery",
    "containerId": "hc-ready-fail"
  }
]

`Liveness Probe (/health/live)`

Determines if the application process is alive or deadlocked.

Action on failure: Kills container and initiates container restart
Never check downstream DBs in liveness probes (prevents cascading crash loops!)
Execution frequency: Every 10-30 seconds

`Readiness Probe (/health/ready)`

Determines if the pod is ready to accept user traffic.

Action on failure: Removes pod from Load Balancer endpoint list (Pod is NOT killed)
Checks database connectivity, warm caches, and memory readiness
Allows pods to gracefully recover during heavy traffic spikes

Express.js Prometheus Metrics Collector with Custom Histograms & Health Probes

An Express.js middleware that records custom Prometheus histograms and exposes health-probe endpoints.

7. Application Performance Monitoring (APM) & Continuous Profiling

APM tools (Datadog, Dynatrace, New Relic) automatically instrument runtime bytecode to generate CPU flame graphs, memory allocation profiles, and automated slow query detection without code changes.

APM Full-Stack Continuous Profiling & Flame Graph Analysis

System Design

Continuous thread profiling, heap leak detection, and automated slow database query diagnosis

100%

Rendering diagram…

flowchart TD
    App[Live Application Process] --> APMAgent[APM Agent / Bytecode Instrumentation]
    
    subgraph Telemetry Captured
        APMAgent --> CPU[Continuous CPU Profiler<br/>Detects thread locks & CPU-heavy loops]
        APMAgent --> Memory[Memory Allocation Profiler<br/>Detects heap leaks & GC pauses]
        APMAgent --> DBTracing[Database Query Inspector<br/>Identifies N+1 queries & unindexed scans]
        APMAgent --> Net[External HTTP / gRPC Call Latency]
    end

    CPU --> Engine["APM Telemetry Backend (Datadog / Dynatrace)"]
    Memory --> Engine
    DBTracing --> Engine
    Net --> Engine

    Engine --> Flame[Interactive Flame Graphs & Slow Trace Drill-Downs]

    classDef appStyle fill:#eff6ff,stroke:#3b82f6,stroke-width:2px,color:#1e40af;
    classDef agentStyle fill:#ecfdf5,stroke:#059669,stroke-width:2px,color:#065f46;
    classDef engineStyle fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px,color:#5b21b6;

    class App,CPU,Memory,DBTracing,Net appStyle;
    class APMAgent agentStyle;
    class Engine,Flame engineStyle;

`Key APM Capabilities in System Design`

Deep runtime visibility inside application runtimes.

Flame Graphs: Visual representation of CPU time spent per function call stack
N+1 Database Query Detection: Flags microservices issuing hundreds of repetitive sequential SQL SELECT queries
Garbage Collection (GC) Pauses: Highlights JVM / V8 stop-the-world latency spikes in memory-intensive workloads

Knowledge Check

1. What is the primary function of Distributed Tracing in microservices?

2. What architectural model does Prometheus use to gather metrics from services?

3. What action does Kubernetes take when a container fails its Liveness Probe?

4. What is the difference between a Readiness Probe and a Liveness Probe?

5. Why should SRE alerting rules focus on symptom metrics rather than cause metrics?