System Design: Reliability and Fault Tolerance

Master high availability, redundancy models, automated failover, disaster recovery (RPO/RTO), chaos engineering, graceful degradation, API idempotency, and SLA/SLO error budgets.

1. High Availability Concepts & Core Reliability Metrics

High Availability (HA) ensures a system remains operational and accessible with minimal unplanned downtime. Availability is quantified as a percentage of uptime ("nines") and measured using the relationship between failure frequency and recovery speed.

High Availability 'Nines' & Incident Lifecycle Metrics (MTBF, MTTD, MTTR)

System Design

Calculating annual downtime thresholds and tracing automated detection and self-healing recovery phases

100%

Loading system design canvas…

[
  {
    "id": "ha-nines",
    "type": "rectangle",
    "x": 30,
    "y": 40,
    "width": 380,
    "height": 260,
    "strokeColor": "#2563eb",
    "backgroundColor": "#eff6ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16000,
    "version": 1,
    "versionNonce": 16001,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ha-nines-text"
      }
    ]
  },
  {
    "id": "ha-nines-text",
    "type": "text",
    "x": 40,
    "y": 115,
    "width": 360,
    "height": 110,
    "strokeColor": "#1e40af",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16002,
    "version": 1,
    "versionNonce": 16003,
    "isDeleted": false,
    "text": "High Availability \"Nines\"\n\n99.0% (Two 9s) -> 3.65 days downtime/yr\n99.9% (Three 9s) -> 8.76 hours downtime/yr\n99.99% (Four 9s) -> 52.6 minutes downtime/yr\n99.999% (Five 9s) -> 5.26 minutes downtime/yr\n\nAvailability = MTBF / (MTBF + MTTR)",
    "fontSize": 11,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "High Availability \"Nines\"\n\n99.0% (Two 9s) -> 3.65 days downtime/yr\n99.9% (Three 9s) -> 8.76 hours downtime/yr\n99.99% (Four 9s) -> 52.6 minutes downtime/yr\n99.999% (Five 9s) -> 5.26 minutes downtime/yr\n\nAvailability = MTBF / (MTBF + MTTR)",
    "containerId": "ha-nines"
  },
  {
    "id": "ha-timeline-box",
    "type": "rectangle",
    "x": 430,
    "y": 40,
    "width": 440,
    "height": 260,
    "strokeColor": "#059669",
    "backgroundColor": "#ecfdf5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16004,
    "version": 1,
    "versionNonce": 16005,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ha-timeline-box-text"
      }
    ]
  },
  {
    "id": "ha-timeline-box-text",
    "type": "text",
    "x": 440,
    "y": 156,
    "width": 420,
    "height": 28,
    "strokeColor": "#065f46",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16006,
    "version": 1,
    "versionNonce": 16007,
    "isDeleted": false,
    "text": "Incident Lifecycle Timeline (MTBF, MTTD, MTTR)",
    "fontSize": 13,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Incident Lifecycle Timeline (MTBF, MTTD, MTTR)",
    "containerId": "ha-timeline-box"
  },
  {
    "id": "ha-up",
    "type": "rectangle",
    "x": 450,
    "y": 85,
    "width": 180,
    "height": 45,
    "strokeColor": "#10b981",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16008,
    "version": 1,
    "versionNonce": 16009,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ha-up-text"
      }
    ]
  },
  {
    "id": "ha-up-text",
    "type": "text",
    "x": 460,
    "y": 93.5,
    "width": 160,
    "height": 28,
    "strokeColor": "#047857",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16010,
    "version": 1,
    "versionNonce": 16011,
    "isDeleted": false,
    "text": "Healthy Operation (MTBF)\nMean Time Between Failures",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Healthy Operation (MTBF)\nMean Time Between Failures",
    "containerId": "ha-up"
  },
  {
    "id": "ha-arr1",
    "type": "arrow",
    "x": 630,
    "y": 107,
    "width": 40,
    "height": 0,
    "strokeColor": "#ef4444",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        40,
        0
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16012,
    "version": 1,
    "versionNonce": 16013,
    "isDeleted": false
  },
  {
    "id": "ha-fault",
    "type": "rectangle",
    "x": 670,
    "y": 85,
    "width": 180,
    "height": 45,
    "strokeColor": "#ef4444",
    "backgroundColor": "#fee2e2",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16014,
    "version": 1,
    "versionNonce": 16015,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ha-fault-text"
      }
    ]
  },
  {
    "id": "ha-fault-text",
    "type": "text",
    "x": 680,
    "y": 93.5,
    "width": 160,
    "height": 28,
    "strokeColor": "#991b1b",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16016,
    "version": 1,
    "versionNonce": 16017,
    "isDeleted": false,
    "text": "Fault Occurs!\nSystem Degraded",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Fault Occurs!\nSystem Degraded",
    "containerId": "ha-fault"
  },
  {
    "id": "ha-mttd",
    "type": "rectangle",
    "x": 450,
    "y": 150,
    "width": 180,
    "height": 55,
    "strokeColor": "#f59e0b",
    "backgroundColor": "#fef3c7",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16018,
    "version": 1,
    "versionNonce": 16019,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ha-mttd-text"
      }
    ]
  },
  {
    "id": "ha-mttd-text",
    "type": "text",
    "x": 460,
    "y": 158.75,
    "width": 160,
    "height": 37.5,
    "strokeColor": "#b45309",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16020,
    "version": 1,
    "versionNonce": 16021,
    "isDeleted": false,
    "text": "MTTD (Detection)\nAlerts & Automated Probes\nTrigger PagerDuty",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "MTTD (Detection)\nAlerts & Automated Probes\nTrigger PagerDuty",
    "containerId": "ha-mttd"
  },
  {
    "id": "ha-arr2",
    "type": "arrow",
    "x": 630,
    "y": 177,
    "width": 40,
    "height": 0,
    "strokeColor": "#f59e0b",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        40,
        0
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16022,
    "version": 1,
    "versionNonce": 16023,
    "isDeleted": false
  },
  {
    "id": "ha-mttr",
    "type": "rectangle",
    "x": 670,
    "y": 150,
    "width": 180,
    "height": 55,
    "strokeColor": "#8b5cf6",
    "backgroundColor": "#f3e8ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16024,
    "version": 1,
    "versionNonce": 16025,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ha-mttr-text"
      }
    ]
  },
  {
    "id": "ha-mttr-text",
    "type": "text",
    "x": 680,
    "y": 158.75,
    "width": 160,
    "height": 37.5,
    "strokeColor": "#5b21b6",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16026,
    "version": 1,
    "versionNonce": 16027,
    "isDeleted": false,
    "text": "MTTR (Recovery)\nAuto-Failover / Rollback / Reboot\nSystem Restored",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "MTTR (Recovery)\nAuto-Failover / Rollback / Reboot\nSystem Restored",
    "containerId": "ha-mttr"
  },
  {
    "id": "ha-goal",
    "type": "rectangle",
    "x": 450,
    "y": 225,
    "width": 400,
    "height": 55,
    "strokeColor": "#059669",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16028,
    "version": 1,
    "versionNonce": 16029,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ha-goal-text"
      }
    ]
  },
  {
    "id": "ha-goal-text",
    "type": "text",
    "x": 460,
    "y": 238.5,
    "width": 380,
    "height": 28,
    "strokeColor": "#065f46",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16030,
    "version": 1,
    "versionNonce": 16031,
    "isDeleted": false,
    "text": "Reliability Goal: Maximize MTBF and Minimize MTTR + MTTD to seconds via automated self-healing.",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Reliability Goal: Maximize MTBF and Minimize MTTR + MTTD to seconds via automated self-healing.",
    "containerId": "ha-goal"
  }
]

`Availability Formula & The 'Nines'`

Availability = MTBF / (MTBF + MTTR). Each additional 'nine' increases engineering complexity and cost exponentially.

99.0% (Two 9s): ~3.65 days downtime/year (Basic web app)
99.9% (Three 9s): ~8.76 hours downtime/year (Standard cloud SLA)
99.99% (Four 9s): ~52.6 minutes downtime/year (High-scale enterprise)
99.999% (Five 9s): ~5.26 minutes downtime/year (Telecom / Critical infrastructure)

`Core Incident Time Metrics`

Key Site Reliability Engineering (SRE) indicators measuring incident response speed.

MTBF (Mean Time Between Failures): Average uptime duration between system faults
MTTD (Mean Time To Detect): Average time elapsed from incident start to alert firing
MTTR (Mean Time To Repair/Recover): Average time taken to restore full service functionality

2. Redundancy Models & Eliminating Single Points of Failure (SPOF)

Reliability requires eliminating Single Points of Failure (SPOF) by duplicating critical components across compute, storage, and networking layers.

Active-Active vs Active-Passive Redundancy Architecture Topologies

System Design

Comparing continuous load sharing against hot/warm standby failover models

100%

Loading system design canvas…

[
  {
    "id": "aa-box",
    "type": "rectangle",
    "x": 30,
    "y": 40,
    "width": 400,
    "height": 270,
    "strokeColor": "#10b981",
    "backgroundColor": "#ecfdf5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16032,
    "version": 1,
    "versionNonce": 16033,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "aa-box-text"
      }
    ]
  },
  {
    "id": "aa-box-text",
    "type": "text",
    "x": 40,
    "y": 161,
    "width": 380,
    "height": 28,
    "strokeColor": "#047857",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16034,
    "version": 1,
    "versionNonce": 16035,
    "isDeleted": false,
    "text": "Active-Active Redundancy",
    "fontSize": 13,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Active-Active Redundancy",
    "containerId": "aa-box"
  },
  {
    "id": "aa-lb",
    "type": "rectangle",
    "x": 140,
    "y": 85,
    "width": 180,
    "height": 45,
    "strokeColor": "#059669",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16036,
    "version": 1,
    "versionNonce": 16037,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "aa-lb-text"
      }
    ]
  },
  {
    "id": "aa-lb-text",
    "type": "text",
    "x": 150,
    "y": 93.5,
    "width": 160,
    "height": 28,
    "strokeColor": "#065f46",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16038,
    "version": 1,
    "versionNonce": 16039,
    "isDeleted": false,
    "text": "Load Balancer (50/50 Traffic)",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Load Balancer (50/50 Traffic)",
    "containerId": "aa-lb"
  },
  {
    "id": "aa-a1",
    "type": "arrow",
    "x": 190,
    "y": 130,
    "width": -90,
    "height": 45,
    "strokeColor": "#059669",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        -90,
        45
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16040,
    "version": 1,
    "versionNonce": 16041,
    "isDeleted": false
  },
  {
    "id": "aa-a2",
    "type": "arrow",
    "x": 270,
    "y": 130,
    "width": 90,
    "height": 45,
    "strokeColor": "#059669",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        90,
        45
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16042,
    "version": 1,
    "versionNonce": 16043,
    "isDeleted": false
  },
  {
    "id": "aa-n1",
    "type": "rectangle",
    "x": 50,
    "y": 175,
    "width": 150,
    "height": 50,
    "strokeColor": "#10b981",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16044,
    "version": 1,
    "versionNonce": 16045,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "aa-n1-text"
      }
    ]
  },
  {
    "id": "aa-n1-text",
    "type": "text",
    "x": 60,
    "y": 186,
    "width": 130,
    "height": 28,
    "strokeColor": "#047857",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16046,
    "version": 1,
    "versionNonce": 16047,
    "isDeleted": false,
    "text": "Node 1 (Active)\nServing 50% Load",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Node 1 (Active)\nServing 50% Load",
    "containerId": "aa-n1"
  },
  {
    "id": "aa-n2",
    "type": "rectangle",
    "x": 260,
    "y": 175,
    "width": 150,
    "height": 50,
    "strokeColor": "#10b981",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16048,
    "version": 1,
    "versionNonce": 16049,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "aa-n2-text"
      }
    ]
  },
  {
    "id": "aa-n2-text",
    "type": "text",
    "x": 270,
    "y": 186,
    "width": 130,
    "height": 28,
    "strokeColor": "#047857",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16050,
    "version": 1,
    "versionNonce": 16051,
    "isDeleted": false,
    "text": "Node 2 (Active)\nServing 50% Load",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Node 2 (Active)\nServing 50% Load",
    "containerId": "aa-n2"
  },
  {
    "id": "aa-desc",
    "type": "rectangle",
    "x": 50,
    "y": 240,
    "width": 360,
    "height": 55,
    "strokeColor": "#059669",
    "backgroundColor": "#ecfdf5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16052,
    "version": 1,
    "versionNonce": 16053,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "aa-desc-text"
      }
    ]
  },
  {
    "id": "aa-desc-text",
    "type": "text",
    "x": 60,
    "y": 253.5,
    "width": 340,
    "height": 28,
    "strokeColor": "#065f46",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16054,
    "version": 1,
    "versionNonce": 16055,
    "isDeleted": false,
    "text": "Zero Failover Delay: If Node 1 dies, LB redirects 100% traffic to Node 2 instantly with zero downtime.",
    "fontSize": 9,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Zero Failover Delay: If Node 1 dies, LB redirects 100% traffic to Node 2 instantly with zero downtime.",
    "containerId": "aa-desc"
  },
  {
    "id": "ap-box",
    "type": "rectangle",
    "x": 460,
    "y": 40,
    "width": 400,
    "height": 270,
    "strokeColor": "#f59e0b",
    "backgroundColor": "#fffbeb",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16056,
    "version": 1,
    "versionNonce": 16057,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ap-box-text"
      }
    ]
  },
  {
    "id": "ap-box-text",
    "type": "text",
    "x": 470,
    "y": 161,
    "width": 380,
    "height": 28,
    "strokeColor": "#b45309",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16058,
    "version": 1,
    "versionNonce": 16059,
    "isDeleted": false,
    "text": "Active-Passive (Standby) Redundancy",
    "fontSize": 13,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Active-Passive (Standby) Redundancy",
    "containerId": "ap-box"
  },
  {
    "id": "ap-lb",
    "type": "rectangle",
    "x": 570,
    "y": 85,
    "width": 180,
    "height": 45,
    "strokeColor": "#d97706",
    "backgroundColor": "#fef3c7",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16060,
    "version": 1,
    "versionNonce": 16061,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ap-lb-text"
      }
    ]
  },
  {
    "id": "ap-lb-text",
    "type": "text",
    "x": 580,
    "y": 93.5,
    "width": 160,
    "height": 28,
    "strokeColor": "#92400e",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16062,
    "version": 1,
    "versionNonce": 16063,
    "isDeleted": false,
    "text": "Health Check / VIP Switch",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Health Check / VIP Switch",
    "containerId": "ap-lb"
  },
  {
    "id": "ap-a1",
    "type": "arrow",
    "x": 620,
    "y": 130,
    "width": -90,
    "height": 45,
    "strokeColor": "#059669",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        -90,
        45
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16064,
    "version": 1,
    "versionNonce": 16065,
    "isDeleted": false
  },
  {
    "id": "ap-a2",
    "type": "arrow",
    "x": 700,
    "y": 130,
    "width": 90,
    "height": 45,
    "strokeColor": "#dc2626",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "dashed",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        90,
        45
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16066,
    "version": 1,
    "versionNonce": 16067,
    "isDeleted": false
  },
  {
    "id": "ap-n1",
    "type": "rectangle",
    "x": 480,
    "y": 175,
    "width": 150,
    "height": 50,
    "strokeColor": "#10b981",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16068,
    "version": 1,
    "versionNonce": 16069,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ap-n1-text"
      }
    ]
  },
  {
    "id": "ap-n1-text",
    "type": "text",
    "x": 490,
    "y": 186,
    "width": 130,
    "height": 28,
    "strokeColor": "#047857",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16070,
    "version": 1,
    "versionNonce": 16071,
    "isDeleted": false,
    "text": "Primary (Active)\nServing 100% Traffic",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Primary (Active)\nServing 100% Traffic",
    "containerId": "ap-n1"
  },
  {
    "id": "ap-n2",
    "type": "rectangle",
    "x": 690,
    "y": 175,
    "width": 150,
    "height": 50,
    "strokeColor": "#94a3b8",
    "backgroundColor": "#f1f5f9",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16072,
    "version": 1,
    "versionNonce": 16073,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ap-n2-text"
      }
    ]
  },
  {
    "id": "ap-n2-text",
    "type": "text",
    "x": 700,
    "y": 186,
    "width": 130,
    "height": 28,
    "strokeColor": "#475569",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16074,
    "version": 1,
    "versionNonce": 16075,
    "isDeleted": false,
    "text": "Standby (Passive)\nIdle / Syncing Logs",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Standby (Passive)\nIdle / Syncing Logs",
    "containerId": "ap-n2"
  },
  {
    "id": "ap-desc",
    "type": "rectangle",
    "x": 480,
    "y": 240,
    "width": 360,
    "height": 55,
    "strokeColor": "#d97706",
    "backgroundColor": "#fffbeb",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16076,
    "version": 1,
    "versionNonce": 16077,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "ap-desc-text"
      }
    ]
  },
  {
    "id": "ap-desc-text",
    "type": "text",
    "x": 490,
    "y": 253.5,
    "width": 340,
    "height": 28,
    "strokeColor": "#92400e",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16078,
    "version": 1,
    "versionNonce": 16079,
    "isDeleted": false,
    "text": "Failover Latency: Requires health probe heartbeat timeout + promotion sequence before standby accepts writes.",
    "fontSize": 9,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Failover Latency: Requires health probe heartbeat timeout + promotion sequence before standby accepts writes.",
    "containerId": "ap-desc"
  }
]

`Active-Active Redundancy`

All redundant nodes concurrently process live client requests via load balancers. If any node fails, healthy nodes instantly absorb the traffic.

Near-zero failover downtime
Optimal hardware resource utilization
Requires multi-master replication and conflict resolution

`Active-Passive (Standby) Redundancy`

Primary node handles 100% of traffic while standby node remains idle, synchronizing state via continuous replication logs.

Simpler consistency model (Single writer)
Standby hardware sits idle during normal operation
Requires heartbeat probing and promotion failover delay

3. Automated Failover Strategies & Split-Brain Prevention

When primary systems crash, automated failover mechanisms promote replicas without human intervention. However, temporary network partitions can cause a catastrophic Split-Brain scenario where two nodes both believe they are the active master.

Automated Failover Sequence & Quorum-Based Fencing (STONITH)

System Design

Visualizing heartbeat timeouts, majority quorum voting (Raft/Consul), and Virtual IP remapping

100%

Rendering diagram…

sequenceDiagram
    autonumber
    participant Client as API Clients
    participant VIP as Virtual IP / Route53 DNS
    participant NodeA as Primary Master (Node A)
    participant Heartbeat as Distributed Quorum (Raft / Consul)
    participant NodeB as Standby (Node B)

    VIP->>NodeA: Normal Traffic Flow
    Note over NodeA: Node A experiences network freeze / crash!
    Heartbeat->>NodeA: Ping Heartbeat Probe (Timeout after 3s)
    Heartbeat->>NodeB: Notify Primary Missing
    Note over Heartbeat,NodeB: Majority Quorum Vote (2/3 Nodes Agree)
    Heartbeat->>NodeA: Issue Fencing Token / STONITH (Shoot Node A)
    Heartbeat->>NodeB: Promote Node B to Primary (Epoch = 2)
    VIP->>VIP: Remap Virtual IP / DNS Record to Node B
    VIP->>NodeB: Route all subsequent Client Requests
    Note over VIP,NodeB: Automated Zero-Touch Failover Complete (MTTR: 4 seconds)

`Split-Brain Hazards & Defense Mechanisms`

If a network partition isolates Node A and Node B, both nodes might accept concurrent writes, causing permanent data divergence and corruption.

Quorum Consensus: Require (N/2) + 1 nodes (e.g. 2 out of 3) to vote before promoting a new leader
Fencing Tokens: Monotonically increasing epoch numbers; storage rejects writes from older epoch tokens
STONITH ("Shoot The Other Node In The Head"): Power off or disconnect unresponsive nodes via IPMI hardware switches before promoting standbys

4. Disaster Recovery (DR) Strategies (RPO vs RTO)

Disaster Recovery (DR) plans for catastrophic regional outages (earthquakes, data center power loss, cloud region failures). Strategies are governed by two business constraints: RPO and RTO.

Cloud Disaster Recovery Strategy Spectrum (Cost vs Recovery Speed)

System Design

Comparing Backup & Restore, Pilot Light, Warm Standby, and Multi-Region Active-Active

100%

Rendering diagram…

flowchart LR
    subgraph DR Spectrum ["Disaster Recovery Patterns (Cost vs Recovery Speed)"]
        direction LR
        P1["1. Backup & Restore<br/><b>RPO:</b> 24h | <b>RTO:</b> 24h<br/>Lowest Cost ($)"]
        P2["2. Pilot Light<br/><b>RPO:</b> Minutes | <b>RTO:</b> 30m<br/>DB Replicated, App Off ($$)"]
        P3["3. Warm Standby<br/><b>RPO:</b> Seconds | <b>RTO:</b> 5m<br/>Scaled-down Replica ($$$)"]
        P4["4. Multi-Region Active-Active<br/><b>RPO:</b> 0 | <b>RTO:</b> 0<br/>Highest Cost ($$$$)"]

        P1 --> P2 --> P3 --> P4
    end

    classDef p1Style fill:#eff6ff,stroke:#3b82f6,stroke-width:2px,color:#1e40af;
    classDef p2Style fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e;
    classDef p3Style fill:#ecfdf5,stroke:#059669,stroke-width:2px,color:#065f46;
    classDef p4Style fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px,color:#5b21b6;

    class P1 p1Style;
    class P2 p2Style;
    class P3 p3Style;
    class P4 p4Style;

`RPO vs RTO Constraints`

Defining the maximum acceptable loss thresholds.

RPO (Recovery Point Objective): Max acceptable data loss measured in time (e.g. 5 minutes of lost transactions)
RTO (Recovery Time Objective): Max acceptable system downtime before restoration (e.g. 15 minutes of outage)

`The 4 DR Architectural Patterns`

Balancing recovery velocity against infrastructure cost.

Backup & Restore: Regular S3 snapshots; slow recovery (Hours/Days, $)
Pilot Light: Core DB continuously replicated; compute spun up on demand (Minutes, $$)
Warm Standby: Scaled-down replica always running in secondary region (Seconds/Minutes, $$$)
Multi-Site Active-Active: Full traffic served across multiple global regions simultaneously (RPO=0, RTO=0, $$$$)

5. Chaos Engineering & Proactive Resilience Testing

Chaos Engineering is the discipline of experimenting on a system in production to build confidence in its capability to withstand turbulent conditions. Pioneered by Netflix with the Simian Army.

Chaos Engineering Continuous Resilience Feedback Loop

System Design

Hypothesis formulation, canary blast radius containment, fault injection, and architectural hardening

100%

Rendering diagram…

flowchart TD
    Start([1. Define Steady State Metrics]) --> Hypo[2. Formulate Resilience Hypothesis<br/>e.g. 'DB replica crash will not cause >1% HTTP 500 errors']
    Hypo --> Blast[3. Limit Blast Radius<br/>e.g. 5% Canary traffic in staging/prod]
    Blast --> Inject[4. Inject Realistic Chaos Fault<br/>Chaos Monkey: Terminate Pods<br/>Latency Monkey: Inject 800ms delays]
    Inject --> Observe{5. Did Steady State Hold?}
    Observe -- Yes --> Scale[6. Increase Blast Radius & Automate]
    Observe -- No / System Broke --> Fix[7. Uncover System Flaw & Harden Architecture]
    Fix --> Start

    classDef stepStyle fill:#eff6ff,stroke:#2563eb,stroke-width:2px,color:#1e40af;
    classDef alertStyle fill:#fee2e2,stroke:#ef4444,stroke-width:2px,color:#991b1b;
    classDef passStyle fill:#ecfdf5,stroke:#10b981,stroke-width:2px,color:#047857;

    class Start,Hypo,Blast,Inject,Scale stepStyle;
    class Fix alertStyle;
    class Observe passStyle;

`Principles of Chaos Engineering`

Chaos experiments reveal hidden bottlenecks before real production outages strike.

Chaos Monkey: Randomly terminates production container pods / EC2 instances during business hours
Chaos Gorilla: Simulates total Availability Zone (AZ) failure to verify auto-rebalancing
Chaos Kong: Simulates entire AWS Region outage to validate global DNS failover
Blast Radius Containment: Always start experiments on a tiny % of canary traffic before scaling up

6. Graceful Degradation & Load Shedding

When under extreme load or partial infrastructure failure, resilient systems avoid catastrophic total outages by shedding low-priority workloads and falling back to degraded, simplified functionality.

Graceful Degradation & Load Shedding Decision Pipeline

System Design

Prioritizing critical transaction checkout flows while dropping non-essential background workloads

100%

Loading system design canvas…

[
  {
    "id": "deg-client",
    "type": "rectangle",
    "x": 30,
    "y": 115,
    "width": 120,
    "height": 60,
    "strokeColor": "#3b82f6",
    "backgroundColor": "#eff6ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16080,
    "version": 1,
    "versionNonce": 16081,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "deg-client-text"
      }
    ]
  },
  {
    "id": "deg-client-text",
    "type": "text",
    "x": 40,
    "y": 131,
    "width": 100,
    "height": 28,
    "strokeColor": "#1e40af",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16082,
    "version": 1,
    "versionNonce": 16083,
    "isDeleted": false,
    "text": "Client Web App\n(High Traffic Spike)",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Client Web App\n(High Traffic Spike)",
    "containerId": "deg-client"
  },
  {
    "id": "deg-a1",
    "type": "arrow",
    "x": 150,
    "y": 145,
    "width": 70,
    "height": 0,
    "strokeColor": "#3b82f6",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        70,
        0
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16084,
    "version": 1,
    "versionNonce": 16085,
    "isDeleted": false
  },
  {
    "id": "deg-gw",
    "type": "rectangle",
    "x": 220,
    "y": 75,
    "width": 190,
    "height": 140,
    "strokeColor": "#8b5cf6",
    "backgroundColor": "#f3e8ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16086,
    "version": 1,
    "versionNonce": 16087,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "deg-gw-text"
      }
    ]
  },
  {
    "id": "deg-gw-text",
    "type": "text",
    "x": 230,
    "y": 107.5,
    "width": 170,
    "height": 75,
    "strokeColor": "#5b21b6",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16088,
    "version": 1,
    "versionNonce": 16089,
    "isDeleted": false,
    "text": "API Gateway / Load Shedder\n\n- CPU > 85% Trigger\n- Drop non-critical telemetry\n- Prioritize Checkout & Auth\n- Return Cached Fallbacks",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "API Gateway / Load Shedder\n\n- CPU > 85% Trigger\n- Drop non-critical telemetry\n- Prioritize Checkout & Auth\n- Return Cached Fallbacks",
    "containerId": "deg-gw"
  },
  {
    "id": "deg-a2",
    "type": "arrow",
    "x": 410,
    "y": 115,
    "width": 70,
    "height": -30,
    "strokeColor": "#10b981",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        70,
        -30
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16090,
    "version": 1,
    "versionNonce": 16091,
    "isDeleted": false
  },
  {
    "id": "deg-a3",
    "type": "arrow",
    "x": 410,
    "y": 175,
    "width": 70,
    "height": 30,
    "strokeColor": "#ef4444",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "dashed",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        70,
        30
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16092,
    "version": 1,
    "versionNonce": 16093,
    "isDeleted": false
  },
  {
    "id": "deg-crit",
    "type": "rectangle",
    "x": 480,
    "y": 60,
    "width": 380,
    "height": 65,
    "strokeColor": "#10b981",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16094,
    "version": 1,
    "versionNonce": 16095,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "deg-crit-text"
      }
    ]
  },
  {
    "id": "deg-crit-text",
    "type": "text",
    "x": 490,
    "y": 78.5,
    "width": 360,
    "height": 28,
    "strokeColor": "#047857",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16096,
    "version": 1,
    "versionNonce": 16097,
    "isDeleted": false,
    "text": "Core Critical Path: Checkout & Payments\nAlways Processed (100% SLA Maintained)",
    "fontSize": 11,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Core Critical Path: Checkout & Payments\nAlways Processed (100% SLA Maintained)",
    "containerId": "deg-crit"
  },
  {
    "id": "deg-noncrit",
    "type": "rectangle",
    "x": 480,
    "y": 175,
    "width": 380,
    "height": 75,
    "strokeColor": "#ef4444",
    "backgroundColor": "#fee2e2",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16098,
    "version": 1,
    "versionNonce": 16099,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "deg-noncrit-text"
      }
    ]
  },
  {
    "id": "deg-noncrit-text",
    "type": "text",
    "x": 490,
    "y": 198.5,
    "width": 360,
    "height": 28,
    "strokeColor": "#991b1b",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16100,
    "version": 1,
    "versionNonce": 16101,
    "isDeleted": false,
    "text": "Non-Critical Path: Recommendations & Reviews\nGracefully Degraded (Static Placeholder / Cached Data)",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Non-Critical Path: Recommendations & Reviews\nGracefully Degraded (Static Placeholder / Cached Data)",
    "containerId": "deg-noncrit"
  }
]

`Graceful Degradation Patterns`

Isolating failures in non-critical components.

Fallback Caching: Serve stale cache data if backend databases are overwhelmed
Feature Flags / Kill Switches: Remotely toggle off expensive UI features (e.g. "Related Items")
Static Fallback UI: Display informative placeholders instead of raw 500 error pages

`Load Shedding Mechanics`

Dropping requests early at the API Gateway.

CPU / Memory Thresholds: Shed non-essential traffic when CPU exceeds 85%
Priority Queues: Process Payment & Auth requests; reject Search & Analytics with HTTP 503 / 429
LIFO Queueing: Under extreme queues, drop oldest requests first since client has likely timed out

7. Idempotency in APIs & Distributed Transactions

An API operation is idempotent if making multiple identical requests has the exact same side effect on server state as making a single request (f(f(x)) = f(x)). This is vital to prevent double charges during network retries.

Idempotency Key Lifecycle & Atomic Distributed State Locking

System Design

Tracing SETNX mutex locks, in-progress conflict resolution, and cached response replaying

100%

Rendering diagram…

sequenceDiagram
    autonumber
    participant Client as Client Application
    participant GW as API Gateway / Middleware
    participant Redis as Redis Cache (Distributed Lock)
    participant DB as Payments Database

    Client->>GW: POST /charges (Header: Idempotency-Key: uuid-123, Amount: $50)
    GW->>Redis: SETNX idempotency:uuid-123 "IN_PROGRESS" EX 120
    alt Key Already Exists (Duplicate Request)
        Redis-->>GW: Key exists! (Status: "COMPLETED" or "IN_PROGRESS")
        alt Status is COMPLETED
            GW-->>Client: 200 OK (Return Cached Response Payload)
        else Status is IN_PROGRESS
            GW-->>Client: 409 Conflict / 425 Too Early ("Request in processing")
        end
    else Key Did Not Exist (First Request)
        Redis-->>GW: Lock Acquired (1)
        GW->>DB: Process Credit Card Charge ($50)
        DB-->>GW: Transaction Successful (Charge ID: ch_999)
        GW->>Redis: SET idempotency:uuid-123 '{"status": 200, "body": {"id": "ch_999"}}' EX 86400
        GW-->>Client: 200 OK (Charge Created: ch_999)
    end

`HTTP Methods & Idempotency Rules`

Standards defined by RFC 7231.

GET, HEAD, OPTIONS: Safe and Idempotent (Read-only, no state mutation)
PUT: Idempotent (Replaces entire resource; repeated calls produce identical state)
DELETE: Idempotent (Deleting resource X multiple times results in X being gone)
POST / PATCH: Non-Idempotent by default; requires client-generated Idempotency-Key headers (UUIDv4) for safe retries

Production-Grade API Idempotency Middleware with Redis Atomic Locking

Middleware that uses atomic Redis locks to safely deduplicate retried API requests.

8. Timeouts, Retries, Exponential Backoff, Jitter & Circuit Breakers

Uncontrolled retries without backoff create Retry Storms (Thundering Herds) that amplify small blips into catastrophic cascading outages. Resilient architectures combine strict timeouts with Exponential Backoff, Full Jitter, and Circuit Breakers.

Circuit Breaker State Transitions & Exponential Backoff Jitter Mechanics

System Design

Preventing cascading microservice collapse via Closed, Open, and Half-Open states

100%

Loading system design canvas…

[
  {
    "id": "cb-box",
    "type": "rectangle",
    "x": 30,
    "y": 40,
    "width": 830,
    "height": 260,
    "strokeColor": "#2563eb",
    "backgroundColor": "#eff6ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16102,
    "version": 1,
    "versionNonce": 16103,
    "isDeleted": false
  },
  {
    "id": "cb-box-text",
    "type": "text",
    "x": 390,
    "y": 121,
    "width": 180,
    "height": 28,
    "strokeColor": "#1e40af",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16104,
    "version": 1,
    "versionNonce": 16105,
    "isDeleted": false,
    "text": "Circuit Breaker State Machine & Exponential Backoff with Jitter",
    "fontSize": 13,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Circuit Breaker State Machine & Exponential Backoff with Jitter",
    "containerId": "cb-open"
  },
  {
    "id": "cb-closed",
    "type": "rectangle",
    "x": 60,
    "y": 100,
    "width": 200,
    "height": 70,
    "strokeColor": "#10b981",
    "backgroundColor": "#d1fae5",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16106,
    "version": 1,
    "versionNonce": 16107,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "cb-closed-text"
      }
    ]
  },
  {
    "id": "cb-closed-text",
    "type": "text",
    "x": 70,
    "y": 107.5,
    "width": 180,
    "height": 55,
    "strokeColor": "#047857",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16108,
    "version": 1,
    "versionNonce": 16109,
    "isDeleted": false,
    "text": "1. CLOSED (Normal)\n\nAll requests pass through.\nTracks error rate %",
    "fontSize": 11,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "1. CLOSED (Normal)\n\nAll requests pass through.\nTracks error rate %",
    "containerId": "cb-closed"
  },
  {
    "id": "cb-arr1",
    "type": "arrow",
    "x": 260,
    "y": 125,
    "width": 120,
    "height": 0,
    "strokeColor": "#ef4444",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        120,
        0
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16110,
    "version": 1,
    "versionNonce": 16111,
    "isDeleted": false
  },
  {
    "id": "cb-txt1",
    "type": "rectangle",
    "x": 265,
    "y": 95,
    "width": 110,
    "height": 25,
    "strokeColor": "#ef4444",
    "backgroundColor": "#fee2e2",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16112,
    "version": 1,
    "versionNonce": 16113,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "cb-txt1-text"
      }
    ]
  },
  {
    "id": "cb-txt1-text",
    "type": "text",
    "x": 275,
    "y": 93.5,
    "width": 90,
    "height": 28,
    "strokeColor": "#991b1b",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16114,
    "version": 1,
    "versionNonce": 16115,
    "isDeleted": false,
    "text": "Error Rate > 50%",
    "fontSize": 9,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Error Rate > 50%",
    "containerId": "cb-txt1"
  },
  {
    "id": "cb-open",
    "type": "rectangle",
    "x": 380,
    "y": 100,
    "width": 200,
    "height": 70,
    "strokeColor": "#ef4444",
    "backgroundColor": "#fee2e2",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16116,
    "version": 1,
    "versionNonce": 16117,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "cb-box-text"
      },
      {
        "type": "text",
        "id": "cb-open-text"
      }
    ]
  },
  {
    "id": "cb-open-text",
    "type": "text",
    "x": 390,
    "y": 107.5,
    "width": 180,
    "height": 55,
    "strokeColor": "#991b1b",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16118,
    "version": 1,
    "versionNonce": 16119,
    "isDeleted": false,
    "text": "2. OPEN (Failing)\n\nFast-fail immediately!\nZero traffic sent to backend",
    "fontSize": 11,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "2. OPEN (Failing)\n\nFast-fail immediately!\nZero traffic sent to backend",
    "containerId": "cb-open"
  },
  {
    "id": "cb-arr2",
    "type": "arrow",
    "x": 580,
    "y": 125,
    "width": 100,
    "height": 0,
    "strokeColor": "#f59e0b",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        100,
        0
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16120,
    "version": 1,
    "versionNonce": 16121,
    "isDeleted": false
  },
  {
    "id": "cb-txt2",
    "type": "rectangle",
    "x": 585,
    "y": 95,
    "width": 90,
    "height": 25,
    "strokeColor": "#f59e0b",
    "backgroundColor": "#fef3c7",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16122,
    "version": 1,
    "versionNonce": 16123,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "cb-txt2-text"
      }
    ]
  },
  {
    "id": "cb-txt2-text",
    "type": "text",
    "x": 595,
    "y": 93.5,
    "width": 70,
    "height": 28,
    "strokeColor": "#92400e",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16124,
    "version": 1,
    "versionNonce": 16125,
    "isDeleted": false,
    "text": "Sleep Window Expired",
    "fontSize": 8,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Sleep Window Expired",
    "containerId": "cb-txt2"
  },
  {
    "id": "cb-half",
    "type": "rectangle",
    "x": 680,
    "y": 100,
    "width": 160,
    "height": 70,
    "strokeColor": "#f59e0b",
    "backgroundColor": "#fef3c7",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16126,
    "version": 1,
    "versionNonce": 16127,
    "isDeleted": false,
    "boundElements": [
      {
        "type": "text",
        "id": "cb-half-text"
      }
    ]
  },
  {
    "id": "cb-half-text",
    "type": "text",
    "x": 690,
    "y": 116.25,
    "width": 140,
    "height": 37.5,
    "strokeColor": "#92400e",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16128,
    "version": 1,
    "versionNonce": 16129,
    "isDeleted": false,
    "text": "3. HALF-OPEN\n\nSends trial probe reqs",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "3. HALF-OPEN\n\nSends trial probe reqs",
    "containerId": "cb-half"
  },
  {
    "id": "cb-arr3",
    "type": "arrow",
    "x": 760,
    "y": 170,
    "width": -600,
    "height": 50,
    "strokeColor": "#10b981",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        -600,
        50
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16130,
    "version": 1,
    "versionNonce": 16131,
    "isDeleted": false
  },
  {
    "id": "cb-arr4",
    "type": "arrow",
    "x": 760,
    "y": 170,
    "width": -280,
    "height": 50,
    "strokeColor": "#ef4444",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "points": [
      [
        0,
        0
      ],
      [
        -280,
        50
      ]
    ],
    "endArrowhead": "arrow",
    "seed": 16132,
    "version": 1,
    "versionNonce": 16133,
    "isDeleted": false
  },
  {
    "id": "cb-bottom",
    "type": "rectangle",
    "x": 60,
    "y": 205,
    "width": 780,
    "height": 75,
    "strokeColor": "#8b5cf6",
    "backgroundColor": "#f3e8ff",
    "fillStyle": "solid",
    "strokeWidth": 2,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "roundness": {
      "type": 3
    },
    "seed": 16134,
    "version": 1,
    "versionNonce": 16135,
    "isDeleted": false
  },
  {
    "id": "cb-bottom-text",
    "type": "text",
    "x": 70,
    "y": 228.5,
    "width": 760,
    "height": 28,
    "strokeColor": "#5b21b6",
    "backgroundColor": "transparent",
    "fillStyle": "solid",
    "strokeWidth": 1,
    "strokeStyle": "solid",
    "roughness": 0,
    "opacity": 100,
    "seed": 16136,
    "version": 1,
    "versionNonce": 16137,
    "isDeleted": false,
    "text": "Exponential Backoff with Full Jitter Formula: t_sleep = min(t_max, random(0, t_base * 2^attempt))\nPrevents Thundering Herd / Retry Storms from crashing recovering downstream databases.",
    "fontSize": 10,
    "fontFamily": 5,
    "textAlign": "center",
    "verticalAlign": "middle",
    "originalText": "Exponential Backoff with Full Jitter Formula: t_sleep = min(t_max, random(0, t_base * 2^attempt))\nPrevents Thundering Herd / Retry Storms from crashing recovering downstream databases."
  }
]

`Exponential Backoff with Full Jitter`

Avoids synchronized retry spikes.

Formula: t_wait = random(0, min(max_delay, base * 2^attempt))
Jitter randomly spreads retries over time, smoothing network traffic curves
Never retry non-retryable 4xx client errors (e.g. 400 Bad Request, 401 Unauthorized)

`Circuit Breaker Pattern`

Protects failing downstream dependencies.

Closed: Normal operation; requests pass through
Open: Error rate threshold tripped; fast-fails immediately without hitting backend
Half-Open: After sleep window, sends small trial probe traffic to test recovery

Resilient HTTP Client with Exponential Backoff, Full Jitter, and Deadline Timeouts

An HTTP client that retries failed requests with exponential backoff, full jitter, and enforced deadlines.

9. Service Level Framework: SLA, SLO, SLI & Error Budgets

Site Reliability Engineering (SRE) balances system reliability against product feature velocity using the SLI / SLO / SLA governance framework.

SRE Service Level Hierarchy & Error Budget Governance Loop

System Design

Connecting quantitative SLI measurements, internal SLO objectives, and feature deployment velocity

100%

Rendering diagram…

flowchart TD
    SLI[1. SLI: Service Level Indicator<br/>e.g. 99.2% of requests latency < 150ms] --> SLO[2. SLO: Service Level Objective<br/>Internal Target: 99.9% over 30 days]
    SLO --> SLA[3. SLA: Service Level Agreement<br/>Customer Contract: 99.5% with financial penalties]
    
    SLO --> EB[4. Error Budget: 100% - SLO<br/>Available Error Budget = 0.1% = 43 minutes/month]
    
    EB --> Check{Is Error Budget Healthy?}
    Check -- Yes: > 20% Remaining --> Deploy[Fast Feature Velocity & New Deployments]
    Check -- No: Exhausted / High Burn Rate --> Freeze[Deployment Freeze!<br/>100% Focus on Tech Debt & Reliability Hardening]

    classDef normalStyle fill:#eff6ff,stroke:#3b82f6,stroke-width:2px,color:#1e40af;
    classDef successStyle fill:#ecfdf5,stroke:#10b981,stroke-width:2px,color:#047857;
    classDef dangerStyle fill:#fee2e2,stroke:#ef4444,stroke-width:2px,color:#991b1b;

    class SLI,SLO,SLA,EB normalStyle;
    class Deploy successStyle;
    class Freeze dangerStyle;

`SLI vs SLO vs SLA & Error Budget Governance`

Aligning engineering incentives with business guarantees.

SLI (Service Level Indicator): Quantitative measurement in production (e.g. Successful HTTP requests / Total requests = 99.94%)
SLO (Service Level Objective): Target goal set by internal engineering (e.g. 99.9% availability over rolling 30-day window)
SLA (Service Level Agreement): Contractual agreement with external clients containing financial refunds/credits if breached (e.g. 99.5%)
Error Budget = 100% - SLO: The acceptable unreliability room (0.1% = ~43 mins/month). If depleted, all feature shipping stops until stability is restored

Knowledge Check

1. What is the mathematical availability of "Five Nines" (99.999%) in annual downtime?

2. What is the primary difference between RPO and RTO in Disaster Recovery?

3. Why is "Full Jitter" added to exponential retry backoff algorithms?

4. How do idempotent APIs prevent duplicate payments when network timeouts occur?

5. What action should engineering take when an Error Budget is completely exhausted?