Business Continuity & Disaster Recovery

Business Continuity & Disaster Recovery plans for losing an entire region or major dependency — backups, failover, and recovery targets, tested in advance rather than assumed.

At a glance

• What it is: backup strategy, defined RTO (Recovery Time Objective) and RPO (Recovery Point Objective) targets, multi-region failover design, and regularly tested DR drills.
• Why it matters: an untested DR plan is really just an assumption — the only way to know a failover works is to have actually run it.
• For developers: know your service’s actual RTO/RPO before an incident, not during one — “we’ll figure it out when it happens” is not a recovery plan.
• When to use it: for any workload where an outage has a real business or customer impact.
• When not to use it: a truly disposable internal tool with no dependents may not need a formal DR plan — but say so explicitly rather than by default.
• Prerequisites: Network Architecture & Connectivity, Infrastructure Observability & Monitoring.

Mental model

graph LR
    Primary["Primary region"] -- replicate --> Secondary["Secondary region"]
    Primary -- fails --> Failover["Failover triggered"]
    Failover --> Secondary
    Secondary -- serves traffic within RTO --> Users["Users"]

Step-by-step walkthrough

1. Define RTO (how quickly service must be restored) and RPO (how much data loss is acceptable) per workload — these drive every other DR decision.
2. Choose a backup and replication strategy that can actually meet the defined RPO (e.g., continuous replication vs. nightly backup).
3. Design the failover mechanism — manual or automated — to a secondary region or standby environment.
4. Document the failover runbook, including who has authority to trigger it.
5. Run a real DR drill on a schedule, and treat any gap it surfaces as a defect to fix, not a footnote.

Worked example

Northwind’s payment gateway has a defined RTO of 15 minutes and RPO of 1 minute, backed by a secondary region with continuous data replication — but the failover had never actually been executed end-to-end.

• Input: documented RTO/RPO targets and a standby secondary region, with no prior real failover test.
• Process: the team schedules a DR drill simulating a full primary-region outage, follows the documented failover runbook to redirect traffic to the secondary region, and measures actual recovery time and data loss against the targets.
• Output: the drill succeeds within 12 minutes — inside the 15-minute RTO — but surfaces that a DNS TTL setting was slowing failover more than expected, and that a manual approval step wasn’t clearly assigned to a specific on-call role. Both are fixed before the next drill, and the plan is now proven rather than assumed. When a real regional outage occurs months later, the team executes the same runbook with confidence. Concretely, this is Azure SQL Database active geo-replication to the secondary region plus Azure Front Door (or Traffic Manager) redirecting traffic on failover — the DNS TTL issue the drill found was Front Door’s endpoint health-probe interval, not a client-side cache.

resource failoverGroup 'Microsoft.Sql/servers/failoverGroups@2023-08-01' = {
  parent: primarySqlServer
  name: 'checkout-failover-group'
  properties: {
    partnerServers: [ { id: secondarySqlServer.id } ]
    readWriteEndpoint: { failoverPolicy: 'Automatic', failoverWithDataLossGracePeriodMinutes: 60 }
    databases: [ checkoutDb.id ]
  }
}

Common use cases

Use case	When to use	Notes
Regulated or customer-facing critical service	Formal RTO/RPO with multi-region failover	Justifies the added cost of standby infrastructure
Internal low-impact tool	Backup-only, no active failover	Restore-from-backup is an acceptable recovery path
Regularly scheduled DR drills	Simulated region-failure exercises	The only reliable way to validate a DR plan actually works

Advanced details

• Active-active (both regions serving traffic) versus active-passive (secondary on standby) failover models trade cost against complexity and recovery speed — active-active gives near-zero RTO but costs more and requires handling data consistency across both regions.
• “Active-passive” actually covers two distinct named strategies: the Well-Architected Reliability pillar’s DR guidance names four DR strategies by increasing cost and decreasing RTO/RPO — Backup and Restore (cheapest, slowest: restore from backups into a new environment), Pilot Light (a minimal, always-on copy of core infrastructure that’s scaled up on failover), Warm Standby (a scaled-down but fully functional secondary always running), and Multi-Site Active/Active (both regions fully serving traffic, matching this page’s “active-active”). Naming which of these four a design actually uses is more precise than the generic active-active/active-passive split.
• DNS-based failover has to account for client-side and resolver caching (TTL) — a low TTL that isn’t actually honored by all clients can silently extend real-world recovery time beyond the measured RTO.
• Backup restore tests should be run on a schedule, separate from full DR drills, since a backup that can’t actually be restored is discovered too late otherwise.
• Region choice isn’t arbitrary: Azure groups regions into paired regions (e.g. East US ↔ West US, North Europe ↔ West Europe) that get platform updates sequenced one at a time and are physically isolated from each other — picking a workload’s secondary region from its documented pair, rather than any region with capacity, is what the Well-Architected Reliability pillar’s DR strategies above actually assume.

Quick reference

Term	Meaning
RTO	Recovery Time Objective — target time to restore service after an outage
RPO	Recovery Point Objective — maximum acceptable data loss, measured in time
Active-active	Both regions serve traffic simultaneously
Active-passive	Secondary region stands by, activated on failover
DR drill	A scheduled, real exercise of the failover process

Troubleshooting

Symptom	Likely cause	Fix
Failover takes far longer than the documented RTO	DNS TTL or manual approval steps not accounted for	Identify and fix each real-world delay found during a drill, then re-test
Restored backup is missing recent data	RPO not actually met by the backup/replication frequency in use	Increase replication/backup frequency to match the defined RPO
Nobody knows who has authority to trigger failover during a real incident	DR runbook doesn’t assign a clear decision-maker	Name a specific on-call role with failover authority in the runbook

Check your understanding

• Why is an untested DR plan effectively just an assumption?
• What’s the practical difference between RTO and RPO, and why do they need separate targets?

Related pages

Part of Introduction to Cloud & Infrastructure Architecture. See also Network Architecture & Connectivity, Infrastructure Observability & Monitoring, and the Glossary.