Resilience & Observability

A car needs both a spare tire and a dashboard of gauges — one keeps you moving when something fails, the other tells you something’s wrong before you’re stranded.

At a glance

• What it is: resilience patterns (retries, timeouts, circuit breakers, graceful degradation) paired with observability (logging, metrics, tracing) to see and survive partial failure.
• Why it matters: distributed systems fail partially and often — without these, small failures cascade into full outages, invisibly.
• For developers: when calling another service, ask what happens if it’s slow or down — and will I know when that happens?
• When to use it: any call to a dependency you don’t fully control (a third-party API, another team’s service, a database under load).
• When not to use it: don’t add circuit breakers/retries to purely in-process, in-memory calls that can’t fail the way a network call can.
• Prerequisites: Event-Driven & Asynchronous Communication — async patterns are one resilience tool.

Mental model

flowchart LR
    A[Call to dependency] --> B{Healthy?}
    B -- Yes --> C[Normal response]
    B -- No / timeout --> D[Circuit breaker opens]
    D --> E[Fallback / degrade gracefully]
    C --> F[Metrics & traces recorded]
    D --> F
    F --> G[Dashboard & alerts]

Step-by-step walkthrough

1. Set an explicit timeout on every call to an external dependency — an unbounded wait is never acceptable.
2. Add retries with backoff for transient failures, but cap them — endless retries can worsen an outage.
3. Wrap unreliable dependencies in a circuit breaker so repeated failures stop hammering a struggling service and fail fast instead.
4. Define a fallback or degraded response for when the dependency is unavailable, rather than failing the whole request.
5. Instrument the call with metrics (latency, error rate) and logs/traces, and put them on a dashboard with alerting — so failures are visible before customers report them.

Worked example

The logistics-tracking service calls a third-party carrier API that is known to be flaky under load, with no timeout or fallback currently in place.

• Input: an unprotected call to a flaky dependency, with no timeout, fallback, or visibility.
• Process: the team adds a timeout and a circuit breaker around the carrier API call, so repeated failures trip the breaker and return a cached “last known status” fallback instead of hanging; dashboards and alerts are added on the breaker state and response latency.
• Output: when the carrier API degrades on Black Friday, the tracking service keeps responding (with slightly stale data) instead of timing out or crashing, and the on-call engineer sees the breaker trip on the dashboard before customers report a problem.

var breaker = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(exceptionsAllowedBeforeBreaking: 5, durationOfBreak: TimeSpan.FromSeconds(30));

try
{
    return await breaker.ExecuteAsync(() => _carrierClient.GetStatusAsync(trackingId));
}
catch (BrokenCircuitException)
{
    return await _cache.GetLastKnownStatusAsync(trackingId); // graceful degradation
}

Azure Application Insights captures the dependency call’s latency/failures and traces automatically, and an Azure Monitor alert rule fires when the circuit-breaker-open metric crosses a threshold, so on-call is paged before customers notice.

Provisioning that Application Insights instance is itself one Bicep resource:

resource appInsights 'Microsoft.Insights/components@2020-02-02' = {
  name: 'logistics-tracking-ai'
  location: resourceGroup().location
  kind: 'web'
  properties: {
    Application_Type: 'web'
    WorkspaceResourceId: logAnalyticsWorkspace.id
  }
}

and the query the on-call engineer runs to confirm the breaker’s cause is KQL over the dependencies table Application Insights already populates:

dependencies
| where name == "CarrierApi"
| where success == false
| summarize FailureCount = count() by bin(timestamp, 5m)

Resilience extends beyond a single instance to where the data itself lives: Azure SQL Database auto-failover groups and Cosmos DB multi-region replication keep a synchronized copy in a paired region, so a regional outage degrades service rather than stopping it outright — the same Leader and Followers replication pattern behind data-tier scaling, applied here for availability instead of throughput. The full failover-group Bicep and paired-region details live in Business Continuity & Disaster Recovery — this page focuses on the application-level pattern (circuit breaker + fallback), not the infra failover mechanics.

Common use cases

Use case	When to use	Notes
Calling a third-party API known to be unreliable	Circuit breaker + fallback	Fail fast and degrade gracefully rather than hanging
Transient network blips	Retry with backoff, capped	Don’t retry indefinitely — it can worsen an outage
Diagnosing “it’s slow sometimes” reports	Add tracing/metrics before guessing	You need visibility before you can fix what you can’t see

Advanced details

• Circuit breaker states: closed (normal), open (failing fast, not calling the dependency), half-open (testing whether the dependency has recovered).
• Graceful degradation: a system can offer a reduced but still useful response (e.g. cached/last-known data) instead of an all-or-nothing failure.
• The three pillars of observability: logs (what happened), metrics (aggregated numbers over time), and traces (the path of one request across services) — each answers a different question.

Quick reference

Term	Meaning
Circuit breaker	A control that stops calling a failing dependency after repeated failures, failing fast instead
Graceful degradation	Returning a reduced but still useful response instead of a hard failure
Backoff	Increasing the delay between retries to avoid overwhelming a struggling dependency
Observability	The ability to understand a system’s internal state from its external outputs (logs, metrics, traces)
Paired region	An Azure region paired with another for sequenced updates and physical isolation, used as a failover target for replicated data

Troubleshooting

Symptom	Likely cause	Fix
One slow dependency causes total request timeouts	No timeout set on the call	Add an explicit timeout, shorter than the caller’s own budget
An outage gets worse as it happens	Retries have no cap or backoff	Add capped, backed-off retries and a circuit breaker
No one notices a failure until customers complain	No metrics/alerting on the call	Instrument latency/error-rate and alert on thresholds

Check your understanding

• What are the three states of a circuit breaker, and what does each one do?
• Why can uncapped retries make an outage worse instead of better?

Related pages

Part of Introduction to Software & Application Architecture. See also Event-Driven & Asynchronous Communication, Performance & Scalability Engineering, Security by Design, and the Glossary.