Resilience & Error Handling in Integration

A resilient integration assumes the other system will fail sometimes, and decides in advance what happens when it does.

At a glance

• What it is: A set of patterns for handling failure in cross-system calls: retries (try again, usually with backoff), dead-letter queues (DLQ — a parking lot for messages that failed repeatedly, so they don’t block everything behind them), circuit breakers (stop calling a failing dependency for a cooldown period instead of hammering it), and compensating transactions (undo an earlier step if a later one in the same process fails).
• Why it matters: Without these, one slow or failing partner doesn’t just fail its own request — it can cascade into retries piling up, queues backing up, and unrelated features breaking.
• For developers: a call to another system with no timeout, no retry limit, and no circuit breaker is a call that can hang your own service when the other side has a bad day — treat every cross-system call as something that will eventually fail.
• When to use it: Every integration that calls another system or processes messages from a broker needs at least a timeout and a retry policy; circuit breakers and compensating transactions are for higher-stakes or higher-volume paths.
• When not to use it: Don’t add unlimited retries to a call that isn’t idempotent — see Data Synchronization & Consistency first.
• Prerequisites: Messaging & Event Brokers and Data Synchronization & Consistency.

Mental model

graph TD
    Call[Call to Partner API] -->|fails| Retry{Retry with backoff}
    Retry -->|still failing| CB[Circuit breaker opens]
    CB -->|cooldown elapses| Call
    Retry -->|exceeds max attempts| DLQ[Dead-letter queue]
    DLQ --> Review[Manual review / alert]
    Call -->|later step fails| Comp[Compensating transaction]

Each layer catches what the layer before it couldn’t recover from — retries handle transient blips, the circuit breaker handles sustained outages, and the DLQ + compensating transaction handle the cases that truly can’t be auto-recovered.

Step-by-step walkthrough

1. Set an explicit timeout on every cross-system call — never rely on the default (which may be “forever”).
2. Add retries with exponential backoff for transient errors, capped at a maximum attempt count, and only for operations confirmed to be idempotent.
3. Add a circuit breaker around calls to a given dependency so sustained failures stop generating load against it, giving it room to recover.
4. Route messages that exceed the retry limit to a dead-letter queue instead of dropping them or blocking the queue behind them.
5. For multi-step processes (see Orchestration vs. Choreography), define a compensating action for every step that has a side effect, in case a later step fails.
6. Alert on DLQ arrivals and circuit-breaker trips so a human finds out before a customer does.

Worked example

During a Black Friday traffic spike, calls from the checkout system to a partner carrier-rate API start timing out intermittently. Without protection, retries pile up, exhausting connection pools and slowing down unrelated parts of checkout.

• Input: a direct, unprotected call to the carrier API on the critical checkout path, with no timeout cap and no isolation from other checkout features.
• Process: add a short timeout and capped, backoff-based retries to the carrier call; wrap it in a circuit breaker so that after a threshold of failures, checkout stops calling the carrier API for a cooldown period and falls back to a cached/default rate; route calls that still fail after retries to a dead-letter queue for later reprocessing; and add a compensating step so that if a payment succeeds but shipment scheduling ultimately fails, the payment is automatically refunded.
• Output: the carrier outage degrades gracefully — checkout keeps functioning with a fallback rate instead of failing entirely, and no customer is charged for a shipment that never gets scheduled.

var policy = Policy.WrapAsync(
    Policy.Handle<HttpRequestException>().WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(Math.Pow(2, i))),
    Policy.Handle<HttpRequestException>().CircuitBreakerAsync(5, TimeSpan.FromSeconds(30)));

try
{
    return await policy.ExecuteAsync(() => _carrierClient.GetRateAsync(request));
}
catch (BrokenCircuitException)
{
    return _cachedRates.GetDefaultRate(request); // graceful fallback
}

Messages that still fail land in the Azure Service Bus queue’s built-in dead-letter sub-queue, and Azure Monitor alerts when it starts filling up.

Common use cases

Use case	Pattern	Notes
Occasional transient network blips	Retry with backoff	Cap the max attempts; only for idempotent operations
A dependency is down for an extended period	Circuit breaker	Stops piling on load; allows a fallback path
A message keeps failing no matter how many retries	Dead-letter queue	Prevents one bad message from blocking the whole queue
A multi-step process fails partway through	Compensating transaction	Undoes the side effects of already-completed steps

Advanced details

• Backoff strategy: use exponential backoff (with jitter) rather than fixed-interval retries, to avoid synchronized retry storms across many callers.
• Circuit breaker states: closed (normal), open (failing fast, no calls made), half-open (a trial call to check recovery) — know which state a dependency is in when debugging.
• DLQ hygiene: a DLQ is only useful if something monitors it — an unmonitored DLQ is just a silent failure with extra steps.

Quick reference

Term	Meaning
Retry with backoff	Retrying a failed call with increasing delay between attempts
Circuit breaker	Stops calling a failing dependency for a cooldown period
Dead-letter queue (DLQ)	Holds messages that failed processing after retries, for review
Compensating transaction	Undoes an earlier step’s effect after a later step fails

Troubleshooting

Symptom	Likely cause	Fix
One slow dependency slows down unrelated features	No timeout/circuit breaker isolating the call	Add a timeout and circuit breaker specific to that dependency
Retry storm makes an outage worse	Fixed-interval retries with no cap, many callers retrying in sync	Use capped exponential backoff with jitter
Failed messages disappear with no trace	No dead-letter queue configured	Route failed messages to a DLQ and alert on arrivals
Customer charged but order never fulfilled	Missing compensating transaction	Add a compensating refund step for that failure path

Check your understanding

• Why is it unsafe to add unlimited retries to a call that isn’t idempotent?
• In the worked example, what would happen without the circuit breaker even with retries in place?

Related pages

Part of Introduction to Integration Architecture. Builds on Orchestration vs. Choreography and Data Synchronization & Consistency; see also the Glossary for term definitions.