Performance & Scalability Engineering

Design for the highway at rush hour, not the empty road on a quiet Sunday morning.

At a glance

• What it is: designing an application to meet performance targets under realistic (and peak) load — caching, async processing, horizontal scaling, load balancing, and performance budgets.
• Why it matters: an app that works fine in a demo can fall over the first time real traffic arrives, at the worst possible moment.
• For developers: for any new endpoint, ask what happens at 10x current load — and can this scale out, not just up?
• When to use it: for any system with unpredictable or growing traffic, and always before a known high-traffic event.
• When not to use it: don’t add scaling complexity (e.g. premature caching layers) for a system with small, predictable load — that’s needless operational cost.
• Prerequisites: Non-Functional Requirements (see Solution Architecture) define the actual targets this topic engineers toward.

Mental model

flowchart LR
    A[Traffic spike] --> B{Can it scale out?}
    B -- Yes: stateless + autoscaling --> C[More instances handle load]
    B -- No: stateful/bottlenecked --> D[Requests queue or fail]
    C --> E[Cache absorbs repeat reads]
    E --> F[Backend load reduced]

Step-by-step walkthrough

1. Establish the performance target first (e.g. p95 latency, requests/sec at peak) — see Non-Functional Requirements.
2. Identify what’s on the critical path of a request and measure it before optimizing blindly.
3. Add caching for expensive, frequently repeated reads; invalidate deliberately, not by guesswork.
4. Design for horizontal scaling: keep request handling stateless so more instances can be added under load.
5. Move non-essential work off the request path with async processing (see Event-Driven & Asynchronous Communication).
6. Load-test against the actual target before a known peak event, not just functionally test.

Worked example

The logistics-tracking API normally handles modest traffic but expects roughly 20x its normal load on Black Friday.

• Input: a service sized for everyday traffic facing a projected 20x spike on a known date.
• Process: the team adds response caching for tracking-status reads (which change infrequently relative to how often they’re polled), makes the service stateless so it can autoscale horizontally, and load-tests at the projected peak weeks in advance.
• Output: the service absorbs the spike by adding instances automatically and serving most reads from cache, instead of every request hitting the backend directly.

public async Task<TrackingStatus> GetStatusAsync(string trackingId)
{
    var cacheKey = $"tracking:{trackingId}";
    if (await _cache.GetAsync<TrackingStatus>(cacheKey) is { } cached)
        return cached; // served from cache, backend untouched

    var status = await _backend.GetStatusAsync(trackingId);
    await _cache.SetAsync(cacheKey, status, TimeSpan.FromSeconds(30));
    return status;
}

On Azure, _cache is backed by Azure Cache for Redis (shared across instances, unlike an in-memory cache), and the service itself runs on Azure Container Apps or AKS with autoscaling rules tied to request rate/CPU so instance count grows automatically ahead of the projected 20x spike.

If the logistics-tracking API instead runs on Azure App Service, the same elastic response is a Bicep autoscale rule that scales the Web App out (more instances) rather than up (a bigger plan tier):

resource autoscale 'Microsoft.Insights/autoscalesettings@2022-10-01' = {
  name: 'logistics-tracking-autoscale'
  location: resourceGroup().location
  properties: {
    targetResourceUri: appServicePlan.id
    enabled: true
    profiles: [
      {
        name: 'CPU-based scale-out'
        capacity: { minimum: '2', maximum: '10', default: '2' }
        rules: [
          {
            metricTrigger: {
              metricName: 'CpuPercentage'
              metricResourceUri: appServicePlan.id
              timeGrain: 'PT1M'
              statistic: 'Average'
              timeWindow: 'PT5M'
              timeAggregation: 'Average'
              operator: 'GreaterThan'
              threshold: 70
            }
            scaleAction: { direction: 'Increase', type: 'ChangeCount', value: '1', cooldown: 'PT5M' }
          }
        ]
      }
    ]
  }
}

The same up-vs-out choice applies one layer down, at the data tier: Azure SQL Database scales up via vCore/DTU tier and scales out reads via read replicas; Azure Cosmos DB scales throughput elastically with autoscale RU/s and can add regions for both read and (with multi-region writes) write scaling; Azure Database for PostgreSQL Flexible Server adds read replicas for read-heavy workloads. None of this requires application code changes as long as the service stays stateless.

Common use cases

Use case	When to use	Notes
Known high-traffic event (sale, launch)	Weeks before, via load testing	Test at the actual projected peak, not an arbitrary number
Expensive, frequently repeated read	Add caching	Define an explicit invalidation strategy up front
Traffic growth over time	Design stateless services from the start	Retrofitting statelessness later is far more expensive

Advanced details

• Scale up vs. scale out: scaling up (a bigger machine) has a ceiling; scaling out (more machines) requires the service to be stateless or to externalize state (e.g. to a shared cache/database).
• Data-tier scaling mirrors the app tier’s up/out choice, on its own schedule: a bigger Azure SQL Database vCore tier (up) or an added read replica (out); Cosmos DB autoscale RU/s absorbs bursts automatically; a PostgreSQL Flexible Server read replica offloads read traffic from the primary. Martin Fowler’s Leader and Followers and Follower Reads patterns describe the mechanism underneath all three: reads are served from replicas (“followers”) to raise throughput without touching the write path.
• Cache invalidation is the hard part: a cache with no invalidation strategy silently serves stale data — decide TTL or event-driven invalidation deliberately.
• Performance budget: setting a target (e.g. “p95 under 300ms”) before writing code prevents “we’ll optimize later” from becoming “we never did.”
• This is the Twelve-Factor App in practice: making the logistics-tracking service stateless so it can scale out is Factor VI (Processes — execute as stateless processes); externalizing state to a shared cache/database is Factor IV (Backing Services — treat them as attached resources); and instrumenting the service for the dashboards mentioned in Resilience & Observability follows Factor XI (Logs — treat logs as event streams, not files to manage). The Twelve-Factor App names the rest of these conventions if a new service needs the full checklist.

Quick reference

Term	Meaning
Scale up	Increasing the resources of a single instance (more CPU/RAM)
Scale out	Adding more instances to share the load
Stateless service	A service that keeps no request-specific state between calls, so any instance can handle any request
Cache invalidation	Deciding when cached data is no longer valid and must be refreshed
Performance budget	A target metric (e.g. p95 latency) set before implementation, not after
Twelve-Factor App	A named set of conventions (stateless processes, externalized backing services, logs as event streams, etc.) for building horizontally scalable services
Autoscale rule	An Azure Monitor rule that changes instance count based on a metric threshold (e.g. CPU > 70%)
RU/s	Request Units per second — Cosmos DB’s elastic throughput scaling unit
Read replica	A read-only, kept-in-sync copy of a database used to scale out read traffic

Troubleshooting

Symptom	Likely cause	Fix
Adding more instances doesn’t help under load	Service holds request-specific state (e.g. in-memory session)	Externalize state to a shared store; make instances interchangeable
Cache serves outdated data after an update	No invalidation strategy	Add explicit TTL or event-driven cache invalidation
System fails only during a known peak event	No load test was run at the actual projected peak	Load-test at target scale before the event, not after an incident

Check your understanding

• Why doesn’t adding more servers help if the service keeps state in memory between requests?
• What’s the risk of caching a frequently changing value without an invalidation strategy?

Related pages

Part of Introduction to Software & Application Architecture. See also Event-Driven & Asynchronous Communication, Resilience & Observability, Security by Design, and the Glossary. For the operational/infra side of scaling (quotas, load balancers, region capacity), see Capacity & Scalability Planning.