Infrastructure Observability & Monitoring

Infrastructure Observability & Monitoring watches the health of the platform itself — compute, network, storage, and node-level signals — so problems are caught before they become application-visible outages.

At a glance

• What it is: metrics, logs, health probes, and alerting focused on the infrastructure layer (CPU, memory, disk, network throughput, node/cluster health), separate from application-level telemetry.
• Why it matters: infrastructure exhaustion — a saturated network link, a full disk, a throttled CPU — is often the earliest warning sign, appearing before application error rates move at all.
• For developers: if your dashboards only show HTTP status codes and app logs, you’re missing the layer where problems usually start — ask what infra metrics exist for the platform your service runs on.
• When to use it: for every environment beyond a disposable sandbox.
• When not to use it: it always applies, though the depth of alerting can scale with the workload’s criticality.
• Prerequisites: Compute & Container Platforms.

Mental model

graph TD
    Infra["Infra metrics: CPU, memory, disk, network, node health"] --> Alerts["Alerting rules"]
    App["App metrics: error rate, latency"] --> Alerts
    Alerts --> OnCall["On-call notified"]
    Infra -. often triggers first .-> App

Step-by-step walkthrough

1. Identify the infra-layer signals that matter for the platform in use (node CPU/memory, disk pressure, network throughput, container restart counts).
2. Set alert thresholds based on early-warning levels, not just “resource is completely exhausted.”
3. Correlate infra alerts with application dashboards so on-call can quickly tell which layer a problem originates in.
4. Route infra alerts to whoever owns the platform, distinct from — but coordinated with — the application on-call rotation.
5. Review and tune thresholds periodically; static thresholds set once tend to become noisy or blind as workloads change.

Worked example

During Northwind’s Black Friday traffic spike, application dashboards for the payment gateway showed no elevated error rate, yet customers reported slow checkouts.

• Input: normal-looking application error-rate and latency dashboards, but real customer-facing slowness.
• Process: after the incident, the team discovers the payment gateway’s underlying compute pool was CPU-throttled and its network link was saturated — neither of which had any alerting configured, since monitoring had only ever been set up for application-level error rates and latency percentiles.
• Output: infra-level alerts are added for CPU throttling, network saturation, and node health, wired to the same on-call rotation; the next high-traffic event surfaces the resource pressure as an early warning well before it affects customer-facing latency. On Azure, this means enabling Azure Monitor Container Insights (or VM insights) for node/pod-level CPU, memory, and network metrics, with alert rules feeding the same on-call rotation as the application’s Application Insights alerts.

resource appInsights 'Microsoft.Insights/components@2020-02-02' = {
  name: 'checkout-appinsights'
  location: location
  kind: 'web'
  properties: {
    Application_Type: 'web'
    WorkspaceResourceId: logAnalyticsWorkspace.id
  }
}

Correlating infra and app telemetry on one timeline (the point below) means querying both from the same Log Analytics workspace:

union AppRequests, AppExceptions
| where TimeGenerated > ago(1h)
| summarize count() by bin(TimeGenerated, 5m), ItemType

Common use cases

Use case	When to use	Notes
Diagnosing “slow but no errors” incidents	Infra metrics often explain what app metrics can’t	Correlate infra and app dashboards on the same timeline
Capacity planning input	Historical infra utilization trends	Feeds into Capacity & Scalability Planning forecasts
Platform-wide health dashboard	Aggregate node/cluster health across all workloads	Gives platform teams a single view independent of any one application

Advanced details

• The four golden signals (latency, traffic, errors, saturation) apply at the infra layer too — saturation in particular is the signal most often missing from application-only monitoring.
• Container/Kubernetes environments need node-level and pod-level signals — a healthy pod can still be starved by node-level resource pressure.
• Correlating infra and app telemetry on a shared timeline — not just side-by-side dashboards — is what actually shortens root-cause time during an incident.

Quick reference

Term	Meaning
Infra metrics	CPU, memory, disk, network, and node/cluster health signals
Saturation	How “full” a resource is relative to its capacity
Health probe	An automated check of whether an instance/node is healthy
Alert threshold	The level at which a metric triggers a notification
Golden signals	Latency, traffic, errors, saturation — the four core signal categories

Troubleshooting

Symptom	Likely cause	Fix
Customers report slowness but app error-rate dashboards look normal	No infra-level (CPU/network/disk) monitoring or alerting exists	Add infra metrics and alerts, correlated with app dashboards
Alerts fire constantly and get ignored	Thresholds set once and never tuned as the workload grew	Review and retune thresholds periodically against real historical baselines
A pod is unhealthy despite the node showing normal averages	Monitoring only at node level, missing pod-level resource pressure	Add pod/container-level metrics alongside node-level ones

Check your understanding

• Why might an application show a normal error rate while users still experience slowness?
• What’s the difference between monitoring at the node level versus the pod/container level, and why do you need both in a container platform?

Related pages

Part of Introduction to Cloud & Infrastructure Architecture. See also Capacity & Scalability Planning, Compute & Container Platforms, Software & Application Architecture’s Resilience & Observability, and the Glossary.