Data Lifecycle Management

Data lifecycle management defines what happens to data from creation through active use, archival, and eventual deletion.

At a glance

• What it is: Retention schedules, archival tiering (moving aging data to cheaper storage), and deletion or expiry policies applied per data domain.
• Why it matters: Keeping everything forever is expensive and increases breach risk; deleting too aggressively can break audits, analytics, or legal obligations.
• For developers: “we’ll just keep it in case we need it later” is a governance decision that needs an owner, not a free default to reach for.
• When to use it: For any data with a defined business or legal shelf life — which is most data that isn’t purely ephemeral.
• When not to use it: Ephemeral cache or session data with no retention requirement doesn’t need a lifecycle policy beyond its natural expiry.
• Prerequisites: Data Privacy, Security & Compliance.

Mental model

graph LR
    Create["Create"] --> Active["Active use"]
    Active --> Archive["Archive\n(cold storage)"]
    Archive --> Delete["Delete / expire"]
    Active -.legal hold.-> Hold["Hold\n(lifecycle paused)"]
    Archive -.legal hold.-> Hold

Data normally moves through creation, active use, archival, and deletion — but a legal hold can pause that progression at any stage, and automated deletion must respect that pause.

Step-by-step walkthrough

1. Classify data by its retention requirement (business need, regulatory obligation, or none).
2. Define a retention period per data domain — not a single blanket policy for everything.
3. Implement archival tiering so aging data moves from hot to cheaper cold storage as it’s accessed less.
4. Automate expiry and deletion jobs rather than relying on manual cleanup.
5. Build in a legal hold mechanism that pauses automated deletion for data under investigation or dispute.

Worked example

Northwind’s shipment-tracking events have been accumulating in the lakehouse for years, and storage costs are climbing steadily with no corresponding business benefit.

• Input: raw shipment events retained indefinitely, most of it never queried after the first 90 days.

• Process: define a lifecycle policy — events older than 90 days move to cold storage automatically, and raw event-level data is deleted after 2 years, retaining only pre-computed aggregate summaries beyond that point.

• Output: storage cost drops significantly, historical trend analysis remains possible from the retained summaries, and the policy is documented so nobody has to guess why raw data older than 2 years isn’t there anymore.

  On Azure, this is implemented as an Azure Blob Storage lifecycle management policy rather than custom code. Since the storage account is provisioned via Azure Bicep alongside the rest of the platform, the lifecycle policy is defined as code too, instead of being a portal click-through step someone forgets to redo after the account is recreated:

resource storage 'Microsoft.Storage/storageAccounts@2023-05-01' existing = {
  name: 'northwindlake'
}

resource lifecyclePolicy 'Microsoft.Storage/storageAccounts/managementPolicies@2023-05-01' = {
  parent: storage
  name: 'default'
  properties: {
    policy: {
      rules: [{
        name: 'shipment-events-tiering'
        type: 'Lifecycle'
        definition: {
          filters: { blobTypes: ['blockBlob'], prefixMatch: ['shipment-events/'] }
          actions: {
            baseBlob: {
              tierToCold: { daysAfterModificationGreaterThan: 90 }
              delete: { daysAfterModificationGreaterThan: 730 }
            }
          }
        }
      }]
    }
  }
}

Common use cases

Use case	When to use	Notes
Log/event retention policy	High-volume operational or event data	Tier to cold storage before considering full deletion
Archival for cost reduction	Data still needed occasionally but rarely	Cold/archive tiers trade retrieval speed for lower storage cost
Legal hold management	Data relevant to litigation, audit, or investigation	Must override automated deletion until the hold is lifted
Scheduled deletion jobs	Data with a defined regulatory or business expiry	Automate rather than relying on manual, easy-to-forget cleanup

Advanced details

• Per-domain vs. per-system policy: retention should be defined per data domain (e.g. “transaction records: 7 years”) and then applied consistently across every system holding that domain, not set independently per system.
• Tiered storage trade-offs: cold/archive tiers are cheaper per GB but slower and sometimes costlier to retrieve — factor retrieval frequency and cost into the tiering decision.
• Legal hold precedence: a legal hold must always take precedence over an automated deletion job — design the deletion process to check for holds before acting, not after.
• Audit trail: retain a record of what was deleted, when, and under what policy — this is often itself a compliance requirement.

Quick reference

Term	Meaning
Retention period	The length of time data must or may be kept
Archival tiering	Moving aging data to cheaper, slower storage as it’s accessed less
Cold storage	Low-cost, lower-performance storage for infrequently accessed data
Legal hold	A requirement to preserve data beyond its normal lifecycle due to litigation or audit
Expiry job	An automated process that deletes or archives data once its retention period ends

Troubleshooting

Symptom	Likely cause	Fix
Storage costs growing steadily with no clear driver	No archival or deletion policy in place	Define and automate a retention/tiering policy per data domain
Data deleted that was under legal hold	Deletion job didn’t check for active holds	Add a hold-check step before any automated deletion runs
No record of what was deleted or when	No audit trail for lifecycle actions	Log every automated archival/deletion action with policy reference

Check your understanding

• Why should retention policy be defined per data domain rather than as one blanket rule for an entire system?
• For a dataset you manage, is there a defined point at which it gets archived or deleted — or does it just accumulate indefinitely?

Related pages

Part of Introduction to Data Architecture. See also Data Privacy, Security & Compliance for the obligations that drive many retention rules, Data Storage Architectures for tiered storage options, and the Glossary for term definitions.