RousselTMLEARNING

Observability

Index Lifecycle Management (ILM): data tiers, rollover and retention

Automate the index lifecycle with ILM: hot, warm, cold and frozen data tiers, rollover by size and age, shrink, force merge, searchable snapshots, deletion and diagnosis with explain.

1. The phases #

Data that ages.

Lifecycle #

Phases of an ILM policy
PhaseTypical actionsNodes
hotrollover, high prioritydata_hot (SSD)
warmshrink, forcemerge, read-onlydata_warm
coldfewer replicas, searchable snapshot (depending on license)data_cold
frozenpartial searchable snapshotdata_frozen + object repository
deletedeletion, possibly after a snapshot-

2. A policy #

In practice.

Policy and diagnosis #

PUT _ilm/policy/logs-30d
{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" } } },
      "warm":   { "min_age": "7d",  "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 } } },
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}

GET logs-nginx-default/_ilm/explain        # phase et étape de chaque index, erreurs éventuelles
Good to know: min_age counts from the index rollover, not from its creation: a « 7-day » index in warm was written for 1 day and then aged 7 days.
Common mistake: Without rollover, the age of a huge index does not change: ILM does nothing useful. Rollover (via a data stream or a write alias) is the foundation of everything else.
Good to know: Some features depend on the version and the license level (Basic, Platinum, Enterprise): check them in the documentation and on the Elastic subscriptions page before using them in production.

Related articles

Transforms and enrich: summarising and enriching data in Elasticsearch

Build entity-centric indices with pivot and latest transforms, and enrich documents at ingest time with enric…

OpenTelemetry and Elasticsearch: compatibility and architecture

Send OpenTelemetry to Elasticsearch: Collector and elasticsearch exporter, managed OTLP endpoint, APM Server,…

Ingesting OpenTelemetry logs into Elasticsearch

Collect and structure logs with the OpenTelemetry Collector (filelog receiver, operators, transform processor…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!