Machine learning: anomaly detection and AIOps

1. Machine learning features #

Overview.

What Elastic offers #

Good to know: Anomaly detection and data frame analytics require a Platinum or Enterprise licence (or the 30-day trial) and nodes with the ml role.

Licence and trial #

GET _license                                   # type et date d'expiration
POST _license/start_trial?acknowledge=true     # essai de 30 jours, une seule fois par version majeure

Common mistake: Building production monitoring on the trial: when it expires, the cluster falls back to the Basic licence and anomaly detection jobs stop. Plan the subscription, or use threshold alert rules (free) for critical needs.

Depending on the version: the exact split of features between licences changes over time (AIOps, trained models): check Elastic's Subscriptions page for your version.

2. Creating a job #

Detectors and datafeed.

Concepts #

PUT _ml/anomaly_detectors/latence-services
{
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [ { "function": "high_mean", "field_name": "transaction.duration.us", "partition_field_name": "service.name" } ],
    "influencers": ["service.name", "host.name"]
  },
  "data_description": { "time_field": "@timestamp" }
}
PUT _ml/datafeeds/datafeed-latence-services
{ "job_id": "latence-services", "indices": ["traces-apm*"] }
POST _ml/anomaly_detectors/latence-services/_open
POST _ml/datafeeds/datafeed-latence-services/_start

3. Reading and using results #

Score, explorer, alerts.

In Kibana #

• Anomaly Explorer: view by influencer and period, colours by score (0 to 100).
• Single Metric Viewer: the actual curve, the expected-behaviour band, anomalies and forecasts (_forecast).
• Anomaly detection alert rule: notify above a score (for example 75) through a connector.

GET _ml/anomaly_detectors/latence-services/results/records?sort=record_score&desc=true&size=5
POST _ml/anomaly_detectors/latence-services/_forecast
{ "duration": "1d" }

Good practice: Let the model learn over several cycles (at least two weeks for a weekly rhythm) before alerting, and start with a high threshold.

Common mistake: Partitioning on a very high-cardinality field (session ID, full URL): model memory explodes and results become unreadable. Partition by service or host, and keep the rest as influencers.

4. Ready-made jobs #

Observability and Security.

Wizards #

Observability apps (APM, logs, infrastructure) and Security offer preconfigured jobs: abnormal APM latency, unusual log rate, rare processes on a host. Elastic Security also uses them as machine learning detection rules.