RousselTMLEARNING

Observability

Observability with OpenSearch: traces, logs and metrics

Turning OpenSearch into an observability platform: Trace Analytics and service map, PPL logs, Prometheus metrics, traceId correlation, investigation notebooks and Query Insights to spot costly queries.

1. One console, three signals #

What OpenSearch offers.

Overview #

Observability in OpenSearch
SignalIngestionUse
TracesOTLP → Data Prepper (otel_traces)Trace Analytics: services, latencies, errors, map
LogsFluent Bit, OTel, Data PrepperDiscover, PPL, patterns
MetricsOTel indexes or external Prometheus sourcePromQL or PPL, visualizations

2. Trace Analytics #

Following a request end to end.

The views #

  • Services: per service, average latency, error rate and throughput; sort to find the slowest.
  • Service map: who calls whom, coloured by latency or errors.
  • Traces: filterable list, then a Gantt chart of a trace's spans.

Spans are ordinary documents: you can query them directly.

source=otel-v1-apm-span-* | where status.code = 2 | stats count() as erreurs by serviceName, name | sort - erreurs | head 10

source=otel-v1-apm-span-* | where serviceName = 'paiement' | stats avg(durationInNanos) / 1000000 as ms_moyen by span(startTime, 5m)
Good to know : status.code = 2 is the error status in OpenTelemetry. Field names depend on the Data Prepper version: check them with GET otel-v1-apm-span-*/_mapping.

3. Correlating #

From symptom to cause.

An investigation #

A typical investigation

The key to correlation is the traceId: if it is written in every log line (OpenTelemetry SDKs do it automatically), you go from a slow trace to its logs in one click, or with a query.

source=otel-logs-* | where traceId = '4bf92f3577b34da6a3ce929d0e0e4736' | fields @timestamp, severityText, body | sort @timestamp
Best practice : Normalize service names everywhere (OpenTelemetry service.name): logs, traces and metrics then join without a mapping table.

Prometheus metrics #

OpenSearch can query an existing Prometheus server as a data source, without copying metrics: you visualize them next to logs and traces.

POST _plugins/_query/_datasources
{
  "name": "prometheus_prod",
  "connector": "prometheus",
  "properties": {"prometheus.uri": "http://prometheus:9090"}
}

4. Query Insights #

The queries that cost the most.

Enabling and reading #

PUT _cluster/settings
{
  "persistent": {
    "search.insights.top_queries.latency.enabled": true,
    "search.insights.top_queries.latency.window_size": "10m",
    "search.insights.top_queries.latency.top_n_size": 10
  }
}

GET _insights/top_queries?type=latency

Query Insights ranks the slowest queries (or the most CPU- and memory-hungry, with type=cpu and type=memory) over a sliding window, with their shape, index and origin. It is the starting point for optimizing a slow dashboard.

Commands to know #

Commands to know
  • GET otel-v1-apm-span-*/_count : number of spans received
  • GET _insights/top_queries?type=latency : slowest queries
  • POST _plugins/_query/_datasources : declare a Prometheus source
  • source=otel-logs-* | where traceId = '…' : logs of a trace

Sources #

Checked in November 2026 (OpenSearch 3.x, Data Prepper 2.x): OpenSearch ships a minor release about every six weeks, check the release notes before relying on a detail.

Related articles

The OpenSearch cluster in depth: nodes, shards and replication

Understanding how an OpenSearch cluster works inside: node roles, cluster manager election, shard allocation,…

Full-text search in OpenSearch: analyzers and relevance

Mastering full-text search: inverted index, analyzers (including French), the _analyze API, match, multi_matc…

Advanced aggregations, PPL and SQL in OpenSearch

Analyzing data in OpenSearch: bucket, metric and pipeline aggregations, date histograms, approximate cardinal…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!