RousselTMLEARNING

Observability

The OpenSearch cluster in depth: nodes, shards and replication

Understanding how an OpenSearch cluster works inside: node roles, cluster manager election, shard allocation, write and read paths, document or segment replication, remote storage and allocation troubleshooting.

1. Node roles #

Who does what in the cluster.

Overview #

Node roles
Good to know : Requests are written in the OpenSearch Dashboards Dev Tools console format (GET _cluster/health). On the command line, the same request is curl -sk -u 'admin:Str0ng!Passw0rd-2026' https://localhost:9200/_cluster/health, with -H 'Content-Type: application/json' -d '…' when there is a body.

The roles #

RoleResponsibilityCritical resource
cluster_managerholds the cluster state: indexes, mappings, shard locationsstability, low load
datastores shards, runs writes and searchesdisk, memory, CPU
ingestruns ingest pipelinesCPU
mlruns ML Commons models (embeddings…)native memory, optional GPU
searchserves searchable snapshots and search replicasdisk cache
remote_cluster_clientqueries other clusters (cross-cluster search)network
coordinating only (node.roles: [])receives requests, fans out, merges resultsmemory, CPU

On a small cluster, every node holds all roles. You specialize as load grows: first three dedicated cluster manager nodes, then coordinating nodes in front of dashboards, then ingest or ml nodes.

2. The cluster manager and quorum #

Why three and not two.

Election #

Among eligible nodes, only one is elected cluster manager. Every decision (create an index, move a shard) requires agreement from a majority of eligible nodes: the quorum.

Eligible nodesQuorumTolerated failures
110
220 (no better than one)
321
532
Best practice : Three cluster-manager-eligible nodes are enough in most cases. An even number adds nothing: 4 nodes tolerate one failure, like 3.

The cluster state #

The cluster state is broadcast to every node on each change. Thousands of indexes, fields or shards make it heavy: that is the first reason to limit the shard count (by default 1,000 per data node, cluster.max_shards_per_node).

GET _cat/cluster_manager?v
GET _cluster/state/metadata?filter_path=metadata.cluster_uuid
GET _cluster/stats?filter_path=indices.count,indices.shards.total

3. Shards and replicas #

Allocation, routing and replication.

Allocation #

Shard allocation over three nodes

OpenSearch places shards to balance the shard count per node, never putting a primary and its replica on the same node. If a node fails, a replica is promoted to primary, then new replicas are rebuilt elsewhere.

Write path #

Write path

The target shard is computed from the ID (or the _routing field): this is why the primary count is fixed at creation. The document is written to the translog (durability) and a memory buffer; it becomes searchable at the next refresh (1 s by default).

Document or segment replication #

ModeHow it worksBenefit
Document (historical default)each replica re-indexes the documentreplicas up to date immediately
Segmentthe primary indexes, replicas copy segmentsup to half the CPU on writes
Remote storesegments and translog written to object storagedurability without relying on replicas
PUT logs-app-000010
{
  "settings": {
    "index.number_of_shards": 3,
    "index.number_of_replicas": 1,
    "index.replication.type": "SEGMENT"
  }
}
Good to know : With segment replication, a replica can lag the primary by a few moments: a read right after a write may not see the document.

4. Troubleshooting allocation #

A yellow or red cluster.

Finding the culprit shard #

GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason&s=state

GET _cluster/allocation/explain
{"index": "logs-app-000010", "shard": 0, "primary": false}

The response explains, node by node, why the shard is not placed:

Reported reasonMeaningAction
same_shardthe primary is already on the only nodeadd a node or go to 0 replicas
disk_thresholddisk above the high watermarkfree space, add a node
max_retry5 allocation failuresPOST _cluster/reroute?retry_failed=true
awareness / filterallocation rule (zone, attribute)check node attributes

Commands to know #

Commands to know
  • GET _cat/nodes?v&h=name,node.role,cluster_manager : roles and elected cluster manager
  • GET _cat/shards?v : shard locations
  • GET _cluster/allocation/explain : why a shard is unassigned
  • POST _cluster/reroute?retry_failed=true : retry failed allocations
  • GET _cat/recovery?v&active_only=true : shard copies in progress

Sources #

Checked in November 2026 (OpenSearch 3.x, Data Prepper 2.x): OpenSearch ships a minor release about every six weeks, check the release notes before relying on a detail.

Related articles

Full-text search in OpenSearch: analyzers and relevance

Mastering full-text search: inverted index, analyzers (including French), the _analyze API, match, multi_matc…

Advanced aggregations, PPL and SQL in OpenSearch

Analyzing data in OpenSearch: bucket, metric and pipeline aggregations, date histograms, approximate cardinal…

Advanced Data Prepper: routing, OpenTelemetry and reliability

Going further with Data Prepper: architecture, buffers and workers, conditional processors, routing to severa…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!