1. Node roles #
Who does what in the cluster.
Overview #
Good to know : Requests are written in the OpenSearch Dashboards Dev Tools console format (GET _cluster/health). On the command line, the same request iscurl -sk -u 'admin:Str0ng!Passw0rd-2026' https://localhost:9200/_cluster/health, with-H 'Content-Type: application/json' -d '…'when there is a body.
The roles #
| Role | Responsibility | Critical resource |
|---|---|---|
| cluster_manager | holds the cluster state: indexes, mappings, shard locations | stability, low load |
| data | stores shards, runs writes and searches | disk, memory, CPU |
| ingest | runs ingest pipelines | CPU |
| ml | runs ML Commons models (embeddings…) | native memory, optional GPU |
| search | serves searchable snapshots and search replicas | disk cache |
| remote_cluster_client | queries other clusters (cross-cluster search) | network |
coordinating only (node.roles: []) | receives requests, fans out, merges results | memory, CPU |
On a small cluster, every node holds all roles. You specialize as load grows: first three dedicated cluster manager nodes, then coordinating nodes in front of dashboards, then ingest or ml nodes.
2. The cluster manager and quorum #
Why three and not two.
Election #
Among eligible nodes, only one is elected cluster manager. Every decision (create an index, move a shard) requires agreement from a majority of eligible nodes: the quorum.
| Eligible nodes | Quorum | Tolerated failures |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 (no better than one) |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
Best practice : Three cluster-manager-eligible nodes are enough in most cases. An even number adds nothing: 4 nodes tolerate one failure, like 3.
The cluster state #
The cluster state is broadcast to every node on each change. Thousands of indexes, fields or shards make it heavy: that is the first reason to limit the shard count (by default 1,000 per data node, cluster.max_shards_per_node).
GET _cat/cluster_manager?v
GET _cluster/state/metadata?filter_path=metadata.cluster_uuid
GET _cluster/stats?filter_path=indices.count,indices.shards.total3. Shards and replicas #
Allocation, routing and replication.
Allocation #
OpenSearch places shards to balance the shard count per node, never putting a primary and its replica on the same node. If a node fails, a replica is promoted to primary, then new replicas are rebuilt elsewhere.
Write path #
The target shard is computed from the ID (or the _routing field): this is why the primary count is fixed at creation. The document is written to the translog (durability) and a memory buffer; it becomes searchable at the next refresh (1 s by default).
Document or segment replication #
| Mode | How it works | Benefit |
|---|---|---|
| Document (historical default) | each replica re-indexes the document | replicas up to date immediately |
| Segment | the primary indexes, replicas copy segments | up to half the CPU on writes |
| Remote store | segments and translog written to object storage | durability without relying on replicas |
PUT logs-app-000010
{
"settings": {
"index.number_of_shards": 3,
"index.number_of_replicas": 1,
"index.replication.type": "SEGMENT"
}
}Good to know : With segment replication, a replica can lag the primary by a few moments: a read right after a write may not see the document.
4. Troubleshooting allocation #
A yellow or red cluster.
Finding the culprit shard #
GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason&s=state
GET _cluster/allocation/explain
{"index": "logs-app-000010", "shard": 0, "primary": false}The response explains, node by node, why the shard is not placed:
| Reported reason | Meaning | Action |
|---|---|---|
| same_shard | the primary is already on the only node | add a node or go to 0 replicas |
| disk_threshold | disk above the high watermark | free space, add a node |
| max_retry | 5 allocation failures | POST _cluster/reroute?retry_failed=true |
| awareness / filter | allocation rule (zone, attribute) | check node attributes |
Commands to know #
Commands to know
GET _cat/nodes?v&h=name,node.role,cluster_manager: roles and elected cluster managerGET _cat/shards?v: shard locationsGET _cluster/allocation/explain: why a shard is unassignedPOST _cluster/reroute?retry_failed=true: retry failed allocationsGET _cat/recovery?v&active_only=true: shard copies in progress
Sources #
Checked in November 2026 (OpenSearch 3.x, Data Prepper 2.x): OpenSearch ships a minor release about every six weeks, check the release notes before relying on a detail.
