RousselTMLEARNING

Observability

Full-text search in OpenSearch: analyzers and relevance

Mastering full-text search: inverted index, analyzers (including French), the _analyze API, match, multi_match, phrase and fuzzy queries, BM25 scoring, highlighting, pagination with search_after and point in time.

1. The inverted index #

Why search is fast.

Principle #

Inverted index

Like a book index, the inverted index maps each term to the list of documents containing it. Searching "refus" means reading one entry, whatever the index size. Everything therefore depends on how text is split into terms: that is the analyzer's job.

Good to know : Requests are written in the OpenSearch Dashboards Dev Tools console format (GET _cluster/health). On the command line, the same request is curl -sk -u 'admin:Str0ng!Passw0rd-2026' https://localhost:9200/_cluster/health, with -H 'Content-Type: application/json' -d '…' when there is a body.

2. Analyzers #

From text to terms.

Anatomy #

Anatomy of an analyzer
StepRoleExamples
char_filterchange raw texthtml_strip, mapping, pattern_replace
tokenizersplit into tokensstandard, whitespace, keyword, ngram
filtertransform tokenslowercase, asciifolding, elision, stop, stemmer, synonym

Testing with _analyze #

POST _analyze
{"analyzer": "standard", "text": "L'Été à Paris"}

POST _analyze
{"analyzer": "french", "text": "L'Été à Paris"}
AnalyzerResulting terms
standardl'été, à, paris
frenchete, pari

The french analyzer removes elision (l'), stop words (à) and reduces words to their stem: "paiements" and "paiement" yield the same term.

A custom analyzer #

PUT produits-v1
{
  "settings": {
    "analysis": {
      "filter": {
        "fr_elision": {"type": "elision", "articles_case": true, "articles": ["l", "d", "j", "qu", "n", "s", "c", "m", "t"]},
        "fr_stem":    {"type": "stemmer", "language": "light_french"},
        "synonymes":  {"type": "synonym", "synonyms": ["pc, ordinateur", "tv, téléviseur"]}
      },
      "analyzer": {
        "francais_sans_accents": {
          "tokenizer": "standard",
          "filter": ["fr_elision", "lowercase", "asciifolding", "synonymes", "fr_stem"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "nom":   {"type": "text", "analyzer": "francais_sans_accents", "fields": {"exact": {"type": "keyword"}}},
      "description": {"type": "text", "analyzer": "francais_sans_accents"},
      "prix":  {"type": "float"},
      "stock": {"type": "integer"}
    }
  }
}
Common mistake : A field's analyzer cannot be changed on an existing index: you must reindex. Test it with _analyze on real examples before creating the index.

3. Full-text queries #

Choosing the right query.

Overview #

QueryBehaviourExample use
matchanalyzes text, searches terms (OR by default)search bar
match + operator: andall terms requirednarrowing down
match_phraseterms in order, slop for tolerance"credit card"
multi_matchseveral fields, ^ boostingname and description
fuzziness: AUTOtolerates typos"computr"
termexact value, no analysiskeyword field only

A product search #

GET produits/_search
{
  "query": {
    "bool": {
      "must": [{
        "multi_match": {
          "query": "ordinatuer portable",
          "fields": ["nom^3", "description"],
          "fuzziness": "AUTO"
        }
      }],
      "filter": [
        {"range": {"prix": {"lte": 1200}}},
        {"range": {"stock": {"gt": 0}}}
      ]
    }
  },
  "highlight": {"fields": {"description": {}}},
  "_source": ["nom", "prix"]
}

nom^3 gives the name three times the weight of the description. Price and stock filters do not count in the score and are cached.

Common mistake : A term query on a text field finds nothing: the searched value is not analyzed while the content was. Use match, or the keyword sub-field.

4. Relevance #

Understanding and tuning the score.

BM25 #

BM25 score
  • Term frequency: the more a term appears in the document, the higher the score, with saturation (parameter k1, 1.2 by default).
  • Rarity: a term rare in the index weighs more than a frequent one.
  • Length: a short field containing the term weighs more than a long text (parameter b, 0.75).

Explaining and adjusting #

GET produits/_search
{"explain": true, "query": {"match": {"nom": "portable"}}}

GET produits/_search
{
  "query": {
    "function_score": {
      "query": {"match": {"nom": "portable"}},
      "functions": [{"field_value_factor": {"field": "stock", "modifier": "log1p", "factor": 0.5}}],
      "boost_mode": "sum"
    }
  }
}

explain details each document's score computation. function_score adds a business criterion (stock, popularity, freshness) to the text score.

5. Paginating without limits #

from/size, search_after and point in time.

The methods #

MethodLimitUse
from + size10,000 results (index.max_result_window)website pages
search_after + PITnone, frozen view of dataexports, deep pagination
scrollolder, costlyto replace with PIT
POST produits/_search/point_in_time?keep_alive=2m

GET _search
{
  "size": 100,
  "pit": {"id": "<pit_id>", "keep_alive": "2m"},
  "sort": [{"prix": "asc"}, {"_id": "asc"}],
  "search_after": [499.0, "sku-1234"]
}

Commands to know #

Commands to know
  • POST _analyze : see the terms an analyzer produces
  • GET <index>/_search {"explain": true} : score details
  • POST <index>/_search/point_in_time?keep_alive=2m : freeze a view to paginate
  • GET <index>/_termvectors/<id> : a document's indexed terms
  • POST <index>/_validate/query?explain : validate and rewrite a query

Sources #

Checked in November 2026 (OpenSearch 3.x, Data Prepper 2.x): OpenSearch ships a minor release about every six weeks, check the release notes before relying on a detail.

Related articles

The OpenSearch cluster in depth: nodes, shards and replication

Understanding how an OpenSearch cluster works inside: node roles, cluster manager election, shard allocation,…

Advanced aggregations, PPL and SQL in OpenSearch

Analyzing data in OpenSearch: bucket, metric and pipeline aggregations, date histograms, approximate cardinal…

Advanced Data Prepper: routing, OpenTelemetry and reliability

Going further with Data Prepper: architecture, buffers and workers, conditional processors, routing to severa…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!