RousselTMLEARNING

Observability

Advanced Data Prepper: routing, OpenTelemetry and reliability

Going further with Data Prepper: architecture, buffers and workers, conditional processors, routing to several indexes, S3 and Kafka sources, OpenTelemetry traces, logs and metrics, dead-letter queue, monitoring and sizing.

1. Architecture #

Sources, buffer, processors, sinks.

Overview #

Data Prepper architecture
ElementRoleKey settings
sourcereceives (http, otel_*, otlp) or pulls (s3, kafka) dataport, ssl, path
bufferabsorbs peaks between source and processorsbuffer_size, batch_size
processortransforms, enriches, filtersworkers, delay
routenames conditions reused by sinksexpressions
sinkwrites to OpenSearch, S3, another pipeline…index, bulk_size, dlq

2. Transforming with conditions #

Processors and expressions.

A complete pipeline #

app-pipeline:
  workers: 2
  delay: 100
  source:
    http:
      port: 2021
      path: /logs
      ssl: false
  buffer:
    bounded_blocking:
      buffer_size: 4096
      batch_size: 256
  processor:
    - parse_json:
        source: message
    - drop_events:
        drop_when: '/level == "DEBUG"'
    - add_entries:
        entries:
          - key: environnement
            value: prod
    - date:
        match:
          - key: time
            patterns: ["yyyy-MM-dd'T'HH:mm:ss.SSSXXX"]
        destination: "@timestamp"
    - delete_entries:
        with_keys: ["message", "time"]
  route:
    - erreurs: '/level == "ERROR" or /level == "FATAL"'
    - paiement: '/service == "paiement"'
  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        username: ingest_dp
        password: "${OS_PASSWORD}"
        insecure: true
        index: logs-erreurs-%{yyyy.MM.dd}
        routes: [erreurs]
    - opensearch:
        hosts: ["https://opensearch:9200"]
        username: ingest_dp
        password: "${OS_PASSWORD}"
        insecure: true
        index: logs-app-%{yyyy.MM.dd}

Errors go to a dedicated index and to the general index (the second sink has no route, it receives everything). Expressions use JSON paths (/level) and the operators ==, !=, and, or, not, =~ (regex).

Useful processors #

ProcessorRole
grok, parse_json, key_value, csvextract fields
date, convert_typetype values
add_entries, rename_keys, delete_entries, copy_valuesreshape the event
drop_eventsdrop by condition
aggregategroup over a window (count, deduplicate)
obfuscatemask personal data
geoiplocate an IP address
anomaly_detectorspot abnormal values in the stream

3. OpenTelemetry #

Traces, logs and metrics.

The path #

OpenTelemetry to OpenSearch

Applications send OTLP to an OTel Collector, which batches and filters, then forwards to Data Prepper. Traces feed the otel-v1-apm-span-* and otel-v2-apm-service-map indexes used by Trace Analytics.

Collector to Data Prepper #

receivers:
  otlp:
    protocols:
      grpc: {endpoint: 0.0.0.0:4317}
      http: {endpoint: 0.0.0.0:4318}
processors:
  batch: {}
exporters:
  otlp/traces:
    endpoint: data-prepper:21890
    tls: {insecure: true}
  otlp/logs:
    endpoint: data-prepper:21892
    tls: {insecure: true}
service:
  pipelines:
    traces: {receivers: [otlp], processors: [batch], exporters: [otlp/traces]}
    logs:   {receivers: [otlp], processors: [batch], exporters: [otlp/logs]}
otel-logs-pipeline:
  source:
    otel_logs_source:
      ssl: false
  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        username: ingest_dp
        password: "${OS_PASSWORD}"
        insecure: true
        index_type: custom
        index: otel-logs-%{yyyy.MM.dd}
Good to know : Data Prepper default ports are 21890 (traces), 21891 (metrics) and 21892 (logs). Recent versions also offer a single otlp source for all three signals: check its availability in your version.

4. Reliability and operations #

Lose nothing, see everything.

Losing nothing #

  • Buffer: size buffer_size to absorb a short OpenSearch outage; beyond that, the source returns an error (HTTP 408 or 429) and the client must retry.
  • Dead-letter queue (DLQ): the opensearch sink's dlq option writes rejected documents (incompatible mapping…) to S3 instead of losing them.
  • Pull sources (S3 with SQS notifications, Kafka): data stays at the source until processed, the most robust option.
    - opensearch:
        hosts: ["https://opensearch:9200"]
        index: logs-app-%{yyyy.MM.dd}
        dlq:
          s3:
            bucket: dp-rejets
            key_path_prefix: app-pipeline/
            region: eu-west-3

Monitoring #

Data Prepper exposes Prometheus metrics on port 4900: records received, written, rejected, buffer fill rate, batch duration.

curl -s localhost:4900/metrics/prometheus | grep -E 'recordsIn|recordsOut|documentErrors|bufferUsage' | head
curl -s localhost:4900/list   # pipelines chargés
Best practice : Alert on two signals: a buffer that stays full (OpenSearch no longer keeps up) and a rising failed-document counter (broken mapping).

Commands to know #

Commands to know
  • docker run -p 2021:2021 -p 21890:21890 -v ./pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml opensearchproject/data-prepper:2 : run Data Prepper
  • curl -s localhost:4900/list : loaded pipelines
  • curl -s localhost:4900/metrics/prometheus : metrics
  • curl -X POST localhost:2021/logs -H 'Content-Type: application/json' -d '[{"message":"…"}]' : send test logs

Sources #

Checked in November 2026 (OpenSearch 3.x, Data Prepper 2.x): OpenSearch ships a minor release about every six weeks, check the release notes before relying on a detail.

Related articles

The OpenSearch cluster in depth: nodes, shards and replication

Understanding how an OpenSearch cluster works inside: node roles, cluster manager election, shard allocation,…

Full-text search in OpenSearch: analyzers and relevance

Mastering full-text search: inverted index, analyzers (including French), the _analyze API, match, multi_matc…

Advanced aggregations, PPL and SQL in OpenSearch

Analyzing data in OpenSearch: bucket, metric and pipeline aggregations, date histograms, approximate cardinal…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!