1. Architecture #
Sources, buffer, processors, sinks.
Overview #
| Element | Role | Key settings |
|---|---|---|
| source | receives (http, otel_*, otlp) or pulls (s3, kafka) data | port, ssl, path |
| buffer | absorbs peaks between source and processors | buffer_size, batch_size |
| processor | transforms, enriches, filters | workers, delay |
| route | names conditions reused by sinks | expressions |
| sink | writes to OpenSearch, S3, another pipeline… | index, bulk_size, dlq |
2. Transforming with conditions #
Processors and expressions.
A complete pipeline #
app-pipeline:
workers: 2
delay: 100
source:
http:
port: 2021
path: /logs
ssl: false
buffer:
bounded_blocking:
buffer_size: 4096
batch_size: 256
processor:
- parse_json:
source: message
- drop_events:
drop_when: '/level == "DEBUG"'
- add_entries:
entries:
- key: environnement
value: prod
- date:
match:
- key: time
patterns: ["yyyy-MM-dd'T'HH:mm:ss.SSSXXX"]
destination: "@timestamp"
- delete_entries:
with_keys: ["message", "time"]
route:
- erreurs: '/level == "ERROR" or /level == "FATAL"'
- paiement: '/service == "paiement"'
sink:
- opensearch:
hosts: ["https://opensearch:9200"]
username: ingest_dp
password: "${OS_PASSWORD}"
insecure: true
index: logs-erreurs-%{yyyy.MM.dd}
routes: [erreurs]
- opensearch:
hosts: ["https://opensearch:9200"]
username: ingest_dp
password: "${OS_PASSWORD}"
insecure: true
index: logs-app-%{yyyy.MM.dd}Errors go to a dedicated index and to the general index (the second sink has no route, it receives everything). Expressions use JSON paths (/level) and the operators ==, !=, and, or, not, =~ (regex).
Useful processors #
| Processor | Role |
|---|---|
| grok, parse_json, key_value, csv | extract fields |
| date, convert_type | type values |
| add_entries, rename_keys, delete_entries, copy_values | reshape the event |
| drop_events | drop by condition |
| aggregate | group over a window (count, deduplicate) |
| obfuscate | mask personal data |
| geoip | locate an IP address |
| anomaly_detector | spot abnormal values in the stream |
3. OpenTelemetry #
Traces, logs and metrics.
The path #
Applications send OTLP to an OTel Collector, which batches and filters, then forwards to Data Prepper. Traces feed the otel-v1-apm-span-* and otel-v2-apm-service-map indexes used by Trace Analytics.
Collector to Data Prepper #
receivers:
otlp:
protocols:
grpc: {endpoint: 0.0.0.0:4317}
http: {endpoint: 0.0.0.0:4318}
processors:
batch: {}
exporters:
otlp/traces:
endpoint: data-prepper:21890
tls: {insecure: true}
otlp/logs:
endpoint: data-prepper:21892
tls: {insecure: true}
service:
pipelines:
traces: {receivers: [otlp], processors: [batch], exporters: [otlp/traces]}
logs: {receivers: [otlp], processors: [batch], exporters: [otlp/logs]}otel-logs-pipeline:
source:
otel_logs_source:
ssl: false
sink:
- opensearch:
hosts: ["https://opensearch:9200"]
username: ingest_dp
password: "${OS_PASSWORD}"
insecure: true
index_type: custom
index: otel-logs-%{yyyy.MM.dd}Good to know : Data Prepper default ports are 21890 (traces), 21891 (metrics) and 21892 (logs). Recent versions also offer a single otlp source for all three signals: check its availability in your version.4. Reliability and operations #
Lose nothing, see everything.
Losing nothing #
- Buffer: size
buffer_sizeto absorb a short OpenSearch outage; beyond that, the source returns an error (HTTP 408 or 429) and the client must retry. - Dead-letter queue (DLQ): the opensearch sink's
dlqoption writes rejected documents (incompatible mapping…) to S3 instead of losing them. - Pull sources (S3 with SQS notifications, Kafka): data stays at the source until processed, the most robust option.
- opensearch:
hosts: ["https://opensearch:9200"]
index: logs-app-%{yyyy.MM.dd}
dlq:
s3:
bucket: dp-rejets
key_path_prefix: app-pipeline/
region: eu-west-3Monitoring #
Data Prepper exposes Prometheus metrics on port 4900: records received, written, rejected, buffer fill rate, batch duration.
curl -s localhost:4900/metrics/prometheus | grep -E 'recordsIn|recordsOut|documentErrors|bufferUsage' | head
curl -s localhost:4900/list # pipelines chargésBest practice : Alert on two signals: a buffer that stays full (OpenSearch no longer keeps up) and a rising failed-document counter (broken mapping).
Commands to know #
Commands to know
docker run -p 2021:2021 -p 21890:21890 -v ./pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml opensearchproject/data-prepper:2: run Data Preppercurl -s localhost:4900/list: loaded pipelinescurl -s localhost:4900/metrics/prometheus: metricscurl -X POST localhost:2021/logs -H 'Content-Type: application/json' -d '[{"message":"…"}]': send test logs
Sources #
Checked in November 2026 (OpenSearch 3.x, Data Prepper 2.x): OpenSearch ships a minor release about every six weeks, check the release notes before relying on a detail.
