RousselTMLEARNING

Observability

The observability trilemma: cost, completeness and performance, and how solutions resolve it

Understanding the observability trilemma (cost, data completeness, query performance), its three trade-offs, the levers to loosen it, and a comparison of ClickHouse, Dynatrace, Datadog, LGTM, Elastic, Splunk, Honeycomb, Prometheus and OpenObserve by what each excels at.

1. The trilemma in brief #

Three desired qualities, two at a time.

Definition #

Every observability platform tries to offer three qualities: controlled cost, complete data (every dimension, over a long period, unsampled) and fast queries on fresh data. The observability trilemma states that as volume grows, you can only fully optimize two at a time: the third becomes the price to pay.

The observability trilemma

Where the idea comes from #

The idea was popularized in 2019 by Ben Sigelman (LightStep co-founder, formerly on Dapper at Google): among several desirable properties (high-cardinality queries, long retention, unsampled data…), you can only get some of them at once without costs exploding. Cindy Sridharan compared it to the CAP theorem of distributed systems. Since then, wording varies between authors, but the tension stays the same: cost, completeness, performance.

Good to know : The trilemma is not a law of physics: it is an economic constraint tied to storage and indexing. Technical progress (columnar storage, object storage) moves the boundary without removing it.

2. The three corners in detail #

What each word covers.

Cost #

Cost is not just the vendor's bill. You must count:

ItemWhat drives it up
Ingestionlog volume, span count, number of metric series
Storageretention, compression ratio, SSD or object storage, replication
Indexingfull inverted indexes, indexed fields, label cardinality
Querieson-demand compute, data scanned, constantly refreshed dashboards, AI agents
Licences and billing modelper host, per GB ingested or indexed, per custom metric, per user
Peoplerunning the platform, upgrades, tuning

Completeness #

  • Cardinality: being able to filter by user_id, pod, trace_id, version, customer… In a time-series system, every label combination creates a series: cardinality is expensive there.
  • Retention: keeping data long enough to compare with the previous week, month or year, and for audits.
  • Fidelity: no sampling or aggregation that erases the rare event, precisely the one you look for during an incident.
  • Coverage: every signal (metrics, logs, traces, profiles, sessions) and every service, not just the most critical ones.

Performance #

  • Query latency: a few seconds at most during an incident, over large periods and volumes.
  • Freshness: delay between emitting an event and seeing it (seconds, not minutes); essential for alerting.
  • Concurrency: handling dozens of dashboards, continuously evaluated alerts and, more and more, AI agents firing hundreds of queries.
Best practice : Measure each corner with a number: cost per GB or per service per month, retention and sampling rate per signal, p95 of dashboard query duration. Without numbers, the trilemma is argued in opinions.

3. The three trade-offs #

Each pair has its price.

Overview #

The three trade-offs
Chosen pairTypical techniqueSacrificeExample
Cost + completenesskeep everything, compressed, on object storageslow or deferred querieslog archives on S3 queried with an on-demand SQL engine
Completeness + performancefull indexes, everything hot, generous resourceshigh costsearch cluster sized for 90 days of indexed logs
Cost + performancesampling, aggregates, limited labels, short retentionincomplete dataPrometheus with 15 days retention and controlled labels; 1% of traces kept

Real-life symptoms #

  • “We do not have the trace for this request”: sampling discarded it (cost + performance).
  • “Cannot filter by customer”: the label was removed to avoid series explosion (cost + performance).
  • “The log bill doubled this quarter”: everything is indexed and kept hot (completeness + performance).
  • “The 6-month query takes 20 minutes”: data is in the cold archive (cost + completeness).
Watch out : Blindly cutting the volume sent (“send less data”) means choosing incompleteness without saying so. Filtering is healthy, provided you know what you lose and decided it.

4. Why the trilemma is getting harder #

More data, more dimensions, more queries.

The causes #

  • Volumes: microservices, Kubernetes and event-driven architectures multiply telemetry-emitting components; a single user request crosses dozens of services.
  • Cardinality: ephemeral pods, container IDs, versions deployed several times a day, business attributes (customer, plan): combinations grow faster than traffic.
  • New signals: continuous profiles, browser sessions, very large LLM application traces (prompts, tokens, evaluations).
  • New consumers: AI investigation agents query telemetry far more often than humans, and need detailed, long-lived data to compare.
  • Billing: per-GB or per-metric models turn every instrumentation decision into a budget decision.

5. Levers to loosen the grip #

Moving the boundary rather than enduring it.

The levers #

The levers
LeverCorner gainedPrice paid
Columnar storage and compression (ClickHouse, Parquet)cost, without losing completenessfull-text search less precise than an inverted index
Separating compute and storage (object storage)cost and long retentionlatency on cold data
Minimal index (labels only, like Loki)ingestion costslower “brute force” queries on large volumes
Hot / warm / cold tierscost, keeping performance on recent datamanagement complexity
Tail samplingcost, keeping errors and slow traces“normal” traces become statistical
Derived metrics (spanmetrics, logs to metrics)dashboard performance and costdimensions fixed in advance
Adaptive aggregation (Adaptive Metrics/Logs, Chronosphere)cost, guided by actual usagedepends on the tool's recommendations
Telemetry pipelines (OTel Collector, Bindplane, Cribl)all three, by routing each piece of data to the right storeone more component to run

Sampling without losing trends #

# OpenTelemetry collector: tail sampling (all errors, slow traces, 5% of the rest)
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow,  type: latency,     latency: { threshold_ms: 1500 } }
      - { name: rest,   type: probabilistic, probabilistic: { sampling_percentage: 5 } }
connectors:
  spanmetrics:            # RED metrics computed on 100% of spans BEFORE sampling
    dimensions: [{ name: http.route }, { name: http.response.status_code }]
service:
  pipelines:
    traces/in:      { receivers: [otlp], processors: [memory_limiter], exporters: [spanmetrics, forward] }
    traces/sampled: { receivers: [forward], processors: [tail_sampling, batch], exporters: [otlphttp/traces] }
    metrics/red:    { receivers: [spanmetrics], processors: [batch], exporters: [prometheusremotewrite] }

RED metrics are computed on all spans, before sampling: dashboards stay exact, and only “ordinary” individual traces are dropped.

Routing by value #

# Route by value: hot logs without debug, everything in compressed archive
exporters:
  otlphttp/hot: { endpoint: https://logs-hot.interne:4318 }
  awss3/archive:
    s3uploader: { region: eu-west-3, s3_bucket: telemetry-archive, s3_prefix: logs }
    marshaler: otlp_proto
    compression: gzip
processors:
  filter/no-debug:
    logs:
      log_record: ['severity_number < SEVERITY_NUMBER_INFO']   # drops DEBUG and TRACE
service:
  pipelines:
    logs/hot:   { receivers: [otlp], processors: [filter/no-debug, batch], exporters: [otlphttp/hot] }
    logs/archive: { receivers: [otlp], processors: [batch], exporters: [awss3/archive] }
Best practice : Archiving 100% of raw logs on object storage costs little: you only pay the “hot” price for what daily investigations need, and you can rehydrate the archive if an audit requires it.

6. Comparing solutions: what each excels at #

None solves the trilemma; each picks its ground.

Positioning #

Solution positioning

Each solution was born from an architecture choice that places it, by default, on one pair of the trilemma. All then offer options to fix their weak spot (cold tiers, sampling, aggregates), but their point of excellence stays tied to that initial choice.

What each one excels at #

SolutionArchitectureExcels atWatch point
ClickHouse (ClickStack, SigNoz)columnar database, SQL, object storage possiblecost at high cardinality: strong compression, filters on any attribute, SQL analyses and joins with business dataa database to operate (unless managed), younger ecosystem
Dynatracesingle OneAgent, index-free Grail data lakehouse (schema on read), DQL language, Davis AIautomation and root-cause analysis: automatic discovery, Smartscape topology, AI-computed causality, little configurationlicence cost (consumption), strong dependency on the agent and platform
Datadogunified SaaS, agent and 900+ integrationsfunctional breadth and time to value: APM, logs, RUM, security, synthetics in one very polished UIhard-to-forecast bill (hosts, custom metrics, indexed logs); Flex Logs for long storage
LGTM (Loki, Grafana, Tempo, Mimir, Pyroscope)specialized components on object storage, minimal indexstorage cost and openness: cheap long retention, open source, Grafana federates dozens of sourcesslower full-text and high-cardinality queries (Loki), several languages (PromQL, LogQL, TraceQL), several components to run
Elastic (Elasticsearch, Kibana)inverted index, ES|QL, hot to frozen tiers (searchable snapshots), LogsDB modefull-text search and log analytics: very fast ad hoc search, aggregations, and observability-security convergence (SIEM)indexing and hot storage cost; index and tier tuning
Splunkindex and SPL language, schema on readad hoc investigation and security: very expressive SPL, mature SIEM ecosystemcost at high volume
Honeycombwide events in columnar storagehigh-cardinality exploration: slice by any field, BubbleUp to find what sets slow requests apartfocused on traces and events; less on infrastructure metrics
Prometheuslocal time-series database, PromQL, pull modelreal-time metrics and alerting: de facto standard, very fast, simple, freelimited cardinality and long retention: Mimir or Thanos to scale
OpenObserveRust, Parquet files on object storage, single binarystorage cost and simplicity: logs, metrics and traces in one light binaryyounger project, smaller community

Indicative grid #

SolutionCostCompletenessPerformanceOperational simplicity
ClickHouse (ClickStack, SigNoz)●●●●●●●●○●●○
Dynatrace●○○●●●●●●●●●
Datadog●○○●●○●●●●●●
LGTM●●●●●○●●○●○○
Elastic●●○●●○●●●●●○
Splunk●○○●●●●●●●●○
Honeycomb●●○●●●●●●●●●
Prometheus●●●●○○●●●●●●
OpenObserve●●●●●○●●○●●●
Watch out : Deliberately simplified grid, built in November 2026 for common uses: every offering evolves fast (cold tiers, adaptive aggregation, managed offerings), and results depend heavily on your volumes and settings. Always run a trial with your own data before deciding.

How each one fixes its weak spot #

  • Elastic: frozen tier on searchable snapshots and LogsDB index mode (up to about 65% less storage according to Elastic) to cut cost.
  • Datadog: Flex Logs and “Logging without Limits” separate ingestion, indexing and storage to keep data longer for less.
  • Dynatrace: Grail removes indexing at ingestion (schema on read) to keep more data without preparing it.
  • LGTM: blooms and structured metadata in Loki, Adaptive Metrics and Adaptive Logs in Grafana Cloud to cut volume and latency.
  • ClickHouse: materialized views, skip indexes and managed offerings (Managed ClickStack) to gain performance and simplicity.
  • Prometheus: Mimir or Thanos add object storage, high availability and long retention.

7. Choosing and arbitrating #

One pair per use case, not one for the whole company.

The method #

  1. Measure: volume per signal and per service, metric cardinality, actual query duration, current cost.
  2. Classify uses: alerting, incident investigation, dashboards, trend analysis, audit and compliance.
  3. Pick a pair per use (table below) rather than one platform for everything.
  4. Instrument with OpenTelemetry to route each signal to the right store and switch tools without re-instrumenting.
  5. Review quarterly: volume per service, unused labels, dashboards never opened, retention actually used.

A pair per use #

UsePair to favourExample implementation
Real-time alertingcost + performancePrometheus or Mimir metrics, controlled labels
Incident investigation (7 to 14 days)completeness + performancedetailed traces and logs hot (ClickHouse, Elastic, SaaS)
Long-term dashboardscost + performanceaggregates (materialized views, derived metrics)
Audit, compliance, one-off analysescost + completenesscompressed archive on object storage
Hybrid architecture

Measuring before deciding #

# Prometheus: the 10 metrics creating the most series
topk(10, count by (__name__) ({__name__=~".+"}))
# Prometheus: total active series
prometheus_tsdb_head_series

# Loki: labels blowing up the number of streams
logcli series '{namespace="prod"}' --analyze-labels

# ClickHouse: size and compression of telemetry tables
SELECT table, formatReadableSize(sum(data_compressed_bytes)) AS disk,
       round(sum(data_uncompressed_bytes) / sum(data_compressed_bytes), 1) AS ratio
FROM system.parts WHERE active GROUP BY table ORDER BY sum(data_compressed_bytes) DESC;

# Elasticsearch: largest indexes
GET _cat/indices?v&s=store.size:desc&h=index,docs.count,store.size

# Dynatrace (DQL): log volume per source over 24 h
fetch logs, from: now() - 24h | summarize n = count(), by: { log.source } | sort n desc

8. Key takeaways #

Summary and checklist.

Summary #

  • The trilemma opposes cost, completeness and performance: as volume grows, you only optimize two at a time.
  • Each solution excels on a pair tied to its architecture: ClickHouse on cost and cardinality, Dynatrace on automation, Datadog on functional breadth, LGTM on storage cost and openness, Elastic on full-text search, Prometheus on real-time metrics.
  • The best answer is rarely a single tool: it is an architecture per use, fed by OpenTelemetry and pipelines routing each piece of data to the right trade-off.

Checklist #

QuestionGood answer
Do you know your cost per service?yes, reviewed monthly
Does sampling keep every error?yes, tail sampling and metrics computed before
Is retention defined per signal?yes, and aligned with actual use
Are high-cardinality labels under control?in metrics, yes; in traces and logs, they are allowed
Can you switch tools without re-instrumenting?yes, thanks to OpenTelemetry

Commands to know #

Commands to know
  • topk(10, count by (__name__) ({__name__=~".+"})) : metrics creating the most series
  • logcli series '{namespace="prod"}' --analyze-labels : Loki label cardinality
  • SELECT … FROM system.parts WHERE active : ClickHouse size and compression
  • GET _cat/indices?v&s=store.size:desc : heaviest Elasticsearch indexes
  • tail_sampling (OTel Collector) : sample while keeping errors and slow traces
  • spanmetrics (OTel Collector) : exact RED metrics despite sampling

Sources #

Checked in November 2026: offerings and billing models change fast, check each vendor's documentation before relying on a detail.

Related articles

Time-based vs event-based SLO: understanding, computing and choosing (with the Qonto example)

Differences between time-based SLOs (time slices, uptime) and event-based SLOs (good events over total): defi…

Installing and deploying ClickStack: all-in-one, Docker Compose, Helm and Managed ClickStack

Deploying ClickStack to fit the need: all-in-one container with persistence, Docker Compose and its variables…

Sending telemetry to ClickStack: OpenTelemetry SDKs, collectors, logs, Kubernetes and browser sessions

Feeding ClickStack: ingestion key, OpenTelemetry SDK variables, forwarding through an existing collector, fil…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!