What you will learn in this TP
- Choose the right instrument
- Configure a view
- Use exemplars
Introduction #
What you will learn in this section
- Choose the right instrument
- Configure a view
- Use exemplars
Context. Management wants to track orders, active carts, stock and API latency. You produce these metrics with OpenTelemetry and check they are usable in Prometheus.
Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container) · Estimated time: 50 min
Hands-on #
Guided steps, with their solution.
1. Start the observability stack
Why? A single image provides an OTLP Collector and every backend: focus on the signal.
Startgrafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.2. Produce metrics with the four instrument families
Why? The instrument type determines aggregation and therefore chart accuracy.
Writeboutique.pythat, in a loop, increments an orders Counter, moves a carts UpDownCounter, records request duration in a Histogram and exposes stock through an ObservableGauge. Export every 5 seconds.3. Find the metrics in Prometheus
Why? The OTLP-to-Prometheus conversion renames metrics: you need to know how to find them.
In Explore > Prometheus, find each metric under its Prometheus name and write the queries: order rate by payment method, open carts, stock per warehouse, p95 latency.4. Fix buckets with a view
Why? Default buckets (0, 5, 10, 25… 10000) are designed for milliseconds: in seconds almost everything falls in the first bucket and p95 is wrong.
Look at the current buckets, stop the script, add a view with buckets in seconds, restart and compare p95.5. Measure cardinality effects
Why? Each distinct attribute value creates a series: an identifier on a metric blows up storage.
Countshop_orders_totalseries, temporarily add a randomuser.idattribute, count again, then remove it.6. Link a latency to its trace
Why? Measurements taken inside a sampled span carry an exemplar with the trace_id: you go from the chart to the exact request.
Display p95 in a Grafana panel with the Exemplars option on, click a high point and open the matching trace in Tempo. If no exemplar shows, check the SDK version andOTEL_METRICS_EXEMPLAR_FILTER.
Difficulty level : ●●●○○ (3/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Quels attributs utiliser pour nommer mes spans HTTP (semconv) ?✓ solved · 1 reply(ies)
- OpenTelemetry Collector : mes traces arrivent mais pas mes métriques✓ solved · 2 reply(ies)
Ask a question about this lesson →See all discussions about this course
Recommended Articles
OpenTelemetry metrics: instruments, temporality, histograms and exemplars
Master the metric signal: MeterProvider, synchronous and asynchronous instruments (Counter, UpDow...
Correlating traces, metrics, logs and profiles with OpenTelemetry
Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...
Tempo: generating metrics from traces (service graph, span metrics, exemplars)
Enable Tempo's metrics-generator: service graphs, RED span metrics, remote write to Prometheus or...
Correlating traces, logs, metrics and profiles in Grafana
Link Tempo to logs (Loki), metrics (Prometheus) and profiles (Pyroscope): tracesToLogs, tracesToM...
Telemetry in ClickHouse: schema, SQL, materialized views and retention
Telemetry in ClickHouse: otel_logs, otel_traces, otel_metrics_* and hyperdx_sessions tables, sort...
OpenTelemetry: automatic vs manual instrumentation
Understand the differences between automatic and manual instrumentation with OpenTelemetry, their...
Data retention in Dynatrace: Grail, buckets and durations by data type
Understand how long Dynatrace keeps logs, traces, metrics, sessions and events, manage retention ...
OpenTelemetry traces in Elasticsearch and Kibana APM
Instrument applications with OpenTelemetry or EDOT SDKs, send traces to Elasticsearch, get Kibana...
Core OpenTelemetry concepts: signals, resources, context and maturity
The essential OpenTelemetry vocabulary: the four signals, API and SDK, resource and instrumentati...