RousselTMLEARNING

TP · OpenTelemetry: From Discovery to Expertise

Lab: Instruments, views and exemplars with the metrics SDK

What you will learn in this TP
  • Choose the right instrument
  • Configure a view
  • Use exemplars

Introduction #

What you will learn in this section
  • Choose the right instrument
  • Configure a view
  • Use exemplars

Context. Management wants to track orders, active carts, stock and API latency. You produce these metrics with OpenTelemetry and check they are usable in Prometheus.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 50 min

Well-typed metrics, correct histograms and a metric linked to a trace.

Hands-on #

Guided steps, with their solution.

  1. 1. Start the observability stack

    Why? A single image provides an OTLP Collector and every backend: focus on the signal.

    Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.
  2. 2. Produce metrics with the four instrument families

    Why? The instrument type determines aggregation and therefore chart accuracy.

    Write boutique.py that, in a loop, increments an orders Counter, moves a carts UpDownCounter, records request duration in a Histogram and exposes stock through an ObservableGauge. Export every 5 seconds.
  3. 3. Find the metrics in Prometheus

    Why? The OTLP-to-Prometheus conversion renames metrics: you need to know how to find them.

    In Explore > Prometheus, find each metric under its Prometheus name and write the queries: order rate by payment method, open carts, stock per warehouse, p95 latency.
  4. 4. Fix buckets with a view

    Why? Default buckets (0, 5, 10, 25… 10000) are designed for milliseconds: in seconds almost everything falls in the first bucket and p95 is wrong.

    Look at the current buckets, stop the script, add a view with buckets in seconds, restart and compare p95.
  5. 5. Measure cardinality effects

    Why? Each distinct attribute value creates a series: an identifier on a metric blows up storage.

    Count shop_orders_total series, temporarily add a random user.id attribute, count again, then remove it.
  6. 6. Link a latency to its trace

    Why? Measurements taken inside a sampled span carry an exemplar with the trace_id: you go from the chart to the exact request.

    Display p95 in a Grafana panel with the Exemplars option on, click a high point and open the matching trace in Tempo. If no exemplar shows, check the SDK version and OTEL_METRICS_EXEMPLAR_FILTER.

Difficulty level : ●●●○○ (3/5)

My progress

  • ⏱ Estimated time : ≈ 50 min

Sign in to track your progress and take notes. Sign in

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Recommended Articles

OpenTelemetry metrics: instruments, temporality, histograms and exemplars

Master the metric signal: MeterProvider, synchronous and asynchronous instruments (Counter, UpDow...

Correlating traces, metrics, logs and profiles with OpenTelemetry

Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...

Tempo: generating metrics from traces (service graph, span metrics, exemplars)

Enable Tempo's metrics-generator: service graphs, RED span metrics, remote write to Prometheus or...

Correlating traces, logs, metrics and profiles in Grafana

Link Tempo to logs (Loki), metrics (Prometheus) and profiles (Pyroscope): tracesToLogs, tracesToM...

Telemetry in ClickHouse: schema, SQL, materialized views and retention

Telemetry in ClickHouse: otel_logs, otel_traces, otel_metrics_* and hyperdx_sessions tables, sort...

OpenTelemetry: automatic vs manual instrumentation

Understand the differences between automatic and manual instrumentation with OpenTelemetry, their...

Data retention in Dynatrace: Grail, buckets and durations by data type

Understand how long Dynatrace keeps logs, traces, metrics, sessions and events, manage retention ...

OpenTelemetry traces in Elasticsearch and Kibana APM

Instrument applications with OpenTelemetry or EDOT SDKs, send traces to Elasticsearch, get Kibana...

Core OpenTelemetry concepts: signals, resources, context and maturity

The essential OpenTelemetry vocabulary: the four signals, API and SDK, resource and instrumentati...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!