Lab: Instruments, views and exemplars with the metrics SDK

What you will learn in this TP

• Choose the right instrument
• Configure a view
• Use exemplars

Introduction #

What you will learn in this section

• Choose the right instrument
• Configure a view
• Use exemplars

Context. Management wants to track orders, active carts, stock and API latency. You produce these metrics with OpenTelemetry and check they are usable in Prometheus.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 50 min

Well-typed metrics, correct histograms and a metric linked to a trace.

Hands-on #

Guided steps, with their solution.

1. 1. Start the observability stack

   Why? A single image provides an OTLP Collector and every backend: focus on the signal.

   Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.

2. 2. Produce metrics with the four instrument families

   Why? The instrument type determines aggregation and therefore chart accuracy.

   Write boutique.py that, in a loop, increments an orders Counter, moves a carts UpDownCounter, records request duration in a Histogram and exposes stock through an ObservableGauge. Export every 5 seconds.

3. 3. Find the metrics in Prometheus

   Why? The OTLP-to-Prometheus conversion renames metrics: you need to know how to find them.

   In Explore > Prometheus, find each metric under its Prometheus name and write the queries: order rate by payment method, open carts, stock per warehouse, p95 latency.

4. 4. Fix buckets with a view

   Why? Default buckets (0, 5, 10, 25… 10000) are designed for milliseconds: in seconds almost everything falls in the first bucket and p95 is wrong.

   Look at the current buckets, stop the script, add a view with buckets in seconds, restart and compare p95.

5. 5. Measure cardinality effects

   Why? Each distinct attribute value creates a series: an identifier on a metric blows up storage.

   Count shop_orders_total series, temporarily add a random user.id attribute, count again, then remove it.

6. 6. Link a latency to its trace

   Why? Measurements taken inside a sampled span carry an exemplar with the trace_id: you go from the chart to the exact request.

   Display p95 in a Grafana panel with the Exemplars option on, click a high point and open the matching trace in Tempo. If no exemplar shows, check the SDK version and OTEL_METRICS_EXEMPLAR_FILTER.

Difficulty level : ●●●○○ (3/5)