Lab: Instruments, views and exemplars with the metrics SDK with correction

What you will learn in this TP

• Choose the right instrument
• Configure a view
• Use exemplars

Introduction #

What you will learn in this section

• Choose the right instrument
• Configure a view
• Use exemplars

Context. Management wants to track orders, active carts, stock and API latency. You produce these metrics with OpenTelemetry and check they are usable in Prometheus.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 50 min

Well-typed metrics, correct histograms and a metric linked to a trace.

Hands-on #

Guided steps, with their solution.

1. 1. Start the observability stack

   Why? A single image provides an OTLP Collector and every backend: focus on the signal.

   Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.
   See the correction

   docker run -d --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 -p 4040:4040 grafana/otel-lgtm
   python3 -m venv .venv && . .venv/bin/activate
   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   # Grafana : http://localhost:3000 (admin / admin)

2. 2. Produce metrics with the four instrument families

   Why? The instrument type determines aggregation and therefore chart accuracy.

   Write boutique.py that, in a loop, increments an orders Counter, moves a carts UpDownCounter, records request duration in a Histogram and exposes stock through an ObservableGauge. Export every 5 seconds.
   See the correction

   pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
   cat > boutique.py <<'EOF'
   import random, time
   from opentelemetry import metrics, trace
   from opentelemetry.metrics import Observation
   from opentelemetry.sdk.resources import Resource
   from opentelemetry.sdk.metrics import MeterProvider
   from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
   from opentelemetry.sdk.metrics.view import View, ExplicitBucketHistogramAggregation
   from opentelemetry.sdk.trace import TracerProvider
   from opentelemetry.sdk.trace.export import BatchSpanProcessor
   from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
   from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
   
   resource = Resource.create({"service.name": "boutique", "service.version": "1.0.0"})
   VUES = []      # étape 4 : ajouter une vue ici
   reader = PeriodicExportingMetricReader(OTLPMetricExporter(), export_interval_millis=5000)
   metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[reader], views=VUES))
   tp = TracerProvider(resource=resource); tp.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
   trace.set_tracer_provider(tp)
   
   meter = metrics.get_meter("tp.boutique", "1.0.0")
   tracer = trace.get_tracer("tp.boutique")
   commandes = meter.create_counter("shop.orders", unit="{order}", description="Commandes validées")
   paniers = meter.create_up_down_counter("shop.carts.active", unit="{cart}", description="Paniers ouverts")
   duree = meter.create_histogram("http.server.request.duration", unit="s", description="Durée des requêtes")
   
   def lire_stock(options):
       for entrepot in ("lyon", "lille"):
           yield Observation(random.randint(50, 500), {"shop.warehouse": entrepot})
   meter.create_observable_gauge("shop.stock.level", callbacks=[lire_stock], unit="{item}")
   
   while True:
       with tracer.start_as_current_span("POST /orders"):
           d = random.choice([random.uniform(0.02, 0.2)] * 9 + [random.uniform(1, 3)])   # 10 % de requêtes lentes
           time.sleep(d / 10)
           duree.record(d, {"http.request.method": "POST", "http.route": "/orders", "http.response.status_code": 201})
       commandes.add(1, {"shop.payment.method": random.choice(["card", "paypal", "transfer"])})
       paniers.add(random.choice([1, 1, -1]))
   EOF
   python boutique.py &      # laisser tourner

3. 3. Find the metrics in Prometheus

   Why? The OTLP-to-Prometheus conversion renames metrics: you need to know how to find them.

   In Explore > Prometheus, find each metric under its Prometheus name and write the queries: order rate by payment method, open carts, stock per warehouse, p95 latency.
   See the correction

   sum by (shop_payment_method) (rate(shop_orders_total[1m]))
   shop_carts_active
   shop_stock_level
   histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket[1m])))

4. 4. Fix buckets with a view

   Why? Default buckets (0, 5, 10, 25… 10000) are designed for milliseconds: in seconds almost everything falls in the first bucket and p95 is wrong.

   Look at the current buckets, stop the script, add a view with buckets in seconds, restart and compare p95.
   See the correction

   # avant : sum by (le) (rate(http_server_request_duration_seconds_bucket[1m]))  -> tout dans le=5
   kill %1
   sed -i 's/^VUES = \[\].*/VUES = [View(instrument_name="http.server.request.duration", aggregation=ExplicitBucketHistogramAggregation([0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5]))]/' boutique.py
   python boutique.py &
   # après quelques minutes : le p95 reflète enfin les 10 % de requêtes lentes

5. 5. Measure cardinality effects

   Why? Each distinct attribute value creates a series: an identifier on a metric blows up storage.

   Count shop_orders_total series, temporarily add a random user.id attribute, count again, then remove it.
   See the correction

   count(shop_orders_total)                                   # 3 séries
   # dans boutique.py : commandes.add(1, {..., "user.id": str(random.randint(1, 5000))})
   count(shop_orders_total)                                   # des centaines, puis des milliers
   # conclusion : identifiants sur les spans et les logs, jamais sur les métriques

6. 6. Link a latency to its trace

   Why? Measurements taken inside a sampled span carry an exemplar with the trace_id: you go from the chart to the exact request.

   Display p95 in a Grafana panel with the Exemplars option on, click a high point and open the matching trace in Tempo. If no exemplar shows, check the SDK version and OTEL_METRICS_EXEMPLAR_FILTER.
   See the correction

   pip show opentelemetry-sdk | grep Version          # exemplars : SDK Python récent
   export OTEL_METRICS_EXEMPLAR_FILTER=trace_based
   kill %1 ; python boutique.py &

Difficulty level : ●●●○○ (3/5)