Lab: OpenTelemetry Operator and auto-instrumentation on Kubernetes

What you will learn in this TP

• Install the Operator
• Declare Collector and Instrumentation
• Instrument through annotation

Introduction #

What you will learn in this section

• Install the Operator
• Declare Collector and Instrumentation
• Instrument through annotation

Context. The platform team wants every cluster application to produce traces without each team changing its images. You validate the Operator approach.

Prerequisites: Docker, kubectl and kind  ·  Estimated time: 50 min

An application instrumented without image changes, whose spans reach an Operator-managed Collector.

Hands-on #

Guided steps, with their solution.

1. 1. Prepare the cluster

   Why? The Operator uses admission webhooks whose certificates are managed by cert-manager.

   Create a kind cluster, install cert-manager then the Operator, and wait until they are available.

2. 2. Declare a Collector

   Why? The Collector is described as a Kubernetes resource: the Operator creates the Deployment, Service and ConfigMap.

   Create a demo namespace and an OpenTelemetryCollector resource in deployment mode with an OTLP receiver, k8sattributes and the debug exporter.

3. 3. Declare instrumentation

   Why? The Instrumentation resource centralises exporter, propagators and sampling for a whole namespace.

   Create an Instrumentation resource that sends OTLP/HTTP to the Collector, with tracecontext and baggage propagators and 100% sampling.

4. 4. Instrument an application through annotation

   Why? The annotation is enough: the Operator injects the Python agent at pod start, without changing the image.

   Deploy a small Flask application in a standard Python image with the instrumentation.opentelemetry.io/inject-python annotation, then look at the injected init container and OTEL_* variables.

5. 5. Check the spans

   Why? The debug exporter prints received spans with the Kubernetes attributes added by k8sattributes.

   Generate a few requests through a port-forward, then look in the Collector logs for the GET / span, service.name and k8s.pod.name.

6. 6. Evolve and clean up

   Why? Changing the Instrumentation resource only affects recreated pods: deployments must be restarted.

   Set sampling to 10%, restart the deployment, check the new OTEL_TRACES_SAMPLER_ARG value, then delete the cluster.

Difficulty level : ●●●○○ (3/5)