Lab: OpenTelemetry Operator and auto-instrumentation on Kubernetes with correction

What you will learn in this TP

• Install the Operator
• Declare Collector and Instrumentation
• Instrument through annotation

Introduction #

What you will learn in this section

• Install the Operator
• Declare Collector and Instrumentation
• Instrument through annotation

Context. The platform team wants every cluster application to produce traces without each team changing its images. You validate the Operator approach.

Prerequisites: Docker, kubectl and kind  ·  Estimated time: 50 min

An application instrumented without image changes, whose spans reach an Operator-managed Collector.

Hands-on #

Guided steps, with their solution.

1. 1. Prepare the cluster

   Why? The Operator uses admission webhooks whose certificates are managed by cert-manager.

   Create a kind cluster, install cert-manager then the Operator, and wait until they are available.
   See the correction

   kind create cluster --name otel
   kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml
   kubectl wait --for=condition=Available deploy --all -n cert-manager --timeout=180s
   kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
   kubectl wait --for=condition=Available deploy --all -n opentelemetry-operator-system --timeout=180s
   kubectl get crd | grep opentelemetry

2. 2. Declare a Collector

   Why? The Collector is described as a Kubernetes resource: the Operator creates the Deployment, Service and ConfigMap.

   Create a demo namespace and an OpenTelemetryCollector resource in deployment mode with an OTLP receiver, k8sattributes and the debug exporter.
   See the correction

   kubectl create namespace demo
   cat <<'EOF' | kubectl apply -f -
   apiVersion: opentelemetry.io/v1beta1
   kind: OpenTelemetryCollector
   metadata: { name: demo, namespace: demo }
   spec:
     mode: deployment
     config:
       receivers:
         otlp: { protocols: { grpc: { endpoint: 0.0.0.0:4317 }, http: { endpoint: 0.0.0.0:4318 } } }
       processors:
         k8sattributes: {}
         batch: {}
       exporters:
         debug: { verbosity: detailed }
       service:
         pipelines:
           traces: { receivers: [otlp], processors: [k8sattributes, batch], exporters: [debug] }
   EOF
   kubectl -n demo get pods,svc     # demo-collector

3. 3. Declare instrumentation

   Why? The Instrumentation resource centralises exporter, propagators and sampling for a whole namespace.

   Create an Instrumentation resource that sends OTLP/HTTP to the Collector, with tracecontext and baggage propagators and 100% sampling.
   See the correction

   cat <<'EOF' | kubectl apply -f -
   apiVersion: opentelemetry.io/v1alpha1
   kind: Instrumentation
   metadata: { name: defaut, namespace: demo }
   spec:
     exporter: { endpoint: http://demo-collector.demo:4318 }
     propagators: [tracecontext, baggage]
     sampler: { type: parentbased_always_on }
   EOF
   kubectl -n demo get instrumentation

4. 4. Instrument an application through annotation

   Why? The annotation is enough: the Operator injects the Python agent at pod start, without changing the image.

   Deploy a small Flask application in a standard Python image with the instrumentation.opentelemetry.io/inject-python annotation, then look at the injected init container and OTEL_* variables.
   See the correction

   cat <<'EOF' | kubectl apply -f -
   apiVersion: apps/v1
   kind: Deployment
   metadata: { name: catalogue, namespace: demo }
   spec:
     replicas: 1
     selector: { matchLabels: { app: catalogue } }
     template:
       metadata:
         labels: { app: catalogue }
         annotations: { instrumentation.opentelemetry.io/inject-python: "true" }
       spec:
         containers:
           - name: app
             image: python:3.12-slim
             command: ["sh", "-c"]
             args:
               - pip install flask >/dev/null && python -c "from flask import Flask; a=Flask('catalogue'); a.get('/')(lambda: 'ok'); a.run(host='0.0.0.0', port=8080)"
             ports: [{ containerPort: 8080 }]
   EOF
   kubectl -n demo rollout status deploy/catalogue --timeout=180s
   kubectl -n demo get pod -l app=catalogue -o jsonpath='{.items[0].spec.initContainers[*].name}{"\n"}'
   kubectl -n demo exec deploy/catalogue -- env | grep ^OTEL_

5. 5. Check the spans

   Why? The debug exporter prints received spans with the Kubernetes attributes added by k8sattributes.

   Generate a few requests through a port-forward, then look in the Collector logs for the GET / span, service.name and k8s.pod.name.
   See the correction

   kubectl -n demo port-forward deploy/catalogue 8080:8080 &
   for i in $(seq 1 5); do curl -s localhost:8080/ ; done ; kill %1
   kubectl -n demo logs deploy/demo-collector | grep -E 'Name +: GET|service.name|k8s.pod.name' | head

6. 6. Evolve and clean up

   Why? Changing the Instrumentation resource only affects recreated pods: deployments must be restarted.

   Set sampling to 10%, restart the deployment, check the new OTEL_TRACES_SAMPLER_ARG value, then delete the cluster.
   See the correction

   kubectl -n demo patch instrumentation defaut --type merge -p '{"spec":{"sampler":{"type":"parentbased_traceidratio","argument":"0.1"}}}'
   kubectl -n demo rollout restart deploy/catalogue && kubectl -n demo rollout status deploy/catalogue
   kubectl -n demo exec deploy/catalogue -- env | grep SAMPLER
   kind delete cluster --name otel

Difficulty level : ●●●○○ (3/5)