RousselTMLEARNING

Article · OpenTelemetry: From Discovery to Expertise

OpenTelemetry on Kubernetes

1. Collector deployment modes #

Agent, gateway, sidecar.

Choosing #

OpenTelemetry architecture on Kubernetes
The per-node agent collects and enriches; the gateway samples, routes and exports.
ModeTypical use
DaemonSet (agent)one Collector per node: local OTLP reception, pod logs, kubelet and host metrics, adding Kubernetes metadata
Deployment (gateway)central point: tail sampling, routing, authentication, export to backends
StatefulSetdistributed Prometheus scraping with the Target Allocator
Sidecara Collector in each pod: strong isolation, but expensive

2. The OpenTelemetry Operator #

Managing collectors and instrumentation through CRDs.

Installing #

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
# ou : helm install opentelemetry-operator open-telemetry/opentelemetry-operator -n opentelemetry-operator-system --create-namespace

A Collector #

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata: { name: agent, namespace: observability }
spec:
  mode: daemonset
  config:
    receivers:
      otlp: { protocols: { grpc: { endpoint: 0.0.0.0:4317 }, http: { endpoint: 0.0.0.0:4318 } } }
    processors:
      memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 25 }
      k8sattributes: {}
      batch: {}
    exporters:
      otlp: { endpoint: gateway-collector.observability:4317, tls: { insecure: true } }
    service:
      pipelines:
        traces: { receivers: [otlp], processors: [memory_limiter, k8sattributes, batch], exporters: [otlp] }

Auto-instrumentation #

Injection by the Operator
After changing the Instrumentation resource, restart the pods.
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: { name: defaut, namespace: shop }
spec:
  exporter: { endpoint: http://agent-collector.observability:4318 }
  propagators: [tracecontext, baggage]
  sampler: { type: parentbased_traceidratio, argument: "0.25" }
  resource: { addK8sUIDAttributes: true }
# puis, dans le template du pod :
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"     # ou -python, -nodejs, -dotnet, -go

When the pod starts, the Operator adds an init container that copies the language agent, and sets the OTEL_* variables (service name derived from the deployment, Kubernetes attributes). No change to the image is required.

Common mistake: Injection only happens when the pod is created: after adding the annotation or changing the Instrumentation resource, restart the deployments (kubectl rollout restart).

3. Kubernetes metadata and metrics #

Enriching and completing.

k8sattributes and receivers #

ComponentContribution
k8sattributes processoradds k8s.pod.name, k8s.namespace.name, k8s.deployment.name, k8s.node.name, labels and annotations to all data
kubeletstats receiverCPU, memory, network, volumes per pod and container
k8s_cluster receiverobject state (deployments, failing pods, nodes)
k8sobjects receiverKubernetes events as logs
Target Allocatordistributes Prometheus targets across several Collectors
Good practice: The k8sattributes processor needs RBAC read permissions on pods, namespaces and ReplicaSets: the Operator and the Helm chart create them, a manual installation must add them.

Helm charts #

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install agent open-telemetry/opentelemetry-collector -n observability --create-namespace \
  --set mode=daemonset --set image.repository=otel/opentelemetry-collector-k8s \
  --set presets.kubernetesAttributes.enabled=true \
  --set presets.logsCollection.enabled=true \
  --set presets.kubeletMetrics.enabled=true

4. Operating in production #

Points to watch.

Checklist #

  • memory_limiter as the first processor and resource limits on each Collector.
  • Gateway with several replicas and loadbalancing by TraceID if tail sampling is used.
  • Collectors' internal telemetry monitored (refusals, send failures, queue).
  • Operator, Collector and agent versions aligned and upgraded together.
  • Instrumentation resources per namespace to adapt sampling to each team.

📄 OpenTelemetry on Kubernetes: Operator, auto-instrumentation and Collector →

My progress

  • ⏱ Estimated time : ≈ 9 min

Sign in to track your progress and take notes. Sign in

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Recommended Articles

OpenTelemetry (OTel) best practices: the guide to modern observability

Master OpenTelemetry with our expert advice: implementing the Collector, following semantic conve...

OpenTelemetry: automatic vs manual instrumentation

Understand the differences between automatic and manual instrumentation with OpenTelemetry, their...

OpenTelemetry Collector: processors (transforming, filtering and protecting telemetry)

Master OpenTelemetry Collector processors: batch, memory_limiter, attributes, resource, filter, t...

Capstone project: the four signals of the OpenTelemetry Demo in the Grafana stack

A capstone project spanning the observability courses: deploy the OpenTelemetry Demo, send metric...

Core OpenTelemetry concepts: signals, resources, context and maturity

The essential OpenTelemetry vocabulary: the four signals, API and SDK, resource and instrumentati...

OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines

Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in Y...

OpenTelemetry Collector: connectors, extensions and observing the Collector

Go further with the Collector: connectors (spanmetrics, servicegraph, count, routing, failover), ...

Grafana Alloy vs OpenTelemetry Collector: Which One Should You Choose?

A detailed comparison between Grafana Alloy and the OpenTelemetry Collector. Discover the strengt...

Sending traces to Tempo: OpenTelemetry instrumentation and Alloy

Produce and route traces to Tempo: Python/Java/Node auto-instrumentation, the manual SDK, OTEL en...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!