1. Collector deployment modes #
Agent, gateway, sidecar.
Choosing #
| Mode | Typical use |
|---|---|
| DaemonSet (agent) | one Collector per node: local OTLP reception, pod logs, kubelet and host metrics, adding Kubernetes metadata |
| Deployment (gateway) | central point: tail sampling, routing, authentication, export to backends |
| StatefulSet | distributed Prometheus scraping with the Target Allocator |
| Sidecar | a Collector in each pod: strong isolation, but expensive |
2. The OpenTelemetry Operator #
Managing collectors and instrumentation through CRDs.
Installing #
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
# ou : helm install opentelemetry-operator open-telemetry/opentelemetry-operator -n opentelemetry-operator-system --create-namespaceA Collector #
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata: { name: agent, namespace: observability }
spec:
mode: daemonset
config:
receivers:
otlp: { protocols: { grpc: { endpoint: 0.0.0.0:4317 }, http: { endpoint: 0.0.0.0:4318 } } }
processors:
memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 25 }
k8sattributes: {}
batch: {}
exporters:
otlp: { endpoint: gateway-collector.observability:4317, tls: { insecure: true } }
service:
pipelines:
traces: { receivers: [otlp], processors: [memory_limiter, k8sattributes, batch], exporters: [otlp] }Auto-instrumentation #
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: { name: defaut, namespace: shop }
spec:
exporter: { endpoint: http://agent-collector.observability:4318 }
propagators: [tracecontext, baggage]
sampler: { type: parentbased_traceidratio, argument: "0.25" }
resource: { addK8sUIDAttributes: true }# puis, dans le template du pod :
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: "true" # ou -python, -nodejs, -dotnet, -goWhen the pod starts, the Operator adds an init container that copies the language agent, and sets the OTEL_* variables (service name derived from the deployment, Kubernetes attributes). No change to the image is required.
Common mistake: Injection only happens when the pod is created: after adding the annotation or changing the Instrumentation resource, restart the deployments (kubectl rollout restart).3. Kubernetes metadata and metrics #
Enriching and completing.
k8sattributes and receivers #
| Component | Contribution |
|---|---|
k8sattributes processor | adds k8s.pod.name, k8s.namespace.name, k8s.deployment.name, k8s.node.name, labels and annotations to all data |
kubeletstats receiver | CPU, memory, network, volumes per pod and container |
k8s_cluster receiver | object state (deployments, failing pods, nodes) |
k8sobjects receiver | Kubernetes events as logs |
| Target Allocator | distributes Prometheus targets across several Collectors |
Good practice: The k8sattributes processor needs RBAC read permissions on pods, namespaces and ReplicaSets: the Operator and the Helm chart create them, a manual installation must add them.Helm charts #
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install agent open-telemetry/opentelemetry-collector -n observability --create-namespace \
--set mode=daemonset --set image.repository=otel/opentelemetry-collector-k8s \
--set presets.kubernetesAttributes.enabled=true \
--set presets.logsCollection.enabled=true \
--set presets.kubeletMetrics.enabled=true4. Operating in production #
Points to watch.
Checklist #
memory_limiteras the first processor and resource limits on each Collector.- Gateway with several replicas and
loadbalancingby TraceID if tail sampling is used. - Collectors' internal telemetry monitored (refusals, send failures, queue).
- Operator, Collector and agent versions aligned and upgraded together.
- Instrumentation resources per namespace to adapt sampling to each team.
My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Quels attributs utiliser pour nommer mes spans HTTP (semconv) ?✓ solved · 1 reply(ies)
- OpenTelemetry Collector : mes traces arrivent mais pas mes métriques✓ solved · 2 reply(ies)
Ask a question about this lesson →See all discussions about this course
Recommended Articles
OpenTelemetry (OTel) best practices: the guide to modern observability
Master OpenTelemetry with our expert advice: implementing the Collector, following semantic conve...
OpenTelemetry: automatic vs manual instrumentation
Understand the differences between automatic and manual instrumentation with OpenTelemetry, their...
OpenTelemetry Collector: processors (transforming, filtering and protecting telemetry)
Master OpenTelemetry Collector processors: batch, memory_limiter, attributes, resource, filter, t...
Capstone project: the four signals of the OpenTelemetry Demo in the Grafana stack
A capstone project spanning the observability courses: deploy the OpenTelemetry Demo, send metric...
Core OpenTelemetry concepts: signals, resources, context and maturity
The essential OpenTelemetry vocabulary: the four signals, API and SDK, resource and instrumentati...
OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines
Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in Y...
OpenTelemetry Collector: connectors, extensions and observing the Collector
Go further with the Collector: connectors (spanmetrics, servicegraph, count, routing, failover), ...
Grafana Alloy vs OpenTelemetry Collector: Which One Should You Choose?
A detailed comparison between Grafana Alloy and the OpenTelemetry Collector. Discover the strengt...
Sending traces to Tempo: OpenTelemetry instrumentation and Alloy
Produce and route traces to Tempo: Python/Java/Node auto-instrumentation, the manual SDK, OTEL en...