RousselTMLEARNING

Article · OpenTelemetry: From Discovery to Expertise

Connectors, extensions and Collector observability

1. Connectors #

Passing data from one pipeline to another.

Principle #

spanmetrics connector
RED metrics (rate, errors, duration) are computed from 100% of spans, before sampling.

A connector is declared as the exporter of one pipeline and the receiver of another. It lets you turn one signal into another, route or duplicate data.

The most useful ones #

ConnectorRole
spanmetricscomputes rate, errors and durations (histograms) per service and operation from spans
servicegraphproduces the metrics of the dependency map between services
countcounts spans, logs or metric points matching an OTTL condition
routingsends data to different pipelines based on an attribute (tenant, environment)
failoverswitches to a backup pipeline if the main one fails
forwardsimply links two pipelines of the same type
connectors:
  spanmetrics:
    histogram: { explicit: { buckets: [5ms, 25ms, 100ms, 250ms, 1s, 5s] } }
    dimensions: [{ name: http.request.method }, { name: http.response.status_code }]
  routing:
    default_pipelines: [logs/standard]
    table:
      - condition: resource.attributes["tenant"] == "premium"
        pipelines: [logs/premium]
service:
  pipelines:
    traces:          { receivers: [otlp], processors: [batch], exporters: [otlp/tempo, spanmetrics] }
    metrics/spans:   { receivers: [spanmetrics], exporters: [prometheusremotewrite] }
Good practice: Place spanmetrics before any tail sampling: metrics stay accurate even if you keep only 10% of the traces.

2. Extensions #

What is not part of the data flow.

Overview #

ExtensionUsage
health_checkHTTP health endpoint for Kubernetes probes and load balancers
pprofprofiling of the Collector itself
zpageslive diagnostic pages (pipelines, recent spans)
file_storagedisk storage for the persistent queue and file read positions
bearertokenauth, oauth2client, basicauthauthentication with backends or clients
opampremote management of the Collector (see Bindplane)
extensions:
  health_check: { endpoint: 0.0.0.0:13133 }
  zpages: { endpoint: 0.0.0.0:55679 }
  file_storage: { directory: /var/lib/otelcol/queue }
exporters:
  otlp/backend:
    endpoint: backend:4317
    sending_queue: { enabled: true, storage: file_storage }   # survit à un redémarrage
service:
  extensions: [health_check, zpages, file_storage]

3. Observing the Collector #

You have to monitor the monitor.

Internal telemetry #

The Collector exposes its own metrics (by default in Prometheus format on port 8888) and can send its internal logs and traces over OTLP. Three questions to watch: is it receiving? is it losing data? is it saturating?

MetricSignals
otelcol_receiver_accepted_spans / _refused_spansdata accepted or refused at the input
otelcol_processor_dropped_*, memory_limiter refusalsdata lost due to lack of memory
otelcol_exporter_sent_* / _send_failed_*successful or failed sends to the backend
otelcol_exporter_queue_size / _queue_capacityqueue filling up: backend too slow or unreachable
service:
  telemetry:
    logs: { level: info }
    metrics:
      level: detailed
      readers:
        - pull: { exporter: { prometheus: { host: 0.0.0.0, port: 8888 } } }
Common mistake: An alert on otelcol_exporter_send_failed_* and on queue filling is better than discovering a gap of several hours in the traces during an incident.
Points where the Collector can lose data
A gap between accepted and sent signals a loss: memory, filter or unreachable backend.

4. Distributions and custom Collector #

Shipping only the components you need.

Choosing #

DistributionContent
otelcol (core)essential components, maintained by the project
otelcol-contribhundreds of community components: ideal for testing
otelcol-k8sa selection suited to Kubernetes
Vendor distributionsGrafana Alloy, Bindplane BDOT, Splunk, Elastic EDOT, AWS ADOT…
# builder-config.yaml pour ocb (OpenTelemetry Collector Builder)
dist: { name: otelcol-maison, output_path: ./dist }
receivers:  [{ gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.115.0 }]
processors: [{ gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.115.0 }]
exporters:  [{ gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.115.0 }]
Good practice: In production, build a distribution with ocb that contains only your components: smaller binary, reduced attack surface, controlled upgrades. Align the version numbers of all modules.

📄 OpenTelemetry Collector: connectors, extensions and observing the Collector →

My progress

  • ⏱ Estimated time : ≈ 8 min

Sign in to track your progress and take notes. Sign in

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Recommended Articles

Grafana Alloy vs OpenTelemetry Collector: Which One Should You Choose?

A detailed comparison between Grafana Alloy and the OpenTelemetry Collector. Discover the strengt...

OpenTelemetry Collector: exporters (sending telemetry to backends)

Configure OpenTelemetry Collector exporters: OTLP gRPC/HTTP, Prometheus, prometheusremotewrite, L...

OpenTelemetry Collector: processors (transforming, filtering and protecting telemetry)

Master OpenTelemetry Collector processors: batch, memory_limiter, attributes, resource, filter, t...

OpenTelemetry (OTel) best practices: the guide to modern observability

Master OpenTelemetry with our expert advice: implementing the Collector, following semantic conve...

OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines

Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in Y...

OTLP explained: understanding the OpenTelemetry protocol

The OTLP protocol explained simply. Understand the differences between gRPC and HTTP for exportin...

OpenTelemetry architecture: understanding the 3 levels

A deep dive into the OpenTelemetry architecture. Learn how data flows from the application to the...

OpenTelemetry on Kubernetes: Operator, auto-instrumentation and Collector

Deploy OpenTelemetry on Kubernetes: Collector deployment modes (DaemonSet, Deployment, sidecar, S...

Grafana Alloy: collecting and transforming telemetry

Learn how to use Grafana Alloy to collect, transform and route logs, metrics and traces. Discover...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!