RousselTMLEARNING

Observability

Bindplane in production: high availability, security, Kubernetes and configuration as code

Operate Bindplane in production: active-active servers, PostgreSQL and event bus, TLS for OpAMP, collector high availability, Kubernetes, CLI, Terraform and monitoring.

1. Server high availability #

The control plane.

Architecture #

Bindplane in high availability
  • Bindplane servers are stateless and run active-active behind a load balancer (which must support WebSockets for OpAMP).
  • Components to provide: load balancer, highly available PostgreSQL, Prometheus for measurements and an event bus between servers (for example Google Pub/Sub in the vendor's reference example).
  • All servers have the same configuration.
Good to know: Reminder: a server outage does not stop collection. Server high availability protects the ability to change configurations, not the data flow.

2. Collection high availability #

The data plane.

Losing nothing #

RiskProtection
Destination unavailableDisk-backed persistent queue and retries in the collector
Gateway downSeveral gateway collectors behind an OTLP load balancer
Volume spikeBatch, memory limiter, gateways sized on measured throughput
Bad configurationBatched rollouts with pause on error, rollback

3. Security #

Protecting the control plane.

Measures #

  • TLS on the interface and on OpAMP (wss://); on Kubernetes, a CA certificate mounted in the collector (OPAMP_TLS_CA), or even mutual TLS (OPAMP_TLS_CERT, OPAMP_TLS_KEY).
  • The collectors' secret key treated like a password and rotated if it leaks.
  • Named accounts, SSO and RBAC (Admin, User, Viewer) depending on the edition; no shared admin account.
  • Destination secrets (tokens, passwords) stored in Bindplane rather than in files on each server.

4. Configuration as code #

Reproducible and reviewed.

CLI and Terraform #

# Exporter tout ce qui a été construit dans l'interface
mkdir -p bindplane/ && for c in $(bindplane get configurations --output json | jq -r '.[].metadata.name'); do
  bindplane get configuration "$c" --export --output yaml > "bindplane/$c.yaml"
done
git add bindplane/ && git commit -m "Export des configurations Bindplane"

# En CI, sur le serveur cible
bindplane apply bindplane/**/*
bindplane rollout start --all

An official Terraform provider also lets you declare configurations, sources, processors and destinations as resources, with code review and a plan before applying.

Common mistake: If both the interface and Git modify the same configurations, the next Git apply will overwrite manual changes. Decide which one is the source of truth, and reserve the interface for exploration and snapshots.
From a reviewed change to a controlled deployment
Applying a configuration and rolling it out are two separate steps. Keep a single source of truth to avoid overwrites between Git and the interface.

5. Monitoring Bindplane #

The pipeline is a critical service.

What to track #

  • Disconnected or failing collectors (Collectors page, or bindplane get agents).
  • Throughput per destination: a sudden drop signals a collection or network problem.
  • Persistent queue size: it grows when a destination is slow.
  • Health of the server, PostgreSQL and Prometheus.
Throughput and queue: fictitious example
A drop in throughput together with a growing queue calls for checking the destination and the network. The indices are fictitious and each series has its own baseline; they give neither a volume nor an alert threshold.

Related articles

Bindplane: managing your OpenTelemetry telemetry pipelines

Understand Bindplane: a control plane for OpenTelemetry collectors via OpAMP, BDOT Collector, configurations,…

OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines

Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in YAML, Grafana…

Installing Bindplane: server, BDOT Collector and OpAMP connection

Install the Bindplane server (Docker Compose, Linux, Kubernetes), connect BDOT collectors via OpAMP on Linux,…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!