Environment variables and OpenTelemetry

1. The key point: Alloy is not an SDK #

Who reads which variable, and why the confusion is common.

Three families of variables not to confuse #

A common misconception: “Alloy is OpenTelemetry compatible, so setting the OTEL_* variables is enough”. That is wrong. These variables are defined by the specification for SDKs, the libraries embedded in your applications. Alloy is a collector: it receives OTLP, but it is configured through its components.

How these results were checked: the behaviours in this article were tested on 4 November 2026 with Grafana Alloy v1.21.3, by sending real OTLP traces. The list of variables changes over time: check the documentation of your version.

2. Variables read natively by Alloy #

Memory, Go runtime, proxy and Linux packages.

Memory and Go runtime #

Alloy is written in Go, so it honours the runtime variables. Together with the proxy variables, they are the only ones the Alloy documentation lists as read automatically.

GOMEMLIMIT is not a hard limit: the documentation says so explicitly. Keep a margin below your container limit, or the process can still be killed (OOM).

Outbound proxy #

Linux packages (systemd) #

With the deb and rpm packages, the service reads its settings from /etc/default/alloy (Debian, Ubuntu) or /etc/sysconfig/alloy (RHEL, Fedora, SUSE). After a change: sudo systemctl restart alloy.

3. Your variables: sys.env() #

Explicit reads, default values and pitfalls.

Reading a variable in the configuration #

Any other variable must be read explicitly with sys.env(). If it does not exist, the function returns an empty string without an error, which is why it is paired with coalesce() to provide a default value.

otelcol.exporter.otlp "tempo" {
  client {
    // host:port for this gRPC exporter; default value when the variable is missing
    endpoint = coalesce(sys.env("TEMPO_ENDPOINT"), "tempo:4317")
    headers  = { "X-Scope-OrgID" = coalesce(sys.env("TEMPO_TENANT"), "anonymous") }
    tls {
      insecure = true
    }
  }
}

Pitfall: without a default, the failure happens later. With endpoint = sys.env("MISSING") on otelcol.exporter.otlphttp, alloy run stops with “at least one endpoint must be specified”. The message does not name the variable: think of it first.

An environment variable is only read when the process starts: changing it means restarting Alloy. For a value that must change at runtime, use local.file or remote.http. Secrets are covered in the Secrets Management lesson.

4. OTEL_RESOURCE_ATTRIBUTES: the variable Alloy can read #

The env detector, the override option and its pitfall.

The resourcedetection component #

The otelcol.processor.resourcedetection component has an env detector, enabled by default (detectors = ["env"]). It reads OTEL_RESOURCE_ATTRIBUTES in the key1=value1,key2=value2 format and adds these attributes to all data passing through. This is how you set the environment or cluster once, at collector level.

The override pitfall #

These four results were measured with the same trace sent by an application named from-app. Three lessons:

• Without the component, the variable has no effect. Setting it for Alloy is not enough.
• override defaults to true: an existing attribute is replaced. If the variable contains service.name, every application going through this Alloy takes that name, and your services merge in Tempo.
• OTEL_SERVICE_NAME is ignored by this detector, unlike an SDK where it takes precedence.

Best practice: on an Alloy receiving several applications, use override = false and limit the variable to site attributes: deployment.environment.name, k8s.cluster.name, cloud.region. Never include service.name.

5. OTEL_* variables on the application side #

Pointing an SDK at Alloy with the standard variables.

General SDK variables #

OTLP exporter variables #

Each one also exists per signal, for example OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, which then overrides the general variable.

The path pitfall: with OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4318, the SDK appends /v1/traces. With the per-signal OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, the URL is used as is: set http://alloy:4318/v1/traces. Another frequent mistake: sending http/protobuf (the default protocol) to the gRPC port 4317.

Full example: an application and Alloy #

The application gets its standard variables; Alloy gets those about the site. Its configuration, validated with alloy validate and tested end to end, explicitly reuses the destination:

otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }
  output {
    traces  = [otelcol.processor.resourcedetection.env.input]
    metrics = [otelcol.processor.resourcedetection.env.input]
    logs    = [otelcol.processor.resourcedetection.env.input]
  }
}

// Adds the OTEL_RESOURCE_ATTRIBUTES attributes without overwriting those of the applications
otelcol.processor.resourcedetection "env" {
  detectors = ["env"]
  override  = false
  output {
    traces  = [otelcol.processor.batch.default.input]
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    traces  = [otelcol.exporter.otlphttp.backend.input]
    metrics = [otelcol.exporter.otlphttp.backend.input]
    logs    = [otelcol.exporter.otlphttp.backend.input]
  }
}

// Explicitly reuses the standard variable: without sys.env it would not be read
otelcol.exporter.otlphttp "backend" {
  client {
    endpoint = sys.env("OTEL_EXPORTER_OTLP_ENDPOINT")
    headers  = { "X-Scope-OrgID" = coalesce(sys.env("BACKEND_TENANT"), "anonymous") }
  }
}

services:
  alloy:
    image: grafana/alloy:latest
    command: run /etc/alloy/config.alloy --server.http.listen-addr=0.0.0.0:12345 --storage.path=/var/lib/alloy/data
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy:ro
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://tempo:4318
      OTEL_RESOURCE_ATTRIBUTES: deployment.environment.name=prod,k8s.cluster.name=eu-west
      GOMEMLIMIT: 900MiB
    deploy:
      resources:
        limits:
          memory: 1G

  app:
    image: my-registry/orders:1.4.0
    environment:
      OTEL_SERVICE_NAME: orders
      OTEL_RESOURCE_ATTRIBUTES: service.version=1.4.0,team=payments
      OTEL_EXPORTER_OTLP_ENDPOINT: http://alloy:4318
      OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
      OTEL_TRACES_SAMPLER: parentbased_traceidratio
      OTEL_TRACES_SAMPLER_ARG: "0.1"

In our test, the trace reached /v1/traces on the backend: otelcol.exporter.otlphttp appends each signal path, like an SDK with the general variable.

Mapping: otelcol.exporter.otlphttp expects a URL (http://tempo:4318) and otelcol.exporter.otlp, over gRPC, a host:port (tempo:4317). Without endpoint, the latter is rejected at load time (“missing required attribute endpoint”), even when OTEL_EXPORTER_OTLP_ENDPOINT is set.

6. The OpenTelemetry engine: alloy otel #

The Collector ${env:VAR} syntax, in public preview.

A Collector YAML configuration #

The Alloy binary also ships an OpenTelemetry engine, started with alloy otel --config=otel.yaml. It reads a YAML file in the Collector format, which uses the ${env:VARIABLE} syntax instead of sys.env(). We checked this substitution and the env detector on Alloy v1.21.3.

receivers:
  otlp:
    protocols:
      http:
        endpoint: ${env:RECEIVER_ENDPOINT}
processors:
  resourcedetection:
    detectors: [env]
exporters:
  otlphttp:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resourcedetection]
      exporters: [otlphttp]

This engine is in public preview: the documentation warns that breaking changes are possible. For production, the default engine (alloy run) remains the stable choice.

7. Kubernetes and best practices #

Helm, Beyla, common mistakes and commands.

On Kubernetes #

# values.yaml of the grafana/alloy Helm chart
alloy:
  extraEnv:
    - name: OTEL_RESOURCE_ATTRIBUTES
      value: "deployment.environment.name=prod,k8s.cluster.name=eu-west"
    - name: GOMEMLIMIT
      value: "900MiB"
  resources:
    limits:
      memory: 1Gi

The Beyla special case: the beyla.ebpf component can pick up, through extra_resource_labels, attributes supplied by the OTEL_RESOURCE_ATTRIBUTES variable of the observed service. The application's variable is what matters, not Alloy's.

Key takeaways #

• Setting OTEL_* for Alloy does not configure Alloy, except OTEL_RESOURCE_ATTRIBUTES with resourcedetection.
• Read your values with sys.env() and provide defaults with coalesce().
• Use override = false on a shared Alloy, and never service.name in its variable.
• Keep OTEL_SERVICE_NAME and OTEL_EXPORTER_OTLP_* for the applications.
• Set GOMEMLIMIT below the container memory limit.

Commands to know #

# Check the configuration with the variables of the current environment
TEMPO_ENDPOINT=tempo:4317 alloy validate config.alloy

# Format a file, then start it
alloy fmt config.alloy
alloy run config.alloy --server.http.listen-addr=127.0.0.1:12345

# See the variables actually received by the process
docker compose exec alloy env | grep -E 'OTEL_|GOMEM'

# Test the pipeline with a minimal trace (OTLP/HTTP)
curl -X POST -H 'Content-Type: application/json' -d @trace.json http://localhost:4318/v1/traces

# OpenTelemetry engine (public preview)
alloy otel --config=otel.yaml

Official sources #

• Grafana Alloy : environment variables
• otelcol.processor.resourcedetection
• otelcol.exporter.otlp
• Fonction sys.env
• Commande alloy otel
• Configurer Alloy sur Linux
• OpenTelemetry : SDK environment variables
• OpenTelemetry : OTLP exporter