RousselTMLEARNING

Article · OpenTelemetry: From Discovery to Expertise

Logs in depth

1. The data model #

A log enriched with context.

LogRecord #

OpenTelemetry does not create a new logging framework: it defines a common data model and bridges to existing libraries. A log record contains:

FieldContent
Timestamp / ObservedTimestamptime of the event / time of its collection
SeverityNumber / SeverityTextnormalized 1-24 (9 = INFO, 13 = WARN, 17 = ERROR) / original text
Bodythe message (text or structure)
Attributescontext (semconv: code.function.name, exception.type…)
TraceId / SpanId / TraceFlagsthe trace and span active when the log was written
Resourcethe emitting service, identical to its traces and metrics
EventNamename of a structured event (« event » logs)
Severity levels
Filtering on severity_number works for all languages, whatever the original text.

Why it is useful #

  • A single format, whatever the language or logging library.
  • Automatic correlation: each log carries the identifier of the current trace.
  • The same resource as the other signals: filter logs, traces and metrics of the same pod in one click.
  • The same transport (OTLP) and the same Collector as the other signals.

2. From the application #

Appenders and the Logs Bridge API.

The principle #

Log collection
An appender plugs your existing logger into the OpenTelemetry SDK, without changing the log calls.

The Logs Bridge API is not meant to be called by your code: it is for authors of appenders (or handlers) that plug Logback, Log4j, java.util.logging, Python's logging module, Serilog or Winston into the SDK. You keep writing logger.info(...).

Examples #

# Python : avec l'auto-instrumentation
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
export OTEL_LOGS_EXPORTER=otlp
opentelemetry-instrument python app.py

# Java : l'agent capture Logback et Log4j par défaut
java -javaagent:opentelemetry-javaagent.jar -Dotel.logs.exporter=otlp -jar app.jar
# Python : configuration manuelle du handler
import logging
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter

provider = LoggerProvider(resource=resource)
provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
set_logger_provider(provider)
logging.getLogger().addHandler(LoggingHandler(level=logging.INFO, logger_provider=provider))
Good practice: Even without exporting logs over OTLP, add trace_id and span_id to your JSON logs: correlation will also work with file-based collection.

3. From files #

The Collector's filelog receiver.

Parsing JSON logs #

receivers:
  filelog:
    include: [/var/log/app/*.log]
    start_at: end
    storage: file_storage                    # reprendre là où on s'était arrêté
    operators:
      - type: json_parser
        timestamp: { parse_from: attributes.time, layout: "%Y-%m-%dT%H:%M:%S%z" }
        severity: { parse_from: attributes.level }
      - type: trace_parser
        trace_id: { parse_from: attributes.trace_id }
        span_id:  { parse_from: attributes.span_id }
      - type: move
        from: attributes.message
        to: body

Kubernetes logs #

On Kubernetes, a Collector deployed as a DaemonSet reads /var/log/pods/*/*/*.log, uses the container operator to decode the runtime format, then the k8sattributes processor to add pod, namespace, deployment and labels. The official Helm chart enables this with the logsCollection preset.

Common mistake: Do not collect the same logs twice (OTLP appender and file): doubled volume and duplicates in the backend. Choose one path per application.

4. Processing and querying #

Reduce, mask, find.

In the Collector #

processors:
  filter/debug:
    logs:
      log_record: ['severity_number < SEVERITY_NUMBER_INFO']      # supprimer DEBUG et TRACE
  transform/pii:
    log_statements:
      - context: log
        statements:
          - replace_pattern(body, "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+", "***@***")

In Loki #

{service_name="checkout"} | severity_text="ERROR"
{service_name="checkout"} | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"
sum by (service_name) (count_over_time({deployment_environment_name="production"} | severity_text="ERROR" [5m]))

📄 Logs with OpenTelemetry: data model, appenders, correlation and collection →

My progress

  • ⏱ Estimated time : ≈ 9 min

Sign in to track your progress and take notes. Sign in

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Recommended Articles

OpenTelemetry traces in depth: spans, status, events, links and sampling

Master the trace signal: anatomy of a distributed trace, span and SpanKind, attributes, status an...

Correlating traces, metrics, logs and profiles with OpenTelemetry

Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...

Profiles, OpenTelemetry's fourth signal: continuous profiling and correlation with traces

Understand OpenTelemetry's new profile signal: continuous profiling, flame graphs, data model and...

OpenTelemetry: automatic vs manual instrumentation

Understand the differences between automatic and manual instrumentation with OpenTelemetry, their...

Datadog APM and OpenTelemetry: traces, services and correlation with logs

Instrument applications for Datadog APM (dd-trace, single step instrumentation, OpenTelemetry), u...

Capstone project: the four signals of the OpenTelemetry Demo in the Grafana stack

A capstone project spanning the observability courses: deploy the OpenTelemetry Demo, send metric...

Ingesting OpenTelemetry logs into Elasticsearch

Collect and structure logs with the OpenTelemetry Collector (filelog receiver, operators, transfo...

Linking traces and profiles: span profiles with Tempo and Pyroscope

Go from a slow trace to the exact CPU profile of that span: span profiles in Go and Java, configu...

Sending traces to Tempo: OpenTelemetry instrumentation and Alloy

Produce and route traces to Tempo: Python/Java/Node auto-instrumentation, the manual SDK, OTEL en...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!