Lab: OpenTelemetry logs correlated with traces and file collection with correction

What you will learn in this TP

• Connect an existing logger to OpenTelemetry
• Correlate logs and traces
• Collect files with filelog

Introduction #

What you will learn in this section

• Connect an existing logger to OpenTelemetry
• Correlate logs and traces
• Collect files with filelog

Context. Billing logs are text files nobody links to traces. You connect the modern service to OpenTelemetry and collect the legacy application's logs without changing it.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 45 min

Structured logs correlated with traces, from both collection paths.

Hands-on #

Guided steps, with their solution.

1. 1. Start the observability stack

   Why? A single image provides an OTLP Collector and every backend: focus on the signal.

   Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.
   See the correction

   docker run -d --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 -p 4040:4040 grafana/otel-lgtm
   python3 -m venv .venv && . .venv/bin/activate
   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   # Grafana : http://localhost:3000 (admin / admin)

2. 2. Connect the logging module

   Why? The LoggingHandler turns each logging call into an OpenTelemetry LogRecord, with the active trace context.

   Write facturation.py: a LoggerProvider with OTLP exporter, a handler added to the root logger, a TracerProvider, and INFO and ERROR logs emitted inside spans.
   See the correction

   pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
   cat > facturation.py <<'EOF'
   import logging, random, time
   from opentelemetry import trace
   from opentelemetry._logs import set_logger_provider
   from opentelemetry.sdk.resources import Resource
   from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
   from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
   from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
   from opentelemetry.sdk.trace import TracerProvider
   from opentelemetry.sdk.trace.export import BatchSpanProcessor
   from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
   
   resource = Resource.create({"service.name": "facturation", "service.version": "2.1.0"})
   logs = LoggerProvider(resource=resource)
   logs.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
   set_logger_provider(logs)
   logging.basicConfig(level=logging.INFO)
   logging.getLogger().addHandler(LoggingHandler(level=logging.INFO, logger_provider=logs))
   traces = TracerProvider(resource=resource)
   traces.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
   trace.set_tracer_provider(traces)
   
   log = logging.getLogger("facturation")
   tracer = trace.get_tracer("tp.facturation")
   for n in range(20):
       with tracer.start_as_current_span("emettre facture"):
           log.info("facture émise", extra={"shop.invoice.id": f"F-{n}", "shop.invoice.amount": random.randint(10, 900)})
           if n % 5 == 0:
               log.error("échec d'envoi au client", extra={"shop.invoice.id": f"F-{n}"})
       time.sleep(0.2)
   logs.shutdown() ; traces.shutdown()
   EOF
   python facturation.py

3. 3. Query and correlate in Grafana

   Why? The trace_id carried by each log lets you jump from the log to the trace and back.

   In Explore > Loki, show the service's errors, expand a line and spot trace_id, span_id, severity and your attributes. Open the linked trace, then from a Tempo span show its logs.
   See the correction

   {service_name="facturation"} | severity_text="ERROR"
   {service_name="facturation"} | shop_invoice_id="F-10"
   sum by (severity_text) (count_over_time({service_name="facturation"}[5m]))

4. 4. Collect a legacy application's logs

   Why? Without touching code, the filelog receiver reads files and parses JSON, severity and trace context.

   Generate a JSON log file, then run a contrib Collector on the same network as the stack, with filelog, json_parser, trace_parser and a resource processor setting service.name.
   See the correction

   mkdir -p logs
   for i in $(seq 1 30); do
     lvl=$([ $((i % 6)) -eq 0 ] && echo ERROR || echo INFO)
     printf '{"time":"%s","level":"%s","message":"traitement lot %d","trace_id":"%s","span_id":"%s"}\n' \
       "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$lvl" "$i" "$(openssl rand -hex 16)" "$(openssl rand -hex 8)" >> logs/app.log
   done
   cat > collector.yaml <<'EOF'
   receivers:
     filelog:
       include: [/logs/app.log]
       start_at: beginning
       operators:
         - type: json_parser
           severity: { parse_from: attributes.level }
         - type: trace_parser
           trace_id: { parse_from: attributes.trace_id }
           span_id: { parse_from: attributes.span_id }
         - type: move
           from: attributes.message
           to: body
   processors:
     resource:
       attributes: [{ key: service.name, value: facturation-historique, action: upsert }]
     batch: {}
   exporters:
     otlphttp: { endpoint: http://lgtm:4318 }
   service:
     pipelines:
       logs: { receivers: [filelog], processors: [resource, batch], exporters: [otlphttp] }
   EOF
   docker network create otel 2>/dev/null ; docker network connect otel lgtm
   docker run -d --name col-logs --network otel -v "$PWD/collector.yaml:/etc/otelcol-contrib/config.yaml" -v "$PWD/logs:/logs" otel/opentelemetry-collector-contrib:latest
   # Loki : {service_name="facturation-historique"} | severity_text="ERROR"

5. 5. Reduce and mask

   Why? The Collector removes noise and personal data before storage.

   Add a filter processor dropping INFO logs and a transform masking e-mail addresses in the body, then check in Loki.
   See the correction

   processors:
     filter/info:
       logs:
         log_record: ['severity_number < SEVERITY_NUMBER_WARN']
     transform/pii:
       log_statements:
         - context: log
           statements:
             - replace_pattern(body, "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+", "***@***")
   # pipeline : processors: [resource, filter/info, transform/pii, batch]
   # docker restart col-logs, puis nettoyage : docker rm -f col-logs

Difficulty level : ●●●○○ (3/5)