Deploy a complete observability pipeline with an Agent and a Gateway with correction

What you will learn in this TP

• Deploy a complete visualization backend (Grafana, Prometheus, Loki, Tempo)
• Configure and start an OTel collector in Gateway mode
• Configure and start an OTel collector in Agent mode
• Instrument an application to send its telemetry to the Agent
• Visualize traces, metrics and logs in Grafana
• Clone the configuration repository
• Start the infrastructure with Docker Compose
• Confirm access to the visualization tools
• Understand the structure of a distributed trace
• Generate traffic and find the matching trace
• Analyze the path of a request
• Understand how the weather API call is made
• Simulate a production release to enable context propagation
• Verify the instrumentation and trace correlation in the console
• Create the gateway's configuration file
• Start the gateway container
• Create the agent's configuration
• Start the agent and the demo application

Target architecture #

What you will learn in this section

• Deploy a complete visualization backend (Grafana, Prometheus, Loki, Tempo)
• Configure and start an OTel collector in Gateway mode
• Configure and start an OTel collector in Agent mode
• Instrument an application to send its telemetry to the Agent
• Visualize traces, metrics and logs in Grafana

The goal is to build a pipeline where an instrumented application sends its data to a **local OTel agent**. This agent then forwards the data to a **centralized OTel gateway**, which processes it and exports it to our storage and visualization backend (stack: Loki, Grafana, Tempo, Prometheus).

Why this Agent + Gateway architecture?

• Resilience: the agent can buffer data if the gateway becomes unavailable.
• Performance: being local, the agent adds very little latency for the application. Heavy processing (filtering, enrichment) is offloaded to the gateway.
• Scalability: you can scale the gateway independently from the applications to handle the load of thousands of agents.
• Security: only the gateway needs credentials to connect to the final backends.

Deploying the observability backend #

What you will learn in this section

• Clone the configuration repository
• Start the infrastructure with Docker Compose
• Confirm access to the visualization tools

We'll start by deploying the backend that will centralize and visualize all our telemetry data.

1. Get the configuration files

   Get the configuration files needed to deploy the backend.

   git clone https://github.com/RousselTM/otel-formation.git 
   cd otel-formation

2. Deploy the backend with Docker Compose

   Start the services: Prometheus for metrics, Loki for logs, Tempo for traces and Grafana for visualization.

   docker compose up -d

3. Check access to the demo application

   Make sure the Java/Wildfly demo application is reachable. Open

   http://localhost:8090

   and generate some traffic by browsing the site.

4. Check access to the Grafana console

   Make sure the Grafana interface is reachable. Open

   http://localhost:3000

   and sign in with admin/grafanapassword.

5. Check the data sources

   Once signed in, let's check that logs, metrics and traces are all present.
   
   Go to **Explore**, pick the relevant data source (Loki for logs, Mimir for metrics, Tempo for traces) and confirm that data shows up.
   
   

   Note: if you already see data, that's expected. The demo application ships pre-instrumented with the OpenTelemetry SDK. It sends its telemetry to a Grafana Alloy gateway (Grafana Labs' take on the OTel Collector), which forwards the data to the backend.
   
   The point of this lab is precisely to reconfigure the application so it sends its data to a collection pipeline that we'll set up ourselves.

   The `docker compose up -d` command deployed a full stack that includes the demo application, the storage backend (Loki, Mimir, Tempo) and a preconfigured collection gateway (Grafana Alloy).

Analyzing a trace #

What you will learn in this section

• Understand the structure of a distributed trace
• Generate traffic and find the matching trace
• Analyze the path of a request

Exploring with Grafana.

1. Reminder: what is a trace?

   A distributed trace represents the full journey of a request across your application's services. It's made up of 'spans', each span representing a unit of work (e.g. an HTTP call, a database query). For a full refresher, we recommend reading this article on OpenTelemetry best practices.

2. Generating and finding a trace

   Open the demo application's weather page at http://localhost:8090/meteo.jsp and look up the weather for Cameroon and France. Then, in Grafana, go to **Explore** > **Tempo**. Use the search to filter by service name (`service.name="wildfly-app"`) and find the trace matching the call to the `/meteo.jsp` page.

   Every interaction with the application generates a unique trace. By filtering on the service name and looking for the root 'span' that matches the entry point of your request (here, the JSP page), you can isolate the trace you're after.

3. Understanding the page's calls

   By examining the trace you found, identify the different internal and external calls made by the `/meteo.jsp` page to build its response. Look at the duration of each span to spot any latency hotspots.

   The trace's waterfall view shows that the `/meteo.jsp` page itself makes several outbound HTTP calls to external services to fetch the weather data. Each call is represented by a child 'span', letting you see the time spent in each remote service.

4. Analyzing attributes (Span vs Resource)

   In a span's details, you'll find **Span Attributes** (specific to that operation) and **Resource Attributes** (shared by every span of that service). Explore the spans related to the database and, using the attributes, find out which type of database was contacted, its IP/port, and the query that was executed.
   
   To learn more about the difference between attributes and resources, check out this article.

   **Resource Attributes** give you the global context (e.g. `service.name`, `telemetry.sdk.language`). **Span Attributes** give you the operation's details (e.g. `db.system`, `db.statement`, `net.peer.name`, `net.peer.port`). Combining both types of attributes is what gives you the full picture.

Context propagation #

What you will learn in this section

• Understand how the weather API call is made
• Simulate a production release to enable context propagation
• Verify the instrumentation and trace correlation in the console

Understand why and how to set up context propagation to link frontend and backend actions.

1. Analyzing the weather API call

   In the earlier traces, we didn't see the call to the weather API. To understand how that call is made, use your browser's developer tools ('Inspect' or 'View page source') on the weather page.

   By inspecting the source of the `meteo.jsp` page, you'll notice that the call to the weather API is made directly in client-side JavaScript (the frontend), not from the Java backend. That's why it doesn't appear in the `wildfly-app` service's trace, which only covers the backend. To link the frontend and backend, you need to set up context propagation.

2. Simulating a production release

   To enable context propagation, we'll simulate a production release by deploying a new version of the application. To do this, go to the project's `app` folder and run the following commands to swap the old version for the new one:

   mv app/ROOT.war app/ROOT-old.war
   mv app/ROOT-new.war app/ROOT.war

   Then restart the application container so the changes take effect:

   sudo docker compose restart wildfly

   This swaps the application's deployment artifact for a new version that includes the instrumentation needed for frontend-to-backend context propagation.

3. Checking the instrumentation

   After restarting the application, go back to the weather page, open your browser's JavaScript console, and check that OpenTelemetry instrumentation is active. You should see OTel-related messages appear.
   
   

   Seeing OpenTelemetry logs in the browser console confirms that the instrumentation code is properly injected on the client side. From now on, when you generate a trace, you'll see that the frontend (JavaScript) and backend (Java) spans are correctly correlated under a single Trace ID, giving you an end-to-end view.

4. Analyzing the instrumentation in Grafana

   After generating traffic on the weather page, go back to Grafana and analyze the new trace. You should now see a full trace that starts with a frontend (JavaScript) span, followed by the backend (Java) spans.

   This correlation is made possible by Trace Context propagation (via the W3C `traceparent` header). The frontend JavaScript SDK creates the trace and its context, which is then passed to the backend on the AJAX call. The backend Java SDK detects that header and continues the existing trace instead of creating a new one.

5. Adding business information to spans

   Automatic instrumentation is powerful, but companies often combine it with manual instrumentation to carry business data. Examine the 'meteo-app-js' trace to find the business information (country, temperature and wind speed) that was injected into the new source code.
   
   

   Real-world use case: in production, this technique is used to inject critical information such as a customer ID, a cart's total amount, a subscription tier, or any other business data. It lets you build dashboards that don't just show technical performance, but answer business questions like: 'What's the average response time for our VIP customers?' or 'What are the most frequent errors for carts over $100?'.

   

   To add attributes, you need manual instrumentation. In the JavaScript code that handles the weather API's response, you grab the active 'span' and add attributes to it with `span.setAttribute('key', 'value')`. Once the changes are applied, you'll see these new attributes (e.g. `weather.country`, `weather.temp`, `weather.wind_speed`) directly in the span's details in Grafana/Tempo.

Configuring the OTel Gateway #

What you will learn in this section

• Create the gateway's configuration file
• Start the gateway container

The gateway is the central collection point. It receives data from every agent, processes it, and exports it to the backend.

1. Configuring the Receivers

   Update the configuration so the gateway listens for OTLP requests over gRPC and HTTP on their default ports.
   
   

   Note: a receiver is the entry point for data into the collector. It defines how the collector receives telemetry (e.g. via the OTLP, Jaeger, or Prometheus protocol).

   Add the following `receivers` section to your `otel-gateway-config.yaml` file:

   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
         http:
           endpoint: 0.0.0.0:4318

2. Configuring a Processor

   Add two processors to your configuration:
   • A `batch` processor to group telemetry data into batches before exporting it, which improves performance and reduces the number of outbound requests.
   • A `resource` processor to enrich all telemetry data with a static attribute. Add the `collector.name` attribute with the value `serveurx`.
   
   

   Note: a processor runs on the data between the moment it's received and the moment it's exported.

   • The batch processor is essential: it groups data into batches before sending it, which optimizes network performance and reduces load on the backends.
   • The resource processor automatically enriches all data with context attributes (e.g. collector name, cloud region, application version), which is crucial for filtering and analysis.

   Add the following `processors` section to your `otel-gateway-config.yaml` file:

   processors:
     batch:
     resource:
       attributes:
         - key: collector.name
           value: "serveurx"
           action: upsert

3. Configuring an Exporter

   Configure exporters to send data to the appropriate backends. Each signal type (trace, metric, log) has its own destination.
   
   
   • For debugging: add a debug exporter to print every received payload directly in the container logs. It's an essential tool to confirm the collector is actually receiving data.
   • Traces: configure an otlp/tempo exporter to send traces to Tempo on its gRPC port (tempo:4317).
   • Metrics: use the prometheusremotewrite exporter to send metrics to Prometheus/Mimir via its remote-write endpoint (the Prometheus URL is http://prometheus:9090).
   • Logs: use the otlphttp/loki exporter to send logs to Loki via OTLP/HTTP (the Loki URL is http://loki:3100).
   
   

   Note: an exporter is the final destination for telemetry data. It defines where and how the collector sends data once it has been received and processed.
   
   The logging exporter has been deprecated and replaced by the debug exporter. While logging may still work, it's recommended to use debug to benefit from the latest improvements and ensure future compatibility.

   Add the following `exporters` section to your `otel-gateway-config.yaml` file:

   exporters:
     # For debugging: prints received data in the container logs (replaces the old 'logging')
     debug:
       verbosity: detailed
   
     # Exports metrics to Prometheus via remote write
     prometheusremotewrite:
       endpoint: http://prometheus:9090/api/v1/write
   
     # Exports traces to Tempo via OTLP/GRPC
     otlp/tempo:
       endpoint: tempo:4317
       tls:
         insecure: true
   
     # Exports logs to Loki via OTLP/HTTP
     otlphttp/loki:
       endpoint: http://loki:3100/otlp

4. Configuring the Extensions

   Add a `health_check` extension to your configuration. It exposes an HTTP endpoint that lets you check whether the collector is healthy.
   
   

   Note: extensions provide capabilities that aren't directly part of the data pipeline (receive, process, export), but that improve the management and monitoring of the collector itself. The health_check extension is crucial in production: it lets orchestrators like Kubernetes know whether the collector is ready to receive traffic, enabling zero-downtime deployments and restarts.

   Add the following `extensions` section to your `otel-gateway-config.yaml` file:

   extensions:
     health_check:
       endpoint: 0.0.0.0:13133

5. Configuring the Services (Pipelines)

   Now that the receivers, processors and exporters are defined, they need to be wired together into pipelines. Each pipeline defines the path a given signal type (trace, metric or log) will follow.
   
   

   Note: the service section, and specifically pipelines, is the heart of the collector's configuration. This is where you define the processing path for each signal type by wiring together the `receivers`, `processors` and `exporters` we defined earlier. Without this section, the collector doesn't know what to do with the data it receives.

   
   Configure the pipelines so that:
   • Traces received via OTLP are processed by the `batch` processor, then sent to `debug` and `otlp/tempo`.
   • Metrics received via OTLP are processed by the `batch` processor, then sent to `debug` and `prometheusremotewrite`.
   • Logs received via OTLP are processed by the `batch` processor, then sent to `debug` and `otlphttp/loki`.
   
   Also enable the `health_check` extension to expose a health endpoint, along with the collector's own internal telemetry to monitor its own state.

   Add the following `service` section to your `otel-gateway-config.yaml` file to wire every component together:

   service:
     extensions: [health_check]
     pipelines:
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [debug, otlp/tempo]
       metrics:
         receivers: [otlp]
         processors: [batch]
         exporters: [debug, prometheusremotewrite]
       logs:
         receivers: [otlp]
         processors: [batch]
         exporters: [debug, otlphttp/loki]
     telemetry:
       logs:
         level: "info"

6. Switching gateways

   You need to edit the compose.yaml file to change the OTEL_EXPORTER_OTLP_ENDPOINT variable so it points to the new gateway 'http://otel-collector:4317', then restart the service.

   - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

Configuring the OTel Agent and the application #

What you will learn in this section

• Create the agent's configuration
• Start the agent and the demo application

The agent runs as close as possible to the application. Its role is simple: collect data and forward it to the gateway. We'll also deploy a demo application that generates traces, logs and metrics.

1. Creating the Agent's configuration

   Create an `otel-agent-config.yaml` file. The agent's configuration is minimal: it receives data from the application (on port 5317, to avoid conflicting with the gateway) and exports it to the gateway.

   Here is the configuration for `otel-agent-config.yaml`:

   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:5317
   
   processors:
     batch:
   
   exporters:
     otlp:
       endpoint: otel-gateway:4317
       tls:
         insecure: true
   
   service:
     pipelines:
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [otlp]
       metrics:
         receivers: [otlp]
         processors: [batch]
         exporters: [otlp]
       logs:
         receivers: [otlp]
         processors: [batch]
         exporters: [otlp]

2. Starting the Agent and the demo application

   Add the `otel-agent` and `app-demo` services to your `docker-compose.yaml` and start them. The application is configured to send its telemetry to the agent.

   Add these two services to your `docker-compose.yaml`:

     otel-agent:
       image: otel/opentelemetry-collector-contrib:latest
       container_name: otel-agent
       command: ["--config=/etc/otel-agent-config.yaml"]
       volumes:
         - ./otel-agent-config.yaml:/etc/otel-agent-config.yaml
       ports:
         - "5317:5317"
       depends_on:
         - otel-gateway
   
     app-demo:
       image: ghcr.io/open-telemetry/opentelemetry-demo-community/otel-demo-client:latest
       container_name: app-demo
       environment:
         - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-agent:5317
         - OTEL_SERVICE_NAME=app-demo
       ports:
         - "8080:8080"
       depends_on:
         - otel-agent

   Then start them:

   docker compose up -d otel-agent app-demo

   Generate traffic by visiting `http://localhost:8080`.

Difficulty level : ●●●○○ (3/5)