Deploy a complete observability pipeline with an Agent and a Gateway

What you will learn in this TP

• Deploy a complete visualization backend (Grafana, Prometheus, Loki, Tempo)
• Configure and start an OTel collector in Gateway mode
• Configure and start an OTel collector in Agent mode
• Instrument an application to send its telemetry to the Agent
• Visualize traces, metrics and logs in Grafana
• Clone the configuration repository
• Start the infrastructure with Docker Compose
• Confirm access to the visualization tools
• Understand the structure of a distributed trace
• Generate traffic and find the matching trace
• Analyze the path of a request
• Understand how the weather API call is made
• Simulate a production release to enable context propagation
• Verify the instrumentation and trace correlation in the console
• Create the gateway's configuration file
• Start the gateway container
• Create the agent's configuration
• Start the agent and the demo application

Target architecture #

What you will learn in this section

• Deploy a complete visualization backend (Grafana, Prometheus, Loki, Tempo)
• Configure and start an OTel collector in Gateway mode
• Configure and start an OTel collector in Agent mode
• Instrument an application to send its telemetry to the Agent
• Visualize traces, metrics and logs in Grafana

The goal is to build a pipeline where an instrumented application sends its data to a **local OTel agent**. This agent then forwards the data to a **centralized OTel gateway**, which processes it and exports it to our storage and visualization backend (stack: Loki, Grafana, Tempo, Prometheus).

Why this Agent + Gateway architecture?

• Resilience: the agent can buffer data if the gateway becomes unavailable.
• Performance: being local, the agent adds very little latency for the application. Heavy processing (filtering, enrichment) is offloaded to the gateway.
• Scalability: you can scale the gateway independently from the applications to handle the load of thousands of agents.
• Security: only the gateway needs credentials to connect to the final backends.

Deploying the observability backend #

What you will learn in this section

• Clone the configuration repository
• Start the infrastructure with Docker Compose
• Confirm access to the visualization tools

We'll start by deploying the backend that will centralize and visualize all our telemetry data.

1. Get the configuration files

   Get the configuration files needed to deploy the backend.

   git clone https://github.com/RousselTM/otel-formation.git 
   cd otel-formation

2. Deploy the backend with Docker Compose

   Start the services: Prometheus for metrics, Loki for logs, Tempo for traces and Grafana for visualization.

   docker compose up -d

3. Check access to the demo application

   Make sure the Java/Wildfly demo application is reachable. Open

   http://localhost:8090

   and generate some traffic by browsing the site.

4. Check access to the Grafana console

   Make sure the Grafana interface is reachable. Open

   http://localhost:3000

   and sign in with admin/grafanapassword.

5. Check the data sources

   Once signed in, let's check that logs, metrics and traces are all present.
   
   Go to **Explore**, pick the relevant data source (Loki for logs, Mimir for metrics, Tempo for traces) and confirm that data shows up.
   
   

   Note: if you already see data, that's expected. The demo application ships pre-instrumented with the OpenTelemetry SDK. It sends its telemetry to a Grafana Alloy gateway (Grafana Labs' take on the OTel Collector), which forwards the data to the backend.
   
   The point of this lab is precisely to reconfigure the application so it sends its data to a collection pipeline that we'll set up ourselves.

   The `docker compose up -d` command deployed a full stack that includes the demo application, the storage backend (Loki, Mimir, Tempo) and a preconfigured collection gateway (Grafana Alloy).

Analyzing a trace #

What you will learn in this section

• Understand the structure of a distributed trace
• Generate traffic and find the matching trace
• Analyze the path of a request

Exploring with Grafana.

1. Reminder: what is a trace?

   A distributed trace represents the full journey of a request across your application's services. It's made up of 'spans', each span representing a unit of work (e.g. an HTTP call, a database query). For a full refresher, we recommend reading this article on OpenTelemetry best practices.

2. Generating and finding a trace

   Open the demo application's weather page at http://localhost:8090/meteo.jsp and look up the weather for Cameroon and France. Then, in Grafana, go to **Explore** > **Tempo**. Use the search to filter by service name (`service.name="wildfly-app"`) and find the trace matching the call to the `/meteo.jsp` page.

3. Understanding the page's calls

   By examining the trace you found, identify the different internal and external calls made by the `/meteo.jsp` page to build its response. Look at the duration of each span to spot any latency hotspots.

4. Analyzing attributes (Span vs Resource)

   In a span's details, you'll find **Span Attributes** (specific to that operation) and **Resource Attributes** (shared by every span of that service). Explore the spans related to the database and, using the attributes, find out which type of database was contacted, its IP/port, and the query that was executed.
   
   To learn more about the difference between attributes and resources, check out this article.

Context propagation #

What you will learn in this section

• Understand how the weather API call is made
• Simulate a production release to enable context propagation
• Verify the instrumentation and trace correlation in the console

Understand why and how to set up context propagation to link frontend and backend actions.

1. Analyzing the weather API call

   In the earlier traces, we didn't see the call to the weather API. To understand how that call is made, use your browser's developer tools ('Inspect' or 'View page source') on the weather page.

2. Simulating a production release

   To enable context propagation, we'll simulate a production release by deploying a new version of the application. To do this, go to the project's `app` folder and run the following commands to swap the old version for the new one:

   mv app/ROOT.war app/ROOT-old.war
   mv app/ROOT-new.war app/ROOT.war

   Then restart the application container so the changes take effect:

   sudo docker compose restart wildfly

3. Checking the instrumentation

   After restarting the application, go back to the weather page, open your browser's JavaScript console, and check that OpenTelemetry instrumentation is active. You should see OTel-related messages appear.
   
   

4. Analyzing the instrumentation in Grafana

   After generating traffic on the weather page, go back to Grafana and analyze the new trace. You should now see a full trace that starts with a frontend (JavaScript) span, followed by the backend (Java) spans.

5. Adding business information to spans

   Automatic instrumentation is powerful, but companies often combine it with manual instrumentation to carry business data. Examine the 'meteo-app-js' trace to find the business information (country, temperature and wind speed) that was injected into the new source code.
   
   

   Real-world use case: in production, this technique is used to inject critical information such as a customer ID, a cart's total amount, a subscription tier, or any other business data. It lets you build dashboards that don't just show technical performance, but answer business questions like: 'What's the average response time for our VIP customers?' or 'What are the most frequent errors for carts over $100?'.

   

Configuring the OTel Gateway #

What you will learn in this section

• Create the gateway's configuration file
• Start the gateway container

The gateway is the central collection point. It receives data from every agent, processes it, and exports it to the backend.

1. Configuring the Receivers

   Update the configuration so the gateway listens for OTLP requests over gRPC and HTTP on their default ports.
   
   

   Note: a receiver is the entry point for data into the collector. It defines how the collector receives telemetry (e.g. via the OTLP, Jaeger, or Prometheus protocol).

2. Configuring a Processor

   Add two processors to your configuration:
   • A `batch` processor to group telemetry data into batches before exporting it, which improves performance and reduces the number of outbound requests.
   • A `resource` processor to enrich all telemetry data with a static attribute. Add the `collector.name` attribute with the value `serveurx`.
   
   

   Note: a processor runs on the data between the moment it's received and the moment it's exported.

   • The batch processor is essential: it groups data into batches before sending it, which optimizes network performance and reduces load on the backends.
   • The resource processor automatically enriches all data with context attributes (e.g. collector name, cloud region, application version), which is crucial for filtering and analysis.

3. Configuring an Exporter

   Configure exporters to send data to the appropriate backends. Each signal type (trace, metric, log) has its own destination.
   
   
   • For debugging: add a debug exporter to print every received payload directly in the container logs. It's an essential tool to confirm the collector is actually receiving data.
   • Traces: configure an otlp/tempo exporter to send traces to Tempo on its gRPC port (tempo:4317).
   • Metrics: use the prometheusremotewrite exporter to send metrics to Prometheus/Mimir via its remote-write endpoint (the Prometheus URL is http://prometheus:9090).
   • Logs: use the otlphttp/loki exporter to send logs to Loki via OTLP/HTTP (the Loki URL is http://loki:3100).
   
   

   Note: an exporter is the final destination for telemetry data. It defines where and how the collector sends data once it has been received and processed.
   
   The logging exporter has been deprecated and replaced by the debug exporter. While logging may still work, it's recommended to use debug to benefit from the latest improvements and ensure future compatibility.

4. Configuring the Extensions

   Add a `health_check` extension to your configuration. It exposes an HTTP endpoint that lets you check whether the collector is healthy.
   
   

   Note: extensions provide capabilities that aren't directly part of the data pipeline (receive, process, export), but that improve the management and monitoring of the collector itself. The health_check extension is crucial in production: it lets orchestrators like Kubernetes know whether the collector is ready to receive traffic, enabling zero-downtime deployments and restarts.

5. Configuring the Services (Pipelines)

   Now that the receivers, processors and exporters are defined, they need to be wired together into pipelines. Each pipeline defines the path a given signal type (trace, metric or log) will follow.
   
   

   Note: the service section, and specifically pipelines, is the heart of the collector's configuration. This is where you define the processing path for each signal type by wiring together the `receivers`, `processors` and `exporters` we defined earlier. Without this section, the collector doesn't know what to do with the data it receives.

   
   Configure the pipelines so that:
   • Traces received via OTLP are processed by the `batch` processor, then sent to `debug` and `otlp/tempo`.
   • Metrics received via OTLP are processed by the `batch` processor, then sent to `debug` and `prometheusremotewrite`.
   • Logs received via OTLP are processed by the `batch` processor, then sent to `debug` and `otlphttp/loki`.
   
   Also enable the `health_check` extension to expose a health endpoint, along with the collector's own internal telemetry to monitor its own state.

6. Switching gateways

   You need to edit the compose.yaml file to change the OTEL_EXPORTER_OTLP_ENDPOINT variable so it points to the new gateway 'http://otel-collector:4317', then restart the service.

Configuring the OTel Agent and the application #

What you will learn in this section

• Create the agent's configuration
• Start the agent and the demo application

The agent runs as close as possible to the application. Its role is simple: collect data and forward it to the gateway. We'll also deploy a demo application that generates traces, logs and metrics.

1. Creating the Agent's configuration

   Create an `otel-agent-config.yaml` file. The agent's configuration is minimal: it receives data from the application (on port 5317, to avoid conflicting with the gateway) and exports it to the gateway.

2. Starting the Agent and the demo application

   Add the `otel-agent` and `app-demo` services to your `docker-compose.yaml` and start them. The application is configured to send its telemetry to the agent.

Difficulty level : ●●●○○ (3/5)