Final Lab: Monitors and SLOs in Terraform

What you will learn in this TP

• Configure the Terraform provider
• Write monitors and SLOs
• Test triggering

Introduction #

What you will learn in this section

• Configure the Terraform provider
• Write monitors and SLOs
• Test triggering

Context. Hand-made monitors keep multiplying with no review and no history. You move to managing them as code.

Prerequisites: Lab 4 completed, Terraform, Datadog application key  ·  Estimated time: 50 min

A reviewed, versioned Datadog configuration.

Hands-on #

Guided steps, with their solutions.

1. 1. Configure the provider

   Why? Keys go through the environment, never into the code.

   Create an application key, export the variables and initialise the project.

2. 2. APM latency monitor

   Why? One monitor per service, on a trace metric.

   Add a monitor on the p95 latency of paiement-api.

3. 3. Log monitor and SLO

   Why? Error logs and an availability objective complete the setup.

   Add a log monitor (more than 20 errors in 5 min) and a metric-based SLO at 95% over 7 days.

4. 4. Apply

   Why? The plan shows exactly what will be created: that is what gets reviewed in a pull request.

   Run terraform plan then terraform apply, and find the objects in the UI.

5. 5. Trigger

   Why? Validate the chain before relying on it.

   Generate 500 requests against the API and watch the state of the monitors and the SLO.

Difficulty level : ●●●○○ (3/5)