RousselTMLEARNING

Observability

Chaos engineering: principles, tools and practice with SLOs

A complete guide to chaos engineering: principles, steady state and hypothesis, blast radius, link with SLOs and error budgets, failure types, tools (Chaos Mesh, LitmusChaos, Chaos Toolkit, AWS FIS, Azure Chaos Studio, Gremlin, Toxiproxy), step-by-step experiments on Kubernetes and running game days.

1. What is chaos engineering? #

Definition, origin and goals.

Definition #

Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production. It is not about « breaking things at random », but about running controlled experiments, driven by a hypothesis, to discover weaknesses before they cause an incident.

The practice was born at Netflix around 2010 with Chaos Monkey, which randomly terminated instances in production to force teams to design fault-tolerant services. The principles were formalized on principlesofchaos.org.

Why do it? #

  • Check that resilience mechanisms (redundancy, failover, retries, timeouts, circuit breakers) really work.
  • Discover hidden dependencies and unexpected failure modes.
  • Reduce time to detect and recover by training teams.
  • Validate alerts, dashboards and procedures (runbooks).
  • Build a culture where failure is expected and studied.
Good to know: A failover that has never been tested often fails on the day you need it. Chaos engineering turns hope into evidence.

2. The principles of an experiment #

Steady state, hypothesis, blast radius.

The five principles #

  1. Define the steady state: a measure of normal behavior visible to users, in practice an SLI (success rate, p99 latency, orders per minute).
  2. Form a hypothesis: « the steady state holds in the experimental group as in the control group ».
  3. Introduce real-world events: server failures, network latency, saturation, dependency errors.
  4. Experiment in production when possible, because real traffic and environments reveal what tests do not show.
  5. Automate and run continuously, while minimizing the blast radius.
Loop of a chaos experiment
Every experiment follows the same loop; an emergency stop interrupts the injection if the SLO degrades.

Controlling the blast radius #

Start small: one pod, one instance, a small percentage of traffic, in pre-production and then in production. Define in advance the stop conditions (the SLI drops below a threshold, a critical alert fires) and a way to immediately cancel the injection.

Blast radius of a chaos experiment
From a pod to all users: widen the scope only once the previous step is under control.
Common mistake: Running an experiment without observability or stop conditions is causing an incident, not running an experiment.

3. Chaos engineering and SRE #

SLOs, error budgets and learning.

Relying on SLOs #

SRE provides the ideal framework: SLIs define the steady state, SLOs set the stop threshold and the error budget tells how much unavailability the team can afford. Some teams explicitly reserve part of the budget for experiments: as long as budget remains, they experiment; once it is exhausted, risky experiments are frozen like releases.

SRE conceptRole in chaos engineering
SLIMeasure of the steady state during the experiment
SLOLimit beyond which the experiment is stopped
Error budgetAllows or suspends experiments depending on consumption
Burn rate alertsMust fire if the injection really degrades the service
Blameless post-mortemFormat for sharing what the experiment taught
SLI during a chaos experiment
In green, the service holds: the SLI stays above the SLO during the injection. Dotted, the hypothesis is refuted: the SLI crosses the stop condition and the injection is cancelled.
Deciding on experiments with the error budget
Example policy: the more error budget remains, the wider the allowed blast radius.

What you learn #

A successful experiment is not one that breaks nothing: it is one that teaches you something. If the hypothesis is refuted, document the weakness, prioritize the fix, then replay the experiment to verify.

4. Which failures to inject? #

Catalog of realistic failures.

Failure types #

CategoryExamplesWhat you check
InfrastructureLoss of an instance, a node, a zoneRedundancy, self-healing, failover
ContainersPod deletion, crash loopsReplicas, probes, PodDisruptionBudget
NetworkLatency, packet loss, partition, failing DNSTimeouts, retries, circuit breakers
ResourcesCPU, memory, disk saturationLimits, autoscaling, graceful degradation
DependenciesSlow database, third-party API returning 5xxCache, degraded mode, queues
Clock and stateClock skew, expired certificateRobustness of validity checks
People and processKey person absent, missing procedureDocumentation, on-call, runbooks
Catalog of failures to inject
Seven failure families to explore around a service, starting with those that have already caused incidents.

5. The tools #

Comparison of the main tools.

Overview #

ToolTypeStrengths
Chaos MeshOpen source (CNCF), KubernetesPodChaos, NetworkChaos, StressChaos, IOChaos resources; dashboard; workflows
LitmusChaosOpen source (CNCF), KubernetesChaosHub of ready-to-use experiments, validation probes
Chaos ToolkitOpen source, multi-platformDeclarative experiments in JSON/YAML, cloud and Kubernetes extensions
AWS Fault Injection ServiceAWS managed serviceExperiment templates on EC2, ECS, EKS, RDS; CloudWatch stop conditions
Azure Chaos StudioAzure managed serviceVM, AKS, network faults; scheduled experiments
Gremlin, SteadybitCommercial platformsScenario library, guardrails, reports
ToxiproxyTest proxyLatency and cuts between an application and its dependencies, ideal in integration tests
Chaos MonkeyNetflix (Spinnaker)Random instance termination, the origin of the discipline
Positioning of chaos engineering tools
For Kubernetes, Chaos Mesh and LitmusChaos are the open source references; cloud managed services and commercial platforms also cover VMs and managed services.

6. Hands-on on Kubernetes #

Chaos Mesh and Chaos Toolkit step by step.

Installing Chaos Mesh #

helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-mesh --create-namespace \
  --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock
kubectl get pods -n chaos-mesh
Chaos Mesh architecture
The Controller Manager turns chaos resources into actions executed by the Chaos Daemon on each node; observability is used to decide whether to stop.

Experiment 1: delete a pod #

Hypothesis: with three replicas and a readiness probe, deleting an API pod does not lower the success rate below 99.9%.

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata: { name: api-pod-kill, namespace: boutique }
spec:
  action: pod-kill
  mode: one                 # un seul pod : rayon d'impact minimal
  selector:
    namespaces: [boutique]
    labelSelectors: { app: api }
kubectl apply -f api-pod-kill.yaml
kubectl get pods -n boutique -l app=api -w     # un pod est recréé
kubectl delete -f api-pod-kill.yaml             # fin de l'expérience
PodChaos experiment in the terminal
One of the three pods is deleted and recreated by the Deployment while the SLI stays stable.

Experiment 2: add network latency #

Hypothesis: if the database responds 200 ms slower, the API p99 stays below 800 ms thanks to the cache.

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata: { name: db-latence, namespace: boutique }
spec:
  action: delay
  mode: all
  selector: { namespaces: [boutique], labelSelectors: { app: postgres } }
  delay: { latency: 200ms, jitter: 20ms }
  duration: 5m
Good practice: During the experiment, watch the latency SLI, the API traces and whether the burn rate alerts fire: an alert that does not fire while users suffer is also a lesson.

A declarative experiment with Chaos Toolkit #

{
  "title": "L'API tolère la perte d'un pod",
  "steady-state-hypothesis": {
    "title": "L'API répond",
    "probes": [{ "type": "probe", "name": "api-ok", "tolerance": 200,
                 "provider": { "type": "http", "url": "https://api.exemple.fr/healthz" } }]
  },
  "method": [{ "type": "action", "name": "supprimer-un-pod",
               "provider": { "type": "python", "module": "chaosk8s.pod.actions",
                             "func": "terminate_pods",
                             "arguments": { "label_selector": "app=api", "ns": "boutique", "qty": 1 } } }],
  "rollbacks": []
}
pip install chaostoolkit chaostoolkit-kubernetes
chaos run experience.json      # vérifie l'hypothèse avant et après l'injection

7. Running game days #

Prepare, run, debrief.

How a game day unfolds #

  1. Prepare: choose a realistic scenario, the hypothesis, the participants (teams involved, on-call), the stop conditions and the communication.
  2. Execute: a facilitator injects the failure, teams detect and respond as in a real incident.
  3. Observe: record time to detect, diagnose and recover, and the alerts received or missing.
  4. Debrief: blameless post-mortem, corrective actions with owners and deadlines.
  5. Replay later to verify the fixes.
How a game day unfolds
The five stages of a game day, from preparation to verifying the fixes.
Progression of a chaos engineering practice
Start with the prerequisites and pre-production experiments before experimenting continuously in production.

Pitfalls to avoid #

  • Experimenting without observability or SLOs: there is no way to know whether the hypothesis holds.
  • Starting with a large-scale failure in production.
  • Not warning teams and support during the first experiments.
  • Not following up on corrective actions: the same weaknesses come back.
  • Confusing chaos engineering with load testing: the two are complementary.
Good practice: Gradually add automated experiments to the CI/CD pipeline (pre-production): every release is then checked against known failures.

Related articles

Time-based vs event-based SLO: understanding, computing and choosing (with the Qonto example)

Differences between time-based SLOs (time slices, uptime) and event-based SLOs (good events over total): defi…

SRE: SLI, SLO and SLA explained simply

The fundamentals of Site Reliability Engineering (SRE): the difference between SLI, SLO and SLA, choosing ind…

The error budget: making decisions with SLOs

Calculate and track an error budget, write an error budget policy and use it to arbitrate between new feature…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!