HAProxy and Docker: building a highly available load-balanced cluster

What you will learn in this TP

• Launch the multi-container infrastructure
• Locate and edit the main file
• Start an HAProxy 3.4 instance with a basic configuration
• Analyse the logs
• Understand the configuration sections
• Understand why modularity matters
• Learn how to include configuration folders
• Understand the important difference between reload and restart
• Configure a frontend dedicated to administration
• Test access to HAProxy's built-in statistics dashboard
• Configure the main frontend for HTTP traffic
• Configure a default backend
• Understand load balancing: Round Robin, Least connection, ...
• Understand the concept of weight
• Understand the concept of X-FORWARD-FOR
• Understand the concept of HA
• Work with ACLs
• Configure a layer 7 check
• Configure pattern matching in the check output
• Limit concurrent connections (Anti-DoS/DDoS)
• Protect against slow attacks (Slowloris)
• Set up a request quota (Rate Limiting)
• Identify and filter automated clients (Anti-bot/Scraping)
• Use silent termination (Silent Drop) for forbidden traffic

Setting up the infrastructure #

What you will learn in this section

• Launch the multi-container infrastructure

Deployment of 3 web servers, an Nginx server and an HAProxy cluster with Docker Compose.

IMPORTANT: You must read this article before starting this lab to master the fundamentals.

We also recommend taking these courses:

• Docker: to better understand the importance of containerisation
• Prometheus: to better understand metrics and how to manage them
• Opentelemetry: to better understand metrics and how to manage them
• Grafana: to better understand how telemetry data is visualised

. This is not essential, though, as we provide every command needed for these products in this lab.

1. Getting the sources

   Clone the project sources from this repository https://github.com/rousseltm/haproxy-formation.git into the folder of your choice.

2. Setting up the infrastructure

   To test our configurations, we need a minimal infrastructure. Run the following commands to deploy it on Docker:

   cd haproxy-formation 
   sudo docker compose -f compose-infra.yaml up -d

   Warning: All docker compose commands must be run from the haproxy-formation folder.

Getting to grips with the configuration #

What you will learn in this section

• Locate and edit the main file
• Start an HAProxy 3.4 instance with a basic configuration
• Analyse the logs
• Understand the configuration sections
• Understand why modularity matters
• Learn how to include configuration folders
• Understand the important difference between reload and restart

Exploring the configuration structure and making it modular.

1. Accessing the base haproxy.cfg file

   The heart of HAProxy lies in its main configuration file: haproxy.cfg. In our Docker environment, this file is mounted as a volume.

   The file is located in config/haproxy/haproxy.cfg. Open it with your usual text editor to look at the sections already present.

2. Starting the base instance

   To test our configurations, we need an HAProxy 3.4 infrastructure with a basic configuration that we will modify later. First, let's check that it works with the basic configuration using the following command:

   sudo docker compose up -d

3. Checking the startup

   The goal is to learn how to identify configuration errors through the system and Docker logs, and to check that the logs generated by HAProxy contain no errors.

   Installation on a virtual or physical machine: logs are usually available in /var/log/haproxy.log or through the command

   journalctl -u haproxy

   Installation with Docker Compose (our case): Use the following command to view the logs of all instances:

   sudo docker compose logs -f haproxy

   or this command to view those of a specific instance (name of container 3: haproxy-formation-haproxy-3)

    sudo docker logs <container_name>

   With the configuration provided, you may encounter errors such as:

   haproxy-1 | [ALERT] (1) : config : parsing [/usr/local/etc/haproxy/haproxy.cfg:71]: Missing LF on last line, file might have been truncated at position 32.

   You must understand the message and fix it.

4. Why modularity matters

   At scale, maintaining a monolithic haproxy.cfg file becomes complex and risky. Modularity lets you separate responsibilities (e.g. one file per application, one for monitoring, one for security, etc.), which makes continuous deployment and code reviews easier and limits human error during updates.
   
   By default, the HAProxy service is started with the -f haproxy.cfg option. You can add as many -f options as you have files, but you can also use the same option with a folder as its parameter: HAProxy will then load every file ending in .cfg in that folder.
   
   Be careful to take the dependencies between modules into account!

   
   
   In our case, with our Docker Compose configuration, use the following command to switch to the conf.d folder (remember to move the configuration file into this folder), which sits at the same level as the haproxy.cfg file. The configuration blocks of the following exercises must therefore be placed in a .cfg file in this folder, for example config/haproxy/conf.d/tp.cfg:

   sudo docker compose down; export HAPROXY_CONFIG=conf.d; sudo -E docker compose up -d

5. Reloading the configuration

   Every change to the file must be picked up by the HAProxy engine.
   

   Reload vs Restart: In production, never use restart because it drops connections. A Reload (HUP signal) loads the new configuration without any service interruption.

   Find the command to reload your server without dropping current connections, both with systemctl and with Docker.

Managing Frontends #

What you will learn in this section

• Configure a frontend dedicated to administration
• Test access to HAProxy's built-in statistics dashboard
• Configure the main frontend for HTTP traffic

Configuring the entry points for statistics and HTTP traffic.

1. Administration entry point

   Create an entry point dedicated to monitoring named 'admin'. It must listen on port 8404 (the default port). Enable the statistics interface, set the root (/) as the access URI and configure an automatic refresh every 5 seconds. Note that you can do this with either a frontend block or a listen block; the result will be the same.

   Since we switched to loading the conf.d folder, add this configuration to a .cfg file in that folder. To apply your changes, use the command export HAPROXY_CONFIG=conf.d; sudo -E docker compose restart haproxy (make sure you do a reload in production).

2. Testing access to the statistics

   Once the 'admin' frontend is configured, check that the monitoring interface is working. You can use your browser or a terminal command to test that the service is reachable on its dedicated port.

3. HTTP frontend

   Configure the main frontend named 'http_in' to receive web traffic from users. It must listen on port 80 and send all requests by default to the 'default' server group (which will be created later).

Managing Backends #

What you will learn in this section

• Configure a default backend
• Understand load balancing: Round Robin, Least connection, ...
• Understand the concept of weight
• Understand the concept of X-FORWARD-FOR
• Understand the concept of HA

Configuring the destination servers that process requests.

1. The default Backend

   Create a server group named 'default'. This backend will handle requests that match no specific rule. It must run in HTTP mode, use the 'roundrobin' algorithm and point to the 'web1' server. Configure a rule so that this backend always serves the 'default.html' file.

2. Hot reload via the configuration file

   Open your statistics interface (port 8404). You should see 1 active server in the default backend. The goal is to add the web2 server to this pool on the fly, i.e. without restarting the HAProxy container, so that current connections are preserved. The steps:
   • Check the creation date of the HAProxy containers: the information is in the 'CREATED' column

     sudo docker compose ps haproxy

   • Edit the file to add the second server and reload the service:

     sudo docker compose kill -s HUP haproxy

   • Check in the interface that the new server is present and that the container creation date has not changed

3. Hot reload via socat

   HAProxy provides a Runtime API reachable through a Unix socket. It lets you change the state of the load balancer (weights, server state, dynamic addition) without any reload.

   To inspect the internal state or run commands directly in your container, open a shell with:
   sudo docker exec -it <nom_du_conteneur> sh
   
   To use the API, the socket must be enabled in the global section of the configuration loaded by HAProxy:
   stats socket /tmp/admin.sock mode 600 level admin

   Recommended reading: To go further into dynamic management of backends and servers through the Runtime API, read our dedicated article: HAProxy: adding and removing backends dynamically.
   Once added, you should see the server appear on your stats page But as you may have noticed, socat is a local command, so it must be run on every instance of your HAProxy cluster. Moreover, as stated in the documentation, adding a server is not enough for it to receive traffic: you have to enable it...

4. The web_servers Backend

   Create a server group named 'web_servers'. This backend must run in HTTP mode and use the 'least connection' balancing algorithm, with each server having a weight equal to its server id. You must also add a rule to pass the client's real IP address to the backend through the 'X-Real-IP' header and declare the 3 application nodes (web1, web2, web3) on port 80 with a health check (check). Finally, update the http_in frontend so that it sends traffic to this backend by default.

5. Scaling HAProxy

   Horizontal scaling means running several HAProxy instances to guarantee high availability (HA). In a real architecture, having a single load balancer instance creates a 'Single Point of Failure' (SPOF): if HAProxy goes down, the whole application becomes unreachable, even if the web servers are healthy. In a Docker environment, scaling spreads the network load and ensures service continuity during updates (rolling updates). Run the Docker Compose command with the --scale haproxy=3 parameter to bring the number of HAProxy instances to 3.
   
   

   Note: Increasing the number of HAProxy nodes is not enough to make it highly available. It needs a VIP: in our case, Nginx does the load balancing to provide full high availability.

6. Synchronising sessions with 'Peering'

   In a highly available HAProxy cluster, a failover between two nodes can cause persistence information to be lost if each instance keeps its data locally. To avoid this, HAProxy includes a native mechanism called Peers that replicates the content of Stick Tables in real time between several HAProxy servers. This feature ensures that session information, rate limiting counters, abuse detection and client affinity data remain consistent after a failover.
   
   

   IMPORTANT: Peering does not synchronise application sessions (PHP, Java, .NET, etc.). It only replicates the data stored in HAProxy Stick Tables. To share application sessions, use an external solution such as Redis or Memcached.

   
   
   

   What can be synchronised with Peers?

   • Session persistence: keeping the affinity between a client and its backend server.
   • Rate Limiting: sharing request counters between HAProxy nodes.
   • Abuse protection: distributed detection of attacks or suspicious behaviour.
   • User tracking: keeping client-related information even after a failover.

   Peers are particularly useful in architectures using Keepalived or VRRP, where a virtual IP address can move from one HAProxy to another.

   Best practices:

   • Use a dedicated or isolated network for peering exchanges to avoid interference with user traffic.
   • Check that the ports used by the Peers are reachable between the cluster nodes.
   • Keep an identical definition of the peers section on every HAProxy server.
   • Monitor the synchronisation state through the Runtime API or the HAProxy statistics page.
   • For stateful applications, still prefer moving sessions out to Redis to avoid any dependency on the load balancer.

   Add a session-sharing section named 'haproxy_cluster' that uses the three HAProxy servers haproxy-formation-haproxy-1/2/3 on the default port. Then enable this session sharing in the 'web_servers' backend with a size of 100k, an expiry after 30m and based on the visitor's source IP.

Advanced routing with ACLs #

What you will learn in this section

• Work with ACLs
• Configure a layer 7 check
• Configure pattern matching in the check output

Redirect requests starting with /api to server 2.

1. ACL changes

   Smart routing lets you send traffic to different backends based on precise criteria. In this exercise, reconfigure HAProxy so that it identifies requests meant for the API (starting with the path /api) and sends them exclusively to the secure_servers backend, which contains the servers of the web service but only from the 2nd to the 10th instance. The remaining traffic must still be distributed by the default backend.
   
   

   By default, HAProxy runs layer 4 tests (TCP - IP:PORT), but you must change the backend to run a layer 7 health check (HTTP - web page) that verifies that the 'secure.html' page does return the string 'Service\ '

   

2. IP whitelisting

   Filtering by IP address is a cornerstone of security. You must now restrict access to the statistics dashboard (port 8404) so that only the internal sources that need it are allowed: in our case the Prometheus server (DNS name prometheus in Docker) and the administrators.
   
   

   Whitelisting vs Blacklisting:

   HAProxy supports both approaches through its ACL rules.

   • Whitelisting: 'Default Deny' strategy. Everything is blocked by default and only known sources are allowed. This is the most robust approach for security.
     Use cases: access to an administration console, an internal API reserved for partners, restricting access to the IPs of a corporate VPN.
   • Blacklisting: 'Default Allow' strategy. All traffic is let through except IPs identified as malicious.
     Use cases: blocking known bots (scrapers), temporary protection against an IP carrying out a brute-force attack, or geo-blocking (blocking entire regions).

Security and Hardening #

What you will learn in this section

• Limit concurrent connections (Anti-DoS/DDoS)
• Protect against slow attacks (Slowloris)
• Set up a request quota (Rate Limiting)
• Identify and filter automated clients (Anti-bot/Scraping)
• Use silent termination (Silent Drop) for forbidden traffic

Protecting the cluster against abuse and network attacks.

1. Protection against DoS and DDoS

   Configure the http_in frontend to limit each source IP to a maximum of 15 concurrent TCP connections (layer 4). Once the configuration is applied, try opening several connections in parallel to check that HAProxy rejects the extra ones.

   Verification: Simulate load with a shell loop that opens many connections in the background:

   for i in $(seq 1 20); do curl -s http://localhost & done

   Look at your HAProxy logs or the dashboard: connections beyond 15 must be rejected immediately.

   DoS vs DDoS: what is the difference?

   • DoS (Denial of Service): the attack comes from a single machine. HAProxy can easily block it by limiting connections per IP.
   • DDoS (Distributed DoS): the attack comes from thousands of sources (botnet). Blocking one IP is no longer enough, because the traffic is distributed.

   HAProxy recommendations (Read the official article):

   • Defence in depth: for massive volumetric attacks (layers 3 and 4), use cloud protection services (Cloudflare, AWS Shield) upstream.
   • Layer 7 filtering: use HAProxy to identify specific attack signatures in headers or URLs with ACLs.
   • Stick Tables: they remain essential to limit the impact of each botnet node on your application resources.

2. Protection against Slowloris

   The Slowloris attack saturates the server by sending HTTP headers very slowly. Implement a protection by limiting the header reception time to 5 seconds.

   Attack simulation: Run this command before the configuration (the connection will hang), then after (HAProxy will close the connection after 5 s):

   curl -X GET "http://localhost:80/" \
        -H "X-Slow: $(head -c 5000 < /dev/zero | tr '\0' 'A')" \
        --limit-rate 1 \
        --verbose

   Parameters explained:

   • -H "X-Slow: ...": generates a huge 5000-byte header (with head and tr) to force a long upload.
   • --limit-rate 1: throttles sending to 1 byte/s. Without protection, the connection would stay open for more than an hour.
   • --verbose: lets you see live the exact moment when HAProxy cuts the session.

   HAProxy recommendations (Read the official article): For optimal protection, HAProxy recommends combining several settings:

   • timeout http-request 5s: limits the waiting time for headers (main defence).
   • timeout http-keep-alive 1s: quickly frees idle connection slots.
   • maxconn: limits the global number of connections to protect the system's file descriptors from exhaustion.

3. Limiting requests (Rate Limiting)

   Set up a quota of at most 50 requests every 10 seconds per IP. Test exceeding the threshold and check that HAProxy does return an HTTP 429 error code.

   Verification: Fire a burst of curl requests to exhaust the quota:

   for i in $(seq 1 60); do curl -I http://localhost; done

   You should see HTTP/1.1 429 Too Many Requests responses after the 50th request.

4. Anti-bot modules and scraping detection

   Block data harvesters (scrapers) and malicious bots by identifying their suspicious User-Agent signatures.

   Bot simulation: Try to access the site while impersonating a known bot with curl:

   curl -I -H "User-Agent: Googlebot-Scraper" http://localhost/

5. Silent request termination

   'Silent Drop' is an advanced defence technique: instead of sending an error code (403/429) that confirms to the attacker that a protection exists, HAProxy simply ignores the request. The attacker wastes resources waiting for a response that will never come.

   Verification: Try to access a forbidden path and observe that curl hangs until the timeout:

   curl -v http://localhost/admin/config

Observability #

Setting up monitoring, metrics exposure and telemetry.

1. Stats Dashboard

   Check that the administration entry point configured earlier still exposes the statistics dashboard at the root of port 8404.

2. Prometheus: exposing metrics

   Configure HAProxy to expose performance metrics in Prometheus format on the /metrics URI. This lets a Prometheus server scrape the state of the load balancer.

   Verification: Test fetching the metrics with curl:

   curl http://localhost:8404/metrics

   You should see the metrics scroll by in plain text format (HELP and TYPE).

3. OpenTelemetry (OTel) support

   Configure native export of telemetry data to an OpenTelemetry collector.

   Note: Native OpenTelemetry support is available from HAProxy version 3.4 onwards.

Difficulty level : ●●●○○ (3/5)