Load-Balanced Cluster with Docker

What you will learn in this TP

• Launch the multi-container infrastructure
• Start an HAProxy 3.4 instance with a basic configuration
• Analyze logs
• Understand the configuration sections
• Understand the importance of modularity
• Learn to manage the inclusion of configuration directories
• Understand the importance of the difference between reload and restart
• Configure a dedicated frontend for administration
• Test the accessibility of HAProxy's native statistics dashboard
• Configure the main frontend for HTTP traffic
• Understand load balancing: Round Robin, Least connection, ...
• Understand the concept of weight
• Understand the concept of X-FORWARD-FOR
• Understand the concept of HA
• Manipulate ACLs
• Configure a layer 7 check
• Configure pattern search in the check output
• Limit simultaneous connections (Anti-DoS/DDoS)
• Protect against slow attacks (Slowloris)
• Implement a request quota (Rate Limiting)
• Identify and filter bots (Anti-bot/Scraping)
• Use silent termination (Silent Drop) for forbidden flows

Infrastructure Setup #

What you will learn in this section

• Launch the multi-container infrastructure

Deployment of 3 web servers, an Nginx server and an HAProxy cluster via Docker Compose.

IMPORTANT: You must read this article before starting this lab to master the fundamentals.

We also recommend following these courses:

• Docker: To better understand the importance of containerization
• Prometheus: To better understand the concept of metrics and their management
• Opentelemetry: To better understand the concept of metrics and their management
• Grafana: To better understand the visualization of telemetry data

. But this is not essential, as we will provide all the commands for these products for this lab.

1. Obtaining sources

   You must clone the project sources from this repository https://github.com/rousseltm/haproxy-formation.git into the folder of your choice.

2. Infrastructure setup

   To test our configurations, we will need a minimal infrastructure. You must run the following commands to deploy it on Docker:

   cd haproxy-formation 
   sudo docker compose -f compose-infra.yaml up -d

   Attention: All docker compose commands must be run in the haproxy-formation directory.

Getting started with configuration #

What you will learn in this section

• Start an HAProxy 3.4 instance with a basic configuration
• Analyze logs
• Understand the configuration sections
• Understand the importance of modularity
• Learn to manage the inclusion of configuration directories
• Understand the importance of the difference between reload and restart

Exploration of the configuration structure and implementation of modularity.

1. Access the base haproxy.cfg file

   The heart of HAProxy lies in its main configuration file: haproxy.cfg. In our Docker environment, this file is mounted as a volume.

   The file is located in config/haproxy/haproxy.cfg. You can open it with your usual text editor to observe the already present sections.

2. Starting the base instance

   To test our configurations, we will need an HAProxy 3.4 infrastructure with a basic configuration that we will modify later. First, we will test its operation with the basic configuration using the following command:

   sudo docker compose up -d

3. Startup Verification

   The goal is to learn how to identify configuration errors through system and Docker logs and to verify the absence of errors in the logs generated by HAProxy.

   Installation on a virtual machine or physical machine: Logs are generally available in /var/log/haproxy.log or via the command

   journalctl -u haproxy

   Installation via Docker Compose (our case): Use the following command to view the logs of all instances:

   sudo docker compose logs -f haproxy

   or this command to view the logs of a specific instance (container name 3: haproxy-formation-haproxy-3)

    sudo docker logs <container_name>

   With the provided configuration, you may encounter errors like:

   haproxy-1 | [ALERT] (1) : config : parsing [/usr/local/etc/haproxy/haproxy.cfg:71]: Missing LF on last line, file might have been truncated at position 32.

   You must understand the message and correct it.

4. The Importance of Modularity

   At scale, maintaining a monolithic haproxy.cfg file becomes complex and risky. Modularity allows for the separation of responsibilities (e.g., one file per application, one for monitoring, one for security, etc.), thus facilitating continuous deployment, code reviews, and limiting human errors during updates.
   
   By default, the HAProxy service is launched with the -f haproxy.cfg option. You can add as many -f options as you have files, but you can also use the same option and pass a directory as a parameter: HAProxy will load all files ending in .cfg in that directory.
   
   Be careful to consider the dependencies between modules!

   
   
   In our case, with our Docker Compose configuration, use the following command to switch to using the conf.d directory (remember to move the conf file to this directory), which is at the same level as the haproxy.cfg file. The configuration blocks for the following exercises should therefore be placed in a .cfg file in this directory, for example config/haproxy/conf.d/tp.cfg:

   sudo docker compose down; export HAPROXY_CONFIG=conf.d; sudo -E docker compose up -d

5. Reloading the configuration

   Each file modification requires the HAProxy engine to take it into account.
   

   Reload vs Restart: In production, never use restart as it cuts connections. Reload (HUP signal) allows loading the new configuration without any service interruption.

   You must find the command to reload your server without cutting current connections via systemctl and via Docker.

Frontend Management #

What you will learn in this section

• Configure a dedicated frontend for administration
• Test the accessibility of HAProxy's native statistics dashboard
• Configure the main frontend for HTTP traffic

Configuration of entry points for statistics and HTTP traffic.

1. Entry point for administration

   Create a dedicated monitoring entry point named 'admin'. This entry point must listen on port 8404 (default port). You must enable the statistics interface, define the root (/) as the access URI, and configure an automatic refresh every 5 seconds. Note that you can achieve this configuration with a frontend block or a listen block; the result will be identical.

   Since we have switched to loading the conf.d directory, add this configuration to a .cfg file in that directory. To apply your changes, use the command export HAPROXY_CONFIG=conf.d; sudo -E docker compose restart haproxy (be sure to do a reload in Production).

2. Test statistics access

   After configuring the 'admin' frontend, verify that the monitoring interface is operational. You can use your browser or a terminal command to test the service's accessibility on the dedicated port.

3. HTTP Frontend

   Configure the main frontend named 'http_in' to receive web traffic from users. It must listen on port 80 and redirect all requests by default to the 'default' server group (which will be created later).

Backend Management #

What you will learn in this section

• Understand load balancing: Round Robin, Least connection, ...
• Understand the concept of weight
• Understand the concept of X-FORWARD-FOR
• Understand the concept of HA

Configuration of destination servers to handle requests.

1. The default Backend

2. Hot reload via config file

   Open your statistics interface (port 8404). You should normally see 1 active server in the default backend. The goal is to add the web2 server to this pool 'hot', i.e., without restarting the HAProxy container, to preserve current connections. The steps are:
   • View the creation date of the HAProxy containers: The information is in the 'CREATED' column

     sudo docker compose ps haproxy

   • You must edit the file to add the second server and reload the service:

     sudo docker compose kill -s HUP haproxy

   • You must verify in the interface that the new server is present and that the container's creation date has not changed

3. Hot reload via socat

   HAProxy has a Runtime API accessible via a Unix socket. It allows modifying the state of the load balancer (weight, server status, dynamic addition) without any reload.

   To inspect the internal state or pass commands directly into your container, connect to the shell via:
   sudo docker exec -it <container_name> sh
   
   To use the API, you must have activated the socket in the global section of the configuration loaded by HAProxy:
   stats socket /tmp/admin.sock mode 600 level admin

   Reading recommendation: To delve deeper into dynamic management of backends and servers via the Runtime API, consult our dedicated article: HAProxy: Dynamic addition and deletion of backends.
   By adding it, you should see the server appear on your stats page But as you may have noticed, the socat command is a local command, so it must be added to all instances of your HAProxy cluster. Furthermore, as indicated in the documentation, adding a server is not enough for it to receive traffic; it must be enabled...

4. The web_servers Backend

   Create a server group named 'web_servers'. This backend must operate in HTTP mode and use the 'least connection' distribution algorithm with each backend having a weight corresponding to its server ID. You must also add a rule to transmit the client's real IP address to the backend via the 'X-Real-IP' header and declare the 3 application nodes (web1, web2, web3) on port 80 with a health check (check). Finally, update the http_in frontend to send traffic to this backend by default.

5. HAProxy Scaling

   Horizontal scaling consists of multiplying HAProxy instances to ensure high availability (HA). In a real architecture, having a single load balancer instance creates a 'Single Point of Failure' (SPOF): if HAProxy goes down, the entire application is inaccessible, even if the web servers are healthy. In a Docker environment, scaling allows for network load distribution and ensures service continuity during rolling updates. You must run the Docker Compose command with the --scale haproxy=3 parameter to increase the number of HAProxy instances to 3.
   
   

   Note: Multiplying the number of HAProxy nodes is not enough to make it highly available. It needs a VIP: in our case, Nginx will do the load balancing to ensure complete high availability.

6. Session Synchronization with 'Peering'

   In a high-availability HAProxy cluster, a failover between two nodes can cause the loss of persistence information if each instance keeps its data locally. To avoid this problem, HAProxy includes a native mechanism called Peers that allows real-time replication of the contents of Stick Tables between multiple HAProxy servers. This feature ensures that session information, rate limiting counters, abuse detections, and client affinity data remain consistent after a failover.
   
   

   IMPORTANT: Peering does not synchronize application sessions (PHP, Java, .NET, etc.). It only replicates data stored in HAProxy Stick Tables. To share application sessions, it is recommended to use an external solution like Redis or Memcached.

   
   
   

   What can be synchronized with Peers?

   • Session persistence: maintaining affinity between a client and its backend server.
   • Rate Limiting: sharing request counters between HAProxy nodes.
   • Abuse protection: distributed detection of attacks or suspicious behaviors.
   • User tracking: retaining client-related information even after a failover.

   Peers are particularly useful in architectures using Keepalived or VRRP where a virtual IP address can migrate from one HAProxy to another.

   Best practices:

   • Use a dedicated or isolated network for peering exchanges to avoid interference with user traffic.
   • Verify that the ports used for Peers are accessible between the cluster nodes.
   • Maintain an identical definition of the peers section on all HAProxy servers.
   • Monitor the synchronization status via the Runtime API or the HAProxy statistics page.
   • For stateful applications, still prefer to externalize sessions in Redis to avoid any dependency on the load balancer.

   You must add a section for session sharing named 'haproxy_cluster' that uses the three HAProxy servers haproxy-formation-haproxy-1/2/3 on the default port. You must therefore enable this session sharing in the 'web_servers' backend for a size of 100k with an expiration after 30m and based on the visitor's source IP.

Advanced routing with ACL #

What you will learn in this section

• Manipulate ACLs
• Configure a layer 7 check
• Configure pattern search in the check output

Redirect requests starting with /api to server 2.

1. ACL Modification

   Intelligent routing allows directing traffic to different backends according to precise criteria. In this exercise, you must reconfigure HAProxy to identify requests intended for the API (starting with the path /api) and send them exclusively to the secure_servers backend, which contains servers from the web service, but only from the 2nd to the 10th occurrence. The remaining traffic should continue to be distributed by the default backend.
   
   

   By default, HAProxy will perform layer 4 tests (TCP - IP:PORT), but you must modify the backend to perform a layer 7 liveness test (HTTP - web page) that verifies that the 'secure.html' page returns the string 'Service\ '

   

2. IP Whitelisting

   Filtering by IP address is a pillar of security. You must now restrict access to the statistics dashboard (port 8404) to allow only internal sources that need it: in our case, the Prometheus server (dns prometheus in docker) and administrators.
   
   

   Whitelisting vs Blacklisting:

   HAProxy is compatible with both approaches via its ACL rules.

   • Whitelisting: 'Default Deny' strategy. We block everything by default and only allow known sources. This is the most robust approach for security.
     Use case: access to an administration console, internal API reserved for partners, access restriction to corporate VPN IPs.
   • Blacklisting: 'Default Allow' strategy. All traffic is allowed except IPs identified as malicious.
     Use case: blocking known bots (scrapers), temporary protection against an IP performing a brute force attack, or Geo-blocking (blocking entire regions).

Security and Hardening #

What you will learn in this section

• Limit simultaneous connections (Anti-DoS/DDoS)
• Protect against slow attacks (Slowloris)
• Implement a request quota (Rate Limiting)
• Identify and filter bots (Anti-bot/Scraping)
• Use silent termination (Silent Drop) for forbidden flows

Protecting the cluster against abuse and network attacks.

1. Protection against DOS and DDOS

   Configure the http_in frontend to limit each source IP to a maximum of 15 simultaneous TCP connections (Layer 4). Once the configuration is applied, try to open several connections in parallel to check that HAProxy rejects subsequent ones.

   Verification: Simulate load with a shell loop to open many connections in the background:

   for i in $(seq 1 20); do curl -s http://localhost & done

   Look at your HAProxy logs or the dashboard: connections beyond 15 must be rejected immediately.

   DoS vs DDoS: What's the difference?

   • DoS (Denial of Service): The attack comes from a single machine. HAProxy can easily block it by limiting connections per IP.
   • DDoS (Distributed DoS): The attack comes from thousands of sources (botnet). Blocking an IP is no longer enough because the traffic is distributed.

   HAProxy recommendations (Read official article):

   • Defense in depth: For massive volumetric attacks (Layers 3 and 4), use upstream Cloud protection services (Cloudflare, AWS Shield).
   • Layer 7 filtering: Use HAProxy to identify specific attack signatures in headers or URLs via ACLs.
   • Stick Tables: They remain essential to limit the impact of each botnet node on your application resources.

2. Protection against Slowloris

   The Slowloris attack saturates the server by sending HTTP headers very slowly. Implement protection by limiting the header reception time to 5 seconds.

   Attack simulation: Run this command before the configuration (the connection will remain blocked), then after (HAProxy will cut the connection after 5s):

   curl -X GET "http://localhost:80/" \
        -H "X-Slow: $(head -c 5000 < /dev/zero | tr '\0' 'A')" \
        --limit-rate 1 \
        --verbose

   Parameter explanation:

   • -H "X-Slow: ...": generates a massive 5000-byte header (via head and tr) to force a long send.
   • --limit-rate 1: throttles the send to 1 byte/s. Without protection, the connection would remain open for over an hour.
   • --verbose: allows observing in real time the exact moment HAProxy interrupts the session.

   HAProxy recommendations (Read official article): For optimal protection, HAProxy recommends combining several settings:

   • timeout http-request 5s: limits the header waiting time (main defense).
   • timeout http-keep-alive 1s: quickly releases inactive connection slots.
   • maxconn: limits the global number of connections to protect system file descriptors from exhaustion.

3. Rate Limiting

   Set a quota of 50 requests maximum every 10 seconds per IP. Test exceeding the threshold and check that HAProxy returns an HTTP 429 error code.

   Verification: Execute a burst of curl requests to saturate the quota:

   for i in $(seq 1 60); do curl -I http://localhost; done

   You should observe HTTP/1.1 429 Too Many Requests responses after the 50th request.

4. Anti-bot modules and Scraping detection

   Block data scrapers and malicious bots by identifying their suspicious User-Agent signatures.

   Bot simulation: Try accessing the site by simulating a known bot via curl:

   curl -I -H "User-Agent: Googlebot-Scraper" http://localhost/

5. Silent request termination

   'Silent Drop' is an advanced defense technique: instead of sending an error code (403/429) that confirms to the hacker the existence of a protection, HAProxy simply ignores the request. The attacker wastes resources waiting for a response that will never come.

   Verification: Test access to a forbidden path and notice that curl remains waiting (hang) until timeout:

   curl -v http://localhost/admin/config

Observability #

Implementation of monitoring, metrics exposition and telemetry.

1. Stats Dashboard

   Verify that the administration entry point configured earlier still exposes the statistics dashboard on the root of port 8404.

2. Prometheus: Metrics exposition

   Configure HAProxy to expose performance metrics in Prometheus format on URI /metrics. This allows a Prometheus server to scrape the load balancer state.

   Verification: Test metrics retrieval via curl:

   curl http://localhost:8404/metrics

   You should see metrics in plain text format scrolling (HELP and TYPE).

3. OpenTelemetry (OTel) Support

   Configure native export of telemetry data to an OpenTelemetry collector.

   Note: Native OpenTelemetry support is available from version 3.4 of HAProxy.

Difficulty level : ●●●○○ (3/5)