Health checks and high availability

1. The basic TCP check #

Simply checking that the port responds.

The check option #

By default, HAProxy tries to establish a TCP connection on the server's port. If it does not respond, the server is marked DOWN.

backend app_servers
    server s1 10.0.0.1:80 check inter 2s fall 3 rise 2

Here: a check every 2s, removal after 3 failures, back in service after 2 successes.

2. The HTTP check (layer 7) #

Checking that the application returns specific content.

Using http-check #

A port can be open while the application returns a 500 error. The HTTP check is more accurate.

backend app_servers
    option httpchk GET /health
    http-check expect status 200
    server s1 10.0.0.1:80 check

3. Backup servers #

Using a standby server only if all the others go down.

The backup keyword #

Ideal to display a maintenance page or use a less powerful standby server.

backend app_servers
    server s1 10.0.0.1:80 check
    server s2 10.0.0.2:80 check
    server s_backup 10.0.0.99:80 check backup

Conclusion #

Zero service interruption.

Constant vigilance #

By combining fine-grained application checks and backup servers, you guarantee maximum availability to your users.

Hands-on: health checks and high availability #

Probes, tolerance and failure tests.

HTTP health checks #

backend app
    option httpchk
    http-check send meth GET uri /health ver HTTP/1.1 hdr Host app.exemple.fr
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 slowstart 30s
    server web1 10.0.0.11:8080 check
    server web2 10.0.0.12:8080 check
    server web3 10.0.0.13:8080 check backup          # serveur de secours

# inter : intervalle, fall : échecs avant DOWN, rise : succès avant UP, slowstart : montée en charge progressive

Simulating a failure #

docker stop web1                                    # ou : systemctl stop app
echo "show servers state app" | sudo socat stdio /var/run/haproxy.sock | awk '{print $4, $6}'
for i in $(seq 1 6); do curl -s -o /dev/null -w "%{http_code} " https://app.exemple.fr/; done; echo   # aucune erreur visible
docker start web1                                   # le serveur revient après `rise` sondes réussies

High availability of HAProxy itself #

# /etc/keepalived/keepalived.conf  (VRRP : une IP virtuelle partagée par deux HAProxy)
vrrp_script chk_haproxy { script "pidof haproxy"  interval 2  weight 2 }
vrrp_instance VI_1 {
    state MASTER            # BACKUP sur le second nœud
    interface eth0
    virtual_router_id 51
    priority 101            # 100 sur le second nœud
    virtual_ipaddress { 192.0.2.10/24 }
    track_script { chk_haproxy }
}

sudo systemctl enable --now keepalived
ip addr show eth0 | grep 192.0.2.10        # l'IP virtuelle est portée par le MASTER
sudo systemctl stop haproxy                # l'IP bascule sur le BACKUP en quelques secondes