Dynamic backend addition and removal

1. Introduction to dynamic management #

Traditionally, changing the structure of backends required reloading the configuration file. The Runtime API now lets you make these changes in memory.

Warning: Adding and removing backends and servers dynamically through the Runtime API requires HAProxy 2.8 or later.

Basic configuration #

To use these commands, your HAProxy instance must expose an administration socket with an admin level in the global section of your configuration:

Classic setup:

stats socket /run/haproxy/admin.sock user haproxy group haproxy mode 660 level admin

and/or through an IP (remote access, but beware of the security risks if you use an IP other than localhost):

stats socket ipv4@127.0.0.1:9999  level admin  expose-fd listeners

2. Glossary of key commands #

To change the infrastructure on the fly, it is essential to understand the role of each Runtime API verb.

help #

The help command displays all the available commands. The syntax is as follows; only the endpoint changes depending on how you enabled the socket:

The path to use with socat may vary depending on your configuration

• Classic:

  echo 'help' | socat stdio /var/run/haproxy.stat

• IP/PORT:

  echo 'help' | socat stdio tcp4-connect:127.0.0.1:9999

add (backend / server) #

The add command creates a new object (backend or server) in memory. A backend can be created from scratch or by inheriting the settings of an existing defaults section (essential for timeouts).

Warning: You may run into resolution problems when adding a server. You can see it by running this command, in which a server has no IP:

$ echo "show servers conn default" | socat stdio /run/haproxy/admin.sock
# bkname/svname bkid/svid addr port - purge_delay served used_cur used_max need_est idle_sess unsafe_nb safe_nb idle_lim idle_cur idle_per_thr[10]
default/web1 4/1 192.168.48.3 80 - 5000 0 0 1 1 0 0 0 -1 0 0 0 0 0 0 0 0 0 0 0
default/web2 4/2 192.168.48.2 80 - 5000 0 0 1 1 0 0 0 -1 0 0 0 0 0 0 0 0 0 0 0
default/web3 4/3 - 80 - 5000 0 0 0 0 0 0 0 -1 0 0 0 0 0 0 0 0 0 0 0

You can avoid this by precomputing the IP:

IP=$(getent hosts web3 | awk '{print $1}'); echo "add server default/web3 $IP:80 check" | socat stdio /run/haproxy/admin.sock

enable (server / health) #

By default, a dynamically added server is in maintenance mode (maint). enable server makes it operational. enable health specifically enables health checks for that server.

publish (backend) #

This is a validation step. As long as a backend is not published, it is invisible to frontends, even though it exists in memory. This lets you fully configure a pool before exposing it to traffic.

map (add / show / del) #

Maps are lookup tables (e.g. URL -> backend) stored in memory. add map adds a routing rule, del map removes it, and show map lets you debug the table's content without reloading HAProxy.

3. Adding a backend and a server #

Here is the workflow to create a new server pool and make it available.

Creation and activation #

The process breaks down into 4 key steps: adding the backend, adding the server, enabling health checks and publishing.

# 1. Ajouter le backend en héritant d'une section defaults
echo "add backend test-backend from mydefaults mode http" | socat stdio unix-connect:/run/haproxy/admin.sock

# 2. Ajouter un serveur au backend
echo "add server test-backend/server1 127.0.0.1:3000 check" | socat stdio unix-connect:/run/haproxy/admin.sock

# 3. Activer le serveur et ses health checks
echo "enable server test-backend/server1" | socat stdio unix-connect:/run/haproxy/admin.sock
echo "enable health test-backend/server1" | socat stdio unix-connect:/run/haproxy/admin.sock

# 4. Publier le backend pour le rendre utilisable par les frontends
echo "publish backend test-backend" | socat stdio unix-connect:/run/haproxy/admin.sock

Dynamic routing with map files #

To send traffic to this new backend without changing the frontend, use in-memory map files:

# Associer le chemin /test au nouveau backend
echo "add map virt@paths.map /test test-backend" | socat stdio unix-connect:/run/haproxy/admin.sock

4. Removing a backend cleanly #

Removal requires following a precise order so as not to break in-flight connections.

Removal procedure #

You must put the server into maintenance and wait until it is 'removable' before physically deleting it.

# 1. Passer le serveur en maintenance
echo "set server test-backend/server1 state maint" | socat stdio unix-connect:/run/haproxy/admin.sock

# 2. Attendre et supprimer le serveur
echo "wait 2s srv-removable test-backend/server1; del server test-backend/server1" | socat stdio unix-connect:/run/haproxy/admin.sock

# 3. Dé-publier et supprimer le backend
echo "unpublish backend test-backend" | socat stdio unix-connect:/run/haproxy/admin.sock
echo "wait 2s be-removable test-backend; del backend test-backend" | socat stdio unix-connect:/run/haproxy/admin.sock

5. Important points to watch #

A few behavior rules of the HAProxy engine you should know.

Publishing and force switch #

A backend referenced by a use_backend is ignored if it is not published. To force its use even when 'unpublished', use the force-be-switch directive.

Memory management #

To allow dynamic additions, HAProxy keeps the defaults sections in memory. If you don't use dynamic backends, you can free this memory with the following global directive:

global
    tune.defaults.purge on

Conclusion #

Towards full automation #

This approach is ideal to couple HAProxy with tools such as Consul or custom controllers, letting you scale your infrastructure without ever causing micro-outages from process reloads.