Processes, memory and performance: monitoring and troubleshooting a Linux server

1. Processes #

What is running and how to control it.

Listing #

ps aux --sort=-%cpu | head       # plus gros consommateurs CPU
ps aux --sort=-%mem | head
ps -ef --forest                 # arbre parent/enfant
pgrep -a nginx ; pidof sshd
top   # P : tri CPU, M : tri mémoire, 1 : par cœur, k : tuer
htop  # version interactive et colorée

States #

Signals #

kill -15 1234     # SIGTERM : demande d'arrêt propre (défaut)
kill -9 1234      # SIGKILL : arrêt immédiat, sans nettoyage
kill -HUP 1234    # SIGHUP : souvent « relire la configuration »
pkill -u bob      # tous les processus de bob
killall -s TERM python3

Good practice: Always send SIGTERM first and wait a few seconds: SIGKILL prevents the application from closing its files and connections cleanly.

Priorities and background #

nice -n 10 tar -czf backup.tgz /srv   # lancer avec une priorité basse
renice +5 -p 1234
ionice -c3 rsync -a /src /dst          # E/S seulement quand le disque est libre
commande &   ; jobs ; fg %1 ; bg %1     # tâches du shell
nohup ./long.sh > long.log 2>&1 &      # survivre à la déconnexion
tmux new -s maintenance                # session détachable (Ctrl+B D, puis tmux attach)

2. CPU, memory and disks #

Reading the right indicators.

Load #

uptime    # load average 1, 5 et 15 min
nproc     # nombre de cœurs

The load counts processes running or waiting (including in D state). It is interpreted relative to the number of cores: 4 on 4 cores = full; 12 on 4 cores = saturation. A high load with little CPU usage often indicates disk waits.

Memory #

free -h

Look at the available column, not free: Linux uses free memory as disk cache (buff/cache), which can be reclaimed immediately. A server with little « free » but plenty of « available » is fine.

When memory really runs out, the kernel triggers the OOM killer, which kills the most memory-hungry process:

journalctl -k | grep -i 'out of memory'
dmesg -T | grep -i oom
cat /proc/1234/status | grep VmRSS

vmstat and iostat #

vmstat 1                 # r, b, si/so, wa
iostat -xz 1             # %util et await par disque (paquet sysstat)
mpstat -P ALL 1          # CPU par cœur
pidstat 1                # consommation par processus
sar -u -f /var/log/sysstat/sa26   # historique (sysstat)

3. Diagnosing a process #

Seeing what it really does.

lsof, strace and /proc #

sudo lsof -p 1234             # fichiers et sockets ouverts
sudo lsof -i :8080            # qui utilise ce port ?
sudo lsof /var/log/app.log
sudo strace -p 1234 -f -e trace=network,file   # appels système en direct
sudo strace -c -p 1234        # statistiques des appels (Ctrl+C)
cat /proc/1234/cmdline | tr '\0' ' '
ls -l /proc/1234/cwd /proc/1234/exe
cat /proc/1234/limits         # limites (fichiers ouverts...)
ulimit -n

Limiting resources #

systemd places each service in a cgroup: it can be limited without touching the application.

sudo systemctl set-property app.service MemoryMax=1G CPUQuota=150%
systemd-cgtop                  # consommation par service

4. A method #

Not searching at random.

USE #

1. uptime: is the load rising?
2. dmesg -T | tail: kernel errors, OOM, disk?
3. vmstat 1: run queue, swap, I/O wait
4. mpstat -P ALL 1: a saturated core?
5. iostat -xz 1: saturated disk?
6. free -h: available memory
7. ss -s, ip -s link: connections, network errors
8. top / pidstat: which process?

Good to know: This « first 60 seconds » checklist comes from Netflix's performance team: it is enough to orient most diagnoses.