RousselTMLEARNING

Article · Linux administration: from discovery to expertise

Processes, memory and performance

1. Processes #

What is running and how to control it.

Listing #

ps aux --sort=-%cpu | head       # plus gros consommateurs CPU
ps aux --sort=-%mem | head
ps -ef --forest                 # arbre parent/enfant
pgrep -a nginx ; pidof sshd
top   # P : tri CPU, M : tri mémoire, 1 : par cœur, k : tuer
htop  # version interactive et colorée

States #

Process states
Many processes in D state drive the load up without consuming CPU.
StateMeaning
Rrunning or ready
Ssleeping, waiting for an event (normal case)
Duninterruptible I/O wait (disk, NFS): cannot be killed
Zzombie: finished, not yet reaped by its parent
Tstopped (Ctrl+Z)

Signals #

kill -15 1234     # SIGTERM : demande d'arrêt propre (défaut)
kill -9 1234      # SIGKILL : arrêt immédiat, sans nettoyage
kill -HUP 1234    # SIGHUP : souvent « relire la configuration »
pkill -u bob      # tous les processus de bob
killall -s TERM python3
Good practice: Always send SIGTERM first and wait a few seconds: SIGKILL prevents the application from closing its files and connections cleanly.

Priorities and background #

nice -n 10 tar -czf backup.tgz /srv   # lancer avec une priorité basse
renice +5 -p 1234
ionice -c3 rsync -a /src /dst          # E/S seulement quand le disque est libre
commande &   ; jobs ; fg %1 ; bg %1     # tâches du shell
nohup ./long.sh > long.log 2>&1 &      # survivre à la déconnexion
tmux new -s maintenance                # session détachable (Ctrl+B D, puis tmux attach)

2. CPU, memory and disks #

Reading the right indicators.

Load #

uptime    # load average 1, 5 et 15 min
nproc     # nombre de cœurs

The load counts processes running or waiting (including in D state). It is interpreted relative to the number of cores: 4 on 4 cores = full; 12 on 4 cores = saturation. A high load with little CPU usage often indicates disk waits.

Memory #

Memory breakdown
The cache is given back immediately to applications that need it.
free -h

Look at the available column, not free: Linux uses free memory as disk cache (buff/cache), which can be reclaimed immediately. A server with little « free » but plenty of « available » is fine.

When memory really runs out, the kernel triggers the OOM killer, which kills the most memory-hungry process:

journalctl -k | grep -i 'out of memory'
dmesg -T | grep -i oom
cat /proc/1234/status | grep VmRSS

vmstat and iostat #

vmstat output
Swap, disk wait and blocked processes: the server is short of memory.
vmstat 1                 # r, b, si/so, wa
iostat -xz 1             # %util et await par disque (paquet sysstat)
mpstat -P ALL 1          # CPU par cœur
pidstat 1                # consommation par processus
sar -u -f /var/log/sysstat/sa26   # historique (sysstat)

3. Diagnosing a process #

Seeing what it really does.

lsof, strace and /proc #

sudo lsof -p 1234             # fichiers et sockets ouverts
sudo lsof -i :8080            # qui utilise ce port ?
sudo lsof /var/log/app.log
sudo strace -p 1234 -f -e trace=network,file   # appels système en direct
sudo strace -c -p 1234        # statistiques des appels (Ctrl+C)
cat /proc/1234/cmdline | tr '\0' ' '
ls -l /proc/1234/cwd /proc/1234/exe
cat /proc/1234/limits         # limites (fichiers ouverts...)
ulimit -n

Limiting resources #

systemd places each service in a cgroup: it can be limited without touching the application.

sudo systemctl set-property app.service MemoryMax=1G CPUQuota=150%
systemd-cgtop                  # consommation par service

4. A method #

Not searching at random.

USE #

USE method
Reviewing each resource avoids forgetting the disk or the network.
  1. uptime: is the load rising?
  2. dmesg -T | tail: kernel errors, OOM, disk?
  3. vmstat 1: run queue, swap, I/O wait
  4. mpstat -P ALL 1: a saturated core?
  5. iostat -xz 1: saturated disk?
  6. free -h: available memory
  7. ss -s, ip -s link: connections, network errors
  8. top / pidstat: which process?
Good to know: This « first 60 seconds » checklist comes from Netflix's performance team: it is enough to orient most diagnoses.

📄 Processes, memory and performance: monitoring and troubleshooting a Linux server →

My progress

  • ⏱ Estimated time : ≈ 9 min

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Backup, monitoring and troubleshooting of a Linux server

Operate a Linux server day to day: 3-2-1 backup strategy with rsync, tar, LVM snapshots and resti...

Linux server security and hardening

Harden a Linux server: least privilege, updates, secure SSH, firewall, fail2ban, SELinux and AppA...

Linux administration best practices

Checklist for administering Linux servers: accounts and sudo, SSH, updates, systemd services, sto...

Users, groups, sudo and permissions on Linux

Manage Linux accounts (useradd, usermod, passwd, chage), groups, sudo and visudo, rwx and octal p...

Storage on Linux: disks, partitions, file systems and LVM

Administer Linux storage: lsblk and blkid, GPT partitioning, ext4 and XFS, mounting and /etc/fsta...

Networking on Linux: configuration, diagnosis, firewall and SSH

Configure and troubleshoot Linux networking: ip, ss, netplan and NetworkManager, DNS, layered dia...

Linux system architecture: kernel, boot and directory tree

Understand how a Linux server works: kernel and user space, the UEFI, GRUB, initramfs and systemd...

Linux vs Unix: the complete guide to commands and distributions

Understand the differences between Linux and Unix, explore distributions (server, desktop, lightw...

The Bash shell and files: navigation, redirections, pipes and text processing

Master the Linux command line: paths, file handling, wildcards, redirections, pipes, searching wi...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!