1. Backing up #
An untested backup does not exist.
Strategy #
| Question | To define |
|---|---|
| RPO | acceptable data loss (e.g. 24 h → daily backup) |
| RTO | acceptable time to restore service |
| Scope | /etc, application data, databases (consistent export), keys |
| Retention | e.g. 7 daily, 4 weekly, 12 monthly |
Tools #
rsync -aHAX --delete /srv/ backup01:/backups/app01/srv/ # miroir (attention, --delete propage les suppressions)
tar -czf /var/backups/etc_$(date +%F).tar.gz /etc
pg_dump -Fc appdb > /var/backups/appdb.dump # base : export cohérent, pas de copie des fichiers à chaud
# restic : dédupliqué, chiffré, incrémental
export RESTIC_REPOSITORY=s3:https://s3.example.com/backups-app01 RESTIC_PASSWORD_FILE=/root/.restic
restic init
restic backup /etc /srv /var/backups --exclude-caches
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune
restic snapshots ; restic check# snapshot LVM : figer un volume le temps de la sauvegarde
sudo lvcreate -s -n snap_data -L 5G /dev/vg_data/lv_data
sudo mount -o ro /dev/vg_data/snap_data /mnt/snap && restic backup /mnt/snap
sudo umount /mnt/snap && sudo lvremove -y /dev/vg_data/snap_dataRestoring #
restic restore latest --target /tmp/restore --include /etc/nginx
restic mount /mnt/restic # parcourir les sauvegardes comme un répertoireGood practice: Schedule a regular restore test on a test machine, and time it: that is your real RTO.
2. Monitoring #
Being warned before users are.
What to monitor #
| Resource | Typical alert threshold |
|---|---|
| Disk | > 85% or full within 24 h (trend) |
| Inodes | > 90% |
| Memory | available < 10%, OOM kills |
| CPU / load | load > 2 × cores for 15 min |
| Services | failed systemd unit, closed port |
| Backup | last success > 26 h |
| Certificates | expiry < 14 days |
| Updates | pending security fixes, reboot required |
Tools #
# métriques : Prometheus node_exporter
sudo apt install prometheus-node-exporter
curl -s localhost:9100/metrics | grep node_filesystem_avail_bytes
# journaux : envoi vers une plateforme centrale (Loki, Elasticsearch...) avec Alloy, Fluent Bit ou rsyslog
# vérification simple d'un service
systemctl is-active app || echo "app arrêtée" | mail -s "ALERTE app01" ops@example.comGood to know: The RousselTM Learning Prometheus, Zabbix, Grafana and Loki courses detail how to monitor a fleet of servers.
3. Troubleshooting #
The most frequent incidents.
The approach #
Quick-reference sheets #
| Symptom | First commands |
|---|---|
| The service does not start | systemctl status app, journalctl -u app -n 50, test the configuration (nginx -t, sshd -t), file permissions, port already taken (ss -tlnp), SELinux (ausearch -m avc) |
| Disk full | df -h, df -i, du -xh --max-depth=1, lsof +L1, log purge |
| Slow server | uptime, vmstat 1, iostat -xz 1, top, dmesg -T |
| Unreachable over the network | ip -br a, ip route, ss -tlnp, firewall, curl -v from the server then from outside |
| SSH connection refused | out-of-band console, sshd -t, journalctl -u ssh, fail2ban, ~/.ssh permissions |
| No longer boots | rescue mode via GRUB, journalctl -b -1 -p err, fstab (findmnt --verify), live system then chroot |
Repairing from a rescue system #
# démarré sur un ISO live ou la console de secours du cloud
lsblk ; sudo mount /dev/sda2 /mnt
for d in dev proc sys run; do sudo mount --bind /$d /mnt/$d; done
sudo chroot /mnt
# corriger /etc/fstab, réinstaller GRUB, réinitialiser un mot de passe...
passwd root ; update-grub ; exit4. Operating over time #
The administrator's routines.
Routines #
- Daily: alerts, successful backups, disk space.
- Weekly: security fixes, authentication logs, unused accounts.
- Monthly: restore test, review of sudo access and SSH keys, version upgrades.
- At every intervention: ticket, commands noted, versioned change (Ansible, Git), rollback planned.
