1. Backing up #
An untested backup does not exist.
Strategy #
| Question | To define |
|---|---|
| RPO | acceptable data loss (e.g. 24 h → daily backup) |
| RTO | acceptable time to restore service |
| Scope | /etc, application data, databases (consistent export), keys |
| Retention | e.g. 7 daily, 4 weekly, 12 monthly |
Tools #
rsync -aHAX --delete /srv/ backup01:/backups/app01/srv/ # miroir (attention, --delete propage les suppressions)
tar -czf /var/backups/etc_$(date +%F).tar.gz /etc
pg_dump -Fc appdb > /var/backups/appdb.dump # base : export cohérent, pas de copie des fichiers à chaud
# restic : dédupliqué, chiffré, incrémental
export RESTIC_REPOSITORY=s3:https://s3.example.com/backups-app01 RESTIC_PASSWORD_FILE=/root/.restic
restic init
restic backup /etc /srv /var/backups --exclude-caches
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune
restic snapshots ; restic check# snapshot LVM : figer un volume le temps de la sauvegarde
sudo lvcreate -s -n snap_data -L 5G /dev/vg_data/lv_data
sudo mount -o ro /dev/vg_data/snap_data /mnt/snap && restic backup /mnt/snap
sudo umount /mnt/snap && sudo lvremove -y /dev/vg_data/snap_dataRestoring #
restic restore latest --target /tmp/restore --include /etc/nginx
restic mount /mnt/restic # parcourir les sauvegardes comme un répertoireGood practice: Schedule a regular restore test on a test machine, and time it: that is your real RTO.
2. Monitoring #
Being warned before users are.
What to monitor #
| Resource | Typical alert threshold |
|---|---|
| Disk | > 85% or full within 24 h (trend) |
| Inodes | > 90% |
| Memory | available < 10%, OOM kills |
| CPU / load | load > 2 × cores for 15 min |
| Services | failed systemd unit, closed port |
| Backup | last success > 26 h |
| Certificates | expiry < 14 days |
| Updates | pending security fixes, reboot required |
Tools #
# métriques : Prometheus node_exporter
sudo apt install prometheus-node-exporter
curl -s localhost:9100/metrics | grep node_filesystem_avail_bytes
# journaux : envoi vers une plateforme centrale (Loki, Elasticsearch...) avec Alloy, Fluent Bit ou rsyslog
# vérification simple d'un service
systemctl is-active app || echo "app arrêtée" | mail -s "ALERTE app01" ops@example.comGood to know: The RousselTM Learning Prometheus, Zabbix, Grafana and Loki courses detail how to monitor a fleet of servers.
3. Troubleshooting #
The most frequent incidents.
The approach #
Quick-reference sheets #
| Symptom | First commands |
|---|---|
| The service does not start | systemctl status app, journalctl -u app -n 50, test the configuration (nginx -t, sshd -t), file permissions, port already taken (ss -tlnp), SELinux (ausearch -m avc) |
| Disk full | df -h, df -i, du -xh --max-depth=1, lsof +L1, log purge |
| Slow server | uptime, vmstat 1, iostat -xz 1, top, dmesg -T |
| Unreachable over the network | ip -br a, ip route, ss -tlnp, firewall, curl -v from the server then from outside |
| SSH connection refused | out-of-band console, sshd -t, journalctl -u ssh, fail2ban, ~/.ssh permissions |
| No longer boots | rescue mode via GRUB, journalctl -b -1 -p err, fstab (findmnt --verify), live system then chroot |
Repairing from a rescue system #
# démarré sur un ISO live ou la console de secours du cloud
lsblk ; sudo mount /dev/sda2 /mnt
for d in dev proc sys run; do sudo mount --bind /$d /mnt/$d; done
sudo chroot /mnt
# corriger /etc/fstab, réinstaller GRUB, réinitialiser un mot de passe...
passwd root ; update-grub ; exit4. Operating over time #
The administrator's routines.
Routines #
- Daily: alerts, successful backups, disk space.
- Weekly: security fixes, authentication logs, unused accounts.
- Monthly: restore test, review of sudo access and SSH keys, version upgrades.
- At every intervention: ticket, commands noted, versioned change (Ansible, Git), rollback planned.
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Processes, memory and performance: monitoring and troubleshooting a Linux server
Understand and control Linux processes (ps, top, signals, nice, jobs), interpret load, memory and...
Backup, Hyper-V and Windows troubleshooting
Back up and restore with Windows Server Backup (including system state and Active Directory), vir...
Linux administration best practices
Checklist for administering Linux servers: accounts and sudo, SSH, updates, systemd services, sto...
Linux server security and hardening
Harden a Linux server: least privilege, updates, secure SSH, firewall, fail2ban, SELinux and AppA...
Users, groups, sudo and permissions on Linux
Manage Linux accounts (useradd, usermod, passwd, chage), groups, sudo and visudo, rwx and octal p...
Storage on Linux: disks, partitions, file systems and LVM
Administer Linux storage: lsblk and blkid, GPT partitioning, ext4 and XFS, mounting and /etc/fsta...
Networking on Linux: configuration, diagnosis, firewall and SSH
Configure and troubleshoot Linux networking: ip, ss, netplan and NetworkManager, DNS, layered dia...
Linux system architecture: kernel, boot and directory tree
Understand how a Linux server works: kernel and user space, the UEFI, GRUB, initramfs and systemd...
Linux vs Unix: the complete guide to commands and distributions
Understand the differences between Linux and Unix, explore distributions (server, desktop, lightw...