RousselTMLEARNING

Article · Linux administration: from discovery to expertise

Backup, monitoring and troubleshooting

1. Backing up #

An untested backup does not exist.

Strategy #

The 3-2-1 rule
An encrypted off-site copy also protects against ransomware and fire.
QuestionTo define
RPOacceptable data loss (e.g. 24 h → daily backup)
RTOacceptable time to restore service
Scope/etc, application data, databases (consistent export), keys
Retentione.g. 7 daily, 4 weekly, 12 monthly

Tools #

restic cycle
Schedule the backup with a systemd timer and monitor the date of the last success.
rsync -aHAX --delete /srv/ backup01:/backups/app01/srv/   # miroir (attention, --delete propage les suppressions)
tar -czf /var/backups/etc_$(date +%F).tar.gz /etc
pg_dump -Fc appdb > /var/backups/appdb.dump                # base : export cohérent, pas de copie des fichiers à chaud

# restic : dédupliqué, chiffré, incrémental
export RESTIC_REPOSITORY=s3:https://s3.example.com/backups-app01 RESTIC_PASSWORD_FILE=/root/.restic
restic init
restic backup /etc /srv /var/backups --exclude-caches
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune
restic snapshots ; restic check
# snapshot LVM : figer un volume le temps de la sauvegarde
sudo lvcreate -s -n snap_data -L 5G /dev/vg_data/lv_data
sudo mount -o ro /dev/vg_data/snap_data /mnt/snap && restic backup /mnt/snap
sudo umount /mnt/snap && sudo lvremove -y /dev/vg_data/snap_data

Restoring #

restic restore latest --target /tmp/restore --include /etc/nginx
restic mount /mnt/restic        # parcourir les sauvegardes comme un répertoire
Good practice: Schedule a regular restore test on a test machine, and time it: that is your real RTO.

2. Monitoring #

Being warned before users are.

What to monitor #

ResourceTypical alert threshold
Disk> 85% or full within 24 h (trend)
Inodes> 90%
Memoryavailable < 10%, OOM kills
CPU / loadload > 2 × cores for 15 min
Servicesfailed systemd unit, closed port
Backuplast success > 26 h
Certificatesexpiry < 14 days
Updatespending security fixes, reboot required

Tools #

# métriques : Prometheus node_exporter
sudo apt install prometheus-node-exporter
curl -s localhost:9100/metrics | grep node_filesystem_avail_bytes
# journaux : envoi vers une plateforme centrale (Loki, Elasticsearch...) avec Alloy, Fluent Bit ou rsyslog
# vérification simple d'un service
systemctl is-active app || echo "app arrêtée" | mail -s "ALERTE app01" ops@example.com
Good to know: The RousselTM Learning Prometheus, Zabbix, Grafana and Loki courses detail how to monitor a fleet of servers.
Monitoring chain
The Prometheus, Loki, Grafana and Alertmanager courses detail each building block.

3. Troubleshooting #

The most frequent incidents.

The approach #

Troubleshooting method
Change one thing at a time, and write down what you did.

Quick-reference sheets #

SymptomFirst commands
The service does not startsystemctl status app, journalctl -u app -n 50, test the configuration (nginx -t, sshd -t), file permissions, port already taken (ss -tlnp), SELinux (ausearch -m avc)
Disk fulldf -h, df -i, du -xh --max-depth=1, lsof +L1, log purge
Slow serveruptime, vmstat 1, iostat -xz 1, top, dmesg -T
Unreachable over the networkip -br a, ip route, ss -tlnp, firewall, curl -v from the server then from outside
SSH connection refusedout-of-band console, sshd -t, journalctl -u ssh, fail2ban, ~/.ssh permissions
No longer bootsrescue mode via GRUB, journalctl -b -1 -p err, fstab (findmnt --verify), live system then chroot

Repairing from a rescue system #

# démarré sur un ISO live ou la console de secours du cloud
lsblk ; sudo mount /dev/sda2 /mnt
for d in dev proc sys run; do sudo mount --bind /$d /mnt/$d; done
sudo chroot /mnt
# corriger /etc/fstab, réinstaller GRUB, réinitialiser un mot de passe...
passwd root ; update-grub ; exit

4. Operating over time #

The administrator's routines.

Routines #

  • Daily: alerts, successful backups, disk space.
  • Weekly: security fixes, authentication logs, unused accounts.
  • Monthly: restore test, review of sudo access and SSH keys, version upgrades.
  • At every intervention: ticket, commands noted, versioned change (Ansible, Git), rollback planned.

📄 Backup, monitoring and troubleshooting of a Linux server →

My progress

  • ⏱ Estimated time : ≈ 8 min

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Processes, memory and performance: monitoring and troubleshooting a Linux server

Understand and control Linux processes (ps, top, signals, nice, jobs), interpret load, memory and...

Backup, Hyper-V and Windows troubleshooting

Back up and restore with Windows Server Backup (including system state and Active Directory), vir...

Linux administration best practices

Checklist for administering Linux servers: accounts and sudo, SSH, updates, systemd services, sto...

Linux server security and hardening

Harden a Linux server: least privilege, updates, secure SSH, firewall, fail2ban, SELinux and AppA...

Users, groups, sudo and permissions on Linux

Manage Linux accounts (useradd, usermod, passwd, chage), groups, sudo and visudo, rwx and octal p...

Storage on Linux: disks, partitions, file systems and LVM

Administer Linux storage: lsblk and blkid, GPT partitioning, ext4 and XFS, mounting and /etc/fsta...

Networking on Linux: configuration, diagnosis, firewall and SSH

Configure and troubleshoot Linux networking: ip, ss, netplan and NetworkManager, DNS, layered dia...

Linux system architecture: kernel, boot and directory tree

Understand how a Linux server works: kernel and user space, the UEFI, GRUB, initramfs and systemd...

Linux vs Unix: the complete guide to commands and distributions

Understand the differences between Linux and Unix, explore distributions (server, desktop, lightw...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!