Final lab: Prepare a server for production and resolve an incident

What you will learn in this TP

• Apply a go-live checklist
• Set up monitoring and backup
• Troubleshoot an incident methodically

Introduction #

What you will learn in this section

• Apply a go-live checklist
• Set up monitoring and backup
• Troubleshoot an incident methodically

Context. A new server must host the internal application. You apply the team's go-live checklist, then an incident occurs: the disk fills up although nobody can find a large file.

Prerequisites: the VM from the previous labs (the app service and key-based SSH in place)  ·  Estimated time: 90 min

A production-ready server, a measured hardening index and a resolved, documented incident.

Hands-on #

Guided steps, with their solution.

1. 1. Measure the starting point

   Why? A before/after audit makes hardening work measurable.

   Install Lynis and note the initial hardening index and main suggestions.

2. 2. Update automatically

   Why? Most compromises exploit already-patched vulnerabilities.

   Apply updates, enable automatic security updates and check whether a reboot is required.

3. 3. Secure access

   Why? Key-based SSH, a firewall and fail2ban form the first line of defence.

   Check the SSH configuration from the networking lab, open only SSH and the application port, install fail2ban for SSH and apply sysctl hardening settings.

4. 4. Monitor

   Why? An unmonitored server fails silently; node_exporter exposes system metrics for Prometheus.

   Install node_exporter, check disk and memory metrics, and add a disk-space check to the health timer.

5. 5. Back up with restic

   Why? restic encrypts, deduplicates and handles retention; the restore test validates the whole chain.

   Initialise a local restic repository (simulating remote storage), back up /etc and /srv, apply retention and restore /srv/app/www to a temporary directory.

6. 6. Incident: the disk is filling up

   Why? A deleted but still open file keeps its space: du no longer sees it, df does.

   Run the incident scenario below, then troubleshoot: compare df and du, find the culprit with lsof and free the space without rebooting the server.

7. 7. Measure and document

   Why? Measuring progress and writing a report turns an incident into learning for the team.

   Re-run Lynis to compare the index, run the quick check from the best-practices article and write a short post-mortem (symptom, cause, resolution, preventive actions).

Difficulty level : ●●●●○ (4/5)