1. Architecture #
Keeping it simple.
Principles #
- Single node as long as possible; cluster when volume or multi-tenancy requires it.
- vmagent for collection, with a disk buffer and two destinations for HA.
- SSD disks, at least 20% free space (part merging).
- Retention set explicitly, daily backups verified by a test restore.
2. Data #
Controlling cardinality.
Rules #
- Cardinality explorer checked every month.
- Relabeling in vmagent to drop useless labels and metrics.
- Stream aggregation for metrics where only the total is useful.
- Dashboards in standard PromQL if they must stay portable.
3. Operations #
Monitoring and upgrading.
Rituals #
- Official dashboards and alerts installed.
- Regular upgrades (changelog read, one instance at a time).
- vmalert with
-remoteRead.urlto keep alert state. - Access protected by vmauth or a reverse proxy.
Reference diagram and quick check #
Checklist.
Checklist #
- Retention and disk sized
- vmagent with disk buffer
- vmalert connected to Alertmanager
- vmbackup backups and restore test
- Monitoring with the official dashboards
- Authenticated access
