Prometheus in Kubernetes: Operator and ServiceMonitor

1. Monitoring Kubernetes: what to collect #

Four sources of metrics.

The sources #

2. Installing kube-prometheus-stack #

All-in-one with Helm.

Installation #

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --set grafana.adminPassword=ChangeMe! \
  --set prometheus.prometheusSpec.retention=15d \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

kubectl -n monitoring get pods
kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80
kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090

The chart installs the operator, Prometheus, Alertmanager, Grafana with ready-to-use Kubernetes dashboards, node-exporter, kube-state-metrics and a standard set of alert rules.

3. The Operator's CRDs #

Declaring what you want to monitor.

ServiceMonitor and PodMonitor #

# Application exposant /metrics sur le port nommé "http-metrics"
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: mon-api
  namespace: monitoring
  labels: { release: monitoring }
spec:
  namespaceSelector: { matchNames: [production] }
  selector:
    matchLabels: { app: mon-api }
  endpoints:
    - port: http-metrics
      path: /metrics
      interval: 30s

# Sans Service : cibler directement les pods
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: workers, namespace: monitoring, labels: { release: monitoring } }
spec:
  namespaceSelector: { matchNames: [production] }
  selector: { matchLabels: { app: worker } }
  podMetricsEndpoints:
    - port: metrics

PrometheusRule #

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: mon-api-alertes, namespace: monitoring, labels: { release: monitoring } }
spec:
  groups:
    - name: mon-api
      rules:
        - alert: ApiHighErrorRate
          expr: sum(rate(http_requests_total{app="mon-api",status=~"5.."}[5m])) / sum(rate(http_requests_total{app="mon-api"}[5m])) > 0.05
          for: 10m
          labels: { severity: critical }
          annotations: { summary: "Plus de 5 % d'erreurs sur mon-api" }

kubectl get servicemonitors,podmonitors,prometheusrules -A
kubectl -n monitoring get prometheus -o yaml | grep -A3 serviceMonitorSelector

Common mistake: If your ServiceMonitor is not picked up, first check that its release label matches the serviceMonitorSelector of the Prometheus object, then that the Service port is named.

4. Kubernetes PromQL queries #

The essentials.

A few useful queries #

# Pods qui redémarrent
increase(kube_pod_container_status_restarts_total[1h]) > 3

# Pods non prêts
kube_pod_status_ready{condition="false"} == 1

# CPU par namespace (cœurs)
sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))

# Mémoire utilisée / demandée
sum by (pod) (container_memory_working_set_bytes{container!=""})
  / sum by (pod) (kube_pod_container_resource_requests{resource="memory"})

# Réplicas indisponibles
kube_deployment_status_replicas_unavailable > 0

# PVC bientôt pleins
kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.1

Good practice: For large clusters, delegate long-term storage to Mimir/Thanos with remoteWrite and reduce Prometheus's local retention.