Kubernetes: reliability and scaling (probes, resources, HPA, PDB)

1. Probes #

Telling Kubernetes whether the application is healthy.

The three probes #

containers:
  - name: api
    image: exemple/api:1.4.2
    startupProbe:
      httpGet: { path: /healthz, port: 8080 }
      failureThreshold: 30
      periodSeconds: 5                  # jusqu'à 150 s pour démarrer
    readinessProbe:
      httpGet: { path: /ready, port: 8080 }
      periodSeconds: 5
      failureThreshold: 3
    livenessProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2

Common mistake: A liveness probe that checks an external dependency (database) causes cascading restarts when that dependency goes down. Keep it simple: « my process responds ».

2. Resources and quality of service #

requests, limits and QoS.

Sizing correctly #

resources:
  requests: { cpu: 250m, memory: 256Mi }      # base du scheduling et du HPA
  limits:   { memory: 512Mi }                 # limite mémoire ; CPU sans limite pour éviter le throttling

kubectl top pods --containers
kubectl describe pod <pod> | grep -E 'QoS|Last State|OOMKilled'
kubectl get pods -o custom-columns=NOM:.metadata.name,QOS:.status.qosClass

Good to know: Under node pressure, the kubelet picks the pods to evict first based on their consumption beyond their requests, then their priority (PriorityClass), then their relative consumption. The QoS class follows from this in practice (a BestEffort pod always exceeds its zero requests), but it is not the direct criterion.

3. Autoscaling #

HPA, VPA and Cluster Autoscaler.

Horizontal Pod Autoscaler #

kubectl autoscale deployment web --cpu-percent=70 --min=2 --max=10
kubectl get hpa -w

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }
    - type: Pods                    # métrique personnalisée : nécessite Prometheus Adapter ou KEDA
      pods: { metric: { name: http_requests_per_second }, target: { type: AverageValue, averageValue: "100" } }
  behavior:
    scaleDown: { stabilizationWindowSeconds: 300 }

# Générer de la charge pour observer le scaling
kubectl run load --rm -it --image=busybox --restart=Never -- sh -c "while true; do wget -q -O- http://web; done"
kubectl get hpa web -w

VPA, Cluster Autoscaler and KEDA #

• VPA: recommends or adjusts requests/limits based on actual consumption (Off mode for recommendations only).
• Cluster Autoscaler / Karpenter: adds or removes nodes when pods stay Pending.
• KEDA: event-driven autoscaling (Kafka/RabbitMQ queue length, cron, Prometheus metrics), down to zero replicas.

4. Availability during operations #

PDB and graceful shutdown.

PodDisruptionBudget and graceful shutdown #

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec:
  minAvailable: 2                  # ou maxUnavailable: 1
  selector: { matchLabels: { app: web } }

kubectl drain <noeud> --ignore-daemonsets --delete-emptydir-data     # respecte les PDB
kubectl cordon <noeud> && kubectl uncordon <noeud>

# Arrêt propre : laisser le temps de terminer les requêtes
terminationGracePeriodSeconds: 45
lifecycle:
  preStop: { exec: { command: ["sh", "-c", "sleep 10"] } }       # laisse l'Ingress retirer le pod avant SIGTERM

Good practice: Combine at least 2 replicas, a readinessProbe, a PodDisruptionBudget and topologySpreadConstraints to survive a node or zone failure.