Lab: Fluent Bit on Kubernetes with metadata

What you will learn in this TP

• Install Loki and Fluent Bit
• Enrich with metadata
• Exclude a pod by annotation

Introduction #

What you will learn in this section

• Install Loki and Fluent Bit
• Enrich with metadata
• Exclude a pod by annotation

Context. The cluster is growing and container logs can only be seen with kubectl logs. You centralise all logs along with their Kubernetes context.

Prerequisites: kind, kubectl, helm  ·  Estimated time: 45 min

All the cluster logs, with namespace and pod.

Setup #

Guided steps, with their solutions.

1. 1. Create the cluster and Loki

   Why? Loki in monolithic mode is enough for this lab.

   Create a kind cluster and install Loki (grafana/loki chart in single binary mode).

2. 2. Install Fluent Bit

   Why? The chart deploys the DaemonSet, the RBAC permissions (for the kubernetes filter) and the configuration.

   Install Fluent Bit with a Loki output and the kubernetes filter.

3. 3. Deploy a chatty application

   Why? You need logs to check the enrichment.

   Start a pod that writes one line per second in the demo namespace.

4. 4. Query with metadata

   Why? Namespace and pod labels make searching immediate.

   Open a tunnel to Loki and search for the logs of the demo namespace.

5. 5. Exclude by annotation

   Why? Teams manage the noise of their own pods without changing the central configuration.

   Annotate the pod with fluentbit.io/exclude: "true" and check that its new lines no longer arrive.

6. 6. Clean up

   Why? To free up resources.

   Delete the cluster.

Difficulty level : ●●●○○ (3/5)