LogQL (1/2): querying logs (selectors, filters and parsers)

1. The stream selector #

Mandatory, and always as precise as possible.

Matchers #

{app="api"}                                  # égalité
{app!="api"}                                 # différent
{app=~"api|web"}                             # expression régulière
{namespace="prod", app=~"checkout-.*"}       # plusieurs labels
{app="api", env!~"dev|test"}                 # exclusion

Common mistake: A selector must contain at least one non-empty matcher: {app=~".*"} is rejected because it is too expensive.

2. Filtering lines #

From the fastest to the most expensive.

Line filters #

{app="api"} |= "error"                   # contient
{app="api"} != "healthcheck"                 # ne contient pas
{app="api"} |~ "timeout|refused"             # regex
{app="api"} !~ "GET /static/.*"              # ne correspond pas à la regex
{app="api"} |= "error" != "healthcheck" |~ "user=\\d+"   # enchaînement
{app="api"} |= ip("10.0.0.0/8")              # filtre sur adresse IP

Good practice: Put |= and != filters before parsers: they are much faster than parsing and then filtering.

3. Parsing lines #

Extracting fields.

json, logfmt, pattern and regexp #

# JSON : {"level":"error","method":"GET","path":"/api","status":502,"duration_ms":340}
{app="api"} | json
{app="api"} | json level="level", http_status="status"      # extraire en renommant
{app="api"} | json | status >= 500 and duration_ms > 300

# logfmt : level=error method=GET path=/api status=502
{app="api"} | logfmt | level="error"

# pattern : lignes d'accès simples (très rapide)
{app="nginx"} | pattern `<ip> - - [<_>] "<method> <uri> <_>" <status> <size>` | status >= 500

# regexp : captures nommées
{app="app"} | regexp `user=(?P<user>\w+) action=(?P<action>\w+)`

Filters on extracted labels #

{app="api"} | json | level="error"
{app="api"} | json | status=~"5.."
{app="api"} | json | duration_ms > 250 and method="POST"
{app="api"} | json | __error__=""                    # ignorer les lignes non parsables
{app="api"} | logfmt | duration > 1s                # durée typée (1s, 500ms)
{app="api"} | json | bytes > 5MB                    # taille typée

4. Formatting #

line_format, label_format and drop/keep.

Reformatting the output #

{app="api"} | json | line_format "{{.method}} {{.path}} -> {{.status}} ({{.duration_ms}} ms)"
{app="api"} | json | label_format route=path, ms=duration_ms
{app="api"} | json | drop user_agent, request_id
{app="api"} | json | keep level, status, path
{app="api"} | json | line_format `{{ .level | upper }} {{ .msg | trunc 80 }}`

Performance best practices #

• Reduce the time range and the number of streams before refining.
• Prefer pattern (or logfmt) to regexp: they are much faster.
• Use structured metadata for identifiers and filter on them without parsing.
• Limit the output with limit (Explore) or --limit (logcli).

logcli query '{app="api"} |= "error" | json | status>=500' --since=30m --limit=50 --output=jsonl
logcli query '{app="api"}' --since=1h --stats          # volumes lus : octets, lignes, flux