RousselTMLEARNING

Article · Cribl: from discovery to expertise

Pipelines and functions

1. Where to place a pipeline #

Three locations.

The three places #

Pipeline places

A pipeline is an ordered list of functions. Each function has a filter (it only applies to matching events) and can be marked Final to stop processing those events. Functions can be grouped for readability and disabled without being deleted.

2. The function catalogue #

The most used.

By use #

Typical pipeline
UseFunctions
ExtractParser (JSON, CSV, key=value, delimited, Grok), Regex Extract, Auto Timestamp, XML / JSON Unroll
ModifyEval, Rename, Flatten, Numerify, Serialize, Code
EnrichLookup, GeoIP, DNS Lookup, Redis
ProtectMask (regular expressions, hashing), dropping sensitive fields
ReduceDrop, Regex Filter, Sampling, Dynamic Sampling, Suppress, Aggregations, Rollup Metrics
MetricsPublish Metrics, Prometheus Publisher, Aggregations
OrganizeChain (call another pipeline), Comment, groups
Good to know : The Tee function has been deprecated since 4.18: use non-final routes or the Output Router to duplicate a flow.

3. Cribl expressions #

JavaScript everywhere.

Examples #

Filters, computed values and file names are written in JavaScript, extended with a C object of helper functions:

ExpressionResult
status >= 500true for server errors
uri.split('?')[0]the URL without parameters
C.Mask.md5(user)MD5 hash of an identifier (pseudonymization)
C.Time.strftime(_time, '%Y-%m-%d')formatted date
C.Lookup('assets.csv', 'ip').match(src_ip, 'owner')owner of the address in a table
C.Net.cidrMatch('192.168.0.0/16', dst_ip)network membership
`${host}:${port}`composed string (template literal)
Watch out : Field values in Eval are expressions: a string is written in quotes ('prod'), otherwise Cribl looks for a field named prod.

4. A complete pipeline #

Apache access logs.

Configuration #

# local/cribl/pipelines/apache_access/conf.yml (excerpt)
functions:
  - id: regex_extract
    filter: "true"
    conf:
      source: _raw
      regex: '/^(?<client>\S+) \S+ \S+ \[[^\]]+\] "(?<method>\S+) (?<uri>\S+) [^"]*" (?<status>\d{3}) (?<bytes>\d+|-)/'
  - id: lookup
    conf:
      file: http_status.csv
      inFields: [{ eventField: status, lookupField: status }]
      outFields: [{ lookupField: category, eventField: status_category }]
  - id: mask
    conf:
      fields: [_raw]
      rules: [{ matchRegex: '/(email=)[^&\s]+/g', replaceExpr: "`${g1}${C.Mask.md5(g0)}`" }]
  - id: eval
    conf:
      add: [{ name: env, value: "'prod'" }]
      remove: [bytes]
  1. Regex Extract creates the client, method, uri, status, bytes fields.
  2. Lookup adds status_category from a CSV table (Processing › Knowledge › Lookups).
  3. Mask replaces e-mail addresses in parameters with a hash.
  4. Eval adds env and removes a useless field.

You build a pipeline in the UI, not in YAML: the excerpt only shows what is saved in local/.

The preview #

  • Load a sample file in the right panel: IN shows the incoming event, OUT the result, with added fields in green and removed ones in red.
  • The statistics icon shows event and byte counts before and after: reduction is measured at design time.
  • Full preview also runs the route and post-processing pipeline to see the event as it will leave.
Best practice : For performance: filter early (Drop first), prefer Parser to long regular expressions, anchor your regexes, and keep the Code function for cases with no equivalent.

Commands to know #

Commands to know
  • C.Mask.md5(user) : pseudonymize a field
  • C.Lookup('assets.csv','ip').match(src_ip,'owner') : enrich from a table
  • C.Time.strftime(_time, '%Y-%m-%d') : format a date
  • /(?<status>\d{3})/ : Regex Extract named group

Sources #

Checked in November 2026 (Cribl 4.20): Cribl ships a minor release about every two months, check the release notes before relying on a detail.

📄 Pipelines and functions: parse, enrich, mask →

My progress

  • ⏱ Estimated time : ≈ 7 min

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search

Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet f...

Cribl Stream in production: distributed architecture, persistent queues and monitoring

Taking Cribl Stream to production: Leader, Worker Groups and processes, commit and deploy, scalin...

Automating Cribl: REST API, Packs, Git and Terraform

Managing Cribl as code: REST API (authentication, group paths, commit and deploy), reusable Packs...

Logstash: enrich, test, monitor and deploy on Kubernetes

Enrich with translate, geoip and elasticsearch, consume Kafka, test pipelines, monitor Logstash a...

Logstash best practices: pipelines, performance, reliability and monitoring

Logstash best practices: multiple pipelines, dissect rather than grok, conditionals, persistent q...

Installing and administering Cribl Stream

Installing Cribl Stream with Docker or on Linux, running it as a service, understanding $CRIBL_HO...

Cribl Edge: collecting close to the sources

Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...

Logstash in production: persistent queues, DLQ and multiple pipelines

Make Logstash reliable in production: persistent queue, dead letter queue, multiple pipelines and...

Feeding OpenSearch: ingest pipelines and Data Prepper

Transform data with ingest pipelines, collect with Data Prepper, receive OTLP and avoid the dynam...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!