Final Lab: Vector unit tests and masking personal data

What you will learn in this TP

• Mask with VRL
• Write unit tests
• Automate in CI

Introduction #

What you will learn in this section

• Mask with VRL
• Write unit tests
• Automate in CI

Context. A GDPR audit found customer e-mail addresses in the logs. You add masking and tests that prevent any regression.

Prerequisites: previous lab completed  ·  Estimated time: 40 min

A log pipeline tested like code.

Setup #

Guided steps, with their solutions.

1. 1. Write the masking

   Why? Masking in the collector protects every destination at once.

   Add a masquage transformation after parse that masks e-mail addresses and tokens in message and removes authorization.

2. 2. Write the tests

   Why? One test per rule: if someone changes the regex, the CI flags it.

   Add a tests section that injects a message containing an e-mail address and a token and checks that they have disappeared.

3. 3. Run the tests

   Why? Tests run without starting a pipeline.

   Run vector test with the Docker image. Deliberately break the regex and run it again.

4. 4. Automate

   Why? CI guarantees that every configuration change is validated and tested.

   Write a CI job that runs vector validate --no-environment then vector test.

Difficulty level : ●●●○○ (3/5)