What is observability

Introduction #

System observability #

To observe a system, you need to collect and analyze the data it emits. Modern systems emit this data automatically (or once enabled in their configuration). Other systems must be made observable. Depending on the system, you add SDKs or plugins, query their APIs... Whatever the method, this data falls into three categories, known as the pillars of observability:

Logs #

Logs are the mechanism a system uses to report on its operating state. In most cases, they are stored in files called log files.

A log file is therefore made of several lines, each representing a change of state, called events:

• application start/stop
• user login/logout
• image added/deleted
• ...

WARNING: an event can be written to a file on a single line or on several lines. Most applications write single-line events, but some, such as errors on Java-based systems, generate events spanning several lines called stack traces.

The information in a log file lets you analyze and understand how a system works. With observability, you use solutions that analyze these files automatically. When configuring these solutions, it is therefore important to declare whether the log file is single-line and/or multiline to get a consistent analysis. Systems help you by generating events formatted as JSON, XML, CSV... or, for unformatted files, by adding delimiters that help identify the beginning, the end and possibly the intermediate parts.

In the end, a good log file must:

• timestamp every event.
• have a log format: the schema that describes the structure of an event.
• have a log format

Metrics #

Metrics are quantifiable measurements that give visibility into how systems are used. Metrics vary from one system to another. Thresholds are usually associated with each metric to trigger corrective actions before reaching the breaking point.

For example, for a system such as a:

• server: you monitor CPU, RAM, disk usage...
• web server: you monitor the number of hits, requests per second, worker status

Test and acceptance servers are often undersized compared to their PRODUCTION counterparts. This makes thresholds hard to define because the value may change from one environment to another. To solve this, whenever possible you work with percentages, which makes the threshold uniform across environments. For example, the CPU usage threshold is set to 80%.

Traces #

Profiles #

Continuous profiling is often considered the 4th pillar of observability. Unlike traces, which follow a request end to end, profiles show how code executes on the processor (CPU) and in memory (RAM), line by line. They answer the question: 'Which function or line of code consumes the most resources in production?'

Tools such as Grafana Pyroscope collect these profiles with a very low performance overhead, making it easier to optimize code, reduce infrastructure costs and improve response times.

Data observability #

Data freshness #

Data freshness measures the time elapsed between the moment data is collected and the moment it is available for consumption. It is crucial to ensure decisions are based on current information. Stale data can lead to errors or delays in decision-making. To monitor freshness, you can measure:

• Data age: time elapsed since the last update.
• Processing latency: delay between new data appearing in the source and its availability in the target system.

Data distribution #

Data distribution shows how values are spread across a dataset. It includes descriptive statistics such as the mean, median, standard deviation, quartiles and outlier detection. It helps detect anomalies or unexpected patterns. Here are some indicators to monitor:

• Mean: average value of the data in a column.
• Standard deviation: measure of how data is spread around the mean.
• Outliers: percentage of values outside an expected range.

Data volume #

Data volume is the total amount of data present in a system or pipeline at a given time. It lets you track the growth or decline of data volumes to identify abnormal behavior such as data loss or duplication. Volume indicators include:

• Number of records: total records in a table or dataset.
• Size in bytes: total size of the data in storage (MB, GB, etc.).

Data schema #

The data schema represents the structure of datasets, including column types, constraints and relationships between tables. Monitoring schema changes ensures that modifications meet expectations and that relationships between tables are respected. Indicators to monitor are:

• Schema validation: check that the current schema matches expectations in terms of column types, names and relationships.
• Schema evolution: track changes in table schemas (columns added/removed).

Data lineage #

Data lineage traces the origin of data, its path through the transformations applied in pipelines, and its final destination. It ensures transparency in data processing and helps diagnose potential errors at the source. Lineage elements include:

• Data source: identify where the data comes from.
• Transformation: record all transformations applied to the data.
• Final destination: check that the data reaches its intended destination.