Observability on Google Cloud: Cloud Monitoring, Logging, Trace, Profiler and Managed Prometheus

1. Overview #

The Google Cloud Observability suite.

The services in the suite #

Google Cloud Observability (formerly Stackdriver, then operations suite) brings together Cloud Monitoring, Cloud Logging, Cloud Trace, Cloud Profiler and Error Reporting, as well as Google Cloud Managed Service for Prometheus, built on Monarch, Google's internal time series database. Metrics for most managed services are collected automatically; platform logs and audit logs arrive in Cloud Logging with no configuration.

Supported pillars #

Good to know: Google Cloud is one of the few clouds to natively offer all four pillars, profiles included, with Cloud Profiler and its low production overhead.

2. The components in detail #

Role of each service.

Cloud Monitoring #

Cloud Monitoring stores system and custom metrics, and offers the Metrics Explorer, dashboards, uptime checks and synthetic monitors, alerting policies (legacy MQL, PromQL now recommended) and SLO monitoring with error budgets and burn rate alerts. Scoping projects (metrics scopes) group metrics from several projects.

gcloud monitoring uptime create api-boutique \
  --resource-type=uptime-url --resource-labels=host=api.exemple.fr,project_id=boutique-prod \
  --path=/healthz --period=1

Cloud Logging #

All logs go through the log router: sinks send them to log buckets (configurable retention), BigQuery, Cloud Storage or Pub/Sub; exclusion filters avoid ingesting what is useless. Log Analytics lets you query upgraded buckets in SQL, with linked BigQuery datasets. Log-based metrics turn patterns into alerting metrics.

gcloud logging sinks create vers-bigquery \
  bigquery.googleapis.com/projects/boutique-prod/datasets/journaux \
  --log-filter='resource.type="cloud_run_revision" AND severity>=WARNING' --use-partitioned-tables

Cloud Trace, Cloud Profiler and Error Reporting #

Cloud Trace stores and analyzes spans (latency, call map, reports). Cloud Profiler continuously collects CPU and memory profiles with very low overhead and compares versions. Error Reporting automatically groups exceptions detected in logs and notifies about new errors.

Ops Agent and collection on Kubernetes #

On Compute Engine, the Ops Agent (based on Fluent Bit for logs and the OpenTelemetry Collector for metrics) collects system metrics and logs, with third-party integrations (NGINX, PostgreSQL…). On GKE, collection of logs and system metrics is built in, and Managed Prometheus collects application metrics with PodMonitoring resources.

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata: { name: api, namespace: boutique }
spec:
  selector: { matchLabels: { app: api } }
  endpoints:
  - port: metrics
    interval: 30s

3. Detailed sheet for each component #

Role, content, data types, retention and internal and external interactions of each building block in the diagram.

Summary view #

The following table summarizes each component of the architecture diagram; the sheets that follow detail them one by one.

Indicative values: Retention periods, limits and pricing models change regularly: check them in the official documentation before sizing an architecture.

Google-Built OpenTelemetry Collector #

The distribution of the OpenTelemetry Collector published and supported by Google.

Ops Agent #

The Compute Engine agent that combines Fluent Bit (logs) and the OpenTelemetry Collector (metrics).

Cloud Monitoring #

Google Cloud's metrics, dashboards, uptime checks and SLO service, backed by Monarch.

Google Cloud Managed Service for Prometheus #

A managed, global Prometheus, stored in Monarch.

Cloud Logging #

Google Cloud's logging service, organized around the log router.

Cloud Trace #

Google Cloud's distributed tracing service.

Cloud Profiler #

Google Cloud's continuous production profiling.

Error Reporting #

Automatic grouping of application exceptions.

Alerts and notification channels #

Cloud Monitoring's alerting engine.

Query APIs (Monitoring, Prometheus, Logging, Trace) #

The APIs that expose all Google Cloud observability data for reading.

Log sinks (Pub/Sub, BigQuery, Cloud Storage) #

The log router outputs, at project, folder or organization level.

4. OpenTelemetry compatibility #

Google collector, Telemetry API and exporters.

Sending OpenTelemetry telemetry to Google Cloud #

Google publishes a Google-Built OpenTelemetry Collector and the googlecloud (traces, metrics, logs) and googlemanagedprometheus exporters from the contrib distribution. The Telemetry API (telemetry.googleapis.com) accepts OTLP directly for traces: a standard collector can use it with Google authentication.

receivers:
  otlp: { protocols: { grpc: {}, http: {} } }
processors:
  batch: {}
  resourcedetection: { detectors: [gcp] }
exporters:
  googlecloud: {}                 # traces et journaux vers Cloud Trace et Cloud Logging
  googlemanagedprometheus: {}     # métriques vers Managed Service for Prometheus
service:
  pipelines:
    traces:  { receivers: [otlp], processors: [resourcedetection, batch], exporters: [googlecloud] }
    metrics: { receivers: [otlp], processors: [resourcedetection, batch], exporters: [googlemanagedprometheus] }
    logs:    { receivers: [otlp], processors: [resourcedetection, batch], exporters: [googlecloud] }

Good practice: Prefer googlemanagedprometheus over googlecloud for metrics: they can be queried in PromQL and are billed by number of samples.

5. Interconnections with external products #

Sharing data.

Exporting and integrating #

Good to know: Managed Prometheus exposes a Prometheus-compatible API: Grafana or any PromQL-compatible tool can query it directly.

6. Costs and best practices #

Controlling ingestion.

Best practices #

• Use exclusion filters and suitable retention per log bucket.
• Centralize an organization's logs with aggregated sinks at folder or organization level.
• Define SLOs in Cloud Monitoring and alert on error budget burn rate.
• Enable Cloud Profiler in production: its overhead is low and it reveals performance regressions.
• Instrument with OpenTelemetry and the Google collector to keep portability.
• Watch the cardinality of custom and Prometheus metrics.

Watch out: Data Access audit logs are disabled by default and can be very large: enable them only for the services that need them.