Observability on AWS: CloudWatch, X-Ray, Application Signals, ADOT, Managed Prometheus and Grafana

1. Overview #

AWS observability building blocks and their role.

An offering built around CloudWatch #

On AWS, observability revolves around Amazon CloudWatch (metrics, logs, alarms, dashboards and many specialized features), complemented by AWS X-Ray for traces, Amazon Managed Service for Prometheus (AMP) for Prometheus-format metrics, Amazon Managed Grafana for visualization and Amazon OpenSearch Service for large-scale log search.

Good to know: CloudWatch automatically collects the basic metrics of most managed services (EC2, RDS, ELB, Lambda…) without an agent. The agent or ADOT is only needed for guest operating system metrics (memory, disk), application logs and traces.

Supported pillars #

Watch out: Profiles remain the least natively covered pillar: AWS has reduced its investment in CodeGuru. For continuous profiling, many teams use Grafana Pyroscope or the profiling solution of their APM tool.

2. The components in detail #

What each service does and when to use it.

CloudWatch Metrics and alarms #

Metrics are organized by namespace (AWS/EC2, AWS/Lambda…), name and dimensions. Standard resolution is one minute; high-resolution metrics go down to one second. Alarms evaluate a metric or an expression (metric math) and trigger an SNS notification, an Auto Scaling or an EC2 action. Composite alarms combine several alarms to reduce noise, and anomaly detection computes a band of expected values.

aws cloudwatch put-metric-alarm --alarm-name api-5xx \
  --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/api/123abc \
  --statistic Sum --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 10 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:eu-west-3:111122223333:astreinte

The Embedded Metric Format (EMF) lets you write metrics into a structured JSON log: CloudWatch extracts them automatically, which is ideal for Lambda.

CloudWatch Logs #

Logs are organized into groups and streams, with a retention configurable per group. Logs Insights queries logs with its own query language (as well as SQL and PPL depending on versions), Live Tail follows them live, metric filters turn patterns into metrics and the Infrequent Access class lowers the cost of rarely queried logs. Data protection automatically masks sensitive data.

aws logs start-query --log-group-name /aws/lambda/commande \
  --start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message | filter level = "ERROR" | stats count() by bin(5m)'

Traces: X-Ray, Transaction Search and Application Signals #

AWS X-Ray receives segments and spans, builds a service map and lets you analyze traces. Transaction Search now stores all spans in CloudWatch Logs (group aws/spans) so they can be searched and aggregated. CloudWatch Application Signals provides an APM view: automatically discovered services, RED metrics (volume, errors, latency), dependencies and SLOs with error budgets, based on OpenTelemetry instrumentation (ADOT).

Good practice: Enable Application Signals on EKS, ECS, EC2 or Lambda to get a per-service dashboard within minutes and define SLOs directly in CloudWatch.

Specialized CloudWatch features #

Managed Prometheus, Managed Grafana and OpenSearch #

Amazon Managed Service for Prometheus stores Prometheus metrics with no servers to manage: you write to it by remote write (Prometheus, ADOT, Grafana Alloy) or with its managed scrapers for EKS, then query it in PromQL. Amazon Managed Grafana provides Grafana workspaces with IAM Identity Center or SAML authentication and CloudWatch, X-Ray, AMP, OpenSearch, Athena and partner data sources. OpenSearch Service is used for log and trace analysis with OpenSearch Dashboards.

3. Detailed sheet for each component #

Role, content, data types, retention and internal and external interactions of each building block in the diagram.

Summary view #

The following table summarizes each component of the architecture diagram; the sheets that follow detail them one by one.

Indicative values: Retention periods, limits and pricing models change regularly: check them in the official documentation before sizing an architecture.

ADOT Collector (AWS Distro for OpenTelemetry) #

The ADOT collector is the distribution of the OpenTelemetry collector maintained by AWS: it receives telemetry from applications, enriches it and exports it to AWS services.

Good to know: Deployment: EKS add-on, ECS sidecar container, Lambda layer or service on EC2.

CloudWatch agent and Fluent Bit (FireLens) #

The CloudWatch agent collects what the platform cannot see from outside a machine; Fluent Bit, often through FireLens on ECS, ships container logs.

CloudWatch Metrics #

The AWS time series store: every service publishes its metrics there, and applications can add their own.

Amazon Managed Service for Prometheus (AMP) #

A Prometheus back end with no servers to manage, PromQL-compatible.

AWS X-Ray, Transaction Search and Application Signals #

The AWS traces and APM building block, now integrated into CloudWatch.

CloudWatch Logs #

The central log store on AWS.

Amazon Managed Grafana #

Grafana workspaces fully managed by AWS.

Query APIs (CloudWatch, Logs, X-Ray, AMP) #

All AWS observability data is accessible through APIs: this is the main « read » exit.

Good to know: For a third-party tool, prefer Metric Streams to periodic polling: lower latency and no GetMetricData call costs.

CloudWatch alarms, SNS and EventBridge #

The chain that turns a condition on the data into an action or a notification.

Amazon Data Firehose (Metric Streams and subscriptions) #

The streaming delivery service that moves metrics and logs out of AWS to other tools.

4. OpenTelemetry compatibility #

ADOT, OTLP endpoints and collector configuration.

AWS Distro for OpenTelemetry (ADOT) #

ADOT is the OpenTelemetry distribution maintained by AWS: collector, SDKs and automatic instrumentation agents (Java, Python, .NET, Node.js) with AWS components (EC2, ECS, EKS, Lambda resource detectors, awsemf and awsxray exporters, sigv4auth authentication). It is available as an EKS add-on, a Lambda layer and a container image.

CloudWatch also exposes OTLP endpoints for traces (X-Ray) and logs: a standard OpenTelemetry collector can send its data there over OTLP/HTTP, signed with SigV4.

extensions:
  sigv4auth:
    region: eu-west-3
    service: aps
receivers:
  otlp:
    protocols: { grpc: {}, http: {} }
processors:
  batch: {}
  resourcedetection: { detectors: [env, eks, ec2] }
exporters:
  awsemf:                      # métriques vers CloudWatch (Embedded Metric Format)
    namespace: Boutique
  awsxray: {}                  # traces vers X-Ray
  prometheusremotewrite:       # métriques vers Managed Prometheus
    endpoint: https://aps-workspaces.eu-west-3.amazonaws.com/workspaces/ws-1234/api/v1/remote_write
    auth: { authenticator: sigv4auth }
service:
  extensions: [sigv4auth]
  pipelines:
    traces:  { receivers: [otlp], processors: [resourcedetection, batch], exporters: [awsxray] }
    metrics: { receivers: [otlp], processors: [resourcedetection, batch], exporters: [awsemf, prometheusremotewrite] }

Good to know: OpenTelemetry SDKs can be used as is: for X-Ray, add the X-Ray ID generator and propagator only if you must interoperate with older services instrumented with the X-Ray SDK; otherwise W3C context is enough.

5. Interconnections with external products #

Sending AWS data to other tools and vice versa.

Exporting from AWS #

Good practice: Prefer Metric Streams to API polling to send metrics to a third-party tool: latency drops from several minutes to a few seconds and the cost of GetMetricData calls disappears.

Centralizing several accounts #

CloudWatch cross-account observability (Observability Access Manager) links source accounts to a monitoring account: metrics, logs, traces and Application Signals can be viewed there without copying data. Combined with AWS Organizations, it provides a single view of the whole organization.

6. Costs and best practices #

Controlling the bill and making observability useful.

Best practices #

• Set a retention per log group (the default is « never expire »).
• Limit the cardinality of custom dimensions: each combination is a billed metric.
• Instrument with OpenTelemetry (ADOT) to stay portable to other back ends.
• Use composite alarms and anomaly detection to reduce noise.
• Sample traces (X-Ray sampling rules or tail sampling in the collector).
• Enable cross-account observability and tag resources to allocate costs.

Common mistake: Logs are often the largest cost item: ingesting all debug logs in production can cost more than the servers themselves.