Arize in production: self-hosted Phoenix, Arize AX, security, costs and monitoring

1. Self-hosted Phoenix #

Persistent and protected.

Configuration #

services:
  phoenix:
    image: arizephoenix/phoenix:latest
    ports: ["6006:6006", "4317:4317"]
    environment:
      PHOENIX_SQL_DATABASE_URL: postgresql://phoenix:${PG_PASSWORD}@db:5432/phoenix
      PHOENIX_ENABLE_AUTH: "true"
      PHOENIX_SECRET: ${PHOENIX_SECRET}          # au moins 32 caractères aléatoires
    depends_on: [db]
  db:
    image: postgres:16
    environment: { POSTGRES_USER: phoenix, POSTGRES_PASSWORD: "${PG_PASSWORD}", POSTGRES_DB: phoenix }
    volumes: ["pg:/var/lib/postgresql/data"]
volumes: { pg: {} }

With authentication enabled, applications send an API key (created in the interface): PHOENIX_API_KEY, passed in the Authorization: Bearer header by phoenix.otel.register.

Good to know: Check your version's documentation for the exact names of the authentication variables and the trace retention policy: the volume of prompts and answers grows quickly.

2. A collector in front of backends #

Mask, sample, route.

Why #

• Mask personal data before any storage (transform or redaction processors).
• Sample high-volume success traces, keep 100% of errors and poorly evaluated traces.
• Send the same traces to Phoenix, Arize AX and the technical tracing backend without changing applications.

3. Costs #

Tokens are the new bill.

Tracking #

Good practice: A smaller model for simple steps (classification, routing) and a large model only for the final answer often cuts the bill with no measurable drop in quality: check it with an experiment.

4. Continuous monitoring #

With Arize AX.

Online evaluations and monitors #

• Online evaluations: evaluators run automatically on a sample of production traffic.
• Monitors: alerts on an evaluation score, latency, cost or error rate, to Slack, PagerDuty or a webhook.
• Dashboards per model, prompt version and feature.
• Sending traces to AX with the arize-otel package (space ID and API key), or from the collector.