RousselTMLEARNING
EN
FrançaisEnglishEspañol
Sign in

Observabilité · Courses

Arize AI: LLM observability and evaluation from discovery to expertise

Complete Arize AI course: Phoenix and Arize AX, OpenInference traces, RAG and agents, LLM-as-a-judge evaluations, datasets, experiments, prompts, costs and production.

8 chapters14 lessonsEN
At your pace, through practice.

By the end of this course you will be able to

  • Explain LLM observability and the role of Phoenix and Arize AX
  • Instrument an application with OpenInference
  • Trace and diagnose a RAG and an agent
  • Automatically evaluate answer quality and calibrate an LLM judge
  • Compare prompt or model versions with experiments
  • Run LLM observability in production: security, costs, monitoring

🎯 Level : Intermediate👥 Who it is for : AI application developers, MLOps and LLMOps engineers, SREs and product teams who run LLMs, RAG or agents in production

LEARNING PATH

Course curriculum

0Course overview1 lesson
  1. Assessment quizQuiz · ⏱ 10 min
1Introduction1 lesson

LLM observability, Phoenix and Arize AX.

  1. Arize AI: LLM observability and evaluationArticle · ⏱ 3 min
2Tracing and OpenInference3 lessons

Conventions, installation and first traces.

  1. OpenInference: tracing AI applicationsArticle · ⏱ 12 min
  2. Installing Phoenix and tracing an applicationArticle · ⏱ 6 min
  3. Lab: Install Phoenix and trace a first LLM applicationHands-on lab · ⏱ 35 min · Level 2/5
3RAG and agents2 lessons

Tracing and diagnosing complex applications.

  1. Tracing a RAG and an agentArticle · ⏱ 6 min
  2. Lab: Trace a RAG and sessionsHands-on lab · ⏱ 45 min · Level 3/5
4Evaluations2 lessons

Code, LLM judge and annotations.

  1. Evaluating with LLM judgesArticle · ⏱ 6 min
  2. Lab: Evaluate answers with code and an LLM judgeHands-on lab · ⏱ 45 min · Level 3/5
5Datasets, experiments and prompts2 lessons

Improve without regressing.

  1. Datasets, experiments and promptsArticle · ⏱ 6 min
  2. Lab: Compare two prompts with an experimentHands-on lab · ⏱ 45 min · Level 4/5
6Production2 lessons

Hosting, security, costs and monitoring.

  1. Arize in productionArticle · ⏱ 6 min
  2. Final Lab: Production LLM observability with PhoenixHands-on lab · ⏱ 1 h · Level 4/5
7Best practices1 lesson

Recommended practices to run the product in production.

  1. Arize and OpenInference best practicesArticle · ⏱ 4 min

LEARN BY DOING

Use cases & applications

Debug a RAG

See the retrieved documents and understand a bad answer.

Compare two prompts

Measure on a dataset before deploying.

Detect hallucinations

Automatically evaluate faithfulness to the context.

Control costs

Track tokens per conversation and feature.

Recommended prerequisites

OpenTelemetry: From Discovery to Expertise

What next?

Grafana Tempo: from discovery to expertiseBindplane: OpenTelemetry telemetry pipelines from discovery to expertiseObservability with eBPF: from discovery to expertise

Frequently asked questions

What is Arize AI?

A vendor of observability and evaluation tools for generative AI: Phoenix (open source) and Arize AX (platform).

What is OpenInference?

OpenTelemetry conventions and instrumentations to trace LLMs, RAG and agents, used by Phoenix and Arize AX.

Do I need an OpenAI key for the labs?

No: a local Ollama model, through its OpenAI-compatible API, is enough. An OpenAI key gives better judges for evaluations.

Phoenix or Arize AX?

Phoenix to develop, evaluate and experiment, self-hosted; Arize AX to monitor production at scale.