Datasets, experiments and prompts with Phoenix

1. Datasets #

Examples that matter.

Building a dataset #

• From traces: select traces in Phoenix (poorly rated, flagged by users, typical cases) and add them to a dataset.
• From a file: questions and expected answers written by business experts.
• Mix common cases, edge cases and cases already failed: a non-regression dataset must contain past mistakes.

import pandas as pd
from phoenix.client import Client

px = Client(base_url="http://localhost:6006")
df = pd.DataFrame({
    "question": ["Comment réinitialiser mon mot de passe ?", "Quels sont vos horaires ?"],
    "answer": ["Mot de passe oublié sur la page de connexion", "9 h à 18 h"],
})
dataset = px.datasets.create_dataset(name="support-faq", dataframe=df,
                                     input_keys=["question"], output_keys=["answer"])

2. Experiments #

Measure before deploying.

Running an experiment #

SYSTEM_V2 = "Tu es l'assistant du support. Réponds en une phrase, avec le lien utile si possible."

def task(input):
    r = client.chat.completions.create(model=model, messages=[
        {"role": "system", "content": SYSTEM_V2},
        {"role": "user", "content": input["question"]}])
    return r.choices[0].message.content

def contains_expected(output, expected):
    return expected["answer"].lower() in output.lower()

def short_enough(output):
    return len(output) < 300

px.experiments.run_experiment(dataset=dataset, task=task,
                              evaluators=[contains_expected, short_enough],
                              experiment_name="prompt-v2")

Each experiment keeps its outputs, scores and traces; Phoenix shows the experiments of the same dataset side by side to see which examples improved and which regressed.

Good to know: Depending on the version, run_experiment is imported from phoenix.experiments or called through the client (px.experiments). Evaluators receive the arguments they declare (input, output, expected, metadata).

3. Prompts #

Versioning what changes the most.

Playground and versions #

• The playground replays an LLM span with another prompt, another model or other parameters, without touching the code.
• Saved prompts are versioned and tagged; the application can load a version by name or tag.
• The template used appears in traces (llm.prompt_template.*): each answer is linked to its prompt version.

Good practice: Simple rule: no prompt or model change in production without an experiment on the reference dataset showing a score at least as good.