1. A RAG is diagnosed step by step #
Most bad answers come from retrieval.
The steps #
| Bad answer caused by... | What the trace shows | Fix |
|---|---|---|
| Irrelevant documents | RETRIEVER span: the passages are off topic | Chunking, embeddings, filters, hybrid search |
| Right document ranked poorly | Low score, document in 5th position | Reranker, number of documents |
| Right context, wrong answer | LLM span: the answer contradicts the context | Prompt, model, temperature |
| Truncated context | Input tokens at the maximum | Fewer documents, shorter passages |
2. Instrumenting an in-house RAG #
When no framework does it for you.
Code #
from opentelemetry import trace
from openinference.semconv.trace import SpanAttributes, OpenInferenceSpanKindValues as Kind
tracer = trace.get_tracer("faq-rag")
def answer(question: str) -> str:
with tracer.start_as_current_span("faq-rag") as root:
root.set_attribute(SpanAttributes.OPENINFERENCE_SPAN_KIND, Kind.CHAIN.value)
root.set_attribute(SpanAttributes.INPUT_VALUE, question)
docs = retrieve(question) # span RETRIEVER (voir l'article OpenInference)
context = "\n".join(d.text for d in docs)
r = client.chat.completions.create( # span LLM créé automatiquement
model=model,
messages=[{"role": "system", "content": "Réponds uniquement à partir du contexte :\n" + context},
{"role": "user", "content": question}])
text = r.choices[0].message.content
root.set_attribute(SpanAttributes.OUTPUT_VALUE, text)
return textGood to know: The root span of kind CHAIN carries the question and the final answer: it is the one that will be evaluated, and the one exported to a dataset.3. Agents #
Deeper traces.
What to look for #
- Each agent turn: the LLM call that decides, the tool chosen and its arguments (
tool_call.function.arguments), the tool's result. - Loops: an agent that calls the same tool ten times is expensive and slow to answer.
- Tool errors: a failing TOOL span that the agent ignored.
Agent frameworks (OpenAI Agents SDK, LangGraph via LangChain, CrewAI, smolagents) have their OpenInference instrumentation: auto_instrument=True is enough if the package is installed.
Good practice: Measure the number of TOOL spans per trace: a sudden increase after a prompt change often reveals an agent going in circles.
4. Sessions and conversations #
Linking exchanges.
Sessions #
from openinference.instrumentation import using_session
with using_session(session_id=conversation_id):
reply = answer(question)Phoenix then groups the traces of the same conversation: you can reread a full exchange, turn by turn, and see when it went off the rails.
