Observability
Arize AI: observability and evaluation of LLM applications and agents
Understand Arize AI: Phoenix (open source) and Arize AX (platform), OpenInference traces, LLM-as-a-judge evaluations, datasets, experiments, prompts and m…
Guides & insights
Observability
Understand Arize AI: Phoenix (open source) and Arize AX (platform), OpenInference traces, LLM-as-a-judge evaluations, datasets, experiments, prompts and m…
Observability
Set up evaluations: code evaluators, LLM judge (create_classifier, evaluate_dataframe), ready-made evaluators, human annotations, judge calibration and re…
Observability
Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function catalogue (Parser, Regex Extract, Eval, Lookup, GeoIP, Mask,…
Observability
Monitor generative AI applications: LLMs, agents, prompts, tokens and evaluations.
Observability
Understand and master Prometheus recording rules: evaluation, naming convention, pitfalls (ratios, empty series), testing with promtool, monitoring and ba…
Cloud computing
Secure access to AWS with IAM: users, groups, roles and JSON policies, permission evaluation, MFA, IAM Identity Center for humans, roles for applications,…
Observability
Detailed guide to OpenInference: OpenTelemetry semantic conventions and instrumentations for LLMs, RAG and agents. Span kinds, llm.*, retrieval.*, tool.* …
Observability
Improve an LLM application methodically: build datasets from traces, run experiments, compare prompt or model versions, and version prompts with the playg…
Observability
Going to production: Phoenix with PostgreSQL and authentication, an OpenTelemetry Collector in front of backends, masking and retention, token cost tracki…
Observability
Checklist for observability and evaluation of LLM applications: OpenInference instrumentation, context, privacy, calibrated evaluations, datasets, experim…
Observability
Evaluate rules and alerts with vmalert, back up with vmbackup, deploy on Kubernetes with the operator and the Helm stack, and monitor VictoriaMetrics.
Observability
Evaluate recording and alerting rules in Mimir, configure the multi-tenant Alertmanager and automate everything with mimirtool (rules, alertmanager, analy…
Learning never stops