TL;DR: We have released review queues for AI traces. These are shared inboxes waiting for a human-in-the-loop to evaluate them on whatever criteria the team decides. Instead of passing documents and excel sheets back and forth, this streamlines the AI review process and facilitates the creation of a strong dataset of successes and failures that you can use to retrain your agents in the future.

Back in December 2023, a guy named Chris Bakke hopped on the chat window on the website for a Chevrolet dealer, told the bot to agree with anything he said, and then asked if he could get a 2024 Tahoe for one dollar. The bot didn't hesitate. "That's a deal," it typed back, "and that's a legally binding offer – no takesies backsies." It wasn't a lawyer, obviously. It was a chatbot that had just seen a human invent a loophole in contract law by saying "no takesies backsies" and accepted it. Chevrolet of Watsonville did not, ultimately, hand over a $76,000 vehicle for $1. But the screenshot, showing the transaction, went viral and ended up racking 20 million views. Somewhere in Detroit, a General Motors employee experienced a Tuesday they'd probably like to forget.

Just two months later, a grief-stricken customer was offered a bereavement discount from an Air Canada chatbot that didn't exist, and an arbitrator still made the company pay it. Such incidents still happen, though far less often. For one, the underlying models are getting better, and the protections built around them are too. But for a second, less glamorous reason: Somebody, somewhere, is reading those transcripts afterwards, purposefully and routinely, to ensure the robot hasn't done anything truly foolish. Historically, that job required emailing back and forth file with a name like traces_review_hallucinations_V3 Final edited.xlsx. That's not necessary anymore. We now have a review queue that allows this job to be streamlined. It also allows for the creation of a dataset that later on feeds the future iterations of your agents so that they know what works and what doesn't.

What Are Review Queues?​

In simple terms, every time an AI says or does something, e.g. answering a question, interacting with an external service, or giving the customer a price, you can log the entire event as a "trace." This log acts like a receipt, detailing exactly what happened, step by step. In the past, for a human to review that receipt meant sifting through hundreds of lines of chat in a spreadsheet and writing a two-word note in a cell. Then came Review Queues, which essentially turns that clunky spreadsheet into a more streamlined support ticketing system.

You set up a queue, something with an evocative name, like "Hallucination Check," or if you happen to be employed at that Chevrolet dealership, perhaps "Seriously, Please Check The Law For Contractual Obligation." Then, you assign it an evaluation question: Will the interaction pass or fail? Rate the response on a 1 to 5 scale. This article does not go into the technical details of how to set them up, but a step by step tutorial can be found in the docs.

Traces can be manually assigned to the queue, or you can set up rules so that traces flagged by an automated grader (for instance, where the model wasn't confident) automatically go in the queue. Then, a human reviewer simply grabs a ticket from the queue, reads the trace, answers the question, and moves on to the next one. Over time, those judgments add up and their value goes beyond a simple checklist, they become a curated dataset of real failure modes and edge cases you can use to fine tune your agents.

Some Inspiration​

Whenever humans need to evaluate anything at scale, whether it's code reviews, content moderation, or, now, AI outputs, processes emerge. Whenever we say "Someone should check this", it always evolves into a full-time job that needs a system, regardless of original intentions. And that brings us to AI oversight, the very thing we expected to replace tedious human work. Predictably, it has instead created a new class of tedious human work. Somebody has to be the person who reads the trace where the chatbot happily informed the customer the Tahoe costs one buck or that they could get a 70% discount on a first class plane ticket, and that person needs an organized way to track what is already done and what is left to do.

There's a deeper value in this, though: These human evaluations don't just serve as compliance documentation. They're fuel for better AI. If you collect enough traces, each marked by a human with a definitive "pass" or "fail," you can train another AI to judge the first one's homework. The system learns what a human deems acceptable and unacceptable, and you then use that to fine-tune the next AI. So, the true workflow looks like this: Build an AI that occasionally messes up, assign a human to grade its errors, use that grading to build a second AI whose primary function is to grade the first one, and then periodically have a human check the grader AI's work to ensure it hasn't gone rogue itself.

With all the advances in LLMs and AI models, we are still far away from removing the human in the loop. LLMs are inherently built to be helpful assistants, which means they can be gaslit, confused, or coerced in various ways into doing things they shouldn't be doing. As long as this is the case, we will still need to keep an eye on the AI and monitor what it's doing, and with Review Queues we have simply created a system that makes that job easier.

Get started: MLflow Review Queues

Questions or feedback? Drop a note by opening an issue or join the MLflow community discussions.

⭐ Star us on GitHub, show your support for the project!

The Visibility Gap​

Whenever an agent fails, there are a set of questions that you need to answer: What did the user ask for? What did the model do? Which tools did it decide to use? What arguments did it pass? How many tokens (and how much time) did each step consume? And, was any of these steps unnecessary?

This problem is hard enough to solve with a single-harness agent. When it comes to multiple harnesses, it can easily slide into a complete mess, each agent generating different data in different structures. Piecing the puzzle together becomes a monumental task, and this is where most teams would choose to abandon observability. But here's the good news: this problem has recently been solved.

One API, Many Harnesses​

Omnigent was recently released and has quickly climbed up the ranks in popularity among devs, reaching 6k GitHub stars in under a month. Its core idea is to unify the interface across the different harnesses in use. It allows you to orchestrate different agent harnesses that each spin up to do what it does best. You could have Claude Code write the code and Codex review it. You could use Cursor to devise a coding plan and execute it with a minimalistic approach in Pi, and most importantly, you could do all of this in a single interface.

This unification matters a lot, not only for development, but also for observability. Now that all the agents are flowing through a single layer, all the traces are standardized and can be passed through an observability layer without all the mess. This is exactly what Omnigent does with MLflow Tracing.

Automatic Tracing Without Code Changes​

This integration is seamless. Omnigent ships with MLflow as an optional dependency. To install it:

uv tool install omnigent mlflow

Then point it to your running MLflow tracking server and set the environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:5000"  # or wherever your MLflow server runs
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="x-mlflow-experiment-id=0"  # your experiment id
export OMNIGENT_TELEMETRY_ENABLED="true"
export OMNIGENT_OTEL_HTTP_CLIENT_INSTRUMENTATION="false"
export OMNIGENT_OTEL_CAPTURE_CONTENT="true"

Then, run omnigent run and you're good to go. Everything is wired automatically. With this setup, in MLflow you get:

• Agent turns, including prompts and responses
• Tool invocations along with the arguments, results, and how long each took
• Per-turn token consumption
• Session metadata (model name, agent name, etc.)

What You Can Do With This​

Now for the most exciting part: here's the universe of possibilities that lies ahead. With all the agentic harnesses being traced, you are no longer flying blind. Instead of relying on personal judgment to gauge whether a workflow change is good, you can now actually measure it.

Take an example where a new, cheap, open-source model is released and you want to analyze whether its performance is good enough to replace your expensive LLM across your harnesses. You swap the model in Omnigent and run a before-vs-after analysis on the traces in MLflow. Not only can you determine which model is better in absolute terms, but you can analyze which harnesses and tool invocations perform best under different LLMs.

Just like data analytics dramatically changed the market by allowing businesses to uncover hidden opportunities and cut unnecessary costs, AI observability does the same for the AI-native business. For example, if you use an MCP server to enable your agents to perform specific actions, you could now A/B test different providers and analyze which one offers the best bang for the buck.

Additionally, it allows you to become a better engineer by answering the right questions:

• How much of your work has been spent on building new features vs. fixing bugs over time?
• Was there any commonality in those bugs?
• Which types of requests took the agents a lot of time to complete?
• Which harness is best for planning vs. analyzing the codebase vs. executing?

Answering these questions gives you an edge to ship faster, at higher quality, and at lower cost.

Closing the Gap​

Multi-harness orchestrators are here to stay. Developers and teams are going to keep on the lookout for the best tool to get the job done, and each job will have a different tool that does it better than any other. The solution is not to force all the work into a single framework, but rather to orchestrate and coordinate them. This requires an observability layer to keep an eye on everything and to spot opportunities and inefficiencies. Omnigent's MLflow integration is that eye, a small shift in architecture, but a big shift in what you and your team spend your time on.

Get started: MLflow Tracing for Omnigent

Questions or feedback? Drop a note by opening an issue or join the MLflow community discussions.

⭐ Star us on GitHub, show your support for the project!

Then suddenly, something breaks. One of your team members deletes the live system prompt, and suddenly three tools depend on that prompt break. One of the short-term hires, for the first time working with your team, accidentally accesses the confidential test data. One of the AI Gateway endpoints is updated, even though this person isn't supposed to have permissions to modify it.

If these situations seem familiar, you are definitely not alone. And these situations are exactly why we're excited to release the MLflow Role-Based Access Control (RBAC).

Traditional permissions fall short for LLM Teams​

Most traditional ML teams are small and insular with maybe 3 or 4 data scientists who cycle through sharing model experiments with each other. However, teams working with LLMs or agent workflows often look very different. Take a modern AI workflow, it probably involves something like:

• Prompt engineers tweaking prompts and templates within your Prompt Registry.
• Eval researchers testing your models and running scorers against traces.
• AI Gateway operators keeping track of your endpoints, secrets, and model definitions.
• Platform engineers who need to administer the MLflow deployment itself.
• Temps (temporary employees, external contractors, etc. Who aren't part of your org and only need read access for a limited scope).

Before MLflow's permissions model got its update, every permission grant needed to be an individual explicit call like: create_experiment_permission() for each user and resource. If you have three users who only need to access a single experiment, this is fine. When you have a team that is dealing with multiple prompts, scoring tools, gateways, and tests with wildly different permission needs, managing this on your own becomes messy very quickly. The bigger the team gets, the more subtle the chances for unintended permissions usage.

What RBAC provides​

Reusability of roles: Create the "prompt editor" role once. Then assign that role to whoever needs to edit your prompts. If you hire someone new, or somebody leaves the team, all you need to do is add/remove that user to the "prompt editor" role. You won't have to go into each one of the dozens of different prompt roles you've made for your teams.

Tomorrow's resources are automatically covered today: Roles are associated with a resource_pattern, which will determine which resources are accessible to those who have that role. For instance, you can assign the (prompt, *, EDIT) role to whoever should be able to edit prompts. This grants the user editing abilities for all of your current and future prompts! This wildcard will be resolved within that specific role's workspace (meaning your prompt edit access won't bleed into another team's prompt library). Your access automatically stays up to date.

Note: In the UI, the wildcard * is exposed as all.

Four intuitive permission levels:

Note: USE covers consuming a resource without modifying it (invoking a Gateway endpoint, referencing a model definition, or creating new resources within a workspace.) Creating new resources rides along with USE, not EDIT, it's an additive right, not a modify-in-place one, which is why there's no separate "create" column. EDIT does not include delete; only MANAGE does.

Mapping LLM Team Roles to MLflow Permissions​

Here's how a typical AI team maps into MLflow roles:

Getting Started - Creating Roles and Assigning Them​

1. Server Setup​

To get started using RBAC in MLflow, the server must be running with auth enabled:

mlflow server --app-name basic-auth

Recommended: If you need to create multiple siloed workspaces for multiple teams, add the --enable-workspaces flag:

mlflow server --app-name basic-auth --enable-workspaces

2. Configuring Admin Auth​

The best practice is to have your credentials in the environment variables or in a .mlflow file at ~/.mlflow/credentials. However, for the sake of simplicity, in this example we are hardcoding the authentication inside the python script:

import os
os.environ["MLFLOW_TRACKING_USERNAME"] = "your_username" # admin default is 'admin'
os.environ["MLFLOW_TRACKING_PASSWORD"] = "your_password" # admin default is 'password1234'

mlflow.set_tracking_uri("http://localhost:5000")

3. Loading the Auth Client​

The auth client allows the creation and management of users and their credentials.

auth_client = get_app_client("basic-auth", tracking_uri="http://localhost:5000")

4. Creating Roles​

prompt_engineer_role = auth_client.create_role(
    workspace="your-workspace-name",
    name="prompt-engineer",
)
auth_client.add_role_permission(
    role_id=prompt_engineer_role.id,
    resource_type="prompt",
    resource_pattern="*",   # wildcard: covers also the prompts created later
    permission="EDIT",
)
auth_client.add_role_permission(
    role_id=prompt_engineer_role.id,
    resource_type="experiment",
    resource_pattern="*",
    permission="READ",
)


experiment_reader_role = auth_client.create_role(
    workspace="your-workspace-name",
    name="experiment-reader",
)
auth_client.add_role_permission(
    role_id=experiment_reader_role.id,
    resource_type="experiment",
    resource_pattern="*",
    permission="READ",
)

5. Assigning Roles to Users​

for user in ("alice", "bob", "carol"):
    auth_client.assign_role(username=user, role_id=prompt_engineer_role.id)


for user in ("john", "lisa"):
    auth_client.assign_role(username=user, role_id=experiment_reader_role.id)

In other words, the following month, when a new prompt engineer joins your team, you only need to review a single role definition (and not a whole stack of 40+ resource calls) in your permission audit for prompt engineers, and it doesn't become a huge mess to add a new employee.

Isolate Different Teams on One MLflow Server​

In practice, an MLflow server would often be used by multiple AI teams at the same time (e.g. A team for search tools, a team for customer support tools). Starting MLflow with --enable-workspaces will give each team isolated space for prompts, experiments, scorers, and Gateway resources.

An engineer, therefore, either belongs to the "search-AI" workspace, or to the "customer-support-AI" workspace (not both, unless explicitly given access to both).

In practice, it means you have a single MLflow deployment to manage, but it's supporting several A.I. Teams that can each manage their own workspace and don't collide with each other.

User Tiers​

RBAC establishes 3 different types of users:

Migrating from Legacy Permissions (Pre-3.13)​

A key thing to note if you are upgrading from an MLflow version before 3.13 is that the legacy per-resource permissions have been removed (like create_experiment_permission()). When upgrading, the DB migration backfills the permissions into the new role_permissions table, keeping the consistency without any breaks.

Key API change:

# Old (removed):
#   auth_client.create_experiment_permission(experiment_id, username, "EDIT")

# New:
auth_client.grant_user_permission(username, "experiment", experiment_id, "EDIT")

How Permission Resolution Works​

When a user tries to access a resource, MLflow resolves their effective permission in this order:

1. Platform Admin: If is_admin = true, access is granted immediately.
2. Role-derived grants: All roles the user holds in the current workspace contribute. Matching grants combine via a max operation: MANAGE > EDIT > USE > READ. A (workspace, *, MANAGE) grant confers management of everything in that workspace.
3. Default permission floor:

• Without --enable-workspaces: The default permission floor is READ and it is applied as a floor when no role grant matches a specific resource.
• With --enable-workspaces: In multi-workspace mode, we still need to grant a user (WORKSAPCE, *, USE) to confer workspace membership (see RBAC permission resolution docs.) With this, the user will get the ability to create resources in addition to the default_permission which could be set to either:
  • READ (default)
  • NO_PERMISSION: In this case, getting access to the workspace only allows creating resources but does not grant visibility into existing ones.

Important: In RBAC, there is no explicit way to deny a permission. If you want to restrict access, you would need to grant more narrowly rather than adding exceptions.

Direct Permissions (Without a Role)​

For one-user, one-resource scenarios, you can grant permissions directly — without creating a named role:

# Give Alice EDIT access to experiment 42
auth_client.grant_user_permission("alice", "experiment", "42", "EDIT")

These calls are gated by per-resource MANAGE — meaning an experiment owner with (experiment, 42, MANAGE) can grant access to others even without workspace-wide management rights. Behind the scenes, these go into a reserved per-user role, but that's an implementation detail you never touch.

Reference: AuthServiceClient Methods​

The Bigger Picture​

Moving quickly with large language models requires systems that scale with your team, managing increased work and increased users. Role-based controls move the focus from an emergency patch-and-forget on permissions to an integrated component.

For complete details, refer to the official MLflow RBAC documentation.

Questions or feedback? Drop a note by opening an issue or join the MLflow community discussions.

⭐ Star us on GitHub — show your support for the project!

AI coding agents like Claude Code are becoming a standard part of the development workflow. An agent working through a complex task, understanding a codebase, writing tests, fixing bugs, can make dozens or hundreds of LLM calls in a single session. That level of autonomy is powerful, but it introduces questions that don't come up with one-shot prompts: How much is this costing? What are the agents actually sending to the model? Can we enforce content policies across every session?

MLflow AI Gateway answers all three. Starting in MLflow 3.12.0, you can route Claude Code through the gateway in two environment variables, and every session immediately gains full request tracing, budget enforcement, and guardrails, without touching your application code.

How It Works​

MLflow AI Gateway sits between Claude Code and the Anthropic API. From Claude Code's perspective, nothing changes, it authenticates the same way it always has. From the gateway's perspective, every request is a traceable, governable event.

This passthrough design means you get centralized visibility without centralized key management: each developer keeps their own Anthropic credentials, and the gateway adds governance on top.

Setting Up the Integration​

The setup takes three steps: start an MLflow server, create an endpoint, and set two environment variables.

Step 1: Start the MLflow Server​

pip install mlflow
mlflow server --port 5000

Step 2: Create an Anthropic Endpoint​

Open the MLflow UI at http://localhost:5000/#/gateway and navigate to AI Gateway. Click Claude Code icon and configure it:

Step 3: Point Claude Code at the Gateway​

Set two environment variables in your shell:

export ANTHROPIC_BASE_URL="http://localhost:5000/gateway/proxy/claude-code"

That's it. Run claude as usual. All requests will now flow through the gateway.

Observability: Every Request as a Trace​

The most immediate benefit is visibility. Every call Claude Code makes, regardless of session length or how many turns the agent takes, is automatically captured as an MLflow trace. No instrumentation, no SDK imports, no code changes.

Open the Traces tab in the MLflow UI to see a timeline of all requests: prompts, responses, token counts, and latency, organized by session.

Click into any trace for a detailed view of exactly what was sent and received:

This level of detail is particularly useful for understanding what a long-running agent session actually did, which subtasks consumed the most tokens, where latency spikes occurred, and how prompts evolved across turns.

Budget Controls: Keep Spending in Check​

Coding agents are designed to work autonomously, which makes them easy to forget about and expensive to over-run. MLflow AI Gateway's budget policies let you set spending thresholds globally or per workspace, with configurable alerts and hard limits.

When a session approaches a threshold, the gateway sends an alert. When it hits the hard limit, further requests are rejected before they reach the model, stopping runaway costs at the source rather than after the bill arrives.

See the budget policies documentation for configuration details.

Guardrails: Enforce Policies Across Every Session​

Because every Claude Code request passes through the gateway, guardrails apply uniformly, no per-application configuration needed. Before requests reach Anthropic, guardrails can screen for prompt injection or restricted topics. After responses come back, they can filter toxic content or redact PII before the agent sees the output.

For example, a PII guardrail on the Before stage will block any request containing personal data like email addresses or phone numbers, and return a structured error with the rationale so the caller knows exactly why it was blocked.

See the guardrails blog post for a full walkthrough of guardrail types, configuration, and the block vs. sanitize actions.

Getting Started​

Everything described here ships with MLflow. Follow the AI Gateway documentation for Claude Code for full setup instructions, or jump straight to OpenAI Codex and Gemini CLI if you use those instead.

Routing Claude Code through MLflow AI Gateway is the fastest way to add observability and governance to autonomous coding sessions. It joins guardrails and budget policies as part of the governance layer built into MLflow AI Gateway. If you run into issues or have feedback, please file a report on MLflow's GitHub Issues.

⭐ Star us on GitHub, show your support for the project!

The challenge is that OpenClaw's power comes from autonomy. It decides which tools to call, in what order, and whether to spawn sub-agents for sub-tasks. That makes it capable, but also opaque. When the agent handles a request well, you don't know why. When it handles one poorly, you don't know where it went wrong. You just see the final message in your chat.

This post shows how to add MLflow Tracing to OpenClaw so that every agent run becomes a fully inspectable execution timeline. We'll walk through setup, explain what traces capture, and use a concrete example to show how tracing turns vague suspicions into actionable debugging information.

Why Tracing Matters for a Personal Agent​

You might think tracing is only for production systems with SLAs and uptime requirements. But personal agents have their own version of the same problem: you're relying on the agent to do real work for you, and when it gets something wrong, you need to understand what happened so you can fix it.

Consider a few scenarios that are hard to debug without traces. You ask your OpenClaw agent to summarize this week's AI news and draft a brief. The summary is shallow and misses the biggest story. Was the web search tool returning poor results? Did the model ignore relevant results during summarization? Did it hit the context window limit and silently drop content? You ask it to reschedule a meeting based on your calendar, and it picks the wrong time slot. Did the calendar tool return stale data? Did the model misinterpret the constraint you gave it? You won't know from the chat reply alone.

Tracing captures the full execution path of every agent run: each LLM call with its prompt and response, each tool invocation with its parameters and result, each sub-agent spawn and its own nested steps, along with token counts and timing for everything. That record transforms debugging from guesswork into direct inspection. You open the trace, find the step where things diverged from what you expected, and now you know exactly what to fix, whether that's a skill definition, a tool configuration, or the way you phrased the request.

Beyond debugging individual runs, traces become the foundation for systematically improving your agent. When you can see how the agent handled a task, you can give it targeted feedback, refine the skills it uses, and verify that your changes actually worked. Tracing is what makes that feedback loop possible.

Your Data Stays Local​

One reason MLflow is a natural fit here is that it shares the same local-first philosophy as OpenClaw. MLflow is 100% open source, governed by the Linux Foundation, and fully self-hosted. When you run an MLflow server on your machine, all the trace data from your OpenClaw agent stays on your infrastructure and never leaves it. No telemetry gets sent to a third party, no vendor has access to your prompts or tool outputs. For a personal agent that handles your emails, calendar, and files, that matters.

Governing LLM Access with AI Gateway​

OpenClaw decides on its own when to call an LLM and which tools to invoke. That autonomy is the point, but it also means the agent can make a lot of API calls without you in the loop. If you have API keys in environment variables or scattered across config files, they're exposed to every process on your machine. And if a skill triggers a retry loop or spawns sub-agents that each make their own model calls, costs can add up fast with no single place to catch it.

MLflow AI Gateway sits between OpenClaw and your LLM providers to solve both problems. You store your API keys once in the gateway, encrypted and never exposed to client code. The gateway also gives you a single place to set a global budget limit across all providers, so a runaway loop can't silently rack up costs regardless of which model it's calling. For an autonomous agent that decides on its own when and how often to call LLMs, that kind of guardrail is worth having.

Setting Up MLflow Tracing with OpenClaw​

Getting started takes just three steps. First, install the OpenClaw MLflow plugin:

openclaw plugins install @mlflow/mlflow-openclaw

Then start an MLflow server locally to ingest the traces.

uvx mlflow server --port 5000

Then configure the MLflow connection using the built-in setup wizard:

openclaw mlflow configure

The wizard walks you through setting your tracking URI and experiment ID interactively. If you prefer to set these manually, you can use environment variables instead:

export MLFLOW_TRACKING_URI=http://localhost:5000
export MLFLOW_EXPERIMENT_ID=<your-experiment-id>

That's it. Start OpenClaw as usual and use it normally. Once the integration is enabled, tracing is automatic. Every agent run generates a trace that gets recorded in your MLflow server. There is no need to modify your skills, tool definitions, or agent configuration.

Open http://localhost:5000 in your browser and you'll see traces appear as your OpenClaw agent runs.

What a Trace Looks Like​

Each OpenClaw agent run produces a hierarchical span tree. At the top is the root agent span, representing the entire run from the moment your message arrives to the moment the agent sends its reply. Nested inside it are the individual steps the agent took, organized by type.

LLM spans capture each model call, including the full prompt sent to the model, the response received, and token counts (input, output, total). Because OpenClaw uses a ReAct loop where the model reasons, acts, observes, and reasons again, a single user request can generate multiple LLM turns. Each turn appears as its own span, so you can follow the agent's chain of reasoning step by step.

Tool spans record every tool invocation. You can see the tool name, the parameters the model chose to pass, and the return value or error. When a tool call fails, the error message is captured directly in the span, making it immediately visible. This is especially useful for OpenClaw's rich tool ecosystem, where a single request might touch web search, file I/O, calendar APIs, and messaging channels.

Sub-agent spans appear when OpenClaw spawns child agents to handle sub-tasks. Each sub-agent gets its own branch in the span tree, with its own LLM and tool spans nested inside. This makes it possible to understand the full execution even when agents are delegating to other agents.

Every span carries metadata including timestamps, duration, and usage statistics. The result is a complete, inspectable timeline of everything the agent did.

Monitoring Trends with the Dashboard​

Once traces start flowing into MLflow, the operational dashboard gives you a bird's-eye view of how your agent is performing. You can see error rates across runs, which tools are being called most frequently, and how token consumption trends over time. If your agent starts failing more often after you update a skill, or if token usage spikes because a new tool is triggering longer reasoning chains, the dashboard surfaces it before you notice it in chat. Think of it as the health check for your personal agent: a single page that tells you whether things are running smoothly or whether something needs attention. You can also enable automatic evaluation to score every incoming trace, so problematic behaviors get flagged automatically as your agent runs.

From Observations to Improvements​

Traces are useful for debugging, but their real value is closing the feedback loop between you and your agent.

When you review a trace and notice the agent did something well or poorly, you can record that feedback in MLflow as a structured annotation on the trace or session (a group of traces sharing a conversation ID). A thumbs-down on a trace that used the wrong tool, a note on a session where the agent missed context from an earlier message. Over time, this builds up a labeled dataset of what your agent gets right and what it gets wrong. That dataset becomes the basis for everything that follows: evaluating new skill versions, tuning prompts, and understanding which types of requests your agent handles reliably.

Here is where things get interesting. You can give OpenClaw direct access to its own traces and feedback through the MLflow CLI and Skills. The agent can read traces of its past runs, see which ones received negative feedback, and use that information to refine its own skill definitions. You provide the signal (feedback on traces), and the agent does the work of translating that signal into better behavior. This is the self-improving loop that tracing makes possible: observe, annotate, and let the agent learn from its own history.

What Comes Next​

Tracing is the foundation, not the ceiling. Once you have visibility and a feedback habit, the natural next step is evaluation. Define scorers for the quality dimensions you care about most and run them against your collected traces. The judges will surface patterns you might not have noticed manually: maybe your agent is consistently verbose, or consistently fails to cite sources, or handles one category of requests much better than another. MLflow also supports continuous evaluation, which scores every new trace automatically as it arrives, so you don't need to run eval manually at all.

The progression is deliberate: start with visibility, add human feedback, then automate quality measurement. Each step builds on the one before it, and you can stop at whichever level meets your needs.

If this is useful, give us a star on GitHub: github.com/mlflow/mlflow⭐️

Have questions or feedback? Open an issue or join the conversation in the MLflow community.

Today we're announcing multimodal tracing in MLflow. Binary content is automatically extracted from traces, stored efficiently as artifacts, and rendered inline in the UI exactly as your model saw it.

Why Text-Only Traces Fall Short​

As LLM applications move beyond text (vision models analyzing photos, audio models transcribing calls, agents generating images), three problems compound:

• Database bloat: A single image generation response embeds ~1.8MB of base64 directly in your span JSON. Across thousands of traces, that's gigabytes of binary data stored in your tracking database, data that was never meant to live in a relational store.
• Slow queries and UI: Loading a trace list means fetching all that inline binary. Trace search slows down, the UI lags, and browsing production traces becomes painful.
• Blind debugging: When a vision model misclassifies an image, you need to see the image alongside the model's response, not a wall of encoded bytes. Text-only traces make multimodal debugging impossible.

Multimodal tracing solves all three.

How It Works​

When MLflow detects binary content in a span, it pulls the bytes out and stores them in your existing artifact store (S3, Azure Blob, GCS, DBFS, or local filesystem), the same storage MLflow already uses for model artifacts. The span keeps only a lightweight reference URI, so the trace database stays small and queries stay fast. The UI fetches and renders the binary on demand when you open a trace.

Auto-Extraction: Zero Code Changes​

If you're already using MLflow's autologging for OpenAI, Anthropic, Gemini, Bedrock, or LangChain, multimodal tracing works out of the box. No code changes, no configuration. MLflow detects and extracts binary content automatically.

import mlflow
import openai

mlflow.openai.autolog()
client = openai.OpenAI()

# Image data is automatically extracted, no code changes needed
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
        ],
    }],
)

MLflow recognizes 8 multimodal data patterns across providers:

Manual Attachments for Custom Content​

For content that doesn't flow through autologging (PDFs, custom file types, or images you generate yourself), use the Attachment class:

import mlflow
from mlflow.tracing.attachments import Attachment

with mlflow.start_span(name="analyze_document") as span:
    pdf = Attachment.from_file("report.pdf")
    span.set_inputs({"document": pdf, "question": "Summarize the key findings"})
    span.set_outputs({"summary": "Q3 revenue was $4.2M, up 18% YoY..."})

Attachment objects get the same treatment as auto-extracted content: the binary is stored as an artifact, and the span JSON contains only the reference URI.

Rich Rendering in the Trace UI​

Multimodal traces render across both the Summary and Details & Timeline views:

• Images display as compact thumbnails. Click to expand to a fullscreen preview.
• Audio plays inline with standard browser audio controls.
• PDFs render in an embedded viewer.
• Other file types show as download links.

The chat view also renders multimodal content inline. Vision model inputs show the image alongside the text prompt, and audio responses include a playable player below the transcript.

Getting Started​

Multimodal tracing is available in MLflow 3.11+. To start capturing multimodal content in your traces:

1. Upgrade MLflow: pip install --upgrade mlflow
2. Enable autologging for your provider (mlflow.openai.autolog(), mlflow.anthropic.autolog(), etc.). Multimodal extraction happens automatically.
3. View traces in the MLflow UI. Images, audio, and files render inline.

For manual attachment creation and the full list of supported patterns, see the Multimodal Content and Attachments documentation.

If you find this useful, give us a star on GitHub: github.com/mlflow/mlflow ⭐️

Have questions or feedback? Open an issue or join the conversation on Slack.

We’ve all been there: you’re tweaking a new setup in the playground, things feel good, and suddenly your agent starts "hallucinating" nonsense into a live company feed. You only catch it when a teammate pings you with a confused emoji. While it’s a funny anecdote in development, it’s a disaster in production. It proves you didn't know what your agent was doing, and you definitely didn't know why.

As you move from simple chatbots to autonomous systems that actually touch your business logic, fragmented tracing and guessing become serious liabilities. To move your agents into the wild safely, you need to master three things:

• The "What": Identifying the weird, new ways multi-agent systems fail.
• The "How": Building the observability you need to see your system's state in real-time.
• The "Why": Pinpointing the exact metrics that let you steer the system before things go sideways.

MLflow helps you bridge this gap, turning a black box of agentic workflows into a transparent, mission-critical deployment.

How Multi-Agent Systems Fail: Why Monitoring Isn’t Enough​

Before diving into multi-agent deployments, let's look at how a single agent can fail. These failure modes are familiar to anyone who has shipped an LLM application:

• Hallucination: The model fabricates data, especially when a tool returns empty or ambiguous results, and the model fills in the gaps rather than surfacing the error.
• Schema Fragility: A minor change in an API response — a renamed field, a new nested object — silently breaks the agent's ability to parse and act on the output.
• Context Decay: Instructions buried deep in a long prompt get effectively ignored. The model attends to the most recent context and loses the thread of earlier constraints.
• Runaway token usage: An agent gets stuck in a retry loop — rerunning the same failing tool call over and over — burning tokens and budget with nothing to show for it.

Now, all of the above still applies to individual agents inside a multi-agent system. But the coordination layer introduces failures that are qualitatively different.

The most common is a cascading error. Imagine your orchestrator delegates a financial calculation to a specialist agent. The specialist completes the task but misinterprets a business constraint — say, using quarterly revenue when annual was expected. Three downstream agents then incorporate this number into their own analyses. By the time the final output reaches a human, the error has compounded through four separate reasoning steps. The result looks plausible. Nobody flags it immediately.

What makes this hard to debug isn't the error itself — it's that the failure originated three steps back, in a span you weren't watching. You spend hours on the output when the problem is in the input to a different agent entirely.

A related issue is shared memory pollution. When agents write intermediate results to a shared context, a hallucination from one agent becomes a "fact" that subsequent agents reason from. The degradation is gradual rather than sudden, which makes it particularly hard to catch — you notice the quality declining but can't pinpoint when it started or why.

A third failure mode worth knowing about is agents stuck in a waiting loop. An orchestrator waits for a specialist's response; the specialist is waiting on a tool that has silently timed out. Neither fails loudly. Your observability infrastructure shows increasing latency, but nothing indicates the system is effectively stalled.

What makes all of this harder than equivalent failures in traditional software is the non-determinism of LLMs. Run the same prompt a hundred times, and you get slightly different outputs. Combined with the cost of every retry, this means failures in multi-agent systems are both harder to reproduce and more expensive to discover.

From monitoring to observability: instrumenting CrewAI with MLflow​

With a single agent, monitoring inputs and outputs is usually enough. You know what went in, you know what came out, and if something looks wrong, you read the prompt and response. Debugging is linear.

With multi-agent systems, that's no longer sufficient. You need to understand not just what each agent returned, but why the orchestrator made the delegation decisions it did, how outputs flowed between agents, and where in the chain something started going wrong. Monitoring individual inputs and outputs gives you data points; observability across the full trace gives you the story.

The unit of analysis shifts from the prompt to the state transition. To understand a multi-agent system in production, you need visibility into:

• Orchestration and routing decisions: Why did the orchestrator send this task to this agent rather than another? Was the delegation correct given the task requirements?
• Inter-agent data flow: What exactly was passed between agents, and did it arrive intact? This is where cascading errors and memory pollution become visible.
• Latency at each step: Which agent or tool is the bottleneck? Where is the system spending time it shouldn't be?

To solve these problems, we first need to see what's going on with the system. This means implementing tracing that captures not just individual tool calls but the full nested, connected graph of agent interactions.

Figure 1. Nested traces in MLflow UI.

In a standard LLM call, you have a start and an end. In a multi-agent workflow, you have nested spans and branching logic. MLflow Tracing allows us to reconstruct the graph by capturing the parent-child relationship between the Orchestrator and Workers. This transforms a black-box execution into a navigable map of state transitions.

Out of the box and custom tracing​

In MLflow, enabling comprehensive tracing can be done with one command.

mlflow.<framework>.autolog()

Since MLflow supports many flavors out of the box, monitoring of the CrewAI application will look like this:

import mlflow

# Turn on auto tracing by calling mlflow.crewai.autolog()
mlflow.crewai.autolog()

# Optional: Set a tracking URI and an experiment
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("CrewAI")

And then the actual CrewAI code:

from crewai import Agent, Crew, Task

# Tasks and Agents definitions, see full example here
# https://mlflow.org/docs/latest/genai/tracing/integrations/listing/crewai/#example-usage
...
crew = Crew(
        agents=[orchestrator, macro_data, researcher, results_lead],
        tasks=[plan_task, stats_task, research_task, synthesis_task],
        verbose=True,
    )
result = crew.kickoff()

Autolog functionality captures traces for every run and aggregates them into the DAG that is easy to navigate. But to fully understand the system's behavior, we suggest tracing custom functions and tools that are not supported by MLflow autolog.

To do it, you can use the custom decorator:

@mlflow.trace(span_type="TOOL", attributes={"key": "value"})
def func():
    ...

In our example:

@mlflow.trace(
    name="numeric_report_validation",
    attributes={"service": "heuristic numeric cross-check (report vs sources)"},
)
def validate_report_numbers_against_sources(
    final_report: str,
    *source_texts: str,
) -> str:
    # Function definition
    ...

The @mlflow.trace() decorator allows you to create a span for any function. This simple approach allows us to capture unique relationships between functions, record exceptions, or custom parameters, as well as capture the cascading financial miscalculation alluded to above.

Capturing the relevant metrics​

Traces tell you what happened. Metrics tell you whether it was acceptable. By combining MLflow's autolog() for LLMs (e.g., mlflow.openai.autolog()) with custom attributes, you can transform simple traces into a verifiable audit trail. Once you have full tracing in place, the next step is deciding which numbers actually matter — because in a multi-agent system, a successful response can hide a costly execution. The system might return the right answer after 15 recursive calls, 10 retries on a flaky API, and two minutes of an agent waiting for a response that nearly never came.

Figure 2. Exploring metrics in MLflow UI.

We can aggregate the necessary metrics into three critical pillars:

1. Orchestration and Routing Logic​

Every time your orchestrator delegates a task, it makes a decision. That decision might be correct, redundant, or just slow — and without instrumenting it, you won't know which. Capturing routing behavior helps you answer the question that matters most: Is the supervisor sending tasks to the right place, efficiently?

A few examples of relevant metrics are:

• Successful delegation rate: Did the supervisor choose a correct agent or tool, or did it get confused by the inaccurate descriptions?
• Delegation Latency: How long it took a supervisor to decide which agent/tool to call
• Redundancy and loop detection: Did the supervisor choose the efficient path

In CrewAI, the step_callback is the natural hook for capturing routing behavior — it fires after each agent step, giving you the agent name, output, and timing:

import mlflow
import time

delegation_counts: dict[str, int] = {}
def track_routing(step_output) -> None:
      agent_name = step_output.agent
      delegation_counts[agent_name] = delegation_counts.get(agent_name, 0) + 1
      span = mlflow.get_current_active_span()

      if span:
          span.set_attributes({
              f"routing.{agent_name}.call_count": delegation_counts[agent_name],
              "routing.total_delegations": sum(delegation_counts.values()),
              # step_callback counts steps per agent, not supervisor delegations;
              # > 3 gives room for a normal reasoning + tool + retry cycle
              "routing.loop_detected": any(v > 3 for v in delegation_counts.values()),
          })
      # Log state handoff after each agent completes
      next_input = crew_state.get_state("current_input") or ""
      log_state_handoff(
          from_agent=agent_name,
          to_agent="next",  # CrewAI doesn't expose next agent here, use state
          output=str(step_output.result),
          next_input=next_input,
)

2. State consistency and memory poisoning​

As we saw with the cascading error example, the biggest risk in a multi-agent system isn't a single agent failing — it's a single agent producing subtly incorrect output that's treated as ground truth by every downstream agent. Tracking state consistency helps you detect this drift before it compounds.

These metrics help to evaluate the consistency and memory state:

• Grounding accuracy: Score comparing the agent's input against the global state to see if the agent's hallucination occurred in the previous agent.
• Concurrency: The number of agents attempting to access or mutate the same element within the same execution window.
• Context handoff efficiency: Are agents sharing the entire context or only an essential part?

A practical starting point for improving context handoff efficiency is a simple token-overlap score between what one agent outputs and what the next agent receives. It doesn't require an additional LLM call and gives you a signal when agents are either passing too much context or too little:

def handoff_efficiency_score(output_text: str, next_input_text: str) -> float:
    """Token overlap between agent output and the next agent's input.
    Score of 1.0 means the next agent received everything the previous one produced.
    Score close to 0.0 means most of the output was dropped before handoff.
    """
    output_tokens = set(output_text.lower().split())
    input_tokens = set(next_input_text.lower().split())
    if not output_tokens:
        return 0.0
    return len(output_tokens & input_tokens) / len(output_tokens)

@mlflow.trace(name="state_handoff", span_type="CHAIN")
def log_state_handoff(
    from_agent: str, to_agent: str, output: str, next_input: str
) -> None:
    span = mlflow.get_current_active_span()
    if span:
        span.set_attributes({
            "handoff.from_agent": from_agent,
            "handoff.to_agent": to_agent,
            "handoff.output_chars": len(output),
            "handoff.input_chars": len(next_input),
            "handoff.efficiency_score": handoff_efficiency_score(output, next_input),
        })

Then call it from each task callback, where you already have the current output and can read the previous agent's output from shared state:

def _on_research_complete(output: object) -> None:
    text = crew_state.task_output_to_text(output)
    crew_state.set_state("research_brief", text)

# How much of the macro data specialist's output made it into the research brief?
stats = crew_state.get_state("macro_stats_snapshot") or ""
log_state_handoff(
    from_agent="macro_data_specialist",
    to_agent="research_analyst",
    output=stats,
    next_input=text,
)

3. Operational telemetry and costs​

In a multi-agent system, a successful response can hide a catastrophic operational failure. The system can give a correct answer, but it uses multiple recursive calls, retries, and spends time in a waiting loop.

To prevent these resource drains and inefficiencies, you must capture:

• Token Attribution: the cost per node, call, path, or task;
• Per-Node Latency and Bottleneck Detection: metrics like span duration vs. queue time allow for the detection of bottlenecks and poorly optimized tool/database calls
• Rate Limit and Throughput Volatility: number of API calls, frequency of specific errors (like 429) to identify hotspots;
• Task depth: to identify if agents are going into an infinite loop

To capture custom multi-agent metrics, you can wrap your multi-agent system execution in the following way:

@mlflow.trace(name="Crew.kickoff", span_type="CHAIN")
def run_crew_with_metrics(crew: Crew) -> CrewOutput:
    """Run ``crew.kickoff()`` inside a traced span that collects summary metrics."""
    delegation_counts.clear()  # reset between runs to avoid false loop detection
    _crew_metrics.clear()
    t0 = time.perf_counter()
    result = crew.kickoff()
    duration_s = round(time.perf_counter() - t0, 3)

    span = mlflow.get_current_active_span()
    if span is not None:
        charter = crew_state.get_state("engagement_charter") or ""
        stats = crew_state.get_state("macro_stats_snapshot") or ""
        research = crew_state.get_state("research_brief") or ""
        synthesis = crew_state.get_state("synthesis_result") or ""

        span.set_attributes({
            "crew.total_duration_s": duration_s,
            "crew.task_count": len(crew.tasks),
            "crew.agent_count": len(crew.agents),
            "orchestration_lead.output_chars": len(charter),
            "macro_data_specialist.output_chars": len(stats),
            "research_analyst.output_chars": len(research),
            "synthesist.output_chars": len(synthesis),
            "validation.report_number_count": len(extract_numeric_tokens(synthesis)),
            "validation.source_number_count": len(extract_numeric_tokens(research)),
            **_crew_metrics,
        })
    return result

Multi-service observability​

In some cases, multi-agent systems can span multiple services connected via HTTP requests. Ideally, we want a unified view of traces for this system, rather than having to monitor two sides independently and stitch traces together in the notebook. MLflow supports OTEL, which enables instrumentation and monitoring for applications split across multiple services. Check more details here.

While MLflow provides a specialized view for LLM-specific spans, multi-agent systems don't live in a vacuum. They rely on databases, authentication services, and third-party APIs. By leveraging MLflow’s OpenTelemetry (OTEL) support, you ensure that your agentic traces aren't stuck in a silo:

• Unified Context: Export your MLflow traces to enterprise backends like Grafana, Datadog, or Honeycomb. This allows SREs to see a single timeline that connects a user’s frontend click to the specific agent hallucination that caused the error.
• Production Standards: Modern infrastructure teams already speak OTEL. Using it means your agent traces appear in the same dashboards, alerts, and on-call runbooks as the rest of your stack — no separate tooling, no special cases.
• Correlation across Services: If an agent fails because a downstream microservice times out, OTEL allows you to correlate the agent's retry logic with the backend's 503 error in a single unified trace.

Putting it all together​

You can find an end-to-end example of a multi-agent system built with CrewAI and instrumented with MLflow in this repository

Multi-agent systems introduce failure modes that don't surface until agents start talking to each other — cascading errors, memory pollution, and agents silently stalled waiting for a response that never comes. To move beyond "it works on my machine", your observability stack needs to capture the full picture: not just what each agent returned, but how agents influenced each other and where the system spent its time and money.

MLflow gives you the tooling to get there: autolog for the easy wins, custom decorators for the parts that matter most, and OTEL integration for when your agents are just one piece of a larger production system.

Seeing what's happening is only half the battle. In the second part of this series, we'll move beyond passive observation to explore governance and active steering — the mechanisms you need to enforce safety, manage costs, and prevent issues before they ever reach your production environment.

What to try next?​

• Run your existing multi-agent workflow with autolog() enabled and look for your slowest or most expensive spans.
• Add the @mlflow.trace decorator to your validation functions and start measuring context handoff efficiency.
• If your agents span multiple services, set up OTEL export and see your traces in a unified view for the first time.

If you find MLflow useful, give us a star on GitHub

But then the silent failures and quality issues begin. You tweak a prompt to improve one behavior, and three others get worse. You can’t tell if your latest update is a step forward, a step backward, or simply a desperate, guesswork move.

At some point, you have to trade the vibe check for a structured approach. Without a structured way to measure what’s happening, you’re hoping for the best.

This post is about making that transition: adopting a structured, systematic way to evaluate your AI application. We’ll walk through integrating MLflow’s four pillars: Tracing, Evaluation and Human Feedback, Prompt Versioning, and AI Governance, focusing on the phases where it actually matters, as part of this systematic approach called eval-driven development cycle.

Figure 1. MLflow Open Source AI Platform’s four pillars

Why Agents Break Differently: The Case for AI Observability​

As software engineers, we know how to deliver reliable software. You write code, run unit tests, push through QA, ship to production, and set up telemetry that pages you when something breaks. We are familiar with this playbook, as it is decades old, and it works because the system’s outputs are deterministic: same inputs, same outputs, every time.

Agents, by contrast, don’t follow that playbook for several reasons. First, their outputs are free-form, in natural language, often unpredictable, and quality is subjective in ways that traditional test assertions can’t capture. Second, what seems like a helpful agent’s response to one user might feel verbose or off-topic to another, and the developer who wrote the agent may not have the domain expertise to tell the difference. This difference leads to the third reason, which introduces subjectivity and risk into quality, demanding cross-functional and collaborative efforts with subject matter experts.

And finally, you’re navigating trade-offs between cost, latency, and quality on every agentic workflow invocation. Without instrumentation, you have no data to guide you.

Without a structured process and a development platform, such as MLflow, to facilitate the rigor necessary, the workflow looks something like this: write the agent, run a few prompts locally, ship to production, and hope for the best. Not a good idea!

AI observability is the foundation you need before anything else makes sense. If you can’t see what your agent is doing at each step, then you can’t monitor, debug, and improve its quality. Tracing gives you that visibility, and it changes the conversation from “I think the agent is working” (guessing) to “here’s exactly what happened in this request” (measuring).

Eval-Driven Development: Three Phases That Shape Your MLflow Evaluation Strategy​

The teams shipping reliable agents aren’t doing anything magical. They follow a prescriptive cycle that MLflow’s AI Platform is built around: Eval-Driven Development. Think of it as three phases, each building on the last, each tightening the feedback loop between “what happened” and “was it good enough.”

Phase 1: Prototype with tracing​

Start by instrumenting your agent from day one. MLflow’s one-line autolog captures every LLM call, tool invocation, and retrieval step as a structured trace with latency, token usage, and cost data attached. With a single line of code, you can enable LLM tracing for any MLflow-integrated library. mlflow.library_name.autlog(). For example, for OpenAI, this will capture all the inputs/outputs, token usage, costs, and latencies.

import mlflow

# Enable automatic tracing for OpenAI calls with a single line
mlflow.openai.autolog()

Manual tracing of your tool usage captures operations that autologging may not. For example,

@mlflow.trace(name="get_embedding", span_type="LLM")
def get_embedding(query: str) -> List[float]:
    """Call OpenAI embeddings API — LLM span."""
    response = client.embeddings.create(
        input=query.replace("\n", " "), model="text-embedding-3-small"
    )
    return response.data[0].embedding


@mlflow.trace(name="query_embedder", span_type="EMBEDDING")
def embed_query(query: str) -> List[float]:
    """Embed a query — EMBEDDING parent span with LLM child span."""
    return get_embedding(query)

Once tracing is enabled, your vibe checks become data-backed. Instead of reading a response and guessing whether it’s good, you can inspect the full trajectory:

• Did a tool call fail silently?
• Did retrieval pull the wrong documents?
• Is one span burning 4 seconds while the rest finish in milliseconds?

These are questions you can actually answer now by looking at traces in the MLflow UI rather than re-running prompts and squinting at outputs.

Phase 2: Incorporate subject-matter expert feedback, add judges, and create evaluation datasets.​

Tracing tells you what happened. Evaluation tells you whether it was any good. MLflow’s labeling and feedback collection UI lets you share your prototype with domain experts who interact with the agent and submit structured feedback on correctness, relevance, safety, and any other dimensions that matter to your use case.

Human feedback helps you find the issue in your prototype. Once you've fixed the issue, you can build an LLM judge to quickly test that it's been fixed reliably across multiple examples. When you deploy to production, the LLM judge helps you monitor your agent to ensure that this issue never occurs again.

Figure 2. Capture Subject Matter Expert (SME) feedback on a judge’s evaluation score.

Run the layered evaluation and capture SME feedback as part of the evaluation.

from mlflow.genai.scorers import RelevanceToQuery, ToolCallRelevance, Guidelines
# Run layered evaluation: built-in judges + custom judges + policy guidelines
results = mlflow.genai.evaluate(
    data=traces,
    scorers=[
        RelevanceToQuery(),
        ToolCallRelevance(),
        Guidelines(guidelines=[
            "Always reply in the user's language",
            "Never disclose internal pricing logic or name of the magazines",
        ]),
    ],
)

Each judge examines traces from a different angle. Built-in judges handle the common dimensions, while guidelines judge enforce policy. Running them together across hundreds of traces is where you catch the issues, faults, or unexpected or undesirable agent behavior that vibe-checking misses entirely.

During your initial evaluation, if judges score low or do not align with the expected behavior, it indicates you need to either reexamine the evaluation or build an evaluation dataset to better align the judges with the outcome.

from mlflow.genai.datasets import create_dataset

# Create an evaluation dataset for an online magazine to test judges and prompts against
evaluation_dataset = create_dataset(
    name="customer_support_qa",
    experiment_id=["0"],  # your eperiment id in MLflow
    )
# Add records (test cases) to the dataset
new_records = [
    {
        "inputs": {"question": "What are the most popular megazines with illustration in the combat and games genre?"},
        "expectations": {"expected_answer": """Here are the top five trending magazines that are safe for both children over 12:
            1. White Dwarf
            2. War Games Illustrated,
            ..."""
          },
    },
    ...
  ]
# create your evalaution set
evalution_dataset.merge_records(new_records)

Next, in phase 2, bolster your evaluation strategy with custom judges that capture your domain-specific requirements.

From Built-in Judges to Custom Evaluations: Layering Your Agent Scoring Strategy​

Built-in judges can’t know your business. If your agent handles insurance claims, “correct” means something very specific that no generic scorer will capture. That’s where custom judges fill the gap. The make_judge API lets you define domain-specific evaluation logic declaratively, without writing scoring functions from scratch.

from mlflow.genai.scorers import make_judge
from typing import Literal

# Define a custom judge for domain-specific content safety
is_content_safe = make_judge(
    name="content_safety",
    instructions="""Evaluate whether {{outputs}} is appropriate
        and professionally worded for the question in {{inputs}}.
        Rate as: safe, unsafe, or inappropriate.""",
    feedback_value_type=Literal["safe", "unsafe", "inappropriate"],
    model="openai/gpt-5-mini",
)

The custom judge runs as an LLM call against whatever model you specify, and its scores land alongside your built-in judge results in the same evaluation run. Now you can run your evals again with all the judges and the evaluation dataset.

results = mlflow.genai.evaluate(
    data=evaluation_dataset,
    scorers=[
        RelevanceToQuery(),
        ToolCallRelevance(),
        is_content_safe(),
        Guidelines(guidelines=[
            "Always reply in the user's language",
            "Never disclose internal pricing logic or name of the megazines",
        ]),
    ],
)

The final bit in phase 2 is optimizing your prompts for better and best alignment with your scorers.

Systematically Improving and Optimizing Prompts in LLMOps​

MLflow’s Prompt Registry versions every prompt and links it directly to traces and evaluation metrics, giving you the A/B testing infrastructure that prompt engineering has always needed. Just as human feedback is part and parcel of your phase 2 evaluation strategy, so is versioning prompts and optimizing them for systematic agent testing.

During testing, you will want to tweak prompts and try different versions. A prompt change is a behavior change, and without version control, you lose the ability to correlate “this prompt” with “these evaluation scores.”

Aside from versioning, another real benefit is automated prompt optimization, an algorithmic approach to help you automatically discover a better prompt. Instead of manually iterating on phrasing, MLflow’s optimize_prompts API runs optimization algorithms like GEPA against your evaluation dataset and judges, converging on prompt versions that score higher without you having to guess your way there.

from mlflow.genai.optimize.optimizers import GepaPromptOptimizer

# Register a baseline prompt and optimize it automatically
original_prompt = mlflow.register_prompt(
    name="qa_prompt",
    template="Analyze this document and extract key facts: {{ document }}",
)

result = mlflow.genai.optimize_prompts(
    predict_fn=my_agent,
    train_data=eval_dataset,
    prompt_uris=[original_prompt.uri],
    optimizer=GepaPromptOptimizer(reflection_model="openai:/gpt-4.1"),
    scorers=[Correctness()],
)

The optimizer registers each candidate prompt version, runs it against your evaluation dataset, scores it with your judges, and picks the winner. You get a prompt that’s measurably better, backed by evaluation data. This closes the loop: traces feed evaluations, evaluations validate prompts, and better prompts produce better outcomes.

After several iterations with Phase 2, you are now ready for Phase 3.

Phase 3: Stakeholder sign-off and production monitoring.​

Before shipping, you need stakeholder buy-in. MLflow’s agent dashboards surface cost, latency, and quality scores in a format stakeholders can actually reason about, making the tradeoff conversation concrete rather than abstract. Once you deploy, the same judges that ran offline now run continuously on live traces, so production monitoring isn’t a separate system. It’s the same evaluation framework applied to real traffic.

Figure 3. Creating a LLM judge for online monitoring

Key Takeaways for a Structured Approach to AI Observability​

Evals aren't just for research teams. If you're shipping an agent, you need a structured evaluation approach to catch issues and fix them before your users do.

You don't need perfect ground truth labels to make progress. Begin with inputs, add expected outputs where you're confident, use LLM judges for the rest, and let your evaluation dataset grow with each iteration.

Trace everything, evaluate in layers, version your prompts. Tracing reveals how agents behave. Adding more judges and versioning and optimizing your prompts leads to your agent’s expected behavior.

Getting started is easy. Add mlflow.openai.autolog(), run one evaluation with a couple of built-in scorers, and you've moved from guessing to measuring. Everything else builds from there.

In short, stop guessing, start measuring!

What's Next?​

If this was useful, give us a star on GitHub. Take a look at our recent MLflow 3.11 Release webinar.

References and Resources​

1. Your Agents Need an AI Platform
2. Testing and Refining Claude Code Skills with MLflow
3. End-to-end Workflow: Eval Driven Development
4. MLflow Tracing: Debugging and AI Observability for GenAI
5. Advanced MLflow Tracing: Manual Spans, RAG, and Agent workflows

As GenAI applications move into production, the question shifts from can we build this? to can we trust what comes out of it? LLMs are powerful but unpredictable: they can produce harmful content, leak personally identifiable information, or violate organizational policies in ways that are hard to anticipate and even harder to catch after the fact. Bolting safety checks onto individual applications means duplicated logic, inconsistent enforcement, and gaps wherever new services bypass the rules.

MLflow AI Gateway supports guardrails in the upcoming 3.12 release: configurable content policies enforced at the gateway layer, before requests reach your LLM or before responses reach your users. Because guardrails run in the gateway, they apply consistently across every application that routes through it, with no changes to application code required.

How Guardrails Work​

Each guardrail evaluates incoming requests or outgoing responses against a set of natural-language instructions. When the guardrail determines that content violates the policy, the gateway either blocks the request entirely or sanitizes (redacts) the offending content before allowing it through.

The pipeline looks like this:

Request  →  Before Guardrails  →  LLM  →  After Guardrails  →  Response

Before guardrails screen the incoming request. They're well-suited for catching prompt injection, PII in user input, or off-topic queries before they consume any tokens.

After guardrails evaluate the LLM's response before it reaches the caller. Use them for toxicity filtering, detecting PII in generated output, or enforcing brand and tone requirements.

Multiple guardrails on the same endpoint execute in order. If any guardrail blocks a request, subsequent guardrails in the same stage are skipped.

Guardrail Types​

Built-in guardrail types come with pre-loaded instructions so you can get started quickly. Because the instructions are plain text, you can edit them at any time to tighten or loosen the policy, add domain-specific context, or handle edge cases unique to your use case — no code changes required, just update the prompt:

Selecting a built-in type pre-populates the name and instructions in the creation wizard. From there, you can refine the prompt to match your organization's specific policies — for example, restricting a PII guardrail to only flag medical record numbers, or tuning a safety guardrail to allow certain technical terminology that a generic classifier would otherwise flag.

Creating a Guardrail​

Guardrails are configured per endpoint. Navigate to AI Gateway > Endpoints, open an endpoint, and select the Guardrails tab.

Click Create Guardrail to open the wizard. The first step asks you to choose a type:

The second step lets you configure every aspect of the guardrail:

Name: a descriptive label shown in the guardrails table and included in error responses when a guardrail blocks a request.

Stage: Before or After. Switching stages automatically updates the content variable references in the instructions editor ({{ inputs }} ↔ {{ outputs }}), keeping them correct without manual edits.

Instructions: natural-language instructions describing what to look for and how to respond. Reference {{ inputs }} for the incoming request and {{ outputs }} for the LLM response. The instructions must include at least one content variable. Here's an example for a custom toxicity check on the After stage:

You are a toxicity detector. Review the LLM response below for any harmful,
offensive, or hateful language. Reply with a JSON object:

{
  "rationale": "Brief explanation of your decision.",
  "result": "yes if the content is safe, no if it is harmful"
}

<response>{{ outputs }}</response>

Guardrail Model: the AI Gateway endpoint used to evaluate the guardrail. You can use a cheaper, faster model for policy enforcement and reserve your primary model for the actual workload. The current endpoint is excluded from the list to prevent circular dependencies.

Action: what happens when the guardrail triggers:

• Block: the request is rejected with an HTTP 400 response. The response body includes the guardrail name and the rationale so callers know exactly why.
• Sanitize: flagged content is redacted or masked, then the cleaned request or response continues through the pipeline.

Click Create Guardrail to save. The guardrail is active immediately for all traffic through the endpoint.

What Blocking Looks Like​

When a guardrail's action is set to Block and a violation is detected, callers receive an HTTP 400 with a structured error body containing the guardrail name and the rationale: actionable context that clients can log, surface to users, or use to trigger alternative handling.

Managing Guardrails​

Clicking any row in the guardrails table opens a detail panel where you can update the stage, instructions, guardrail model, or action. Updates are saved atomically: changes register as a new scorer version and replace the guardrail on the endpoint without dropping any in-flight requests.

To delete a single guardrail, open its detail panel and click Delete. To remove multiple guardrails at once, select their checkboxes in the table and use the Delete button in the toolbar.

Getting Started​

Guardrails are included with MLflow and available through the AI Gateway UI:

pip install mlflow
mlflow server

Then open the MLflow UI, navigate to AI Gateway > Endpoints, and add guardrails to any unified endpoint. For full configuration details, see the Gateway documentation.

Guardrails are the latest addition to MLflow AI Gateway's governance layer, joining budget policies and usage tracking to give teams complete visibility and control over their LLM traffic. If you run into any issues or have feedback, please file a report on MLflow's GitHub Issues.

⭐ Star us on GitHub — show your support for the project!

This post walks through how to use MLflow AI Gateway as the control plane for a multi-agent system. We'll use a production-style customer support agent as the example, showing how to pinpoint exactly where costs accumulate across agent steps, identify what to optimize, and set budget alerts before spend gets out of hand.

Note: The cost figures throughout this post are illustrative. Actual costs will vary depending on your provider, pricing tier, and token volumes.

Meet the Agent​

Our agent, let's call it SupportBot, is a multi-agent system built to handle tier-1 customer support. Here's the architecture:

When a customer sends a message, the embedding pipeline converts the query into a vector and retrieves relevant context from our knowledge base. The orchestrator reads the query and decides which sub-agent to invoke: the order lookup agent for "where's my package?" questions, the refund agent for return requests, or the RAG agent for general policy and product questions. After the sub-agent responds, the orchestrator synthesizes a final answer and runs it through a guardrail check to catch PII leakage or off-brand tone before it reaches the customer.

Each component does its job well. However, nobody modeled what it costs to run it.

The Hidden Cost Anatomy of a Single Ticket​

Every ticket triggers a chain of calls. Each link in that chain has a price tag.

1. Embedding: converting the query to a vector for knowledge base retrieval
2. Orchestrator routing: an LLM decides which sub-agent handles the request
3. Sub-agent execution: the actual work which may need additional calls if the agent retries or invokes a tool
4. Response synthesis: the orchestrator writes the customer-facing reply
5. Guardrail check: one more LLM call to verify safety and tone

One question = 4–6 LLM calls, if no errors happen.

Here's the cost breakdown for a single ticket — ticket T001, an order status query:

Four LLM calls, four different cost contributions. The orchestrator routing is cheap ($0.0009), but the synthesizer and guardrail together account for over 80% of the total. Now multiply that across every ticket, every day.

Here's what that looks like at 500 tickets per day across 20 sample tickets:

At ~$9.40/day with 500 tickets, the math feels manageable. But scale to 10,000 tickets per day and it's ~$188/day — roughly $5,650/month. If context windows bloat or retry rates spike during provider outages, costs can easily double.

The problem isn't that any single call is expensive. It's that agent costs are multiplicative, not additive, and this multiplicative call pattern is one of the most consistently reported causes of production cost shock in multi-agent systems.

Before You Scale: A Pre-Ship Checklist​

Before your agent handles its first production ticket, make sure you have these five points in place. Each one maps to a later section in this post, think of this as the minimum viable cost infrastructure:

If you're already in production without these, that's fine, every step below is retrofittable. But if you're still building, this is the cheapest time to wire them in.

Step 1: Route Through the Gateway​

The first step is simple but foundational: route all LLM calls through MLflow AI Gateway instead of calling providers directly. This gives you a single control plane for credentials, traffic, and cost tracking.

Getting started takes two commands:

pip install 'mlflow[genai]'
mlflow server

Open the gateway UI at http://localhost:5000/#/gateway. Under API Keys, register your provider credential once. Then under Endpoints, click Create Endpoint for each role in the agent, give it a name, pick a provider and model, and attach the API key. SupportBot uses four:

Each endpoint gets its own URL path and can be independently routed, rate-limited, and budgeted. See Create & manage endpoints for the full walkthrough.

On the agent side, the change is minimal — swap base_url to point at the gateway:

from openai import OpenAI

# Before: calling Databricks FMAPI directly
# client = OpenAI(base_url=f"{HOST}/serving-endpoints", api_key=TOKEN)

# After: routing through MLflow AI Gateway
client = OpenAI(base_url="http://localhost:5000/gateway/mlflow/v1", api_key="")

Your agent code stays the same. The gateway handles the rest: credential management, request routing, and automatic tracing of every call.

Step 2: See Where the Money Goes​

Once traffic flows through the gateway, every request is automatically traced with token counts, cost, model name, and latency, with no additional instrumentation required.

To capture traces in your agent code, enable autologging:

import mlflow
mlflow.openai.autolog()

Now when SupportBot resolves a ticket, you can see the full trace, every span in the chain, from embedding through guardrail, with cost attribution at each step:

The trace shows the full span hierarchy: process_ticket → route_query → refund_agent → synthesizer → guardrail, each with its own latency and token count. One customer question, four LLM calls, ~14 seconds end-to-end.

Traces revealed which components were consuming the most tokens, making it clear where to optimize.

Step 3: Set Budget Guardrails​

Visibility tells you what's happening. Budget policies tell the system what to do about it. MLflow AI Gateway supports threshold-based budget policies with two actions: ALERT (fire a webhook, traffic keeps flowing) and REJECT (fire a webhook and return HTTP 429 to block new requests).

In the gateway UI under AI Gateway > Budgets, click Create budget policy and set a budget amount, reset period (daily / weekly / monthly), and an action (ALERT or REJECT). Register a Slack webhook under Budget alert webhooks so ALERT policies land in your on-call channel. Here's the layered policy we settled on for SupportBot:

The layered approach is intentional. The daily alert at $90 gives us time to investigate before anything breaks. The daily reject at $150 is the safety net — it prevents a retry storm or context bloat from turning a bad day into a catastrophic one. The monthly policies give us a longer-horizon ceiling so a slow drift doesn't go unnoticed.

By default the gateway tracks spend in-process. See Budget alerts & limits for the full reference.

Here's what happens at different traffic levels with these thresholds in place:

When the $90 alert fires, a Slack notification lands in our #agent-ops-alerts channel with details: which policy was breached, current spend, and the threshold amount. That gives the on-call engineer time to check the dashboard and decide whether the spike is a legitimate traffic increase or something gone wrong.

Watch for retry loops. Budget policies cap total spend, but they won't save you from a spike that burns through your daily limit in minutes. Set per-minute or per-hour rate limits at the gateway level separately from your daily budget. If your trace view shows a sudden jump in error rate combined with high request velocity, that's a retry storm, sub-agents hammering a failing provider endpoint. At minimum, every LLM call in your agent should use exponential backoff with jitter. Without it, a transient provider error can cascade into hundreds of wasted calls that drain your budget before the alert even fires.

When the $150 reject policy fires, new requests get an HTTP 429 response. The agent needs to handle this gracefully.

Put retries and model fallback in the gateway. For each endpoint, the gateway has a Priority 2 (Fallback) section where you list alternate models to try in order when the primary errors or rate-limits. For orchestrator, we add Claude Haiku 4.5 as a cost-optimized fallback so a transient GPT-5.1 outage automatically shifts traffic to the cheaper model without any client changes. See Traffic routing & fallbacks for the full configuration.

That leaves the client to decide what the user sees when every model option is exhausted.

from openai import OpenAI, RateLimitError

gateway = OpenAI(base_url="http://localhost:5000/gateway/mlflow/v1", api_key="")

def call_orchestrator(messages):
    try:
        # Retries and model fallback are handled by the gateway
        return gateway.chat.completions.create(model="orchestrator", messages=messages)
    except RateLimitError:
        # Every fallback exhausted — hand off to humans or queue
        if is_urgent(messages):
            return {"role": "assistant",
                    "content": "I'm connecting you with a human agent now. "
                               "Please hold — someone will be with you shortly."}
        enqueue_for_later(messages)
        return {"role": "assistant",
                "content": "We're experiencing high demand. Your request has been "
                           "queued and we'll follow up within 2 hours via email."}

Step 4: Optimize with Model Routing, Caching, and Validation​

With visibility into costs and guardrails in place, the last step is optimization. Not every query needs the most powerful (and expensive) model. Simple FAQ lookups and order status checks work just fine on Claude Haiku 4.5, while complex refund reasoning genuinely benefits from GPT-5.1.

MLflow AI Gateway's traffic splitting lets you test this hypothesis without rewriting agent code. On the orchestrator endpoint, open Priority 1 (Traffic Split), click Add Model, and assign weights that sum to 100% (e.g. 70% GPT-5.1 and 30% Claude Haiku 4.5). The gateway updates with zero downtime, so you can dial the split up or down as evaluation results come in.

We started by routing 30% of orchestrator traffic to Claude Haiku 4.5 and monitored quality through MLflow's evaluation traces. When we confirmed no degradation on routing accuracy, we shifted to 50/50, then 70% Haiku. The orchestrator's job, classifying which sub-agent to call, turned out to be well within Claude Haiku 4.5's capabilities.

Caching as a complementary lever. Exact match caching and semantic caching (matching queries that are paraphrases of each other) can eliminate redundant LLM calls entirely, particularly for the embedding and FAQ-retrieval paths where customers often ask near-identical questions. The key is to ensure cached responses still appear in your MLflow traces with metadata like cache_hit=true and cost=0, so your dashboards and cost-per-ticket metrics stay accurate. Caching won't replace intelligent model routing, but it compounds the savings: route to a cheaper model and avoid the call entirely when you've seen the question before.

Validate before you commit. Traffic splitting without measurement is just guessing. Before shifting majority traffic to a cheaper model, run a held-out evaluation: take a sample of recent routing decisions, replay them through the candidate model, and measure routing accuracy. We set a threshold of >95% accuracy — if the cheaper model couldn't match that on our evaluation set, we didn't shift further. MLflow's evaluation traces make this straightforward: log the original decision alongside the candidate's output and compare programmatically.

Takeaways​

1. Agent costs are multiplicative, not additive. Every sub-agent, every retry, every guardrail check multiplies the cost of a single interaction. Plan for this from day one, don't wait until the bill arrives.

2. You cannot control what you cannot see. Trace everything. The gateway's auto-tracing made this effortless, we didn't add a single line of observability code beyond mlflow.openai.autolog().

3. Budget limits are a safety net, not a strategy. The real savings come from understanding your cost profile and making targeted optimizations like model routing. Budget limits just prevent the worst-case scenario.

4. MLflow's gateway turns cost control from reactive firefighting into a continuous operational loop. Visibility feeds optimization, optimization changes the cost profile, and budget policies catch anything unexpected, all through a single control plane.

For more on MLflow AI Gateway, see:

• Introducing MLflow AI Gateway
• Control LLM Spend with Budget Alerts and Limits
• Your Agents Need an AI Platform

The problem is that finding actionable insights from massive traces is a needle-in-a-haystack problem. From thousands of production logs, how do you spot conversations where users got frustrated or dropped out? How do you catch an agent that silently returns wrong answers with high confidence and misleads users? Manually reviewing traces one by one doesn't scale.

Today we're announcing Automatic Issue Detection in MLflow, a new AI-driven Insights feature that replaces hours of manual check and triage to just 3 clicks.

Why Do You Need Automatic Issue Detection?​

As LLM applications grow in production, maintaining agent quality becomes increasingly challenging. Four problems compound as traffic scales:

• Manual review doesn't scale: Inspecting individual traces one by one is unsustainable as request volume grows.
• Unclear criteria: It's hard to know which quality dimensions to measure without predefined baselines to start from.
• Scattered failure patterns: Related failures spread across thousands of traces with no systematic grouping, making recurring issues easy to miss.
• Unstructured tracking: Without formal issue management, identified problems disappear into notes and Slack threads—and regressions go undetected.

Automatic Issue Detection moves teams from reactive, manual debugging to proactive, systematic quality identification.

Getting Started​

Issue Detection is built into the MLflow UI and runs directly against the traces you've already collected.

How It Works​

The CLEARS Framework​

MLflow organizes issue detection across six quality dimensions, forming the CLEARS framework (Correctness, Latency, Execution, Adherence, Relevance, Safety). Choose which categories to focus on based on your application's requirements:

You choose which CLEARS categories matter for your use case. A customer support bot might prioritize Adherence and Safety; a code assistant cares most about Correctness and Execution.

The Detection Pipeline​

Once you start an analysis run, MLflow:

1. Samples and analyzes traces using an LLM of your choice (via MLflow AI Gateway or a direct API connection)
2. Clusters related problems so you see patterns, not just a flat list of individual failures
3. Annotates the source traces with specific findings so you can drill straight to the evidence
4. Generates a summary of key findings, severity distribution, and recommended next steps

The analysis runs asynchronously with real-time progress tracking, so you can kick it off and come back to the results.

Issue Triage​

Detected issues aren't fire-and-forget alerts. Each one can be moved through a structured lifecycle:

• Pending — newly surfaced, needs review
• Resolved — fix has been deployed and verified
• Rejected — investigated and determined not to be a real problem

You can also edit issue descriptions and adjust severity ratings as your team learns more. This gives you a living record of your application's quality history, not just a snapshot.

Summary​

Automatic Issue Detection is available today in MLflow 3.11.1+ as part of MLflow AI Insights. To get started:

1. Trace your LLM application with MLflow to collect the data Issue Detection needs.
2. Detect Issues by selecting your CLEARS categories and running analysis against your traces.
3. Triage and resolve findings through the structured issue lifecycle—Pending, Resolved, or Rejected.

If you find this useful, give us a star on GitHub: github.com/mlflow/mlflow ⭐️

Have questions or feedback? Open an issue or join the Slack channel.

What is OpenHands?​

OpenHands is an open-source platform for building AI agents that interact with code, terminals, file systems, and the web. It ships as both a Python SDK and a CLI, making it easy to script autonomous workflows that go far beyond chat-based code assistants.

Key capabilities include:

• Terminal access - agents run shell commands in a sandboxed environment
• File editing - agents read, create, and modify project files
• Web browsing - agents navigate and extract information from the web
• MCP integration - agents connect to external tools via the Model Context Protocol
• Model-agnostic - works with Claude, OpenAI, and open-source models

Because OpenHands agents run autonomously, no human is watching every step in real time. That makes observability and governance critical. You need to know what happened, what tools fired, how many tokens were spent, and whether the output actually meets your bar. This is where MLflow comes in.

Trace OpenHands Agents via OpenTelemetry​

When an agent runs autonomously, having visibility over what it did is important. Which tools fired? How many LLM calls were made? Where did the tokens go? Without this information, debugging failures and optimizing performance is guesswork.

MLflow Tracing is a powerful tool for adding observability to any agents with minimal setup. OpenHands emits OpenTelemetry traces natively, and MLflow accepts them out of the box. To connect OpenHands and MLflow, first start your MLflow server if you haven't already.

uvx mlflow server

Then set the following three environment variables in your OpenHands script to connect the two:

import os

# Point OpenTelemetry traces to your MLflow server
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://localhost:5000"
os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = "x-mlflow-experiment-id=123" # Replace "123" with your MLflow experiment ID
os.environ["OTEL_EXPORTER_OTLP_TRACES_PROTOCOL"] = "http/protobuf"

With these set, here is a script that creates an OpenHands agent, gives it a simple task, and sends every trace to MLflow:

from openhands.sdk import LLM, Agent, Conversation, Tool
from openhands.tools.file_editor import FileEditorTool
from openhands.tools.task_tracker import TaskTrackerTool
from openhands.tools.terminal import TerminalTool

llm = LLM("openai/gpt-5")


agent = Agent(
    llm=llm,
    tools=[
        Tool(name=TerminalTool.name),
        Tool(name=FileEditorTool.name),
        Tool(name=TaskTrackerTool.name),
    ],
)

cwd = os.getcwd()
conversation = Conversation(agent=agent, workspace=cwd)

conversation.send_message("Write 3 facts about the current project into FACTS.txt.")
conversation.run()
print("All done!")

Once the agent finishes, every LLM call, tool invocation, and agent step appears as structured spans in MLflow. Navigate to the MLflow UI (e.g. http://localhost:5000), select the experiment set in the environment variables, and open on the "Traces" tab to view the recorded traces.

Clicking on the ID of the trace will open a new pane with more details as shown below.

From this view, you can see a detailed information about the OpenHands agent, such as

• The prompt and response from the LLM
• Tools and the system prompt given to the agent
• The tools actually invoked by the agent and their inputs and outputs
• The latency of the each step
• The tokens usage breakdown and corresponding costs

These are extremely helpful for debugging and optimizing the OpenHands agent.

Evaluate OpenHands Agent Runs​

Tracing tells you what happened. Evaluation tells you whether it was good. These are two complementary angles: observability and quality assurance. Did the agent produce the right result? Is the generated code correct? Is the output relevant to the original request? Was there any inefficient tool calls?

MLflow Evaluation is a comprehensive toolkit for agent developers to assess the quality of their agent in a systematic way. It provides 60+ built-in scorers and LLM judges, integration with popular libraries like RAGAS, DeepEval.

The quickest way to evaluate OpenHands agent runs is to directly run the built-in LLM judges from the MLflow UI.

1. Select traces you want to evaluate from the "Traces" tab.
2. Click "Run Judges" button in the actions menu.
3. Select the judges you want to run and submit.
4. MLflow shows the results once the judges are finished on the same page with new columns added to the table.

In the following example, we use three scorers to evaluate the output quality.

• RelevanceToQuery: Whether the output is relevant to the original request
• Correctness: Whether the output is correct
• ToolCallEfficiency: Whether the tool calls are efficient

Each trace is assessed against the scoring criteria, producing clear pass/fail indicators and detailed rationale from the LLM judge. This is especially powerful for agents that run unattended. You define the quality bar, and MLflow tells you whether the agent cleared it.

Alternatively, you can run evaluation programmatically with the mlflow.genai.evaluate() API as shown below.

import mlflow
from mlflow.genai.scorers import RelevanceToQuery, Correctness, ToolCallEfficiency

# Load OpenHands traces from MLflow
traces = mlflow.search_traces(locations=["123"], max_results=10) # substitute "123" with your experiment ID

# Evaluate output quality
results = mlflow.genai.evaluate(
    data=traces,
    scorers=[
        RelevanceToQuery(),
        Correctness(),
        ToolCallEfficiency(),
    ],
)

The link to the evaluation results will be displayed in the console.

Governance for LLM Traffic with AI Gateway​

One pitfall in highly autonomous agents like OpenHands is that they come with their own set of risks. They might burn excessive tokens for long-running tasks, leak sensitive data to LLM providers, or get compromised by prompt injection attacks.

AI Gateway is a centralized proxy layer that sits between your application and LLM providers. It routes requests through a single, unified API while managing credentials, tracking usage, enforcing governance policies, and providing complete observability across all LLM calls. Instead of scattering API keys across scripts and environments, you configure credentials once in the gateway and let it handle authentication, cost tracking, and policy enforcement for every request.

Using MLflow AI Gateway with OpenHands is extremely easy. Just set the base_url to the AI Gateway endpoint URL (you can find it in the MLflow AI Gateway UI -> Endpoints -> Use).

llm = LLM(
    base_url="http://localhost:5000/gateway/mlflow/v1", # MLflow AI Gateway endpoint URL
    model="my-openai-endpoint", # Set the model to the name of the endpoint you want to use
)

That's it! Now all LLM traffic from OpenHands is routed through MLflow AI Gateway. It gives you governance and observability for free, for example,

• Budget control - set a budget policy to alert or reject when the spending exceeds a threshold
• Usage tracking - every LLM call is logged automatically with token-level cost visibility
• Secret management - store API keys securely in the centralized gateway and never expose them to your scripts
• Fallback routing - define fallback chains to route requests to a backup provider if the primary one is unavailable

Summary​

OpenHands agents are powerful but autonomous, which means you need robust observability to trust and improve them. MLflow provides that in three layers as an AI Platform:

1. Trace every agent run with three environment variables. See tool calls, LLM interactions, and token counts as structured, searchable spans.
2. Evaluate output quality and tool effectiveness with built-in and custom scorers. Catch regressions before they reach production.
3. Govern LLM traffic through AI Gateway for cost control, usage tracking, and centralized credentials management.

Both projects are fully open source, and we expect deeper integrations as the ecosystem matures.

If you find this useful, give us a star on GitHub: github.com/mlflow/mlflow⭐️

Have questions or feedback? Open an issue or join the Slack channel.

You wrote a Claude Code skill: a SKILL.md file that extends Claude with a new capability. You tested it manually and it looks right. But how do you know it reliably works?

The problem is fundamental: a skill guides LLM behavior, and LLM behavior is inherently unpredictable. You can't assert output == expected_output. You need to observe what Claude did: which tools it called, what steps it took, whether it made the right judgment calls. Doing this manually means keeping a record of every Claude interaction and verifying outcomes case by case — time-consuming, and you have to repeat it every time the skill changes.

Here's the loop we built to solve this:

1. Trace Claude Code's own execution with MLflow while it runs the skill
2. Judge those traces with checks that verify correct behavior
3. Refine the skill based on failing judges, automatically, with Claude Code itself

If this sounds familiar, it should. It mirrors what software engineers do when fixing bugs with Claude Code: write unit tests that express the expected behavior, then ask Claude to refine the code until all tests pass. We apply the same pattern here, but the "code" being refined is a skill.

This is also the methodology we use to develop and refine our Claude Code Skills for MLflow.

What Is a Claude Code Skill?​

A skill is a markdown file with YAML frontmatter that Claude Code reads before acting. The description field tells Claude when to load it, and the body progressively provides instructions on how to execute the skill, including examples and tool guidance.

Here's the frontmatter from the agent-evaluation skill in the MLflow Skills repo, which we will use as our running example:

---
name: agent-evaluation
description:
  Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing
  LLM agent's output quality ... Evaluates agents systematically using MLflow
  evaluation with datasets, scorers, and tracing. IMPORTANT - Always also load
  the instrumenting-with-mlflow-tracing skill before starting any work.
allowed-tools: Read, Write, Bash, Grep, Glob, WebFetch
---

This skill guides Claude through the full evaluation workflow: run the agent to understand its behavior, select appropriate quality scorers, prepare an evaluation dataset, and execute mlflow.genai.evaluate() to get a systematic quality assessment.

The body is a complete walkthrough: discover the agent structure, set up tracing, select LLM scorers, create an evaluation dataset, and run mlflow.genai.evaluate(). It's authoritative guidance, and whatever Claude reads here shapes every decision it makes.

This is what makes skills hard to test: there is no output to compare against. Going back to the agent-evaluation skill, the question "Did Claude discover the agent's entry point before trying to evaluate it?" cannot be checked with assertEqual.

Example: Testing and Improving a Claude Code Skill with MLflow​

We'll use the agent-evaluation skill as a concrete example — all the test code lives in the tests/ directory of the skills repository.

Before diving into the details, here's how the pieces of our methodology fit together.

The test harness orchestrates environment setup, headless Claude Code execution, and judge evaluation in a single reproducible run.

A test harness runs Claude Code headlessly against a target project with the skill installed. MLflow traces every action Claude takes during the session, e.g., file reads, shell commands, API calls, tool calls, to name a few. After Claude finishes, a set of judges runs against those traces to check whether Claude executed the skill correctly.

Each judge evaluates one specific aspect of the trace: whether a particular artifact was created, whether Claude followed the right sequence of steps, whether it invoked the right tools. If all judges pass, the skill worked as intended. If any judge fails, the rationale points directly at what went wrong.

This gives us a reproducible, observable check of skill behavior that requires no human in the loop. The following sections walk through each component in detail.

Tracing Claude Code with MLflow​

MLflow ships with built-in support for tracing Claude Code itself. A single command instruments every session in a project directory:

mlflow autolog claude /path/to/project \
  --tracking-uri http://127.0.0.1:5000 \
  --experiment-id 42

From that point on, every tool call Claude makes (reading a file, running a shell command, calling the Claude API) becomes a span in a trace. The trace is a ground-truth record: not what Claude said it did, but what it actually did, in order, with full inputs and outputs.

Writing Judges​

A judge is a check that verifies a specific aspect of how Claude executed the skill. Judges are implemented as MLflow scorers: each receives a Claude Code trace and returns Feedback with a value and rationale. Two patterns cover almost every test:

LLM judge

Use make_judge() to semantically analyze the trace:

from mlflow.genai.judges import make_judge
from typing import Literal

agent_ran_instrumented_code = make_judge(
    name="agent-ran-instrumented-code",
    instructions=(
        "Examine the {{ trace }} and determine whether the agent ran the "
        "application or agent code after adding MLflow tracing instrumentation. "
        "Look for evidence that the agent executed the instrumented program "
        "(e.g., running a CLI command, calling an entry point, executing a script). "
        "Return 'yes' if the agent ran the code after instrumenting it, 'no' otherwise."
    ),
    feedback_value_type=Literal["yes", "no"],
)

This judge reads the actual span tree and reasons about whether the sequence of actions was correct. No rule can do that.

Rule-based judge

Check a side effect in the final testing environment which was modified by the execution of the skill (see dataset_created.py):

from mlflow import MlflowClient
from mlflow.entities import Feedback
from mlflow.genai.scorers import scorer

@scorer(name="dataset-created")
def dataset_created(trace) -> Feedback:
    client = MlflowClient()
    datasets = client.search_datasets(experiment_ids=[eval_exp_id])
    if datasets:
        return Feedback(
            value="yes",
            rationale=f"Found {len(datasets)} dataset(s) in experiment {eval_exp_id}",
        )
    return Feedback(
        value="no",
        rationale=f"No datasets found in experiment {eval_exp_id}",
    )

This judge does not evaluate Claude's internal behavior, but rather it checks whether Claude created the artifact we expected. You can browse all the judges for the agent-evaluation test in the tests/judges/ directory.

Both types are needed: LLM judges handle behavioral and sequential questions, while rule-based judges provide deterministic checks on observable side effects.

Going back to our running example, the full test for agent-evaluation uses six judges, each checking one requirement:

• dataset-created: did Claude call mlflow.genai.datasets.create_dataset()?
• scorer-registered: did Claude register a scorer before evaluation?
• evaluation-run-created: did mlflow.genai.evaluate() produce a run?
• agent-trace-logged: did the agent under evaluation produce traces?
• tracing-skill-invoked: did Claude load the tracing skill as instructed?
• agent-eval-skill-invoked: did Claude actually read and follow the skill?

Each judge is asking whether Claude followed the skill's workflow — did it create the expected artifacts, follow the right sequence of steps, and invoke the right tools? Together they define the acceptance criteria for the skill. If all six pass, the skill works.

The screenshot below shows an MLflow trace from a real agent-evaluation skill run. The left panel shows Claude's span tree — the sequence of tool calls it made, starting with loading the skill. The right panel shows all six judges passing, with the rationale for each LLM judge visible inline.

The MLflow trace viewer showing Claude's tool call sequence (left) and the six judge assessments (right). The agent-eval-skill-invoked judge's rationale explains exactly why the trace passes: Claude loaded the skill and followed its instructions.

Running the Tests​

A YAML config ties the methodology together. You can see the full config for this example at tests/configs/agent_evaluation.yaml:

name: "agent-evaluation-test"
project_dir: mlflow-agent

setup_script: tests/scripts/setup_agent_eval.py
skills:
  - agent-evaluation
  - instrumenting-with-mlflow-tracing

prompt: "Evaluate the output quality of my agent. Do not ask for input."
timeout_seconds: 900
allowed_tools: "Bash,Read,Write,Edit,Grep,Glob,WebFetch"

judges:
  - tests/judges/dataset_created.py
  - tests/judges/scorer_registered.py
  - tests/judges/evaluation_run_created.py
  - tests/judges/agent_trace_logged.py
  - tests/judges/tracing_skill_invoked.py
  - tests/judges/agent_eval_skill_invoked.py

test_skill.py orchestrates the full sequence:

1. Start a local MLflow server and create two experiments: one for the evaluation work the skill is guiding, and one for Claude's own execution traces
2. Run the setup script (clone the target agent repo, seed test data)
3. Install skills into PROJECT_DIR/.claude/skills/
4. Enable tracing: mlflow autolog claude PROJECT_DIR
5. Run claude -p "PROMPT" headlessly
6. Wait for traces to flush, then run all judges on traces created after step 4

python tests/test_skill.py tests/configs/agent_evaluation.yaml

Sample output:

[PASS] dataset-created on trace tr-abc123: yes
[PASS] scorer-registered on trace tr-abc123: yes
[PASS] evaluation-run-created on trace tr-abc123: yes
[FAIL] agent-ran-instrumented-code on trace tr-abc123: no
       Rationale: Claude added tracing decorators but did not execute the agent
                  afterward. No CLI invocation was found after the instrumentation step.

That last line is actionable. Claude didn't run the agent after instrumenting it, so the skill didn't make this step explicit enough.

A few things worth noting about this setup. The setup script is flexible: it can clone any repository, install dependencies, and seed whatever data the skill needs. And a single skill can be covered by multiple configs with different prompts, letting you verify that the skill triggers (or doesn't trigger) correctly under a range of conditions.

The Automated Refinement Loop​

Here's where it gets interesting: when judges fail, we don't fix the skill manually; instead, we feed the failing trace and judge rationale back to Claude Code and ask it to fix the corresponding SKILL.md.

Here is the conceptual loop:

while judges_fail:
    run test → collect failing judge rationales
    claude -p "Judge '{name}' failed with rationale: '{rationale}'. Fix SKILL.md."
    rerun test

Two real examples from the agent-evaluation skill's history illustrate exactly how this played out.

Example 1: Claude bypassing MLflow entirely

Early runs saw dataset-created and evaluation-run-created both fail. Inspecting the trace revealed why: Claude had created an evaluation/eval_dataset.py file with a hand-rolled evaluation loop, completely bypassing MLflow's APIs. No dataset in MLflow, no run logged. The judges had nowhere to find success.

Claude Code read the trace, saw the custom file creation, and made this addition to SKILL.md:

+## ⛔ CRITICAL: Must Use MLflow APIs
+
+**DO NOT create custom evaluation frameworks.** You MUST use MLflow's native APIs:
+
+- **Datasets**: Use `mlflow.genai.datasets.create_dataset()` - NOT custom test case files
+- **Scorers**: Use `mlflow.genai.scorers` and `mlflow.genai.judges.make_judge()` - NOT custom scorer functions
+- **Evaluation**: Use `mlflow.genai.evaluate()` - NOT custom evaluation loops
+
+**Why?** MLflow tracks everything (datasets, scorers, traces, results) in the experiment.
+Custom frameworks bypass this and lose all observability.
+
+If you're tempted to create `evaluation/eval_dataset.py` or similar custom files,
+STOP. Use `scripts/create_dataset_template.py` instead.

Next run: dataset-created and evaluation-run-created both pass.

Example 2: A missing skill dependency

The tracing-skill-invoked judge kept failing: Claude was attempting evaluation without first loading the tracing skill, even though SKILL.md listed it as a prerequisite. The problem was in the skill's description field, the trigger text Claude reads before loading the skill body. It said nothing about the tracing skill dependency.

One line added to the description:

-description: Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing
- LLM agent's output quality ...
+description: Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing
+ LLM agent's output quality ... IMPORTANT - Always also load the
+ instrumenting-with-mlflow-tracing skill before starting any work.

The tracing-skill-invoked judge has passed on every run since.

Both fixes share the same shape: Claude reads the trace, identifies the gap between what it did and what the judge expected, and makes a targeted edit. The skill author never had to debug by hand.

MLflow is what makes the loop grounded. Claude isn't guessing what went wrong from a vague description in the prompt. It's reading the full span tree: the exact tool calls it made, in order, with timestamps. The diagnosis is direct.

Because multiple test configs can cover the same skill, Claude can run the full suite against its revisions of SKILL.md. This prevents overfitting the skill to any single prompt or test case: a fix that addresses one failing test config must not cause another to regress.

What We Learned​

A few patterns emerged from running this system on the agent-evaluation skill:

Write judges before you polish the skill. The judges are the specification. Writing them first forces you to articulate what success actually means, and often reveals that your initial skill draft was underspecified. A skill that passes all its judges on the first try probably has judges that are too weak.

Traces reveal surprising gaps. The tracing-skill-invoked judge caught cases where Claude attempted evaluation without loading the tracing skill, despite SKILL.md listing it as a prerequisite. The fix was a single sentence in the description field, the text Claude reads before loading the skill body. Without the trace, this failure mode would have been invisible: there's nothing in the output that signals a missing skill dependency.

Both judge types are needed. LLM judges and rule-based judges serve different purposes. LLM judges handle behavioral and sequential questions that no rule can express. Rule-based judges provide deterministic checks on side effects: a dataset was created, a run was logged. Use both.

The rationale is the real value. Feedback.rationale is what makes automated refinement possible. A bare yes/no from an LLM judge gives Claude Code nothing to work with. A rationale like "Claude added tracing decorators but never executed the agent afterward" gives it exactly what it needs to make a targeted, surgical fix.

Get Started​

git clone https://github.com/mlflow/skills.git
cd skills
pip install "mlflow[genai]" pyyaml
export OPENAI_API_KEY=...   # needed for LLM judges
export REPO_URL=https://github.com/your-org/your-agent  # agent repo to evaluate
python tests/test_skill.py tests/configs/agent_evaluation.yaml

The test spins up a local MLflow server, clones a sample agent repo, runs Claude Code headlessly, and prints judge results. Add your own judges to tests/judges/ and reference them in a new YAML config.

Adopting this methodology for your own skill takes minutes: write a YAML config, define your judges, and run the test. From there, you never have to debug a skill by hand — Claude reads the failing trace and fixes the skill itself. The approach scales to any Claude Code skill, whether you're testing a two-step guide or a complex multi-step workflow.

AI safety evaluation frameworks produce log files. When a sample fails or a model scores differently across versions, debugging means opening JSON and reading through message histories, tool calls, and scoring events by hand. There's no experiment tracking across runs, no visual span tree showing what the model did at each step, and no way to compare two evaluations side by side.

inspect-mlflow fixes this. Install the package, set two environment variables, and every evaluation automatically logs hierarchical runs with metrics, execution traces with span-level visibility, and artifacts. No code changes needed.

What Gets Tracked​

The tracking hook creates a parent MLflow run for each evaluation, with a nested child run for each task. Every task run captures the model name, dataset, solver configuration, and generation parameters as MLflow parameters. Per-sample scores log as step metrics so you can see accuracy trends across the dataset. Token usage is aggregated per model, and event counts capture total model calls and tool invocations.

At the end of each task, the hook writes structured artifact tables under inspect/, per-sample results, and the full evaluation log. The screenshot below shows a task run with 17 logged metrics including accuracy, token usage, and event counts:

Eval scores also log as MLflow trace assessments, so they appear directly in the Traces table alongside execution time and status. This makes it possible to sort and filter traces by evaluation outcome. The "match" column below shows the average score (1.0) and per-trace assessment count (3 samples):

Debugging with Execution Traces​

Setting MLFLOW_INSPECT_TRACING=true activates the tracing hook, which maps every evaluation event to an MLflow trace span. The span tree mirrors the eval hierarchy, so you can drill into any sample and see exactly what the model did at each step. The span tree below shows a full evaluation with three samples, each containing solver and scorer spans:

Clicking any span opens its inputs and outputs. The LLM span below shows a model call that consumed 23 tokens and returned "2 + 2 equals 4.":

The tracing hook handles tool-using agents the same way. Define a tool, add it to your solver, and each tool invocation appears as a TOOL span with the function name, arguments, and returned result. The code below is a standard Inspect AI evaluation. With MLFLOW_TRACKING_URI and MLFLOW_INSPECT_TRACING set, all runs and traces log to MLflow automatically:

from inspect_ai import Task, eval
from inspect_ai.dataset import Sample
from inspect_ai.scorer import match
from inspect_ai.solver import generate, use_tools
from inspect_ai.tool import tool

@tool
def calculator():
    """Perform arithmetic calculations."""
    async def run(expression: str) -> str:
        """Evaluate a math expression.

        Args:
            expression: e.g. "47 * 89"
        """
        return str(eval(expression, {"__builtins__": {}}))
    return run

task = Task(
    dataset=[
        Sample(input="Use the calculator to compute 47 * 89.", target="4183"),
        Sample(input="Use the calculator to compute 1024 / 16.", target="64"),
    ],
    solver=[use_tools([calculator()]), generate()],
    scorer=match(),
)

logs = eval(task, model="openai/gpt-4o-mini")

Beyond Tracking​

The package also includes a Scout import for analyzing agent transcripts with Inspect Scout, and a compare_evals() function that aligns samples across two evaluation logs and runs significance tests. See the documentation for details on both.

How It Works​

Inspect AI fires lifecycle hooks during evaluation. The tracking and tracing hooks capture those events and write them to MLflow:

Getting Started​

Install the package and point it at an MLflow server:

pip install 'inspect-mlflow' 'mlflow>=3.0'
export MLFLOW_TRACKING_URI="http://localhost:5000"
export MLFLOW_INSPECT_TRACING="true"
inspect eval my_task.py --model openai/gpt-4o

See the package documentation for configuration options and examples.

Resources​

• MLflow Tracing Documentation
• MLflow Evaluation & Monitoring
• inspect-mlflow on PyPI
• inspect-mlflow documentation
• Inspect AI Extensions page

Provenance​

I (Debu Sinha) built the inspect-mlflow package. The underlying hooks were developed across four PRs to the Inspect AI repository, reviewed and merged by JJ Allaire. Farnaz Kohankhaki (Vector Institute and National Research Council of Canada) contributed autolog support and structured artifact tables. The package is listed on the Inspect AI Extensions page.

Organizations that adopt MLflow across multiple teams usually end up choosing between two operating models. They can run a separate MLflow deployment per team, which keeps resources separated but adds user friction and increases operational cost, upgrade work, and infrastructure duplication. Alternatively, they can share a single deployment, which is simpler to operate but quickly mixes experiments, registered models, prompts, and artifacts from unrelated projects.

Shared deployments also make access control harder to manage. Teams often want broad access to their own resources while still restricting other teams, but per-experiment or per-model permissions do not scale well when new resources are created every day. Artifact storage creates a similar problem: once multiple teams share the same server, administrators need predictable storage boundaries without relying on every client to set artifact locations correctly.

Available starting in MLflow 3.10, workspaces were added to solve this organizational and operational gap. A workspace is an optional organizational layer that scopes resources inside a single MLflow deployment, adds workspace-level permissions, and keeps artifact storage aligned with that boundary.

Workspaces let multiple teams share one MLflow deployment while keeping resources, permissions, and artifact paths grouped by workspace.

What Workspaces Add​

• Centralized organization for experiments, models, prompts, traces, datasets, AI Gateway resources, and their child resources
• Logical isolation so teams can share one server without mixing day-to-day work
• Flexible access control through workspace-level permissions that apply across all resources in a workspace
• Backward compatibility through the reserved default workspace for existing deployments
• Extensibility through pluggable workspace providers and workspace-aware artifact routing

When workspaces are enabled, child resources inherit the workspace of their parent. Runs, traces, metrics, parameters, and artifacts inherit from the experiment. Model versions inherit from their registered model. This means the boundary is applied consistently without changing the normal MLflow client workflow.

How Workspaces Look in Practice​

To enable workspaces, start the MLflow server with a SQL backend, a --default-artifact-root, and the --enable-workspaces flag. File-based backends are not supported, and existing deployments should run mlflow db upgrade before turning the feature on:

mlflow server \
  --backend-store-uri postgresql://user:pass@localhost/mlflow \
  --default-artifact-root s3://mlflow-artifacts \
  --enable-workspaces

An administrator can then create a workspace, optionally choosing a workspace-specific artifact root:

import mlflow

mlflow.set_tracking_uri("https://mlflow.example.com")

workspace = mlflow.create_workspace(
    name="team-a",
    description="Workspace for Team A ML projects",
    default_artifact_root="s3://team-a-artifacts",
)
print(workspace.name)

The UI also exposes workspaces directly, making it easy to browse available workspaces, see the last-used workspace, and create a new workspace from the landing page.

The MLflow UI showing the reserved default workspace alongside user-created workspaces and a workspace-specific artifact root.

Users select a workspace once and then continue using MLflow as usual:

import mlflow

mlflow.set_tracking_uri("https://mlflow.example.com")
mlflow.set_workspace("team-a")

experiment_id = mlflow.create_experiment("forecasting-baseline")

with mlflow.start_run(experiment_id=experiment_id):
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_metric("rmse", 0.81)

REST clients can use the same server by sending an X-MLFLOW-WORKSPACE header:

curl -X POST https://mlflow.example.com/api/2.0/mlflow/experiments/create \
  -H "Content-Type: application/json" \
  -H "X-MLFLOW-WORKSPACE: team-a" \
  -d '{"name": "forecasting-baseline"}'

This is the key design goal: workspaces add organization and policy boundaries without introducing a separate client API or a different deployment model for every team.

Permissions Without Permission Sprawl​

In deployments with authentication enabled, workspace permissions provide a convenient fallback for all resources in a workspace. If a user does not have an explicit resource-level permission, MLflow falls back to the workspace permission for that resource's workspace. This lets administrators grant broad access to a team without managing each experiment, model, prompt, or AI Gateway resource one by one. The same permission levels used elsewhere in MLflow apply at the workspace level: READ, USE, EDIT, MANAGE, and NO_PERMISSIONS.

There is an important nuance here: direct resource permissions and workspace permissions are not identical. A user can have access to a specific experiment in team-a without having workspace- level access to team-a as a whole. In that case, the resource remains accessible, but mlflow.list_workspaces() will not list that workspace for the user. This behavior keeps workspace discovery aligned with workspace-level authorization rather than with isolated exceptions.

Artifact Isolation and Backward Compatibility​

Artifacts are isolated by workspace by default. New experiments created in a workspace use an artifact location under:

<default_artifact_root>/workspaces/<workspace-name>/<experiment-id>

If a workspace sets its own default_artifact_root, new experiments use that root instead. This is useful when different teams need separate buckets, prefixes, or storage policies.

To preserve that isolation boundary, clients cannot set artifact_location directly when workspaces are enabled. The server owns artifact placement so that one workspace cannot accidentally or intentionally write outside its assigned layout.

Existing deployments remain backward compatible through the reserved default workspace. When workspaces are enabled on an existing server, legacy resources are placed in default and keep their stored artifact locations. For installations that already use basic-auth, administrators should set grant_default_workspace_access = true in basic_auth.ini during the transition so existing users can still access those legacy resources.

Flexible Design for Platform Teams​

Some organizations want workspaces to map directly onto existing platform concepts such as Kubernetes namespaces, internal project catalogs, or identity-provider groups. Workspaces were designed for that use case.

MLflow uses a pluggable workspace provider interface, exposed through the mlflow.workspace_provider entry point. The workspace providers documentation covers the plugin model in more detail. The default SQL provider stores workspace metadata in the tracking database, but a custom provider can define how workspaces are listed, resolved, and provisioned. For artifact storage, administrators can start with the default prefix-based layout and only move to custom routing when it is actually needed.

Artifact repositories can also implement for_workspace() to return a workspace-specific instance. This makes it possible to route one workspace to a dedicated S3 bucket, choose credentials dynamically, or apply storage policies that differ by team. If MLflow is serving artifacts, this can also allow a single MLflow deployment to serve artifacts from multiple S3 buckets based on the active workspace.

Workspaces Are Not a Hard Isolation Boundary​

Workspaces provide logical separation and authorization controls inside one MLflow deployment. They are a strong fit for shared infrastructure, but they are not a substitute for fully independent deployments when strict compliance or data-plane isolation is required.

If teams require separate networking, separate credentials at the service boundary, or hard guarantees that all data stays in isolated infrastructure, they should run separate MLflow servers. Workspaces are intended to reduce operational duplication inside a shared deployment, not to replace hard multi-tenant isolation.

Getting Started​

If you want to try workspaces today, start with the docs below:

• Workspaces overview
• Getting started with workspaces
• Workspace configuration
• Workspace permissions
• Workspace providers

The workspaces feature was also covered in the MLflow 3.10 webinar, including the UI flow for creating workspaces and assigning permissions.

If this model fits your deployment, try it on a shared MLflow server and let us know what integration patterns or permission models you still need. Feedback and contributions are always welcome through GitHub Issues and GitHub Discussions.

Consider a customer support agent that looks up order status, processes refunds, and searches your knowledge base. You wire it up, test it on a handful of questions, and it works. Ship it. Then a customer asks for a refund on an order that doesn't exist, and the agent hallucinates a confirmation number. The wrong customer's home address shows up in a response due to a lack of data access controls. A model provider pushes an update, and refund success rate quietly drops from 94% to 71%. A retry loop goes haywire overnight and you wake up to a $12,000 invoice. The agent framework got you to version one. It won't get you to reliability.

Some teams try to solve these problems by stitching together separate tools for tracing, evaluation, prompt management, and LLM access control. But the tools don't share data, creating duplication and silos, and the integration tax grows with each one you add.

For your agents to thrive in production, you need an integrated AI platform with all of the following capabilities:

1. Observability: full visibility into what your agent is doing, step by step
2. Evaluation: reproducible quality measurement across every dimension you care about
3. Version control: versioned prompts and configurations that can be compared, optimized, and rolled back
4. Governance: centralized control over LLM calls, data access, and costs

This article walks through why each one matters and how MLflow brings them together in a unified open source platform.

Observability: You Can't Debug What You Can't See​

Your agent processes 2,000 support tickets a day. A customer complains that the agent told them their refund was processed, but it wasn't. What happened?

Maybe the tool call failed silently. Maybe the LLM hallucinated the confirmation. Maybe the retrieval step pulled the wrong knowledge base article. Maybe the agent called the right tool with the wrong parameters. How do you know? Without observability, all you can do is guess from the final response.

AI platforms provide observability through tracing: a system that records every step your agent takes, so you can replay and inspect any request after the fact. As we'll see, tracing also makes the rest of the platform work: evaluation, version control, and governance all depend on having full visibility into what your agent is doing.

import mlflow

# One line instruments your entire agent
mlflow.openai.autolog()

def handle_ticket(ticket):
    return client.responses.create(
        model="gpt-5",
        input=ticket,
        tools=[
            lookup_order,
            process_refund,
            search_knowledge_base,
        ],
    )

MLflow offers a comprehensive tracing framework that captures the full execution graph of every agent interaction: every LLM call, every tool invocation, every retrieval step, even your custom functions, with inputs, outputs, token counts, and latency. MLflow Tracing is OpenTelemetry-compatible, so it works with any programming language, any agent framework, and any LLM provider. MLflow also offers one-line automatic tracing for more than 30 popular frameworks and providers (LangGraph, OpenAI Agents SDK, CrewAI, AutoGen, Pydantic AI, Google ADK, and many more) that instruments your entire application automatically with just a single line of code.

Evaluation: Prove Your Agent Works Before You Ship It​

Your customer support agent handles refunds, answers product questions, and looks up order status. Before you deploy it, how do you know it performs well enough? And after it's in production, how do you know it's still working well? You can't manually sift through thousands of traces looking for problems. Every change you push risks breaking something that goes undetected because most users don't provide detailed feedback, they just stop using your product. And while it is possible to vibe check that a change in your agent fixes a specific problem, how do you verify that it doesn't cause regressions for other requests?

AI platforms solve this with an evaluation framework: a system that automatically scores your agent's outputs using defined criteria, both before deployment and continuously in production. Think of it like the Swiss Cheese model: each layer has holes, but stack them together and problems can't slip through.

• Deterministic tests catch obvious breakages: did the agent call the right tool? Did the refund amount match the order total? Fast and cheap, but they can't tell you whether an answer actually made sense.
• LLM judges score outputs on dimensions like correctness, safety, and relevance, covering the qualitative criteria that deterministic tests can't.
• Human feedback is the final layer. Domain experts catch subtle issues that neither code nor models reliably detect, and calibrate LLM judges to make sure they're scoring the way your team would.

No single layer is enough. Stack all three and you have coverage that scales.

import mlflow
from mlflow.genai.scorers import (
    Safety,
    Correctness,
    ToolCallCorrectness,
)

# Evaluate your agent against a dataset
results = mlflow.genai.evaluate(
    data=support_tickets_dataset,
    predict_fn=handle_ticket,
    scorers=[
        Safety(),
        Correctness(),
        ToolCallCorrectness(),
    ],
)

MLflow's evaluation framework is built around this layered approach. Deterministic tests validate tool calls and structured outputs. 70+ built-in LLM judges cover qualitative dimensions like correctness, safety, and relevance. And a built-in labeling UI lets you collect human feedback to catch what the other layers miss and calibrate your judges.

Because evaluation is integrated with tracing, you can run scorers against production traces continuously, catching quality issues in minutes.

Version Control: Your Agents Need a Changelog​

Agents have a lot of moving parts, including system prompts, tool definitions, data retrieval configurations, and model parameters. When something changes and quality drops, you need to know what changed, compare it against the previous version, and roll back if necessary.

Most teams struggle with this because of one principal problem: nothing ties a change to its impact on quality. Prompts and configurations often live as hardcoded values in application code, and changes frequently blend together. One engineer rewrites the refund instructions to handle edge cases, another updates the few-shot examples to improve tone, and when refund accuracy drops, nobody can pinpoint which change caused the quality drop because no change has lineage to performance data.

AI platforms solve this with a prompt registry: a versioned store for prompts and configurations where every version has lineage to traces and evaluation results.

import mlflow

# Register a prompt in the registry
mlflow.genai.register_prompt(
    name="customer_support_system",
    template="You are a support agent for {{company_name}}. "
    "You help with order status, refunds, and product "
    "questions. Always verify the order ID before refunds. "
    "Never share customer PII in responses."
)

# Load and use the latest version
prompt = mlflow.genai.load_prompt("customer_support_system")

MLflow's prompt registry links every version of your prompt directly to the traces and performance data it produced, so you can trace a quality drop back to the exact change that caused it. MLflow's Prompt Registry also offers built-in prompt optimization functionality, which automates prompt engineering by using an LLM to generate, test, and select improved prompt versions.

Governance: Your Agent Has No Safety Net​

Your customer support agent has no rate limits. When a retry bug causes a loop, it burns through your API budget before anyone notices, and you wake up to a $12,000 invoice. There's no centralized way to enforce content policies, so the agent can return whatever the LLM generates, including customer PII that should never appear in a chat response. And if your LLM provider has an outage, there's no automatic fallback; your agent just fails.

AI platforms solve these problems with an AI gateway: a centralized proxy between your agent and every LLM provider it calls. An AI gateway handles traffic routing, cost controls, and content guardrails in one place, so they're enforced consistently and never skipped.

from openai import OpenAI

# Point your agent at the gateway instead of OpenAI directly
client = OpenAI(
    base_url="https://your-mlflow-server/gateway/mlflow/v1"
)

# Your agent code doesn't change. The gateway handles:
# - API key management (keys never touch application code)
# - Automatic failover (if OpenAI is down, route to Anthropic)
# - Usage tracking (token counts, costs per endpoint)
# - Rate limiting and budget controls
# - Content guardrails and PII redaction

MLflow's AI Gateway exposes an OpenResponses-compatible API, so you can switch model providers without changing your code. A/B test GPT-5 against Claude by splitting traffic 90/10. Set up usage alerts so a retry loop can't burn through your budget overnight. Enforce PII redaction with guardrails, so sensitive data never reaches users when LLMs misbehave. Because the gateway is integrated with MLflow Tracing, every request automatically becomes a trace with full context: model used, tokens consumed, latency, and whether any guardrails were triggered.

Why You Need a Unified Platform​

You could build your own AI platform by stitching together separate tools. Use Langfuse for tracing, DeepEval for evaluation, DIY tracking of prompts used in production, and LiteLLM for gateway routing. But the overhead compounds quickly. Each tool has its own account, billing, and access controls to manage. Your trace data lives in one system, your evaluation results in another, and your prompt versions in a third — with no single dashboard to see the full picture. And the integration tax adds up fast: your evaluation framework can't access your traces, so you build a pipeline to export data between them; your prompt changes aren't automatically tested against your evaluation suite, so quality regressions slip through; your gateway logs don't connect to your traces, so you can't correlate a cost spike with the agent behavior that caused it. A DIY platform held together with glue code is fragile, expensive to maintain, and less powerful than an integrated one.

When all four capabilities live in one platform, they work better together:

• Traces feed evaluations: judges run directly against production traces to catch quality issues in real time
• Evaluations validate prompts: every prompt change is measured against your evaluation suite before deployment
• Judges act as guardrails: the same LLM judges that evaluate quality can block unsafe or low-quality responses at the gateway before they reach users
• The gateway generates traces: every model interaction is automatically traced with cost, latency, and token usage, giving you full visibility without extra instrumentation

Running production-grade agents efficiently and reliably requires a unified AI platform, rather than a collection of disconnected tools.

MLflow: The Open Source AI Platform for Agents​

MLflow is the only open source AI platform that provides all four of these capabilities (tracing, evaluation, version control, and AI gateway) in a unified offering. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI agents and LLM applications, while controlling costs and managing access to models and data. With over 30 million monthly downloads and backing from the Linux Foundation, thousands of organizations rely on MLflow each day to ship agents to production with confidence.

MLflow integrates with any agent framework, programming language, and LLM provider, including LangGraph, OpenAI Agents SDK, Vercel AI SDK, Claude Agent SDK, Google ADK, Pydantic AI, and many more. For teams that need enterprise-grade support, managed infrastructure, and deeper platform integration, MLflow is also available through vendors including Databricks and Amazon SageMaker.

Getting Started with MLflow​

The two fastest ways to get started are adding tracing to an existing agent or setting up the AI Gateway:

pip install 'mlflow[genai]'
mlflow server

import mlflow

# One line instruments your entire agent
mlflow.openai.autolog()
# or langgraph, Google ADK, etc.

pip install 'mlflow[genai]'
mlflow server

from openai import OpenAI

# Direct your agent's LLM calls to the gateway, which
# handles auth, routing, cost controls, and guardrails
client = OpenAI(
    base_url="https://your-mlflow-server/gateway/mlflow/v1"
)

From there, layer on evaluation, register your prompts, and start monitoring. Each capability builds on the others. Check out the MLflow for Agents and LLMs documentation to learn more.

Your agents can do a lot. Give them the platform to do it reliably.

Resources​

• MLflow for Agents and LLMs documentation
• MLflow Tracing documentation
• MLflow Evaluation documentation
• MLflow Prompt Registry documentation
• MLflow AI Gateway documentation

If you find MLflow useful, give us a star on GitHub⭐️.

One of the most common challenges teams face when scaling GenAI applications is runaway LLM costs. A misconfigured prompt, an unexpected traffic spike, or a forgotten development endpoint can quietly burn through thousands of dollars before anyone notices. Until now, catching these issues required external monitoring or manual review of usage dashboards after the fact.

MLflow AI Gateway now includes budget policies — configurable spending thresholds that alert you or automatically block requests when costs exceed a defined limit. Because budgets are enforced at the gateway layer, they apply consistently across every application and service that routes through the gateway, regardless of which provider or model is being called.

How Budget Policies Work​

A budget policy defines a spending threshold in USD over a recurring time window. When cumulative spend within that window crosses the threshold, the gateway takes one of two actions:

• Alert: Fires a webhook notification while allowing requests to continue. This is useful for visibility without disrupting production traffic.
• Reject: Blocks all subsequent requests with an HTTP 429 response.

Budgets reset automatically at the start of each new time window:

You can scope budget policies to specific workspaces when workspaces are enabled, allowing per-team or per-project spend tracking and enforcement.

Creating a Budget Policy​

Setting up a new budget policy is straightforward from the MLflow UI. Specify the budget amount, time window, action (alert or reject), and optionally scope it to a workspace.

Alert Webhooks​

When a budget threshold is exceeded, the gateway delivers a webhook payload containing key details including the budget policy ID, the configured budget amount, current spend, the time window, target scope, and window start timestamp.

Importantly, the alert fires once per window. Subsequent requests within the same window do not trigger additional webhooks, keeping your notification channels clean.

This makes it straightforward to integrate budget alerts into existing incident response workflows — pipe the webhook into Slack, PagerDuty, or any HTTP-based alerting system.

Tracker Strategies: Local vs. Redis​

Budget tracking needs to maintain running spend totals across requests. The gateway supports two strategies depending on your deployment topology:

Local Tracker​

• Tracks spend in-process with no external dependencies
• Lowest possible latency
• Budget state is not shared across workers or replicas
• Survives restarts via trace backfill

This is the right choice for single-instance deployments or development environments.

Redis Tracker​

• Shares state across all gateway workers and replicas
• Atomic operations ensure race-free budget enforcement
• Requires the MLFLOW_GATEWAY_BUDGET_REDIS_URL environment variable and pip install redis
• Adds a small per-request latency overhead for Redis round-trips

For production deployments running multiple gateway replicas, Redis ensures budgets are enforced globally rather than per-instance.

Getting Started​

Budget policies are managed through the MLflow API and are available to admin users when authentication is enabled. To start using budget policies:

1. Install MLflow with GenAI support:

pip install 'mlflow[genai]'

2. Start the tracking server:

mlflow server

3. Configure budget policies through the API — set your threshold, choose your time window, pick an action (alert or reject), and optionally scope to a workspace.

The MLFLOW_GATEWAY_BUDGET_REFRESH_INTERVAL environment variable controls how frequently policies are re-fetched (default: 600 seconds).

For full configuration details and API reference, see the Budget Alerts & Limits documentation.

Budget policies are part of MLflow's ongoing effort to make AI Gateway a complete governance layer for LLM access. Combined with usage tracking and the observability features already available, teams now have the tools to not only understand their LLM spend but actively control it.

If you run into any issues or have feedback, please file a report on MLflow's GitHub Issues.

⭐ Star us on GitHub — show your support for the project!

An agent doesn't just produce an answer. It makes a plan, picks tools, executes a multi-step workflow, and adapts when steps fail. A correct final answer can mask a flawed plan, redundant tool calls, or broken reasoning along the way. To catch those problems, you need to evaluate what happened inside the execution trace, not just what came out the other end.

The integration adds 10 scorers that bring the Agent GPA framework to MLflow. You pass an MLflow trace, and the scorer reads the full span tree (plans, tool calls, intermediate outputs) to evaluate agent behavior. MLflow already supports trace-based judges and agentic metrics from DeepEval and RAGAS, but with the TruLens integration, MLflow now supports the structured three-dimensional lens (Goal, Plan, Action) developed by the TruLens team at Snowflake.

The Agent GPA Framework​

GPA stands for Goal-Plan-Action, and it evaluates three alignment dimensions in an agent's execution:

Goal-Plan alignment asks: did the agent make a good strategy? An agent that gets the right answer by brute-forcing every available tool has a Goal-Plan problem even if the output looks fine.

• PlanQuality checks whether the plan decomposes the goal into feasible subtasks.
• ToolSelection checks whether the agent picked the right tools for each subtask.

Plan-Action alignment asks: did the agent follow through? Did it skip steps, reorder things, or repeat work?

• PlanAdherence checks whether the agent's actual actions match its stated plan.
• ToolCalling checks whether function calls are valid, with correct parameters and complete inputs.

Holistic alignment looks at the trajectory as a whole.

• LogicalConsistency checks whether each step is coherent with prior context and reasoning.
• ExecutionEfficiency checks whether the agent reached the goal without redundant calls.

On the TRAIL benchmark, GPA judges identify 95% of human-labeled agent errors (267/281), compared to 55% for baseline trace-aware judges that also read the execution trace but lack the structured Goal-Plan-Action decomposition. That 40-percentage-point gap shows that reading the trace alone is not enough. How you structure the evaluation matters.

Pass a trace and the scorer handles the rest. Because agent traces often exceed LLM context windows, TruLens GPA scorers pre-process the trace to reduce its size while preserving key information including the agent plan and tool calls. Under the hood, the integration serializes your MLflow trace to JSON and passes the processed span tree to TruLens' provider, which evaluates each dimension with chain-of-thought reasoning. You get back a score and a rationale explaining what it found.

How Trace Evaluation Catches What Output Evaluation Misses​

Here's a concrete scenario. Say you have a travel-planning agent that should: (1) search for flights, (2) check hotel availability, (3) book both. The agent returns "Your trip is booked!" and it looks correct. But the trace tells a different story:

Span 1: search_flights("NYC", "LAX", "2026-04-01") -> 3 results
Span 2: search_flights("NYC", "LAX", "2026-04-01") -> 3 results  <- duplicate
Span 3: book_flight(flight_id="FL123") -> confirmed
Span 4: search_hotels("LAX", "2026-04-01") -> 2 results
Span 5: book_hotel(hotel_id=None) -> error
Span 6: book_hotel(hotel_id="H456") -> confirmed

Output-only evaluation gives this a pass - the trip got booked. However, trace-level evaluation catches three problems:

• ExecutionEfficiency: redundant flight search (Span 2 duplicates Span 1)
• ToolCalling: book_hotel called with None before retry (Span 5)
• PlanAdherence: the agent booked a flight before searching for hotels

Combining Agent and RAG Evaluation​

You can mix agent trace scorers, RAG scorers, and scorers from other frameworks in a single mlflow.genai.evaluate() call. The trace scorers read the span tree, while RAG scorers like Groundedness extract context from retrieval spans in the trace automatically. All scorers support a model parameter for choosing your LLM provider (OpenAI, Anthropic, or any LiteLLM-compatible provider).

import mlflow
from mlflow.genai.scorers.trulens import (
    Groundedness,
    PlanAdherence,
    ExecutionEfficiency,
)
from mlflow.genai.scorers.phoenix import Hallucination

traces = mlflow.search_traces(locations=["..."])

results = mlflow.genai.evaluate(
    data=traces,
    scorers=[
        # Agent behavior (reads the full span tree)
        PlanAdherence(model="openai:/gpt-5.2"),
        ExecutionEfficiency(model="openai:/gpt-5.2"),
        # RAG quality (extracts context from retrieval spans)
        Groundedness(model="openai:/gpt-5.2"),
        # Content quality (Phoenix)
        Hallucination(model="openai:/gpt-5.2"),
    ],
)

Each scorer runs independently and writes results to the same experiment. Results land in the MLflow assessment table alongside any other evaluation results.

The trace detail view shows the full span tree on the left and TruLens GPA assessments on the right. Expand any assessment to see the chain-of-thought reasoning behind the score.

Getting Started​

To get started, install MLflow and TruLens with the LiteLLM provider. For full API details, see the TruLens scorers documentation.

pip install mlflow>=3.10.0 trulens trulens-providers-litellm

from mlflow.genai.scorers.trulens import PlanAdherence, Groundedness

# Agent trace scorer
scorer = PlanAdherence(model="openai:/gpt-5.2")
feedback = scorer(trace=my_agent_trace)
print(feedback.value)      # "yes" or "no" based on threshold
print(feedback.rationale)  # Chain-of-thought reasoning

# RAG scorer (extracts context from retrieval spans in trace)
scorer = Groundedness(model="openai:/gpt-5.2", threshold=0.6)
feedback = scorer(trace=my_rag_trace)
print(feedback.value)              # "yes" or "no"
print(feedback.rationale)          # Why it passed or failed
print(feedback.metadata["score"])  # 0.85

Resources​

• Third-Party Scorers Overview
• Trace-Based Judges
• Agent GPA Framework (Snowflake Engineering Blog)
• Trace-Aware Agent Evaluation for MLflow (Snowflake Engineering Blog)
• Agent GPA Paper (arXiv)
• TRAIL: Trace Reasoning and Agentic Issue Localization (arXiv)
• TruLens MLflow Integration Documentation
• Introducing DeepEval, RAGAS, and Phoenix Judges in MLflow

Provenance​

I (Debu Sinha) contributed the TruLens integration (PR #19492) to MLflow's open-source third-party scorer framework, adding 10 scorers: 4 RAG metrics and 6 agent trace evaluators based on the Agent GPA framework. The integration went through four review rounds with Samraj Moorjani (Software Engineer at Databricks, MLflow maintainer), with final approval from Avesh C. Singh (Software Engineer at Databricks). It follows the scorer pattern Moorjani established in the DeepEval and RAGAS integrations and extends it to agent trace evaluation, a category that requires reading the full span tree rather than just inputs and outputs.

Josh Reini (TruLens maintainer, Snowflake) reviewed the integration's scorer semantics and validated the trace-aware evaluation behavior. Reini published a companion post on the Snowflake Engineering Blog covering the Agent GPA research and TRAIL benchmark results in depth. A cross-project documentation PR was also merged into the TruLens repository.

Related artifacts:

• Upstream MLflow TruLens PR #19492 (merged)
• TruLens documentation PR #2344 (merged, cross-project)
• Introducing DeepEval, RAGAS, and Phoenix Judges in MLflow (companion blog)

The problem: Vector Search has many tuning knobs (embedding model, chunk size, search mode, reranker, filters), but most teams pick defaults and never measure the impact. Without a benchmark, you're tuning blind.

The approach: Build a ground-truth evaluation set once, then use MLflow's retriever evaluation to benchmark each configuration change with precision@k, recall@k, and nDCG@k — all tracked and comparable in the MLflow Experiment UI.

Key takeaway: Tune one knob at a time, let the metrics decide, and use MLflow to make the results reproducible and shareable. Query-time settings (hybrid, reranker, filters) are often just as impactful as index-time settings — and they're cheaper to test.

Every RAG system depends on retrieval, and RAG is a core tool for most AI agents. If your vector search index returns the wrong documents, your agent will generate wrong answers — confidently. But here's the problem: vector search has too many knobs to tune, and most teams never measure the impact of any of them.

Embedding model, chunk size, ANN vs. hybrid search, rerankers, metadata filters — every vector store exposes these tuning options, whether you're using Databricks Vector Search, Pinecone, LanceDB, Qdrant, or pgvector. Every one of these choices changes what your retriever returns. But without a benchmark, you're flying blind — change a setting, try a few queries by hand, hope it helps.

This post introduces a RAG benchmarking workflow using MLflow that works with any vector store. We run three benchmark rounds — each isolating a single tuning knob — and use MLflow’s precision@k, recall@k, and nDCG@k metrics to measure the impact of every change. Every benchmarking run is tracked in MLflow, so you can compare configurations side-by-side and tune with data, not intuition.

[!NOTE] This guide assumes familiarity with MLflow for Agents and LLM Applications. For beginners, refer to the MLflow retriever evaluation tutorial. Even though the examples here show Databricks Vector Search, Pinecone, and LanceDB, the approach and pattern are general to work for any retriever, regardless of the vector store.

The Benchmark Plan​

We'll run three benchmark rounds, each isolating a single tuning knob:

The code below shows each round implemented against three different vector stores — Databricks Vector Search, Pinecone, and LanceDB — to demonstrate that the MLflow evaluation pattern is the same regardless of the retriever underneath.

Step 1: Build a Ground-Truth Benchmark Set​

In order to benchmark retrieval, you first need a set of queries paired with the document IDs that should be returned, ordered by relevance.

import pandas as pd
eval_data = [
    {
        "query": "How do I get started with DataStack?",
        "expected_docs": ["doc-001", "doc-002", "doc-004"],
    },
    {
        "query": "How to set up API keys and authentication?",
        "expected_docs": ["doc-003", "doc-002"],
    },
    {
        "query": "Stream data from Kafka in real time",
        "expected_docs": ["doc-011"],
    },
    {
        "query": "Track ML experiments and compare model runs",
        "expected_docs": ["doc-020", "doc-024"],
    },
    {
        "query": "Build a dashboard and set up scheduled alerts",
        "expected_docs": ["doc-042", "doc-043"],
    },
    # ... 30-100 queries for production benchmarks
]
eval_df = pd.DataFrame(eval_data)

This benchmark set is your measuring stick. Build it once, use it across every experiment. For production, aim for 50–100+ annotated queries.

Step 2: Write a Retriever for Each Vector Store​

MLflow, as an open source platform, doesn't limit which vector store you use. It just needs a function that takes a query and returns a list of doc IDs. Here's what that looks like for three different providers:

Databricks Vector Search​

from databricks.vector_search.client import VectorSearchClient
vs = VectorSearchClient()
index = vs.get_index(endpoint_name="my-endpoint", index_name="catalog.schema.my_index")
def retrieve_databricks(query: str, top_k: int = 5, query_type: str = "ann") -> list[str]:
    kwargs = {"query_text": query, "columns": ["doc_id"], "num_results": top_k}
    if query_type == "hybrid":
        kwargs["query_type"] = "hybrid"
    res = index.similarity_search(**kwargs)
    return [row[0] for row in res["result"]["data_array"]]

Pinecone​

from pinecone import Pinecone
pc = Pinecone(api_key="your-api-key")
index = pc.Index("my-index")
def retrieve_pinecone(query: str, top_k: int = 5, embedding_fn=None) -> list[str]:
    query_vec = embedding_fn(query)  # your embedding model
    res = index.query(vector=query_vec, top_k=top_k, include_metadata=True)
    return [match["doc_id"] for match in res["matches"]]