How to Evaluate AI Agents Before Deployment

An AI support agent says it has resolved a ticket. The reply sounds helpful, but the ticket is still open. A demo that checks only the conversation could count this as a success. The customer will discover the failure later.

Agent evaluation turns a task into a repeatable experiment: prepare the starting conditions, run the agent, and inspect evidence against explicit success criteria. This guide shows how to build a small evaluation suite and a Python grader before giving an agent access to a real workflow.

Define success before choosing a score

Start with one workflow. For a support agent, an example task might be: “The authenticated customer has confirmed that the issue is fixed. Mark ticket DEMO-42 resolved.”

Write down the expected result and the conditions that permit it:

These criteria describe our example application. Your workflow may require different checks, such as verifying that a generated file opens or that a booking exists.

Anthropic’s guide to agent evaluations distinguishes the execution transcript from the outcome in the environment. That distinction explains why an agent’s statement that it completed a task needs independent verification.

Inspect the steps that matter

An agent’s trajectory is the sequence of steps it takes during a task. Inspecting tool selection and arguments helps locate failures that a final-answer score can hide. LangChain’s evaluation documentation separates final-response, single-step, and trajectory evaluations.

For our ticket example, record whether the write targeted DEMO-42 and whether the tool returned success. If your application requires a permission check before a write, verify that ordering too.

Avoid prescribing every incidental step. An extra read could be a sensible recovery after a timeout. LangChain’s documentation also notes that exact trajectory matching can reject alternative correct paths. Prefer requirements tied to actual workflow constraints.

Build a small set of realistic cases

For an initial suite, I would use a few dozen carefully reviewed cases covering the workflow’s main branches. The number is a starting point for development, not evidence of production readiness.

Include situations where the right action is to stop or ask for information:

• Normal completion: the customer confirms resolution and supplies a valid ticket identifier.
• Missing identifier: the agent asks which ticket to update.
• Unresolved issue: the agent continues troubleshooting instead of closing the ticket.
• Tool failure: the write fails, and the agent reports that the update did not complete.
• Repeated request: an already resolved ticket does not trigger duplicate downstream work.
• Wrong account: access controls reject another customer’s ticket, and the agent does not claim success.
• Conflicting context: an old conversation summary says the issue was fixed, but the latest message says it persists.

The last case connects to agent memory versus RAG: remembered context needs to be tested against current evidence.

Give every case its own starting state. Reusing a ticket that a previous trial already resolved can make a broken agent appear effective. Keep the expected outcome and grading rules outside the agent’s writable environment.

Run a minimal Python outcome grader

The following example demonstrates grading with two hand-written fixtures. Both runs contain the same confident reply. Only one includes a successful tool operation and the expected final state.

Save this as grade_agent_runs.py. It uses Python’s standard library and makes no model or network calls.

import json


def grade(run):
    target = "DEMO-42"
    successful_write = any(
        call["name"] == "resolve_ticket"
        and call["args"].get("ticket_id") == target
        and call["result"].get("ok") is True
        for call in run["tool_calls"]
    )
    target_resolved = (
        run["after"].get(target, {}).get("status") == "resolved"
    )
    other_ids = (set(run["before"]) | set(run["after"])) - {target}
    unrelated_unchanged = all(
        run["before"].get(ticket_id) == run["after"].get(ticket_id)
        for ticket_id in other_ids
    )
    checks = {
        "successful_write": successful_write,
        "target_resolved": target_resolved,
        "unrelated_unchanged": unrelated_unchanged,
    }
    return {"passed": all(checks.values()), "checks": checks}


before = {
    "DEMO-42": {"status": "open"},
    "DEMO-99": {"status": "open"},
}
runs = [
    {
        "id": "completed",
        "reply": "Ticket DEMO-42 is now resolved.",
        "before": before,
        "after": {
            "DEMO-42": {"status": "resolved"},
            "DEMO-99": {"status": "open"},
        },
        "tool_calls": [
            {
                "name": "resolve_ticket",
                "args": {"ticket_id": "DEMO-42"},
                "result": {"ok": True},
            }
        ],
    },
    {
        "id": "claimed_only",
        "reply": "Ticket DEMO-42 is now resolved.",
        "before": before,
        "after": before,
        "tool_calls": [],
    },
]

passed = 0
for run in runs:
    result = grade(run)
    passed += int(result["passed"])
    print(run["id"], json.dumps(result, sort_keys=True))
print(f"Passed: {passed}/{len(runs)}")

Run it with:

python3 grade_agent_runs.py

The completed fixture passes. The claimed_only fixture fails the write and outcome checks. The final line is:

Passed: 1/2

This result demonstrates the grader’s behavior; it is not a measurement of a real agent. The example also assumes that the task’s authorization and starting conditions have already been established.

To evaluate your agent, replace the fixtures with records captured by a trusted test runner. Read before and after directly from the test store, and collect tool calls at the execution boundary. Letting the agent supply its own alleged database snapshot would defeat the outcome check.

This grader covers three narrow checks. It does not assess the reply, permission handling, transient side effects, or external notifications. Add separate checks for those requirements when they apply.

Use a model judge for questions that need judgment

Code can verify a ticket identifier or state change. Reviewing whether an explanation is clear and supported by evidence needs a different rubric.

For our example, a reviewer could check whether the final reply accurately states the observed status, avoids promising a failed update, and gives a useful next step when a tool fails. A model judge can assist with those judgments, provided its decisions are checked against human-reviewed examples.

Anthropic describes code-based, model-based, and human grading as complementary approaches, and notes that model graders need human calibration. Keep a failed required state check as a failure even if the wording receives a high quality score. That is a deliberate rule for this workflow.

Compare versions across repeated trials

Run each real-agent case several times from a clean state. Anthropic recommends multiple trials because model outputs can vary between runs. Record the model version, prompt revision, tool configuration, and dataset version so that comparisons remain interpretable.

Track task completion and required constraint violations separately. Also record latency, tool calls, and resource use. A higher average completion rate can conceal a newly introduced wrong-account write.

For an initial comparison, use the same cases and trial counts for the current agent and the proposed change. Inspect failures by case, including tasks that the current version already handles. Keep some reviewed cases out of prompt tuning to check whether improvements extend beyond the examples used during development.

Decide what must pass before deployment

For this support workflow, a practical release review would require:

1. Correct outcomes on the required regression cases.
2. No observed writes to unrelated or unauthorized tickets.
3. Accurate reporting when a tool fails.
4. Acceptable completion time and resource use for the intended workload.
5. Human review of unexplained failures and a way to disable the workflow if live behavior deteriorates.

Choose acceptance thresholds before comparing candidates. Passing a small suite provides evidence about those tested conditions; production monitoring must still check how the workflow behaves with real requests.

Start by testing one task’s answer, actions, and resulting state. A small suite with trusted evidence makes failures easier to diagnose and gives each prompt, model, or tool change a concrete standard to meet.