An AI agent evaluation framework is a set of outcome-level acceptance criteria that judge whether an agent did the whole job — not whether one reply read well. Score it on task completion, correctness, policy adherence, escalation, cost, and latency, and measure offline tests and live runs separately.
Most agent evaluations answer the wrong question. They grade the response: was it fluent, was it relevant, did it sound helpful. That tells you the model can write. It does not tell you the agent issued the right credit, applied the correct discount tier, or knew to stop and ask a human when the amount crossed a threshold.
Business work is not a single response. It is a sequence of steps — read the context, use the right tools, make the change, and know where its own authority ends. An agent can produce a beautifully worded answer and still get the job wrong at every one of those steps. So the acceptance criteria that matter are the ones a business owner would recognize: did the work complete, was it correct, did it stay inside policy, did it escalate when it should have, and did it do all of that at a cost and speed you can live with.
This post is that scorecard. It is deliberately vendor-neutral on the metrics — the six dimensions apply to any agent — and it names where an Insulin agent gives you the controls to hold an agent to them.
What is an AI agent evaluation framework?
An AI agent evaluation framework is a defined set of outcome-level acceptance criteria — the specific, checkable conditions an agent must meet to be judged “good enough for real work.” It measures the end-to-end task, not the isolated response, because the response is only one step of the work.
The distinction is the whole point. A developer test suite asks did the model return the expected string. A business acceptance criterion asks did the agent complete the task the way a competent teammate would — using the correct tools, honoring policy, and escalating the edge cases. Both matter, but only the second tells a program owner whether to put the agent in front of customers or the finance close.
Two rules keep an evaluation honest, and both are easy to skip:
- Score the trajectory, not just the ending. An agent can reach the right final answer by a wrong path — guessing instead of retrieving, or skipping an approval it should have hit. That agent will fail the next task that does not happen to guess right. Read what it did, not only what it returned.
- Keep offline results and live performance in separate columns. A curated test set tells you the agent’s ceiling under known inputs. It does not tell you how it behaves on the messy, ambiguous requests real users send. Treat a passing offline suite as a gate to production, never as evidence of production quality.
The scorecard: six dimensions
Score each dimension on its own, because they trade off against each other. Rate each on a simple 1–5 or pass/fail scale per task, then aggregate across a representative task set — one great task tells you nothing.
| Dimension | The question it answers | How to measure it |
|---|---|---|
| Task completion | Did the agent finish the job end to end, not just start it? | Rate of tasks completed without a human finishing them; count of abandoned or half-done runs |
| Correctness | Was the outcome factually and operationally right? | Accuracy against a ground-truth answer key; for grounded work, whether claims trace to cited sources |
| Policy adherence | Did it stay inside the rules and its granted scope? | Rate of actions within allowed tools and policy; count of out-of-scope attempts or violations |
| Escalation | Did it stop and ask a human at the right boundary? | Rate of correct hand-offs; false-negatives (should have asked, didn’t) and false-positives (asked needlessly) |
| Cost | What did the work cost per completed task? | Tokens and tool calls per successful task, and their money cost — measured per task, not per message |
| Latency | Was it fast enough for where it runs? | End-to-end time to a completed task; interactive work and overnight jobs have different bars |
A seventh dimension, user trust, is worth tracking as a lagging indicator: do the people who receive the agent’s work accept it, edit it, or quietly redo it by hand? Trust is downstream of the six above, but it is the signal that tells you whether the numbers translated into work people actually rely on.
Task completion and correctness: the outcome, not the reply
Task completion measures whether the agent finished the whole job; correctness measures whether the finished job was right. Grade them separately, because an agent that completes 100% of tasks incorrectly is worse than one that completes 60% correctly and hands off the rest.
For correctness on grounded work — anything where the agent answers from your documents rather than general knowledge — the strongest signal is traceability. An Insulin agent attached to a knowledge base returns cited sources, so responses are grounded in your company’s information rather than the model’s guesswork. That gives you a concrete correctness check: open the citation and confirm the claim is actually there. An answer with no traceable source is not a correct answer you can trust; it is a plausible one you have to re-verify by hand, which defeats the point.
The habit to build: define the ground truth before you run the evaluation. Decide what “done and correct” looks like for each task in your set, then grade against it. Grading correctness by reading the output and asking “does this seem fine” is how a confident, wrong agent passes.
Policy adherence: scope is the acceptance criterion
Policy adherence measures whether the agent stayed inside the rules and the tools it was actually granted. It is where response-quality evaluations are silent and where business risk concentrates — a fluent agent that reads a record it should never have touched has failed, however good the reply read.
The control that makes this measurable is scoping. An Insulin agent is configured with an allowlist of the integrations it may use, and scope is the boundary — an agent cannot reach data outside its job. That turns policy adherence from a hope into a testable property: grant the deal-desk agent only the systems the deal desk needs, then score how often it operates within that boundary and count any out-of-scope attempts. Because tool execution runs through approval workflows in chat — you see the plan and approve it before anything happens — you can inspect the intended actions during evaluation, not just the after-effects.
Write your policy criteria as explicit lines to check, for example:
- Did every tool the agent invoked sit inside its allowlist?
- Did it respect the role-based limits of the person it acted for?
- On any action that writes or sends, did it route through an approval rather than acting unilaterally?
Escalation: measuring the hand-off
Escalation measures whether the agent stopped and asked a human at the correct boundary — neither barreling through a decision it should not own, nor pestering for approval on trivia. It is a two-sided error: escalating too little is dangerous, escalating too much is useless, and a good evaluation scores both.
This is why the act-versus-ask design matters as a measurable property. With an Insulin agent, tool execution can be held for approval — nothing runs until you approve — so the reversible steps (reading, drafting, planning) run unattended while the irreversible ones (sending, writing) wait for a person. To evaluate escalation, build a task set that deliberately includes edge cases past the agent’s authority — an unusually large amount, an ambiguous request, a missing input — and score whether it hands off on exactly those and proceeds on the routine ones. The same principle governs when an agent should act versus ask: let the safe work run, keep the consequential decision a human choice.
A useful failure count here is the silent one: tasks where the agent should have escalated, didn’t, and produced a confident wrong outcome instead. Those are the runs that erode trust fastest, so weight them heavily.
Cost and latency: measure them per task, not per message
Cost and latency measure the price and speed of completed work, not of individual model calls. The unit is the task: an agent that answers each message cheaply but takes eight tool calls and three retries to finish one job is not a cheap agent.
Two choices keep these honest:
- Right-size the model to the work. With an Insulin agent you pick the model that suits the work — a quick triage agent and a deep analysis agent do not need the same one. So cost and latency are not fixed properties of “the agent”; they are dials you set per use case. Evaluate them against the bar for that use case, not one global target.
- Match the latency bar to where the agent runs. An interactive chat agent lives or dies on responsiveness. An overnight scheduled job can take minutes and nobody notices, so trading latency for a stronger model or more thorough retrieval is often correct there. Grade latency against the surface, not in the abstract.
The run history behind Insulin jobs and agents is what makes cost and latency observable after the fact — each run records what happened, so you measure real completed-task cost instead of estimating from a single call.
Offline tests versus live performance: keep them apart
An offline evaluation runs the agent against a fixed, curated task set with known correct answers; a live evaluation measures the agent on real, in-production traffic. They answer different questions, and reporting one as if it were the other is the most common way agent quality gets overstated.
- Offline is your gate and your regression net. Because the inputs and the ground truth are fixed, an offline suite is repeatable — you can re-run it after every prompt, model, or knowledge-base change and catch a regression before it ships. Its blind spot is that real users do not send curated inputs.
- Live is the only measure of production quality. Real requests are ambiguous, adversarial, and full of cases your test set never imagined. Track the same six dimensions on live runs — completion, correctness, policy, escalation, cost, latency — and treat divergence from your offline numbers as the signal to expand the test set.
State both, side by side, whenever you report a result. “Passes the offline suite at 94%” and “resolves 78% of live requests without a hand-off” are both true and both necessary; presenting only the first is how a demo-ready agent gets mistaken for a production-ready one.
Frequently asked questions
What is an AI agent evaluation framework? It is a defined set of outcome-level acceptance criteria that judge whether an agent did the whole job — not whether one reply read well. Score it on task completion, correctness, policy adherence, escalation, cost, and latency across a representative task set.
Why isn’t response quality enough to evaluate an AI agent? Response quality grades one reply; business work is a sequence of steps. An agent can write a fluent answer and still use the wrong tool, breach policy, or fail to escalate. Evaluate the end-to-end task and the actions it took, not just the text it returned.
How do I measure whether an agent stays inside policy? Scope the agent to an allowlist of integrations and score how often it operates within that boundary, counting any out-of-scope attempts. An Insulin agent cannot reach data outside its granted scope, and tool execution runs through approvals you can inspect.
How should I evaluate an agent’s escalation behaviour? Build a task set that includes edge cases past the agent’s authority and score whether it hands off on those and proceeds on routine work. Escalation is two-sided: weight silent failures — cases where it should have asked, didn’t, and acted wrongly — most heavily.
What is the difference between offline and live agent evaluation? Offline runs the agent against a fixed task set with known answers — a repeatable gate and regression net. Live measures real production traffic, the only true quality signal. Report both side by side; an offline pass is not evidence of production quality.
Takeaways
- Evaluate the end-to-end task, not the isolated response. A fluent reply is one step; the job is completion, correctness, policy, escalation, cost, and latency.
- Score the six dimensions separately — they trade off. An agent that completes every task incorrectly is worse than one that completes fewer correctly and escalates the rest.
- Policy adherence is measurable when scope is an allowlist: an Insulin agent cannot reach data outside its granted scope, and tool execution runs through approvals you can inspect.
- Escalation is two-sided. Test edge cases deliberately, and weight silent failures — should-have-asked-but-acted — most heavily.
- Measure cost and latency per completed task, right-sized to the model and surface — an interactive chat and an overnight job have different bars.
- Keep offline test results and live performance in separate columns. An offline pass is a gate to production, never proof of it.
Ready to hold an agent to criteria like these? See how an Insulin agent is scoped, approved, and grounded, how jobs record every run for after-the-fact measurement, or get a demo.
Stay Updated
Get the latest Cloud GTM insights, product updates, and marketplace strategies delivered to your inbox.