How to Measure AI Answer Quality

AI answer quality metrics are the small set of numbers that tell you whether a grounded answer is right, not merely well-written — whether it came from your documents, cited the ones it used, and said “I don’t know” when the answer was not in the corpus. Fluency is not on the list, because fluency is free.

Most teams evaluate an AI knowledge base the way they read a well-formatted email: they skim it, it sounds right, they move on. That is exactly the failure mode a grounded system is built to prevent, and skimming defeats it. A model will produce a confident, fluent paragraph whether or not retrieval found the right passage — so a paragraph that reads correct is not evidence of anything.

To know whether an answer is any good, you have to measure the parts a reader cannot see: where the content came from, whether the citation supports it, whether the retrieval even surfaced the right document, and how the system behaves when the answer is not in your corpus at all. This post is the shortlist of metrics that cover those, and how to compute each one on your own knowledge base.

Start from what “quality” means for a grounded answer

A grounded answer has three jobs, and each maps to a different metric. It should draw only from your documents (groundedness). It should draw from the right documents, and enough of them (source coverage). And it should retrieve those documents in the first place (retrieval recall). An answer can pass one and fail the others — fluently citing one real document while inventing two claims it did not support is a groundedness failure that a coverage check misses.

So “quality” is not a single score. It is a small vector, and the point of measuring each dimension separately is that each one tells you which part broke. A correctness number alone tells you an answer was wrong; it does not tell you whether retrieval missed the passage or the model ignored the passage it got. Those need different fixes.

The reason this is checkable at all is citations. In Insulin, every grounded answer returns the source documents behind it, so groundedness is not a judgement call — you open the citation and read the passage. A system that only returns prose forces every metric below to be a subjective vote instead of a measurement.

The metrics that actually tell you something

Here is the shortlist. None of them requires a research budget; every one is computable from a set of test questions you write, the answers the system gives, and the citations attached to those answers.

Metric	What it measures	How to compute it
Groundedness / faithfulness	Whether every claim in the answer is supported by a cited source	For each answer, count claims that a cited passage supports ÷ total claims. Open the citations and check — this is why cited sources make it measurable rather than a vote.
Source coverage	Whether the answer cited enough of the relevant documents, not just one	Of the documents that should inform the answer, count how many were cited ÷ how many exist. A single citation on a four-fact answer is low coverage even if that one citation is correct.
Retrieval recall	Whether the right passage was retrieved at all, before the model wrote anything	Run a test search for each question; count questions where the correct passage appears in the results ÷ total questions. If it never surfaced, no prompt fixes the answer.
Retrieval precision	How much of what was retrieved was actually relevant	Of the passages returned by a test search, count relevant ÷ total returned. Low precision buries the right passage under adjacent ones and dilutes the answer.
Answer correctness	Whether the final answer is factually right against a known-good key	For a set of questions you can grade, count correct answers ÷ total. Grade against your own reference answer, not the model’s confidence.
Citation accuracy	Whether each citation actually says what the answer claims it says	Sample cited claims; count citations whose passage supports the claim ÷ citations checked. A citation to a real-but-wrong passage is an accuracy failure, not a groundedness win.
”I don’t know” rate	Whether the system declines when the answer is outside the corpus	Ask questions you know are not in your documents; count honest abstentions ÷ out-of-corpus questions. A confident answer here is the most dangerous failure of all.

Two of these do more work than the rest. Groundedness is the one that separates a grounded system from a chatbot, and it is only cheap to measure because the citation is right there. The “I don’t know” rate is the one teams forget entirely — and it is where an ungrounded model does the most damage, because a fluent answer to a question your documents cannot answer looks identical to a correct one.

Measure retrieval separately from the answer

The single most useful move in evaluation is to stop grading conversations and start grading retrieval on its own. An answer is the product of two stages — find the passage, then write from it — and a conversation blends both, so a wrong answer never tells you which stage failed.

A test search splits them cleanly. In Insulin, a test search against a knowledge base returns the matching content, its source document, and a relevance score per result — before any answer is generated. That lets you compute recall and precision directly: ask a question you know the answer to, and check whether the correct passage came back and how strongly the search ranked it. If it did not surface, the problem is retrieval, and no amount of prompt-writing will save the answer. If it surfaced and the answer was still wrong, the problem is downstream, in the model or the instructions.

That relevance score is also the dial you tune against. Because Insulin’s retrieval is hybrid — keyword plus vector — a low recall number on exact identifiers and a low recall number on loosely-worded questions have different fixes: one wants more keyword weight, the other more vector weight. Reading the relevance score is how you tell which, and how you confirm a weight change actually moved the right passage up rather than just feeling better.

Test the out-of-corpus case on purpose

Most evaluation sets are built from questions the documents answer, which measures the system at its easiest and skips the case that hurts. The dangerous question is the one your corpus cannot answer — because a system that always produces something will produce something here too, and it will read exactly like a real answer.

So build a deliberate out-of-corpus set: questions adjacent to your material but not covered by it. The metric is simple — what fraction does the system decline, versus answer anyway — and it is often the number that changes a rollout decision. An assistant that is right on in-corpus questions but confidently wrong on the three questions a week that fall outside it is worse than no assistant, because people cannot tell the two situations apart. Here again the citation is the tell: an answer that should have abstained but did not will either cite nothing or cite something that, when you open it, does not support the claim.

This matters most where the answer travels. An interactive question you asked yourself is low-stakes — you have context and will notice something off. The same grounded system running inside a shared channel or an unattended job produces output that reaches people who were not present to judge it, which is why the evaluation scorecard for agents treats honest abstention and escalation as first-class dimensions, not nice-to-haves.

Frequently asked questions

What are the key metrics for AI answer quality? Groundedness, source coverage, retrieval recall and precision, answer correctness, citation accuracy, and the rate at which the system declines out-of-corpus questions. Fluency is not one of them — a wrong answer reads just as well as a right one.

What is groundedness? Groundedness is the share of claims in an answer that a cited source actually supports. It is measurable because the answer returns its sources: you open each citation, read the passage, and check that it backs the claim.

How do I measure retrieval quality separately from the answer? Run a test search instead of a conversation. In Insulin it returns the matching content, its source document, and a relevance score per result — so you can see whether the right passage was retrieved at all before any answer is written.

Why measure the “I don’t know” rate? Because a system that always produces an answer will answer questions your documents cannot, and that answer looks identical to a correct one. Ask out-of-corpus questions and measure how often it honestly declines.

What is source coverage? Source coverage is how many of the relevant documents an answer cited, out of how many exist. A single correct citation on a four-fact answer is low coverage — the other three facts came from somewhere you did not check.

Does a citation mean the answer is correct? No. A citation makes the answer checkable, not correct — the passage can be real but wrong for the claim. That is why citation accuracy is a separate metric from groundedness and answer correctness.

Takeaways

Quality is a vector, not a score. Measure groundedness, source coverage, retrieval recall and precision, correctness, citation accuracy, and out-of-corpus honesty separately, because each names a different failure.
Fluency tells you nothing. A wrong answer reads as well as a right one, so grade the parts a reader cannot see.
Cited sources are what make groundedness a measurement instead of a vote — you open the citation and check it.
Grade retrieval on its own with a test search and its relevance score, so a wrong answer tells you whether the passage was missed or ignored.
Test out-of-corpus questions on purpose. The “I don’t know” rate is the number a rollout decision most often turns on.

Insulin knowledge bases use hybrid search and return cited sources on every grounded answer, so these metrics are things you can measure rather than estimate. Explore Insulin knowledge bases or get a demo to run a test search on your own documents.

How to Measure AI Answer Quality and Source Coverage