Evaluating AI Output Quality (LLM Evals): How to Test Chatbots and AI Agents Before and After Release

Many AI projects stall at an impressive demo. The chatbot answers three sample questions well, everyone applauds, then the system is released to customers and turns out to answer with confidence but wrongly, invent policies that do not exist, or change behavior after a model update. The cause is almost always the same: there is no measurable way to know whether the AI system is good enough before and after release.

Evaluating the quality of AI output, often called LLM evals, is the practice of measuring that systematically. For a business, evaluation is not a technical luxury. It is the guardrail that decides whether an AI feature is safe to hand to customers, employees, or your financial processes. This article covers what to measure, how to build a test set from real data, scoring methods, and how to monitor quality after release.

Summary

• A good demo is not proof of quality: without a fixed test set you cannot tell whether a prompt or model change made results better or worse
• The best test sets come from real questions, documents, and edge cases in your business, labeled by people who understand the domain
• Use a mix of automated scoring, scoring by another model, and human review, each for a different kind of quality
• Set pass thresholds before release, run evaluations whenever the prompt, model, or data changes, and monitor quality in production
• Start with 50 to 100 representative test cases, not an expensive evaluation platform

Why AI cannot be tested like ordinary software

In conventional software, testing is relatively easy: for the same input the expected output is the same, so one automated test either passes or fails. Large language models work differently. The answer is free text, can be correct in many wordings, and can vary slightly on every run. The same question phrased with slightly different words can produce answers of different quality.

So AI testing does not ask whether the output is identical, but whether it meets the quality criteria we set: factually correct, consistent with the sources, polite, free of data leaks, and formatted as needed. This shift from a small testing chore to a discipline of its own is what gave rise to the practice of evaluation.

The risks are real for a business. An assistant that misstates a price or warranty term can cause a dispute. A system that summarizes a contract while missing one important clause can mislead a decision. And a system that is good today can degrade tomorrow because the model provider updated its version without a notice you noticed. Without evaluation, these problems only become visible after customers complain.

Defining what quality means

Before building a test set, first define the quality dimensions that matter for your use case. Not all dimensions are equally important, and stating them explicitly prevents endless debate.

• Factual accuracy: whether the answer is correct according to the company's official sources
• Groundedness: whether the answer uses only information from the supplied documents, not the model's inventions
• Completeness: whether important points are not missed
• Format compliance: whether the output matches the structure other systems need, for example valid JSON
• Safety and compliance: whether it refuses requests it should refuse and does not leak personal data
• Tone and language: whether it fits the brand voice and the user's language
• Latency and cost: whether it is fast enough and cheap enough per request

For a customer service assistant, factual accuracy and groundedness are usually primary. For a marketing draft tool, tone and creativity matter more than any single fact. For an agent that performs actions in systems, compliance with authority limits is an absolute requirement, a topic discussed further in the article on AI application security, prompt injection, and data leakage.

Building a test set from real data

A test set, often called a golden set or evaluation dataset, is a collection of cases with inputs and expected answers or criteria. Its quality determines the quality of the entire evaluation, so it deserves to be treated like a business asset.

Where test cases come from

The best source is data that really happened. Take real questions from support ticket logs, WhatsApp chat history, the customer service mailbox, or sales team notes. Real questions are usually messier, shorter, and more ambiguous than those invented by the technical team, and that is where systems often fail. If the project has not started yet, ask the team that answers customers most often to write down the twenty questions that appear most frequently and the twenty that are hardest.

Make sure personal data in examples is anonymized before use, in line with the principles of personal data protection. Test sets are often copied into various tools and providers, so it is safer if they are clean from the start.

A balanced composition

A test set containing only easy questions gives a false sense of safety. Build a composition that mirrors reality:

• Common cases that represent most of the traffic, to confirm baseline performance
• Edge cases, such as ambiguous questions, bilingual questions, or questions with typos
• Cases whose answer is genuinely not in the documents, to test whether the system dares to say it does not know
• Risky cases, such as requests to give a discount outside policy or to reveal another customer's data
• Attack cases, such as hidden instructions inside a document uploaded by a user

To begin, 50 to 100 well chosen cases are already very useful. Add a new case every time a production failure is found, so the test set grows with experience and the same failure does not quietly recur.

Labeling by people who know the domain

Reference answers or scoring criteria must be written or approved by people who truly understand the business: customer service staff, accountants, warehouse staff, or an internal legal expert. Labeling left entirely to a vendor often misses company nuances. Also record who labeled and when, because policies change and stale labels produce misleading evaluations.

Scoring methods: a combination beats any single approach

No single scoring method suits every dimension. Practices that work combine three layers.

1. Automated rule-based scoring

Some qualities can be checked with simple code and the result is certain: whether the output is valid JSON, whether it contains required fields, whether its length is within limits, whether it contains forbidden words or a card number pattern. These checks are cheap, fast, and can run on every change. Use as many as possible for anything that can be defined firmly.

2. Scoring by another model (LLM as judge)

For subtler dimensions such as groundedness or clarity, another language model can be asked to grade answers against a written rubric, for example a one to five scale with a definition for each number. This is far more scalable than grading one by one manually, but it has weaknesses: a model judge can be biased toward longer answers, can favor the style of its own model, and is not always consistent. So calibrate the judge by comparing its scoring with human scoring on a small sample, and refine the rubric until agreement is adequate.

3. Human review

Humans remain the gold standard for matters of risk and nuance. Take a random sample of outputs, plus all cases rated poorly by the automated judge, and have reviewers who know the domain grade them. Human review is expensive, so use it in a targeted way: before major releases, after significant changes, and periodically with a small sample in production.

For document retrieval based systems such as a company knowledge chatbot with RAG, evaluation should separate two parts: whether retrieval finds the right documents, and whether the model answers correctly based on them. Separating them speeds up diagnosis, because the fix for a retrieval problem is very different from the fix for an answer problem.

Setting pass thresholds and release gates

Evaluation is only useful if connected to decisions. Before release, set pass thresholds agreed by the business owner, not only the technical team.

• Set a minimum score per dimension, for example factual accuracy of at least 95 percent on common cases
• Set zero tolerance for critical categories, for example no leaking of another customer's data in any single attack case
• Set a regression limit, for example a change must not lower any score by more than two points compared with the previous version
• Decide who is authorized to approve exceptions and how the reason is recorded

Such a gate resembles automated testing in ordinary software development. If you already have a flow of automated testing and CI/CD, AI evaluation can run as an additional step in it: every change to the prompt, model, or knowledge base triggers an evaluation, and the release is held if scores fall below the threshold.

Which changes should trigger re-evaluation

An AI system has more parts that can change than ordinary software, and each change can affect quality.

• Changes to the prompt or system instructions, even seemingly trivial ones such as one extra sentence
• Replacing or updating the model version, including automatic updates by the provider
• Changes to the knowledge base: new documents, revised policies, or deleted old documents
• Changes to settings such as temperature, context length, or the number of documents retrieved
• Adding or changing tools the agent can call
• Changes in user patterns, for example a new product that creates new kinds of questions

Pin the model version where possible, so that changes happen only when you decide and have tested them. Similar considerations arise when choosing between self-hosted LLMs and cloud APIs, because control over the model version differs between the two choices.

Monitoring quality after release

The test set catches problems you have already imagined. Production will reveal problems you have not. So evaluation does not stop at release.

• Record the input, output, retrieved documents, and prompt and model versions for each interaction, with reasonable data protection
• Provide a simple feedback button for users, such as thumbs up or down, then review everything rated poorly
• Take a random sample of interactions every week for human review
• Watch proxy indicators: the rate of escalation to humans, the rate of repeated questions, the share of I do not know answers, and complaints
• Track cost per interaction and response time, because declining efficiency is often an early sign of trouble
• Feed every real failure into the test set so it does not recur

This practice aligns with the principles of observability, logging, and error handling: you cannot fix what you cannot see.

Common mistakes in AI evaluation

• Testing only with questions invented by the development team, so the test set is too easy and does not mirror real customers
• Changing the prompt repeatedly and judging by gut feeling from three examples instead of a fixed test set
• Relying on one average score that hides serious failures in a small but risky category
• Using a model judge without ever calibrating it against human scoring
• Leaking test cases into the prompt or training data so scores look high but are misleading
• Failing to update the test set when policies, prices, or products change
• Treating evaluation as a one-off project rather than an ongoing process

Steps to start in two to four weeks

1. Pick the single most important or riskiest AI use case, not all of them at once.

2. Write the quality definition on one page and agree on it with the business owner.

3. Collect 50 to 100 real, anonymized cases, including edge cases and risky cases.

4. Have domain staff write reference answers or scoring rubrics.

5. Run the current system against the test set to get a baseline score, even if the result is disappointing.

6. Build automated scoring for what is certain, and a calibrated model judge for what is subtle.

7. Set pass thresholds and run the evaluation before every change is released.

8. Turn on logging and feedback in production, then add real failures to the test set.

This approach fits with a company AI usage policy and AI governance: evaluation results become evidence that the AI system is supervised, not merely trusted.

Frequently asked questions

Do we need a dedicated evaluation platform? Not to begin with. A spreadsheet of test cases and one script that calls the system and scores the results is enough to see the value of the practice. A dedicated platform becomes useful once volume and team size grow.

Can scoring by another model be trusted? It can, provided the rubric is clear and results are calibrated against human scoring on a sample. Do not make it the sole decision maker for high risk decisions.

How often should evaluations run? Whenever the prompt, model, data, or tools change, plus a scheduled weekly or monthly check to catch degradation caused by external factors.

Does evaluation guarantee the AI is never wrong? No. Evaluation reduces and measures risk, it does not eliminate it. So for high impact decisions, still provide human review and an escalation path.

Conclusion

Reliable AI is not the result of a clever prompt alone, but of a disciplined measurement process. With a clear quality definition, a test set from real data, a combination of automated and human scoring, pass thresholds before release, and monitoring afterward, a business can use AI with well founded confidence. Start small with one use case and a few dozen real examples. The greatest value appears when every change can be answered with numbers instead of gut feeling.