Complete documentation
Recognize the required documents without inventing gaps.
How to evaluate an AI solution
AI responds well.But does the work improve?
Compare the solution with the current routine. Look at what it gets right, what needs correcting and how much effort it takes to reach a usable result.
The same task.
Two ways to do it.
Define the input, the expected output and who will use the result. Compare under equivalent conditions, including preparation, checking and correction.
Current approachA person, a rule
A person, a rule
or a tool.
Solution under evaluationAI with the
AI with the
human support it needs.
Quality
Correct results, errors and impact.
Total effort
Do the work, check and correct it.
Feasibility
Cost and conditions of use.
Choose the cases
before seeing the answers.
Include what happens often and what can go wrong. A good average can hide an important failure.
- Common cases
The routine the solution needs to handle.
- Exceptions
Situations that call for a different decision.
- Insufficient information
Cases that require asking for help.
How should you prepare the sample and assess errors?
Record the origin, permission to use the cases, their versions and limitations. Set aside a portion for the final evaluation, separate from the examples used to adjust the solution.
Define in advance what counts as a correct result or an error. Depending on the task, distinguish omissions, unsupported claims, unauthorized access and unnecessary referrals. Consider frequency and severity.
Sample evaluation plan
Check a request
and its attachments.
The solution should identify outstanding issues and state the rule applied. These are criteria to check when comparing it with the current review process.
Missing document
Identify what is missing and which rule requires the document.
Unreadable attachment
Refer it for review, without automatic approval.
Compare missed issues, incorrect alerts and review effort. Agree on acceptable limits with the person responsible for the process.
Agree in advance
on when to move forward.
No single pass rate suits every task. The decision depends on the value observed and the impact of errors.
Move forward
The criteria were met within the scope evaluated.
Adjust
The flaws can be corrected and another comparison makes sense.
Stop
The benefit does not justify the cost or risks.
Not enough evidence? Record what is missing before drawing a conclusion.
Who checks the work, and what does checking cost?
Choose reviewers who know the task, give clear instructions and agree on how to resolve disagreements. Measure the time spent checking and correcting. If someone needs to redo the work to trust the answer, include that effort.
What can the test establish?
Separate observed costs and results from estimates of future use. Record the limitations of the sample and where the result has yet to be verified. Moving forward with the evaluation does not mean the solution is ready for production.
Before you start
The plan comes down
to five questions.
- Which task and current approach will we compare?
- Which cases and exceptions belong in the sample?
- Which errors matter, and how will they be judged?
- Who reviews the work, and how will total effort be measured?
- What allows us to move forward, adjust or stop?
References for further reading
Google Cloud · Generative AI evaluation describes criteria, metrics and the analysis of scores alongside individual responses.
NIST AI RMF 1.0 · Measure 2.5 calls for documenting the limits of generalization.
Talk to Gaulett
Which solution do you
need to put to the test?
Bring the task, the question and the examples available. Let’s discuss what to compare to decide on the next investment.