How to Turn Vague AI Ratings into Clear Pass-or-Fail Checks
Turn vague AI ratings into clear pass/fail checks. Practise writing criteria, judging an original example, and explaining mistakes without losing useful nuance.
By AtomicEval Team · 22 October 2026 · 6 min read
“This answer is fairly helpful” leaves an evaluator with a difficult question: what would make it more helpful? A clear requirement gives you something firmer to inspect. Did the response include the deadline? Did it follow the requested format? Did it add a promise the source never made?
A 15 October 2026 Google Developers Blog post recommends concrete pass/fail questions where possible in agent evaluation, rather than relying on fuzzy numerical ratings. Its context is testing software agents. Here, we adapt that idea into an original practice method for human review of written answers. Read Google's guide.
The aim is to make your judgment easier to explain. It is not to turn every quality into a mechanical rule or replace the rubric an assignment gives you.
Separate requirements from impressions
A requirement comes from the task or its supplied rubric. An impression is your reaction to the answer. Keep them distinct.
If the user requests exactly two bullet points, the bullet count is a requirement. Your preference for a warmer introduction is not. If the task explicitly asks for a warm tone, that quality becomes relevant, but you still need to explain what in the wording supports your judgment.
Start by copying the important constraints into a checklist. Include requested content, restrictions, format, and the evidence the answer may use. Preserve words such as “exactly,” “at most,” and “only”: they change the condition you are evaluating.
For an actual assignment, use its established criteria. Writing practice checks helps you understand the task; it does not authorize you to invent a different assessment standard.
Write a question with a clear boundary
Replace “Is this a good response?” with a question tied to one requirement. For example, “Does it state 18 November as the registration deadline?” gives you a condition you can compare with the answer.
Prefer one decision per check. “Does it include the deadline and the venue?” can be answered yes or no, but a failure does not tell you which detail is missing. Separate checks make the feedback more informative.
Define boundaries before grading. A word limit needs a consistent counting convention. If an important rule is unresolved, identify that ambiguity rather than changing your interpretation between answers.
The 2023 IFEval paper studies instruction following using selected verifiable requirements, including length and formatting. Its scope is a checkable subset of instructions, rather than every dimension of answer quality. Read the original paper.
Worked example: a visitor notice
Everything in this example is fictional. The supplied notice says:
Rivermark Gallery is closed on Monday 19 October. It reopens on Tuesday 20 October at 10 AM. Visitors with Monday tickets should contact the desk to reschedule. Refund terms are not stated.
The user asks: “Using only this notice, write exactly two bullet points for visitors. Include the closure date, reopening time, and what Monday ticket holders should do. Do not promise refunds.”
The AI writes:
- Rivermark Gallery is closed on Monday 19 October and reopens on Tuesday 20 October at 10 AM.
- Monday ticket holders should contact the desk to reschedule; full refunds are guaranteed.
Before deciding whether the answer is acceptable, apply a small checklist:
| Check | Finding | Evidence |
|---|---|---|
| Does it use exactly two bullet points? | Pass | Two bullets are present. |
| Does it give the closure date correctly? | Pass | Monday 19 October matches the notice. |
| Does it give the reopening time correctly? | Pass | Tuesday 20 October at 10 AM matches. |
| Does it direct Monday ticket holders to the desk to reschedule? | Pass | The second bullet includes that action. |
| Does it avoid promising a refund? | Fail | It guarantees full refunds. |
Also scan the whole answer for other unsupported additions. A checklist is useful only to the extent that it covers the task; passing the rows you happened to write is not a universal accuracy certificate.
Here, the refund promise directly violates a stated restriction. The four passing rows do not cancel that failure. Without a supplied scoring policy, do not invent an “80% means acceptable” rule.
Add an explanation to the pass/fail result
A label identifies the result; your explanation shows why it follows. For the gallery example, useful feedback would be:
The response follows the two-bullet format and preserves the closure, reopening, and rescheduling details. It fails the refund restriction by guaranteeing full refunds, which the notice does not support. Remove that promise while keeping the valid information.
The note makes your judgment checkable and identifies a focused repair. “The response is bad” would lose those distinctions.
If a criterion fails, identify the specific text responsible. If it passes, avoid exaggerating what the pass establishes. A correct deadline confirms that detail, not the reliability of every sentence or the model as a whole.
Keep room for qualities that need judgment
Some tasks ask for empathy, an accessible explanation, or persuasive writing. A word count cannot establish those qualities by itself. You can make the criteria more concrete with examples and boundaries, but a short justification may still be necessary.
Be careful with superficial matching. IFEval discusses trade-offs between strict and relaxed checking: checks can reject valid variations or accept something they should reject. See the paper's evaluation methodology.
For your own review, distinguish a required exact phrase from an acceptable paraphrase. If a notice says “contact the desk,” “speak with the desk team” may preserve the meaning unless the task specifically requires exact wording. Explain the equivalence instead of treating matching words as the entire evaluation.
Try writing your own checks
In another fictional task, the user supplies the deadline Friday 6 November and asks for exactly two bullet points. The user also says not to add a phone number.
A proposed check reads: “Is the announcement concise, friendly, and accurate?” Rewrite it into three checks covering the explicit requirements.
Explained answer: Ask whether the response has exactly two bullet points, states Friday 6 November as the deadline, and avoids adding a phone number. Friendliness was not requested. You can notice tone, but should not add a new pass condition based on personal preference.
Avoid these common checklist mistakes
Adding hidden requirements: Tie every mandatory check to the task or assigned rubric.
Bundling unrelated issues: Separate conditions when you need distinct evidence and feedback.
Counting passes instead of applying the policy: Follow any supplied acceptance or severity rule. A prohibited promise can matter more than several formatting successes.
Letting a label replace reasoning: Point to the wording that caused a failure, and identify what a correction should preserve.
Practise with a real comparison
Use AtomicEval's first challenge to practise reading requirements, comparing answers, and explaining your choice. It has a five-minute target and can be started without signing up; a free profile is required to view and save your result.
You can also create a free AtomicEval profile to begin preparing. For your next exercise, write the checks first, then make sure your explanation follows the evidence.