How to Write Specific, Useful AI Evaluation Feedback

“The answer needs improvement” tells a reader that you disliked something. It leaves them to discover the actual problem.

Useful AI evaluation feedback makes your judgment inspectable. Another reviewer should be able to find the passage you mean, understand the requirement it misses, and see why your conclusion follows. When a task requests a correction, your feedback should also make the next edit clear.

You do not need a long paragraph or impressive terminology. You need a precise link between the response, the evidence, and the task.

A recent example of feedback tied to a failure

An AWS post published on October 16, 2026 describes turning agent traces and observed failure patterns into targeted prompt and tool-description recommendations. It illustrates the value of connecting an observed problem to a particular change. Read the AWS post.

Earlier research offers a related, narrower finding. In selected self-refinement experiments involving code optimisation, sentiment reversal, and acronym generation, the 2023 Self-Refine paper found targeted feedback more effective than generic feedback variants. The model generated its own critiques; these results do not validate a human assessment template. Madaan et al., section 4.

The practice format below is our original adaptation for human response evaluation. Use the instructions and rubric supplied with your assessment to decide what to include.

Make your feedback answer four questions

What exactly needs attention?

Identify a specific claim, omission, instruction failure, or unsupported inference. Point to a short phrase or a clearly named part of the answer.

“The response is inaccurate” is too broad when one sentence reverses a deadline. “The transfer-eligibility sentence reverses the deadline” gives the reader a place to look. Describe the response rather than guessing what the model intended or understood.

What evidence establishes the issue?

Connect your observation to the prompt, supplied source, tool result, or allowed external evidence. Keep quotations short and relevant.

If the rule says “at least 48 hours before,” preserve that qualification. Replacing it with “within 48 hours” changes the meaning. If the evidence does not settle a claim, say what remains unverified rather than labelling it false automatically.

Why does it matter for this task?

Explain the consequence: a wrong eligibility decision, an unanswered question, a broken format requirement, or another rubric-relevant failure.

Separate factual errors from preferences. You may prefer a warmer tone, but that alone does not establish a violation when the task only asks for a neutral answer. Prioritise the issue that actually changes the requested outcome.

What focused change would address it?

When the task asks for improvement advice, suggest the smallest correction that resolves the problem while preserving valid content.

“Rewrite everything” is unhelpful when one conclusion needs changing. “Replace the eligibility conclusion and retain the recording option” is more usable. If the task requests only a comparative justification, you can stop after explaining the decisive difference; a full replacement answer may be unnecessary.

Worked example: an eligibility answer with the deadline reversed

Fictional practice scenario. The organisation, rules, and responses are invented.

The Elm Workshop Club supplies this policy:

• Session transfers require a request at least 48 hours before the workshop starts.
• Later requests receive access to the recording instead.

The user says: “I requested a transfer 36 hours before the start. Can I move to the next session?”

The AI answers: “Yes. Because you requested the change within 48 hours, you qualify for a transfer. Ask the club to move your booking to the next session.”

A vague critique would be:

The answer misunderstands the policy. It should be clearer and more accurate.

This names a broad concern but does not explain the actual reversal. “Clearer” might even encourage a more polished version of the same wrong decision.

A useful critique would be:

The response says the user qualifies because the request was “within 48 hours.” The policy requires at least 48 hours, and this request came 36 hours before the start, so it does not qualify. Replace the transfer recommendation with an explanation that the user receives recording access instead.

That feedback identifies the passage, supplies the controlling rule, explains the incorrect decision, and offers a focused repair. It does not invent an exception or a penalty.

You can compress it when space is limited:

The response reverses the deadline: 36 hours does not meet the “at least 48 hours” requirement. It should explain that recording access applies instead of recommending a transfer.

The shorter version preserves the evidence and correction. Concision works when it retains the reason for your judgment.

Check your critique before submitting it

Read the original response again after writing your feedback. Does it actually contain the error you describe? Have you overlooked a sentence that already answers the concern?

Then mentally apply your proposed fix. Would the result still obey the other requirements? A correction that adds unsupported details or breaks the required format creates another problem.

Finally, remove claims about hidden causes. You may observe a reversed deadline; you cannot infer from that sentence alone that training data, a particular prompt, or deliberate deception caused it. Keep the diagnosis at the level the evidence supports.

Short exercise: choose the more useful feedback

Fictional exercise. The user supplies two handover facts: deployment is planned for Tuesday, and testing is incomplete. They request exactly two bullet points.

The AI responds with two bullets:

• Deployment is planned for Tuesday.
• Testing is complete.

Which feedback is better?

A: “This summary needs more detail. Add another bullet explaining the testing process.”

B: “The second bullet says testing is complete, contradicting the supplied note that testing is incomplete. Correct that status while keeping the requested two-bullet format.”

Explained answer: B is better. It locates the false statement, identifies the evidence, and proposes a correction that preserves the format. A misses the factual contradiction and asks for an extra bullet, which would violate the user's instruction. The Tuesday bullet is supported and does not need changing.

Common mistakes

• Giving a verdict without identifying the decisive passage.
• Listing minor style preferences ahead of a wrong task outcome.
• Suggesting a rewrite that changes correct facts or required formatting.
• Calling an unverified claim false without checking the available evidence.
• Adding guessed causes or criticism of the model's intentions.

When a response meets the task, explain what it gets right. Useful evaluation does not require inventing an error.

Practise defending your judgment

Try writing one short note that identifies the decisive issue and connects it to evidence before looking at an explained answer.

Take your first AtomicEval challenge to practise comparing responses and explaining your choice. It has a five-minute target and can be started without signup. A free profile is required to view and save your result.

When you want to continue preparing, create a free AtomicEval profile.

Sources

• AWS: Building trust in AI agents, October 16, 2026.
• Madaan et al.: Self-Refine, submitted March 30, 2023; revised May 25, 2023.