Ten Common Beginner AI Evaluation Mistakes—and How to Fix Them

An AI answer can sound excellent and still fail the task. For a beginner, the challenge is learning to explain that difference without relying on a vague feeling that one answer is better.

Use these ten mistakes as a review checklist. The fixes below are a practical teaching method, not an official scoring standard. If an evaluation task supplies a rubric, rating scale, or tie policy, follow those instructions. Your aim is to make a judgment another person can understand and check.

1. Choosing a winner before reading the full request

You notice a polished opening and decide the response is strong. Only later do you discover that the user asked for three bullet points, a source restriction, or an answer for a child.

Fix: Extract the requirements first: requested outcome, audience, format, constraints, and supplied evidence. Turn them into a short checklist before looking for a winner. When practising on AtomicEval, give the challenge instructions the same attention as the candidate responses.

2. Treating a longer response as a better response

A longer answer gives you more material to read, but extra paragraphs may repeat the same idea or bury the requested answer. A short response may also omit something essential.

Fix: Ask what each additional sentence contributes to this task. Research on MT-Bench examined verbosity and position biases in tested LLM judges; those findings do not establish how human evaluators behave. MT-Bench paper

For your own practice, reward useful coverage and obey any length requirement.

3. Accepting specific claims because they sound confident

A response names a date, policy, study, or statistic with certainty. You treat that precision as evidence, even when the provided material does not support it.

Fix: Identify claims that affect the answer. Check them against the supplied reference, or use authoritative external sources if the task permits browsing. Record an unresolved claim as unresolved. Without enough evidence, distinguish “unsupported” from “false”; the two labels make different assertions.

4. Checking the final answer but skipping the reasoning

The conclusion looks right, so you overlook an invalid calculation or an explanation that contradicts it. You might also penalize an answer for not displaying reasoning that the user never requested.

Fix: Check any reasoning that is present and relevant. For arithmetic, recompute the key step. For an explanation, check whether the stated evidence supports the conclusion. Judge visible work; do not speculate about an AI system's hidden thought process.

5. Replacing the user's preferences with your own

You prefer a friendly tone, detailed explanation, or particular recommendation. That preference becomes the reason for your rating, even though the user requested something else.

Fix: Connect each criticism to a requirement or a concrete effect on usefulness. “Too casual for the requested formal letter” is defensible. “I dislike casual writing” is personal taste. Where the prompt leaves room for style, acknowledge acceptable alternatives instead of inventing a rule.

6. Applying different criteria to the two responses

You examine A's factual accuracy closely, then praise B mainly for readability. The verdict compares different qualities.

Fix: Evaluate both answers against the same task requirements. HELM's original evaluation used multiple metrics and standardized conditions to expose trade-offs. HELM paper

A beginner can use a simple table: requirement, evidence in A, evidence in B, and significance. That table is our practice adaptation, not HELM's scoring procedure.

7. Counting errors without considering their impact

One response has two awkward phrases. Another has one error that changes the advice. Counting defects gives the second answer an advantage it may not deserve.

Fix: Explain what each defect changes for the user. Does it block the requested outcome, alter a key fact, or merely add friction? Use the task's stated severity rules when available. Avoid inventing numerical weights just to make a preferred response win.

8. Forcing a winner when the evidence supports a tie

Two responses meet the request equally well, but you pick the one with a more attractive first sentence. Alternatively, you call a tie because comparing them feels difficult.

Fix: Check whether the task permits ties. If it does, use a tie when there is no meaningful difference under its criteria. If it requires a preference, choose using the permitted criteria and describe how small the advantage is. Difficulty alone is not evidence of equivalence.

9. Writing feedback that cannot be checked

“B is more professional” leaves the reader guessing. It does not identify the relevant passage, requirement, or consequence.

Fix: Write a compact justification: verdict, decisive evidence, and impact. Mention the strongest relevant trade-off rather than listing every sentence. AtomicEval's A/B challenge offers a way to practise a preference together with a justification. Aim for evidence another reviewer could locate, even if they disagree with your verdict.

10. Turning one response comparison into a model-wide claim

One answer makes an error, so your feedback declares that its model is unreliable or incapable of the whole task category.

Fix: Keep the conclusion within the evidence. Say “A omits the requested limitation in this example.” A broader claim needs broader evidence across suitable tasks and conditions. Your comparison can be decisive about these responses while staying modest about everything else.

Worked example: the attractive promise

This is a fictional teaching example. A community repair café's notes say: Saturday, 10 AM to noon, community hall, bring a broken small household item, repairs are not guaranteed. The prompt asks for a two-sentence announcement using only those notes.

A: “Bring a broken small household item to our repair café at the community hall this Saturday, 10 AM to noon. We guarantee a successful repair while you wait.”

B: “The repair café runs at the community hall this Saturday, 10 AM to noon; bring a broken small household item. Repairs are not guaranteed.”

Prefer B. Both meet the sentence count and give the time and location. A reverses an explicit limitation and invents a service promise. B preserves the notes. A's appealing tone does not resolve its decisive factual failure.

Your turn: make the feedback precise

In another fictional task, the notes say that a seed swap starts at 2 PM and participants should bring labelled seed packets. The user asks for one sentence using only those notes.

The response says: “Bring labelled seed packets to the seed swap at 2 PM, where every participant receives a free gardening kit.”

What is the most useful feedback?

Explained answer: The time, requested item, and sentence count are correct. The free gardening kit is unsupported by the notes and could create a false expectation. Recommend removing that clause. Calling the entire answer “wrong” would obscure what it already does well.

Before your next practice session, read these ten headings, evaluate one pair, and reread your justification. Check whether it names evidence and stays within the task.

If you want to try the process, take your first AtomicEval challenge. It has a five-minute target and can be started without signing up. To view and save your result, create a free AtomicEval profile.

Sources and further reading

• Zheng and colleagues, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023.
• Liang and colleagues, Holistic Evaluation of Language Models, submitted 2022; revised 2023.