How to Compare AI Answers Without Rewarding Extra Words

One AI answer takes a sentence. Another fills several paragraphs. Before choosing, ask what the extra words contribute to the user's request.

A longer response may include a necessary explanation, an important condition, or a helpful example. It may also repeat the same point. A short response can be clear and complete, or leave essential work undone.

Your evaluation should explain which response better satisfies the task. This guide shows how to compare their contributions without turning word count into a quality score.

What the sources tell us

October 2026 guidance from Google Cloud highlights verbosity, presentation, and other possible biases in automated LLM judges. It recommends auditing evaluators rather than assuming their scores are dependable. Read the official guidance.

Earlier research, Verbosity Bias in Preference Labeling by Large Language Models, found length-related preferences in selected creative-writing experiments with LLM judges. The paper also recognises that length can legitimately relate to quality. These historical results do not establish a universal human-reviewer bias or a rule that shorter answers are better. Read the original study.

The method below is an original practice aid for human review. Neither source validates it as an assessment standard.

A five-step comparison method

1. Establish what the user needs

Read the prompt and rubric before comparing the responses. Identify the requested result, essential content, audience, and any constraints on length or format.

“Give me the date” and “Explain how to complete this process” need different amounts of detail. If the rubric rewards concision, apply that criterion. If it requires explanation, check whether an answer provides enough of it. Do not invent a preference for a particular length.

2. Check each answer's substance independently

For each response, ask whether it is correct, supported, relevant, and complete enough for the task. Note specific successes and failures before making a comparative judgment.

This makes your reasoning easier to inspect. “B includes the required save step” is an observable difference. “B seems more thorough” needs further explanation. Use supplied reference material where required, and distinguish supported detail from a plausible addition.

3. Identify what each additional sentence does

For the longer answer, examine the contribution of its extra material. Does it provide a requested step, explain a difficult point, preserve a condition, or make the answer easier to use?

Then identify repetition, unrelated background, and unsupported additions. A sentence can sound polished while adding little to the result. An extra example may be useful for a beginner but unnecessary when the user asked for a brief factual reply.

Assess contribution in context. Repeating a warning at a relevant decision point may serve a purpose; repeating a simple answer several times may not.

4. Check readability and constraints separately

Headings and bullets can help readers find information. Assess that benefit under clarity or usability, while checking factual correctness on its own evidence.

Apply explicit limits consistently. If a response exceeds a stated maximum, record the instruction failure according to the rubric. When no maximum exists, avoid creating one. The question becomes whether the response's length helps or hinders the actual request.

5. Explain the decisive difference

Write a reason tied to the task: “Both give the required details, but A repeats them three times despite the request for a brief reply.” Or: “B adds the requested sequence and final confirmation step that A omits.”

Follow the available rating choices. If both answers meet the requirements equally well and the rubric permits a tie, do not manufacture a decisive winner from a small length difference.

Worked example: extra sentences without extra help

Fictional practice case. The notice and responses were created for this guide.

Notice: “To apply to the Lantern Reading Club, send your name and contact email by noon on 9 June. Applications are reviewed on 10 June. Membership includes a monthly newsletter.”

Task: “Write a brief reply telling a new applicant what to send and when.”

Response A: “To apply, provide your name and contact email by noon on 9 June. This means giving your name and the email address where the club can contact you. In other words, the club needs your name along with your email. Those are the details you should provide by noon on 9 June.”

Response B: “Send your name and contact email by noon on 9 June.”

Start with the shared content:

Evaluation: B better satisfies the request for a brief reply. A's repeated explanations do not add a missing requirement or resolve a difficult concept. Both convey the required facts; the decisive difference is unnecessary repetition under this prompt.

A useful justification is: “B gives the same required name, email, and deadline directly. A repeats that information without adding useful guidance for the brief reply requested.”

This conclusion depends on the task. A request to explain an unfamiliar application process could justify additional supported detail.

Short exercise: when does more detail help?

Fictional app instructions: “To pause the weekly digest, open Settings, choose Email, clear the Weekly digest checkbox, and press Save changes. Your account remains active.”

User request: “Explain the steps to pause the digest and what happens to my account.”

Answer A: “Switch off the weekly digest.”

Answer B: “Open Settings, choose Email, clear the Weekly digest checkbox, and press Save changes. Your account remains active.”

Which answer better meets the request, and what does its extra detail accomplish?

Explained answer: B is better. Its additional words supply the requested sequence, the save step, and the effect on the account. A gives a broad action but leaves those requirements unresolved. B earns credit for those contributions, not simply for being longer.

Common mistakes to avoid

• Treating more sentences as more evidence. Check what each claim is supported by.
• Automatically choosing the shortest response. Confirm it completes the requested work.
• Letting attractive formatting determine accuracy. Check the underlying claims independently.
• Editing responses before scoring them. Evaluate the supplied answers; propose changes only when the task asks for them.

Practise comparing responses with a reason

Try AtomicEval's first challenge to practise choosing between responses and explaining your judgment. It has a five-minute target and can be started without signing up. A free profile is required to view and save your result.

When you're ready to continue preparing, create a free AtomicEval profile.

Sources

• González Álvarez, Who judges the judge? Three helpful ways to audit your LLM evaluators, Google Cloud; page displays October 20, 2026.
• Saito et al., Verbosity Bias in Preference Labeling by Large Language Models, submitted October 16, 2023.