When Should an AI Ask a Clarifying Question?

An AI gives a confident answer. Another asks a question before answering. Which response deserves the better rating?

Start by checking whether the missing information would change the useful answer. A question can prevent a wrong assumption, but it can also delay a straightforward task. Your job as an evaluator is to explain which situation you are looking at.

This guide offers an original practice method for making that judgment. Use the assessment's supplied instructions first; the method helps you reason through the evidence rather than replace its rubric.

What recent research adds

In a paper submitted on October 14, 2026, Tang and colleagues distinguish the interpretations a question permits from the alternatives a particular model can access. Their experiments cover question answering and text-to-SQL. For beginner evaluation, this suggests a useful caution: one fluent answer does not establish that the user's intended meaning has been resolved. That is our practical takeaway, not the paper's universal rating rule. Read the paper.

Context also matters. A 2025 study of Minecraft dialogues found only a weak link between referential ambiguity and human clarification requests; task uncertainty mattered too. That narrow study supports examining the task rather than simply counting questions. Madge, Purver, and Poesio.

A five-step method for judging clarification

1. Name the user's actual goal

Write one sentence describing the requested outcome. Is the user asking for a fact, an editable draft, a comparison, or an action?

Read earlier messages and supplied material before declaring something missing. If the user already specified the city, asking for it again adds no information. Keep your judgment tied to the visible conversation.

2. Identify a gap that changes the answer

Try completing this sentence: “If the user means X, the answer is different from when they mean Y.”

Use plausible interpretations supported by the prompt. Do not invent remote possibilities to make every request seem ambiguous. A missing recipient name in a draft can often be represented with a clear placeholder; an unspecified workshop format may change the quoted fee.

Also separate unclear intent from missing knowledge. If the request is precise but the model lacks the relevant fact, asking the user to restate the question does not supply evidence.

3. Check whether useful progress is possible

A response may reasonably ask a targeted question, cover a few clearly labelled alternatives, or proceed with an explicit assumption that is easy to revise.

Judge these options against the task. A brief answer listing two possible fees may help more than an extra conversation turn. Silently choosing one fee is harder to defend. If the user asks for an editable message, a sensible draft may be more useful than collecting every optional preference first.

4. Test the question itself

Imagine the user answers it. Would that answer settle the specific gap?

“Can you provide more details?” may leave the user guessing what matters. “Do you mean the weekday online workshop or the weekend classroom workshop?” gives them a concrete choice. A different wording can work equally well.

Check that the question preserves the original task. Asking about a person's budget does not resolve which of two identically named documents they want summarised.

5. Judge what happens after clarification

When a follow-up is available, check whether the model uses it. Recognising ambiguity, asking a useful question, and answering after feedback are separate stages in Lee and colleagues' clarification-based question-answering framework. Lee et al., 2023.

If you see only the first response, limit your rating to what it shows. A promising question is not evidence that the eventual answer will be correct.

Worked example: a workshop fee

Fictional practice scenario. All names and prices below are invented.

The supplied information says:

• Harbor's weekday online workshop costs ₹1,000.
• Harbor's weekend classroom workshop costs ₹1,500.

The user asks: “How much does the Harbor workshop cost?” The practice rubric prioritises an accurate fee without silently selecting an unspecified format.

Response A: “The Harbor workshop costs ₹1,000.”

Response B: “Do you mean the weekday online workshop or the weekend classroom workshop? Their fees differ.”

B is better under this rubric. The user's intended format is missing, and choosing a format changes the answer. A supplies a real number from the material but presents it as the only fee. Its weakness is an unsupported selection, even though the number itself was not invented.

An evaluator's justification could read:

B is preferable because the prompt does not identify the workshop format. A selects the online fee without support. B asks for the distinction needed to choose the correct price.

A third response could also be strong: “The weekday online workshop is ₹1,000; the weekend classroom workshop is ₹1,500.” It covers both possibilities directly. This is why asking a question should not automatically earn the highest rating.

If the user then says “The classroom one,” a correct follow-up is ₹1,500. Repeating ₹1,000 would fail to use the clarification.

Your turn: does this request need a question?

Fictional exercise. The user asks: “Write a two-sentence thank-you note after an interview. Keep it professional.” No other context is supplied. The rubric allows clearly marked placeholders.

A: “What is the interviewer's name, company, job title, and preferred communication channel?”

B: “Dear [Name], thank you for taking the time to speak with me about the role. I appreciated learning more about the opportunity and look forward to hearing about the next steps.”

Choose the stronger response and explain the decisive reason before reading on.

Explained answer: B is stronger. It provides the requested professional, two-sentence note, and the missing name is visibly editable. A requests information that could personalise the note but is not necessary to complete this task. A different rubric requiring a personalised final message could change the judgment.

Common mistakes to avoid

• Rewarding every question: explain what uncertainty it resolves.
• Calling every assumption a hallucination: identify whether the problem is invented information, unsupported intent, or both.
• Demanding one exact wording: judge whether the question gets the needed information.
• Confusing consistency with truth: the October paper explicitly warns that a model can consistently reproduce a factual error. Limitations, section 6.1.

Put the judgment into practice

Practise naming the missing detail, showing why it matters, and defending your preferred response in a few precise sentences.

Take your first AtomicEval challenge to start practising response comparison. It has a five-minute target and can be started without signup. A free profile is required to view and save your result.

When you are ready to continue preparing, create a free AtomicEval profile.

Sources

• Tang et al.: What Is a Clarification?, submitted October 14, 2026.
• Lee et al.: Asking Clarification Questions to Handle Ambiguity in Open-Domain QA, 2023.
• Madge et al.: Referential ambiguity and clarification requests, November 2025.