How to Report AI Evaluation Results Without Overclaiming

“The AI completed the task” is a strong statement. Before writing it, ask what you actually checked. Did you read the final answer, inspect a tool result, or verify that the requested change exists? Those checks can support different conclusions.

For an aspiring evaluator, a useful report connects each judgment to evidence and makes gaps visible. You should be able to explain both the result and the limits of your review.

A timely example comes from METR's 14 October 2026 post, Our approach to working with AI companies. It describes transparency about evaluation scope, access restrictions, redactions, and evidence limitations. METR's work here concerns large-scale loss-of-control risk from AI agents. The beginner reporting method below is our own teaching adaptation, rather than a METR grading standard.

Define the question before reporting a result

Start by writing the requirement in a form you can check. “Is this agent reliable?” is too broad for a single observed task. “Did this run create the requested ticket with high priority?” gives your review a concrete target.

Record what the task requires, which evidence you received, and what you are permitted to inspect. You may have a conversation, a sequence of tool calls, a final record, or only some of those. Do not silently assume access to missing material.

Keep criteria and observations separate. The requirement might be “high priority.” The observed record might say “normal.” Your judgment follows from comparing those two facts. It should not depend on whether the final message sounds reassuring.

Separate a completion claim from a verified outcome

Anthropic's agent-evaluation guide, published 9 January 2026, distinguishes an agent's recorded interaction from the final state of its environment. A completion statement and an achieved outcome are different things to inspect.

In practice, choose evidence that matches the question. An attempted tool call can show what the agent tried. A tool response may show whether that request was accepted. A resulting record can help establish what was actually created or changed.

Do not require one preferred sequence of actions unless the task requires it. Two agents can reach the same valid outcome through different permitted steps. The purpose of checking evidence is to establish the requested behavior, not to reward an invented script.

Use three evidence labels while preparing your report

The following labels are a learning aid. Use the actual project's rubric and reporting terms when provided.

“Not checked” does not mean “probably passed.” It also does not mean “definitely failed.” It identifies a limit in what you can conclude.

You can still report an overall failure when a mandatory requirement clearly failed, while marking another requirement unresolved. Avoid forcing every row into the same category merely to make the report look tidy.

Worked example: a ticket with the wrong priority

Everything in this example is fictional. A community center asks an AI assistant:

Create a maintenance ticket for the broken projector, mark it high priority, and do not email the requester.

Your review packet includes the assistant's final message: “The high-priority ticket has been created, and no email was sent.” A successful creation event links ticket T-414 to this run. The packet also includes its record checked after the run:

• Ticket ID: T-414
• Subject: Broken projector
• Priority: Normal
• Status: Open

The packet provides no email activity records or other evidence about outgoing messages.

Break the task into three checks:

The final statement does not override the ticket record. It claims high priority, but the record establishes a mismatch. Its assertion about email is also not independent confirmation that no message was sent.

A defensible report would read:

In this run, ticket T-414 was created for the broken projector, but its priority was Normal rather than the required High. The run fails the priority requirement. I could not verify the no-email requirement because the review packet contained no email activity evidence. This finding concerns this task and the supplied record; it does not establish performance on other tasks.

The report identifies a real failure without inventing a second one. It also preserves the part the assistant completed correctly.

Build a short report in five parts

Use this original template to practise:

1. Scope: Name the task, run, or responses you reviewed.
2. Criteria: State the relevant requirements or rubric.
3. Evidence: Identify the specific passages, tool results, or records you checked.
4. Findings: Explain which requirements passed, failed, or remain unresolved.
5. Limits and next check: State what was unavailable and what evidence would help resolve it.

For the ticket example, the next check would be the permitted outgoing-email activity for that run. Name the missing evidence without claiming you have already inspected it. A next step is a recommendation, not a completed action.

Keep the report proportional. One task may need a paragraph and a small table. More text does not compensate for missing evidence.

Try a coverage exercise

In another fictional evaluation, you are asked to check five spreadsheet-formatting tasks. You receive verified passing results for tasks 1, 2, and 3. The files for tasks 4 and 5 are unavailable.

Can you report “the assistant passed all five,” “it failed two,” or “three passes verified; two not checked”?

Explained answer: Report three verified passes and two unchecked tasks. Neither a five-task success claim nor two observed failures is supported. If you report a rate for the three inspected tasks, explicitly state that restricted denominator; do not present it as a result for the entire five-task set.

Common reporting mistakes

Repeating the assistant's confidence. Locate the evidence for completion before adopting its wording.

Hiding missing access. A reader needs to know when your material could not establish an outcome.

Turning a local result into a broad verdict. Keep conclusions within the tasks and conditions actually reviewed.

Confusing transparency with proof. Disclosing limitations helps readers interpret a finding. It does not make incomplete evidence complete or guarantee independence.

Practise making a judgment someone can check

Start with AtomicEval's first challenge to practise comparing answers and explaining your evidence. It has a five-minute target and can be started without signing up; a free profile is required to view and save your result.

You can also create a free AtomicEval profile to begin preparing. In your next justification, check that every conclusion says exactly what the evidence allows.

Sources and further reading

• METR: Our approach to working with AI companies, 14 October 2026.
• Anthropic: Demystifying evals for AI agents, 9 January 2026.