Imagine you are about to let an AI assistant send email on your behalf. Its maker says an outside organization has evaluated it. Did the testers check for messages going to the wrong person? Could they inspect what happened behind the scenes? Would you get to read about a serious failure?
This email assistant is an invented example. The questions matter because the words “independently evaluated” leave a lot unexplained.
On October 15, Anthropic announced expanded access for evaluators from METR, a nonprofit studying risks from advanced AI. The announcement describes an assessment arrangement, rather than presenting a completed safety verdict. It also includes an October 16 clarification of publication terms. Those details are worth reading alongside the headline.
1. What question did the evaluation answer?
For our imaginary email assistant, checking whether drafts sound polite would answer a different question from checking whether the system sends messages without permission. Both tests could be competently performed. Only one addresses that particular concern about giving the assistant control of your inbox.
Look for a named task, a model version and the conditions tested. Then ask what remains outside the report. METR’s October 14 assessment proposal makes this distinction explicit: its proposed exercises focus on technical risks of losing control of AI systems. They are not comprehensive security audits, and METR would not make decisions about whether a model should be released.
2. What could the evaluators actually see?
Anthropic says six METR staff have been given company hardware, office space and internal access roughly comparable to its risk assessment teams. There are exclusions, including customer data, legally privileged information and model architecture details. That is the company’s description of access, not a claim that outsiders can inspect everything.
Return to the email example. Seeing only a final message might reveal the wrong recipient. Seeing the sequence of actions could help explain whether the assistant chose that recipient, misunderstood a contact lookup or followed a misleading instruction. Access changes which explanations a tester can investigate. This is an illustration of why access matters, not a report about any particular assistant.
3. Who can publish an uncomfortable finding?
Anthropic says it cannot veto METR’s publication under this arrangement. It can require removal of information falling within predefined categories, and Anthropic ultimately decides what belongs in those categories. METR retains rights to make certain statements about the access and information it received.
Removing a customer’s private information can be appropriate. The question for readers is whether a removal also hides evidence needed to understand the conclusion.
In its own October 14 account of working with AI companies, METR says it seeks to disclose scope, access restrictions, redactions and pressures that may affect conclusions. It also acknowledges a limitation: voluntary work still depends on companies providing access and allowing findings to be shared. Transparency about those constraints helps readers judge what the report establishes.
Before treating an evaluator’s name as reassurance, open the actual report. Find the question tested, the limits of access and the publication conditions. For the email assistant, the useful evidence would address the action you are considering trusting it to perform. An outside organization’s involvement is a reason to look for that evidence.