Anthropic은 공개를 막지 않는다—다만 무엇이 기밀인지는 결정한다

Anthropic says six evaluators from the nonprofit METR can work inside the company with employee-like access and publish findings without asking Anthropic for permission. That is a meaningful step beyond testing a public chatbot from outside. It is not the same as an auditor having complete control over what can be disclosed.

The agreement allows Anthropic to demand deletion of material that falls within predefined confidential categories, and Anthropic ultimately decides whether information belongs to those categories. The company says it cannot veto a report merely because the conclusions are unflattering. The tension is therefore precise: publication approval is independent, but part of the boundary around publishable evidence remains controlled by the organization being evaluated.

That does not make the project fake. Sensitive laboratories genuinely hold customer data, security details, employee information, and model capabilities that could create harm if released carelessly. The test is whether confidentiality protects legitimate secrets without becoming a broad channel for removing inconvenient findings.

What employee-like access changes

External evaluators normally receive selected checkpoints, API accounts, documentation, or time-limited test environments. They can probe model behavior but may not see internal mitigations, incident records, monitoring systems, deployment decisions, or the tools used by the safety team.

METR’s six-person team can use Anthropic equipment, office space, and internal tools. That can reveal how risk management works in practice rather than how it is described in a policy document. Evaluators can examine workflows, ask employees questions, and test systems closer to the point where release decisions are made.

Access also reduces a familiar problem in AI evaluation: the company can otherwise choose the model, interface, safeguards, and evidence bundle presented to outsiders. An evaluator operating inside the environment has a better chance of observing inconsistencies between the public product, internal model, and organizational controls.

The rights that matter

Independence is not one switch. It is a bundle of rights: access to relevant systems, freedom to choose tests, control over methodology, authority to preserve evidence, the ability to publish negative conclusions, and protection against retaliation or future exclusion.

Under Anthropic’s description, METR does not need company approval to publish results. Anthropic also says it has no general veto. Those are stronger terms than a conventional commissioned review in which the client owns the final report.

The limitation is redaction. If Anthropic identifies material as covered by a previously defined confidentiality category, it can require removal. Predefinition matters because it makes it harder to invent a new category after a damaging result appears. Yet the arrangement still gives the evaluated company the final call on whether the category applies.

Why confidentiality cannot simply disappear

A frontier-model audit may encounter exploit chains, unreleased capabilities, security controls, private customer data, or details that would help an attacker reproduce a dangerous behavior. Publishing every artifact could increase the risk the evaluation is meant to reduce.

Other audit fields solve this through layered disclosure. A regulator or accredited auditor may receive full evidence, while the public report summarizes the finding and withholds operational details. Disputes about secrecy can be escalated to an independent authority rather than resolved solely by the inspected company.

AI evaluation lacks that mature structure. METR and Anthropic are effectively testing a governance model while testing models. Their agreement should be judged not only by the first report but by how it handles the first serious disagreement over what the public deserves to know.

Questions the eventual report must answer

The report should list the categories of information that could be removed, at least in summarized form. It should disclose how many redaction requests Anthropic made, how many METR disputed, and whether any finding changed materially because supporting evidence could not be published.

Readers also need to know who selected the systems and risks examined, how long access lasted, whether METR could preserve records, and whether Anthropic can end future access after criticism. Methodology should be detailed enough for another evaluator to reproduce the nonconfidential parts.

The strongest protection would be a dispute mechanism outside Anthropic. A trusted third party could review contested material and decide whether the confidentiality definition was applied correctly. Without that, the public must rely on the evaluator to signal when a restriction impaired the work.

What this means for AI governance

The arrangement recognizes that surface-level red teaming is no longer sufficient. Frontier systems are embedded in infrastructure, policies, and monitoring processes. Evaluating them requires organizational access similar to financial audits, safety inspections, or on-site regulatory examinations.

It also exposes a gap between voluntary evaluation and regulation. A company can grant excellent access to one nonprofit while competitors provide less, change terms later, or decline entirely. Durable oversight needs common minimum rights, reporting standards, conflict rules, and consequences when access is withheld.

Voluntary projects can still build the evidence for those standards. They show which access matters, which confidentiality problems arise, and what evaluators need to publish credible conclusions.

Verdict

Anthropic’s METR arrangement is more independent than a company grading its own homework and less independent than a regulator with statutory authority. Employee-like access and publication without prior approval are real advances. Company-controlled confidentiality judgments remain a real limitation.

The correct question is not whether the audit is perfectly independent. It is whether the structure makes hidden problems more likely to be discovered and disclosed than before—and whether future versions transfer disputed secrecy decisions away from the company under review. The answer to the first question may be yes. The second remains unfinished.