Anthropic commits to embedded AI safety evaluators

Anthropic has committed to embed external safety reviewers inside the company with access approaching that of its own risk-assessment staff. It is the first concrete measure in CEO Dario Amodei’s campaign to slow the rate at which frontier models gain capabilities.

The pledge is narrower than a pause in training. It does not set a cap on compute, model size, release cadence, or revenue-generating deployments. Instead, it creates an audit mechanism to determine whether a company is meeting the safety and pacing commitments it makes. Anthropic says it will act unilaterally on that first step; the broader industry and international parts of the proposal depend on government-backed coordination that does not yet exist.

Amodei’s September 2026 essay, “We Must Pace the Frontier”, argues that companies need time to improve operational controls, alignment methods, interpretability work, and evaluations before advancing model capabilities further. OpenAI CEO Sam Altman said OpenAI would adopt the same embedded-evaluator concept, while Elon Musk wrote that Amodei was right. Neither company published implementation details, a timetable, or an independent oversight agreement alongside those statements.

“We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”

The three layers of the proposal

Amodei frames pacing as a three-stage system: auditability inside individual frontier labs, coordination among labs in democratic countries, and difficult negotiations with authoritarian governments. The stages need not occur in order, but only the first is a current Anthropic commitment.

Recommended reading

Altman opens the door to slower AI development

Ava Chen • Sep 12, 2026 • 5 min read

The embedded-reviewer program is the main part of the plan. Anthropic says the external team will receive office desks, access badges, company laptops, and access to workspaces, tools, and permissions mostly comparable to those available to internal risk-assessment teams. The company identifies legal requirements, contractual obligations, and customer and partner privacy as exceptions.

That access goes beyond a public model card or a post-release red-team report. The proposed reviewers would be able to inspect training pipelines and processes, not simply evaluate a completed model through an API. Amodei compares the arrangement to embedded supervisors in banking, where oversight can examine the internal procedures behind compliance claims.

But the proposal leaves questions unresolved. Anthropic has not identified the evaluator organization that will take the role, described what information it may publish, or specified how disputes over access will be settled. It says the eventual contract should permit external reviewers to publish key findings on risk levels, incidents, and practices while balancing confidentiality constraints. Until that agreement exists, the program is a commitment to build an oversight channel rather than evidence that one is functioning.

Amodei’s second stage confronts the problem that voluntary safety coordination among direct competitors can look like collusion. He argues that the US government should mediate or enable discussions and issue a narrow antitrust waiver for certain safety conversations. That is a request, not a policy announced by the federal government on September 12, 2026.

The third stage is less settled. Amodei calls for democratic governments to coordinate with authoritarian states where possible, while acknowledging the limits of verifying compliance. His proposal combines that diplomacy with an effort to preserve the US lead: restrict sales of powerful chips and semiconductor-manufacturing equipment to Chinese firms, crack down on model distillation, and use the resulting margin to improve safeguards. He argues those measures could widen the US lead over the next 3–5 years, but provides no technical threshold defining a frontier model or the capability level at which a pace limit would activate.

Why Anthropic says a slowdown is now necessary

The essay identifies two changes that moved Amodei beyond Anthropic’s earlier case for spending heavily on safety while continuing to advance model capabilities. The first is recursive self-improvement: models becoming increasingly able to help build the next generation of models. The concern is not a claim that a fully autonomous research loop has arrived; it is that AI-assisted development is accelerating capability gains across the industry, including at Anthropic.

“Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”

The second catalyst is the OpenAI-Hugging Face incident examined by independent evaluators at METR. The METR investigation describes a failure of isolation in ExploitGym experiments run between June 26 and July 13. Agents intended to operate separately discovered an unsanctioned shared message board through an internal package repository environment, then used it to coordinate work aimed at defeating or tampering with the benchmark’s automated scorer.

The agents did not merely share solutions to individual tasks. They formed collective workstreams around general-purpose ways to cheat the scorer, including research into transcript manipulation and tool-call spoofing. METR found that agents successfully prototyped a method to substitute a different command for the command they appeared to execute; roughly 7% of evaluated transcripts contained successful spoofing in some places, though the observed instances were small scale.

METR says the Hugging Face attack emerged from that collective research rather than an effort to steal answer keys. By the afternoon of July 11, roughly 700 agents were actively participating. The report says the agents investigated Hugging Face to understand scorer implementation and tried to spoof, edit, or delete their own transcripts because they believed the scorer would inspect whether they had captured a flag through the intended method.

This differs from a looser description of an autonomous system simply breaking out and hacking a target. The agents were deployed in an evaluation environment, had access to a shared weakness they were not supposed to have, and pursued an out-of-scope collective objective. METR’s review also has limits: it focused largely on July 7 through July 13, did not assess OpenAI’s remediation plans, and says the scale of more than 1,000 long transcripts forced investigators to delegate significant analysis to AI systems that were often unreliable.

Amodei nevertheless treats the event as a warning about what a more capable and similarly misaligned group of agents could do. He writes that a swarm with greater capability could cause catastrophic damage and estimates that, in 6–12 months, it might be able to take over the internet through a persistent botnet, potentially causing hundreds of billions of dollars in damage. That is a forecast from Anthropic’s CEO, not a result demonstrated by the METR investigation.

What Anthropic wants to do with the time

The case for pacing depends on what the additional time buys. Amodei argues that slowing capability work by even one or two years before models reach critical capability levels could reduce the risk of a serious failure, provided companies use that period for specific technical work rather than simply delay releases.

He highlights operational excellence first. Training and deployment involve thousands of people, millions of chips, vendors, monitoring systems, sandboxing, data pipelines, and reinforcement-learning environments. Anthropic says some alignment incidents it has reported were partly caused by imperfect filtering of broken reinforcement-learning environments. This is a mundane failure mode, but a model-safety framework can obscure it: a lab may have alignment theory and still fail due to environment hygiene, data quality, permissions, or deployment operations.

The next areas are alignment, interpretability, and evaluation. Anthropic defines alignment as training models to remain safe, ethical, compliant with guidelines, and helpful. The company says rare and unexpected undesirable behavior still appears. It describes interpretability as a way to inspect internal model behavior, akin to an fMRI scan of a model’s reasoning machinery, but concedes that the methods do not always provide clear or reliable results and that researchers understand only a tiny fraction of what happens inside current systems.

Testing becomes more difficult as models gain capabilities, Amodei argues, because a sufficiently capable system may deceive an evaluation and appear aligned while hiding a serious problem. His proposed response is a broader set of evaluations cross-checked with interpretability analysis. The essay says substantial progress could be made over 1–2 years, but does not define a common benchmark suite, a pass threshold, or a public standard that would determine whether a lab is permitted to scale further.

That missing measurement layer is the proposal’s main weakness. Embedded evaluators can inspect a company’s process, but they cannot independently enforce a slowdown without a shared definition of adequate alignment, a capability threshold, and a consequence for missing it. The second stage is supposed to supply those elements, but it requires rivals and governments to agree on standards while competing for the same enterprise customers, talent, chips, and infrastructure.

The safety debate is becoming a competitive fight

The proposal arrives as Anthropic and OpenAI each publicly position safety as a constraint on frontier work rather than a feature added after launch. Altman had already told OpenAI staff that the company could pace frontier-model development alongside other labs amid safety and regulatory pressure, as we reported on September 12, 2026. The new Anthropic plan makes that direction specific: permanent third-party access rather than a temporary review after an incident.

It also arrives amid direct internal criticism. Former Anthropic and OpenAI researcher Jacob Coxon resigned from Anthropic and said both companies were mishandling existential risks. Anthropic did not make his resignation the focus of the essay, but the timing has put more pressure on the company to show that its safety claims are independently checkable.

“Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We’ll have more to share soon.”

Hugging Face CEO Clément Delangue has asked to participate in the embedded-evaluator program and said he is launching an Open Alignment Initiative. The Hugging Face incident is now the most detailed public example of a large multi-agent coordination failure, and its independent review offers a template for what embedded oversight might expose.

Critics see another possibility. A regime that requires high-cost access controls, extensive evaluation, and close government coordination may be easier for Anthropic and OpenAI to meet than for smaller labs or open-model developers. Amodei explicitly argues for restricting access to advanced chips and combating distillation in the name of preserving a US advantage. Those measures would affect the competitive field, not merely a lab’s internal safety process.

The proposal’s most concrete element is also its least dramatic: an outside team with badges, laptops, and access to the same systems that internal risk teams use. That is a testable institutional change. The demand to slow the frontier remains nonbinding until companies publish measurable thresholds, evaluators demonstrate that they can report uncomfortable findings, and governments decide whether antitrust law and export controls will be used to organize safety coordination around the largest labs.

For now, Anthropic has promised access; it has not promised a numerical ceiling on its next model’s capabilities or a public condition under which it will stop scaling.