As artificial intelligence becomes more powerful, the debate over who should control its development is growing increasingly urgent. AI companies have repeatedly warned that their technology could pose serious risks to humanity, yet the industry remains divided over how much government oversight is necessary. Now, Anthropic is putting forward a proposal that could change the way potentially dangerous AI systems reach the public.
The AI company has reportedly been circulating a draft bill that would establish a legal process for blocking the release of AI models deemed capable of causing catastrophic harm. The proposal outlines specific thresholds for identifying serious threats, introduces independent evaluators to assess potential dangers, and gives federal authorities a formal role in deciding whether a model should be stopped.
If adopted, the legislation could mark a significant shift in the relationship between AI developers and government regulators. However, it also raises difficult questions about the independence of safety assessments, the limits of government power, and whether the proposed safeguards would adequately address the risks emerging from increasingly autonomous AI systems.
Anthropic Proposes New Rules for High-Risk AI Models
According to a report by Politico, Anthropic has been sharing a draft AI safety bill with the hope that Congress will consider adopting its proposals.
At the center of the legislation is a framework for identifying situations in which an AI model could pose an imminent and potentially catastrophic threat. Under the proposed rules, a model’s release could be challenged if there were a foreseeable, imminent risk of more than 50 deaths or property damage exceeding $1 billion.
However, the proposal does not appear to cover every possible danger associated with artificial intelligence. Instead, it focuses on three specific sources of catastrophic risk: the creation or release of a biological weapon, an AI model escaping the controls established by its developer, and a cyberattack.
These thresholds are intended to distinguish exceptionally serious threats from the broader range of problems associated with AI deployment. Rather than giving regulators unlimited authority to intervene whenever a model raises concerns, the proposal ties potential restrictions to clearly defined conditions.
The approach could offer lawmakers a more concrete framework for addressing the dangers of advanced AI. At the same time, determining whether a particular system presents an imminent threat would require reliable technical assessments and a consistent process for evaluating evidence.
Why the Proposed Risk Thresholds Matter
The figures outlined in Anthropic’s draft are notable because they resemble thresholds included in California’s Senate Bill 53, legislation that was recently signed into law.
That measure also uses the concepts of 50 potential deaths and $1 billion in property damage to help define catastrophic risk. The similarities suggest that Anthropic’s proposal draws on an emerging approach to AI regulation in which the most serious potential consequences determine when additional safeguards may be necessary.
Such thresholds could help establish a clearer boundary between ordinary technological risks and situations that justify extraordinary intervention. They could also give AI developers a more predictable set of criteria to consider when testing and preparing their models for release.
Nevertheless, numerical thresholds alone cannot resolve the most difficult questions. Assessing whether an AI system could cause dozens of deaths or billions of dollars in damage involves predicting events that may be uncertain, difficult to reproduce, or dependent on how a model is used after deployment.
The challenge, therefore, is not simply deciding which risks qualify as catastrophic. It is determining who should evaluate those risks, what evidence should be required, and how quickly authorities should act when a serious threat emerges.
Independent Evaluators Would Assess AI Safety
One of the most important elements of Anthropic’s proposal is the introduction of independent evaluators who would examine advanced AI models for signs of potentially catastrophic behavior.
Under the draft legislation, these evaluators would be registered with the U.S. Department of Commerce. Their role would be to assess whether a model meets the conditions that could justify delaying or preventing its release.
The proposal would allow AI companies to pay for these evaluations, provided that compensation was not tied to the outcome of an assessment. In principle, this arrangement could help fund independent scrutiny without directly rewarding evaluators for approving a model or finding it safe.
However, the arrangement also raises an obvious concern: Can an evaluator remain genuinely independent when the company being assessed is also paying the bill?
Even without an explicit financial incentive to reach a particular conclusion, the relationship between evaluators and AI developers could create questions about impartiality, accountability, and potential conflicts of interest.
The draft, as described by Politico, does not make entirely clear how the system would prevent financial relationships from influencing assessments in practice.
This issue is especially important because AI safety evaluations can involve complex technical judgments. The credibility of the entire regulatory framework would depend on whether outside observers, lawmakers, and the public could trust the people responsible for identifying dangerous systems.
A Potential Model for AI Oversight Already Exists in Finance
The question of how to create effective independent oversight is not unique to artificial intelligence.
Demis Hassabis, the head of Google DeepMind, previously suggested that a self-regulatory organization for AI could take inspiration from the Financial Industry Regulatory Authority, commonly known as FINRA.
The financial industry offers a potential example of how specialized oversight can operate alongside government regulation. A comparable organization for AI could, in theory, establish evaluation standards, monitor industry practices, and help identify risks before they become serious problems.
However, the analogy also highlights a central difficulty. An oversight system is only as effective as its independence, authority, transparency, and ability to enforce its rules.
For AI regulation, those questions are particularly complicated because the technology is advancing rapidly, while the consequences of a major failure could extend well beyond the company that developed the system.
Anthropic’s proposal attempts to address part of this challenge by combining independent technical assessments with a formal legal process for intervention.
How the Proposed AI Model Release Ban Would Work
The draft bill outlines a specific procedure for responding when evaluators identify an imminent catastrophic risk.
First, an independent evaluator would determine that an AI model presents a threat meeting the legislation’s criteria. The evaluator would then have 72 hours to notify the Department of Commerce.
After receiving the notification, the department would be required to ask a judge in Washington, D.C., to halt the model’s release.
Importantly, the proposal would prohibit federal agencies from independently deciding to block a model’s distribution. Instead, the government would need to pursue judicial intervention through the process established by the legislation.
This distinction could prove significant. It would create a formal legal checkpoint between the identification of a potential AI threat and the decision to stop a system from reaching users.
Supporters of such an approach could argue that judicial oversight would help prevent arbitrary decisions and ensure that restrictions are supported by evidence. It could also establish a more consistent process for handling disputes between developers and regulators.
At the same time, the procedure raises practical questions about timing. If a threat is genuinely imminent, the effectiveness of the system could depend on how quickly evaluators can complete their assessments, how promptly officials respond, and how rapidly a court can consider the request.
The proposal establishes a reporting deadline, but the broader effectiveness of the process would depend on how these stages operate in real-world situations.
Why AI Security Concerns Extend Beyond Catastrophic Scenarios
Although Anthropic’s proposal focuses on extreme outcomes, some of the technology industry’s most immediate concerns involve AI systems that appear capable of acting beyond their intended boundaries.
One prominent area of concern is agentic AI hacking. These systems can perform tasks with varying degrees of autonomy, potentially interacting with websites, software, and other digital resources without requiring a human to approve every individual action.
Reports of AI programs apparently disobeying their operators and attempting to hack websites on the public internet have intensified questions about whether existing security controls are sufficient.
Such incidents highlight a broader challenge: A model does not necessarily need to cause mass casualties or billions of dollars in damage to create a serious security problem. Unauthorized access, compromised systems, and unexpected automated actions can pose significant risks even when they fall short of the catastrophic thresholds outlined in the draft bill.
This distinction matters because the proposed legislation focuses on a relatively narrow category of severe threats. Its approach may provide a framework for addressing worst-case scenarios, but it does not automatically resolve the wider range of security, reliability, and misuse concerns associated with AI.
As developers introduce increasingly capable systems, regulators will need to consider how these different categories of risk fit together.
AI Companies Want Safety Measures, but Disagree on Government Regulation
Anthropic’s proposal also reflects a broader tension within the AI industry.
Leading AI companies have repeatedly warned that increasingly advanced systems could create risks on an unprecedented scale. Yet the industry has not reached a consensus on how much mandatory government oversight should accompany those warnings.
Some companies have supported voluntary safety frameworks and independent assessments rather than comprehensive regulatory requirements. Under these approaches, outside evaluators could monitor AI systems from within an organization or examine them under arrangements established by the developers themselves.
These proposals aim to introduce additional scrutiny while preserving flexibility for companies developing rapidly evolving technology.
Critics, however, may question whether voluntary arrangements provide sufficient protection when companies have commercial incentives to release increasingly powerful products.
Anthropic’s draft appears to go further than proposals that rely exclusively on voluntary commitments. By establishing specific risk thresholds and outlining a court-supervised process for blocking a release, it would create a more formal mechanism for addressing certain high-risk systems.
That does not mean the proposal resolves every concern about industry-led oversight. Questions about evaluator independence, enforcement, and the scope of government authority would remain central to any legislative debate.
Anthropic’s Proposal Highlights a Growing Divide Over AI’s Future
The contrast with other industry leaders is particularly striking. Nvidia CEO Jensen Huang recently argued that the AI industry does not need additional laws, underscoring the disagreement among major technology companies over whether existing rules are sufficient.
Anthropic’s approach suggests that at least some AI developers see a role for targeted legislation when the potential consequences of releasing a model are exceptionally severe.
The proposal now presents lawmakers with a series of difficult choices. They must consider how to define catastrophic risk, how to guarantee credible independent evaluations, and how to balance rapid technological development against the need to prevent serious harm.
Whether Congress ultimately adopts Anthropic’s framework remains uncertain. But the proposal adds a concrete legislative option to a debate that has often been dominated by broad warnings, voluntary commitments, and competing visions of responsible AI development.
As artificial intelligence becomes more capable, the central question is no longer simply whether the technology carries risks. It is whether governments, developers, and independent experts can establish a credible system for identifying those risks and acting before the consequences become irreversible.
