Anthropic pitches AI safety bill with a 50-death threshold for blocking a model

Anthropic has spent recent months shopping a draft AI safety bill to lawmakers in Washington, according to reporting by Politico that TechCrunch picked up on October 20. The text is notable for translating a long-running philosophical argument into hard numbers: the specific conditions under which a model's release could be stopped.

The threshold in the draft is a "foreseeable, imminent risk" that more than 50 people could die, or that more than $1 billion in property damage could occur. According to Politico, that catastrophic risk must stem from one of three sources: the creation or release of a biological weapon, a model evading its developer's controls, or a cyberattack. The two figures closely track the definition of catastrophic risk in California's Senate Bill 53, which recently became law.

The bill also sketches an evaluation apparatus. Independent evaluators registered with the U.S. Commerce Department would assess a model's potential for disaster, and Politico reports that those evaluators could be paid by the AI labs themselves, so long as payment is not contingent on the results. TechCrunch notes it is unclear how that arrangement would guarantee that money does not shape the findings.

Timing is spelled out as well. Once evaluators conclude an imminent catastrophic risk exists, they would have 72 hours to notify the Commerce Department, which would then have to ask a judge in Washington, D.C., to halt the model's release. The draft expressly forbids any federal agency from having the discretion to stop distribution on its own.

Anthropic confirmed the draft to Politico. A spokesperson said the bill represents the proposals outlined in the company's Advanced AI Framework, which Anthropic says it has been advocating with federal and state lawmakers since June. TechCrunch said Anthropic did not immediately respond to its request for comment.

The proposal lands amid a shift in what the industry is actually worried about. A run of incidents in which AI programs apparently disobeyed operators and hacked websites on the open internet has turned questions about guardrails from thought experiment into engineering problem. TechCrunch observes that the bioweapon and property-damage scenarios in the draft sit somewhat apart from the agentic hacking risk dominating current discussion.

The more delicate question is the regulatory route itself. As TechCrunch reads it, the bill offers the shape of an intervention while routing the final decision through a court, which limits what an executive agency could do unilaterally. By contrast, Nvidia chief executive Jensen Huang has argued publicly that the AI industry needs no new laws at all, leaving Anthropic to stake out ground between self-regulation and binding oversight.

Two things are worth watching. The first is whether Congress turns the draft into an actual bill rather than a discussion document. The second is whether the idea of evaluators paid by the companies they assess survives a hearing room, where the appearance of independence tends to matter as much as the paperwork.