The draft bill Anthropic pitched to lawmakers would only allow the government to block artificial intelligence models if they could kill or seriously injure 50 people or more or cause $1 billion in damage.

That threshold — which has ricocheted across Capitol Hill and social media since POLITICO first reported the company was shopping it — may look familiar.

Anthropic’s proposal closely tracks California’s landmark SB 53, a model for legislation elsewhere. But that law uses the damage threshold to define how AI companies must assess and disclose catastrophic risks to state officials, not to decide when a model’s release should be blocked.

Such a shift puts a consequential question at the center of the debate over advanced AI: How much potential harm should it take for the government to intervene? Anthropic’s proposal, first reported by POLITICO, would help set that bar.

“I think many people are appropriately finding it unnerving that companies seem to be contemplating situations in which their models might be involved in really severe catastrophes,” said Nathan Calvin, general counsel of AI safety organization Encode AI, which cosponsored SB 53. “An imminent risk of 50 people dying or $1 billion in damages rightfully strikes many people, including myself, as a frighteningly high bar.”Anthropic did not respond to a request for comment.

Under the bill, a federal judge, at the request of the Commerce Department, would only be able to block model release if there is imminent, foreseeable enumerated risk. A model reaches this threshold if it is believed to be able to cause more than $1 billion in damages or kill or seriously injure more than 50 people via a cyberattack, the model evading its developer’s controls or the creation or release of a biological weapon.

It was designed to mirror Anthropic’s Advanced AI Framework, published in June, that lays out the company’s vision for how to regulate the emerging technology, according to an Anthropic spokesperson.

The risk language mirrors similar risk thresholds in SB 53, the California bill that Gov. Gavin Newsom (D) signed into law last year, the RAISE Act, which Gov. Kathy Hochul (D) approved in December and Illinois’ SB 315 signed by Gov. JB Pritzker (D). Each of the bills built on one another, using similar verbiage.

Anthropic’s own head of federal affairs, Anthony Cimino, said on social media that the definitions were “pulled directly from existing standards in California law.”Anthropic previously endorsed SB 53 and other state laws around the country.

The laws in California, Illinois and New York were centered on fostering transparency and requiring companies to report incidents of risk. Anthropic’s federal version, meanwhile, would use similar criteria to dictate when a model can be blocked by a federal court.

Given the difference in how these definitions are used, there is a “big question on whether the threshold is still appropriate,” said Michelle Lopes Maldonado, the associate director of AI policy at the Information Technology and Innovation Foundation, a think tank whose corporate donors include Anthropic. Alphabet, Meta and Microsoft.

California State Sen. Scott Wiener (D), the primary sponsor on SB 53, said on social media that using the same definition to decide whether models should be blocked “dramatically limits the situations in which the federal government can stop model deployments.”“Misuse of that law’s definition would effectively gut the emergency shutdown provisions in this proposal,” he said. Others, however, think the threshold is fine as-is.

“I think it’s a decent way of doing it,” Charlie Bullock, Institute for Law and AI’s senior policy director, said in an interview.

The goal with the definition, Bullock and other experts POLITICO spoke with said, is to help narrow what the bill is intended to do.

AI chatbots have been blamed for some user deaths, including by allegedly encouraging minors to commit suicide or self harm. If the threshold were to be lowered, it may allow federal judges to block the model if they believe it could cause harm in a smaller subset of its users, which is not the intention of bills focused on catastrophic risk, Bullock said. But with a higher threshold, it focuses the court’s judgment on whether the model could cause widespread harm.

The definition, Bullock said, “does a rough and ready job of distinguishing between very serious public safety, national security risks from AI, and more short-term, not quite as serious risks.”Causing over $1 billion in damage may seem too high a bar to clear, Lopes Maldonado said. But “you can get there pretty quickly,” she said, adding that she is still undecided about whether a lower threshold should be applied.

Adam Kovacevich, the founder and CEO of tech industry coalition Chamber of Progress, sees Anthropic’s use of the definition as a way to “defang” opposition to preemption.

Some lawmakers, with the support of much of the tech industry, have proposed preempting state AI laws with a federal standard, thus eliminating a patchwork of individual state laws that companies say is difficult to comply with. But such proposals have faced stiff opposition, particularly from Democrats, who do not want to see strong state laws neutered by a watered-down federal version.

If their bill paralleled laws passed in California, New York and Illinois, Kovacevich said, Anthropic may be able to make the case to lawmakers that their bill was as strong as the state laws, so preemption would simply shift the protections from the state to the federal level. But Kovacevich, Bullock and others said that Anthropic’s proposal is comparatively weaker than other proposals, including the FRONTIER Act, introduced earlier this summer by Reps. Lori Trahan (D-Mass.) and Jay Obernolte (R-Calif.). Cimino said the company would continue to tinker with the preemption language.

“We circulated these ideas because it’s important to get early feedback from policymakers,” Cimino said. “At its core, our proposal is intended to be the strongest on the table.”