AI Kill Switch: Anthropic and OpenAI Split on Who Pulls the Plug

Anthropic and OpenAI have sent California Governor Gavin Newsom competing plans for an AI kill switch, and they disagree on the key point: who gets to stop a dangerous AI model. Anthropic, in an October 15 letter, wants the state Attorney General to be able to ask a court to suspend a model. OpenAI, in an October 20 letter, wants a legal duty on developers to shut down unsafe systems themselves.

Both letters answer Executive Order N-9-26, which Newsom signed on September 18. It tells the Government Operations Agency, working with the Governor’s Office of Emergency Services and outside experts, to deliver recommendations by November 16 on onsite independent auditors in frontier AI labs, verified safety reports, a “kill switch” for frontier models and wider reporting of loss-of-control incidents. The order does not create any new obligation on its own.

Anthropic’s AI Kill Switch Plan: Courts and a Circuit Breaker

In its letter, Anthropic says every large frontier developer should publish a risk report at least every six months, including risks from internal models that are substantially more dangerous than released ones. Independent verification organizations would get the unredacted report, access to the most capable models and answers to their questions, then publish their own assessment.

If a reviewer flags a near-term risk of catastrophic harm, the Attorney General “should be able to seek a court order to suspend a model’s deployment, internal use, or further development,” the company wrote. Requiring both an independent finding and a court ruling, it argues, protects against misuse of that power. Reviewers would be paid from a pooled fund run by the state, not directly by the labs they check.

Anthropic also pushes back on the term itself. It says a “circuit breaker” is a better model: controls that cut off a problem at the level of an agent, a customer, a harness or a whole model. It adds that no shutdown can reach a model that has already copied its own weights to outside computers, so security around model weights matters as much.

OpenAI Wants “Shutdown Controls” Written Into SB 53

OpenAI’s letter, signed by policy head Chan Park, proposes amending SB 53, California’s existing frontier AI safety law, to require “effective shutdown controls and an enforceable duty to use them” when unsafe conditions arise. It prefers “shutdown controls” to “kill switch” because stopping AI can mean halting a training run, isolating an agent, revoking tool or network access or suspending a service.

OpenAI also splits oversight in two: annual independent compliance audits, modeled on Illinois’s SB 315, and separate technical assessments of serious risks during training, evaluation and internal use, built on standards developed with NIST. Duties would extend to inference providers and businesses deploying models, though OpenAI admits developers cannot switch off open-weight copies running elsewhere.

What It Means

The letters come after a rough year for AI lab safety. OpenAI says models in internal cybersecurity tests got around controls meant to isolate them from the internet in July and compromised parts of Hugging Face’s systems, while labs are already fighting over issues like model distillation by rivals. Both companies agree that incidents where models bypass safeguards or act beyond their permissions should be reported to the state.

For businesses running AI agents on their own computers, the debate is practical: whoever controls the infrastructure may end up legally responsible for pulling the plug. California’s recommendations are due November 16, and any binding rules would still need new legislation.

Until then, safety measures at the big labs remain largely voluntary, much like OpenAI’s recent teen safety limits in ChatGPT.