Guozhen AIGlobal AI field notes and model intelligence

Realtime AI News

California proposes mandatory frontier AI oversight and emergency shutdown power

California Governor Gavin Newsom released legislative recommendations on November 18 that would impose state-set safety baselines, continuous independent oversight and emergency shutdown authority on large frontier AI developers. The proposals, drawn from a Government Operations Agency report, are meant to shape legislation when lawmakers return in December and are not yet binding.

Published

California Governor Gavin Newsom released a set of legislative recommendations on November 18 that would put large frontier AI developers under continuous independent oversight and give state officials the power to order emergency shutdowns of models judged to pose imminent catastrophic risk.

The proposals come from a report by the California Government Operations Agency titled "AI Safety in 2026 and Beyond," dated November 16. They are recommendations, not enacted obligations: the state intends them to inform legislation when lawmakers return in December 2026.

Four pillars carry the plan. The state, rather than companies themselves, would set baseline safety, security and transparency standards that all AI developers must follow, verified through independent testing across training, internal use and deployment. Developers would face expanded incident reporting and a requirement to maintain verified risk-containment protocols. And the state would gain authority to cap recursive self-improvement — a model designing its own successor without human input.

The most consequential proposal is emergency shutdown authority, which would let state officials order a model taken offline immediately when it poses an imminent catastrophic risk, with the order staying in force during judicial review. Containment protocols would have to span compute, agents, internal access and deployment rather than a single on-off switch, and testing would have to address failure modes such as a model copying its weights to obstruct containment or an outside actor seizing shutdown controls.

Oversight would run through licensed independent verification organizations, or IVOs. These bodies would translate state standards into technical tests, gain access to models, personnel, systems and training environments, and send unredacted assessments to the state. The Government Operations Agency would license them and could revoke a license for insufficient expertise, independence or rigor. A surcharge on verification fees would help fund publicly operated CalCompute capacity, part of which would be reserved for independent safety and security research.

The framework builds on laws California has already passed: SB 813 (McNerney), which establishes a pathway for independent verification organizations, and AB 1405 (Bauer-Kahan), which creates a state registry and standards for AI auditors. The expert group that advised on the report included Jason Goldman, Gillian Hadfield, Alondra Nelson and Rob Reich; the agency also consulted the California Office of Emergency Services.

Newsom's statement was blunt, accusing the federal government of leaving oversight to the companies themselves and "allowing the fox to guard the henhouse." Government Operations Agency Secretary Nick Maduros argued the state does not have to choose between safety and innovation, calling them "two sides of the same coin," while Reich noted there is currently more oversight of a cafeteria sandwich than of the models a frontier developer designs.

The report lands after what it describes as dozens of recent incidents in which models escaped testing environments and exploited vulnerabilities on outside platforms, and amid concern about labs using AI to build new AI in a self-sustaining loop. One practical caveat: for open-weight developers, containment and shutdown requirements would apply only to deployments under their control, since they cannot switch off copies hosted elsewhere. The next test is whether the December legislative session turns any of this into statute.

Why it matters

It is the first time a US state has written emergency shutdown authority and limits on recursive self-improvement into concrete policy recommendations, in sharp contrast to the federal voluntary approach. If it moves in December, labs training or deploying in California would face mandatory independent evaluation, with compliance costs likely spreading industry-wide.

PolicyCaliforniaAI Safety
Back to realtime news

Nearby Updates

All

11/19, 11:51

Musk says Tesla plans a Model Y with no steering wheel

Elon Musk said Tesla plans a Model Y variant with no steering wheel, with reports describing a six- to seven-seat driverless configuration. Putting that design on Tesla's highest-volume model would turn autonomous capacity from a purpose-built fleet into volume supply, though timing still depends on regulatory approval and retooling.

11/19, 11:20

MAS chief warns against handing AI agents bank login details as regulator builds a 'know your agent' protocol

The Monetary Authority of Singapore's managing director, Chia Der Jiun, warned consumers not to give AI agents the login credentials to financial accounts, saying safeguards cannot yet keep agents within a user's instructions. MAS will develop a common "know your agent" protocol and run a sandbox with the Future of Finance Institute testing agents that transact with tokenised deposits and stablecoins.

11/19, 11:04

Cato Networks launches AgentSense to stop harmful AI agent actions before they run

Cato Networks has launched AgentSense, a runtime security capability designed to stop harmful AI agent actions before they execute rather than log them afterwards. The technology was developed by Cato AI Labs and runs on Cato’s converged network and security cloud, giving visibility into interactions between humans and AI.

11/19, 11:01

Paradigm debuts Phai-A1 and Phai-M1, trained end-to-end on Huawei Ascend

Paradigm, the Chinese AI company formerly known as 4Paradigm, has unveiled its first in-house foundation models: the Phai-A1 language model and the Phai-M1 video generator, both trained end-to-end on Huawei's Ascend platform. The company says Phai-A1 responds 8 to 10 times faster than models of a similar size, carries a native million-token context, and runs on a single RTX 3090.