California proposes mandatory AI safety standards and emergency shutdown power

Governor Gavin Newsom recommended that the state, rather than companies, set baseline safety standards every AI company must follow. The package calls for independent safety testing and verification across training, development and deployment, plus expanded incident reporting. Companies would also need verified risk-containment protocols that allow the state to order an immediate shutdown in an emergency.
The release builds on California's existing AI safety framework and adds limits on recursive self-improvement, a scenario in which a model designs its own successor without human input. Newsom criticized the federal government for leaving AI oversight largely to the companies themselves, and the full report is published with the announcement.
For founders building in California, this points toward compliance work beyond disclosure, since independent testing, shutdown procedures and documented containment could become baseline requirements. Teams building agents that modify or deploy models should map which of their systems would fall under these standards.