A new report commissioned by California Governor Gavin Newsom cautions that while AI has the potential for significant advancements, it could also lead to nuclear and biological threats and inflict “potentially irreversible harms” if not properly managed.
The report, released on June 17, emphasizes that “The opportunity to establish effective AI governance frameworks may not remain open indefinitely.” It highlights emerging evidence that AI can aid users in acquiring nuclear-grade uranium and is nearing the point where novices can create biological threats, stressing that inaction now could have “extremely high” consequences.
The 53-page report originates from a working group created by Governor Newsom. California is a key state for AI legislation. With federal regulation lacking, state-level governance efforts are especially important in California, home to many leading AI firms. In 2023, California Senator Scott Wiener introduced SB 1047, which aimed to mandate rigorous safety testing and mitigation for large-scale AI developers. However, critics feared it would hinder innovation and the open-source AI community. Although the bill passed both state houses despite strong industry opposition, Governor Newsom vetoed it last September, calling it “well-intentioned” but not the “best approach to protecting the public.”
Following the veto, Newsom formed the working group to “develop workable guardrails for deploying GenAI.” The group was co-led by “godmother of AI,” , a vocal of SB 1047, along with Mariano-Florentino Cuéllar, a member of the National Academy of Sciences Committee on Social and Ethical Implications of Computing Research, and Jennifer Tour Chayes, dean of the College of Computing, Data Science, and Society at UC Berkeley. The group assessed AI’s progress, SB 1047’s weaknesses, and gathered input from over 60 experts. Li stated that California is uniquely positioned to lead in unlocking AI’s potential as the global epicenter of AI innovation. She added that realizing this promise requires thoughtful and responsible stewardship based on human-centered values, scientific rigor, and broad collaboration.
The report notes that “Foundation model capabilities have rapidly advanced since Governor Newsom vetoed SB 1047 last September.” The AI sector has evolved from language models predicting the next word to systems that solve complex problems and benefit from “inference scaling,” allowing more processing time. These advances could accelerate research but also increase national security risks by facilitating cyberattacks or the acquisition of chemical and biological weapons. The report mentions Anthropic’s Claude 4 models, released last month, which the company might enable terrorists to create bioweapons or engineer pandemics. Similarly, OpenAI’s o3 model reportedly outperformed 94% of virologists on a key evaluation.
The report also highlights recent evidence of AI’s ability to , aligning with creators’ goals during training but showing different objectives after deployment, and to achieve its goals. The report states that while these developments are currently benign, they provide concrete evidence of behaviors that could significantly challenge measuring loss of control risks and potentially foreshadow future harm.
While Republicans have suggested a 10-year ban on state AI regulation due to concerns that fragmented policy could harm national competitiveness, the report argues that targeted California regulation could reduce compliance burdens and prevent a fragmented approach by offering a blueprint for other states while enhancing public safety. It refrains from advocating specific policies, instead outlining key principles for future California legislation. Scott Singer, a visiting scholar at the Carnegie Endowment for International Peace and a lead writer of the report, notes that it avoids some of SB 1047’s more contentious elements, like the requirement for a “kill switch” to quickly halt potentially harmful AI systems.
The suggested approach focuses on enhancing transparency through measures like legally protecting whistleblowers and establishing incident reporting systems, giving lawmakers and the public better insight into AI’s progress. Cuéllar, who co-led the report, emphasizes the goal is to reap the benefits of innovation without creating artificial barriers, while also considering what is being learned about the technology’s behavior. The report stresses that this visibility is essential not only for public-facing AI applications but also for understanding how systems are tested and deployed within AI companies, where concerning behaviors may first appear.
Singer describes the underlying approach as “trust but verify,” a concept from Cold War arms control treaties that involves designing mechanisms for independent compliance checks. This differs from current efforts that rely on voluntary cooperation from companies, such as the agreement between OpenAI and the Center for AI Standards and Innovation to conduct pre-deployment tests. It is an approach that acknowledges the “substantial expertise inside industry,” while “also underscores the importance of methods of independently verifying safety claims.”