California governor vetoes contentious AI safety bill

WASHINGTON/SAN FRANCISCO (Reuters) – California Governor Gavin Newsom on Sunday vetoed a hotly contested artificial intelligence safety bill after the tech industry raised objections, saying it could drive AI companies from the state and hinder innovation.

Newsom said the bill “does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data” and would apply “stringent standards to even the most basic functions — so long as a large system deploys it.” 

Newsom said he had asked leading experts on generative AI to help California “develop workable guardrails” that focus “on developing an empirical, science-based trajectory analysis.” He also ordered state agencies to expand their assessment of the risks from potential catastrophic events tied to AI use.

Generative AI – which can create text, photos and videos in response to open-ended prompts – has spurred excitement as well as fears it could make some jobs obsolete, upend elections and potentially overpower humans and have catastrophic effects.

The bill’s author, Democratic State Senator Scott Wiener, said legislation was necessary to protect the public before advances in AI become either unwieldy or uncontrollable.

The AI industry is growing fast in California and some leaders questioned the future of these companies in the state if the bill became law.

Wiener said Sunday the veto makes California less safe and means “companies aiming to create an extremely powerful technology face no binding restrictions.” He added “voluntary commitments from industry are not enforceable and rarely work out well for the public.”

“We cannot afford to wait for a major catastrophe to occur before taking action to protect the public,” Newsom said, but added he did not agree “we must settle for a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities.”

Newsom said he will work with the legislature on AI legislation during its next session. It comes as legislation in U.S. Congress to set safeguards has stalled and the Biden administration is advancing regulatory AI oversight proposals.

Newsom said “a California-only approach may well be warranted – especially absent federal action by Congress.”

Chamber of Progress, a tech industry coalition, praised Newsom’s veto saying “the California tech economy has always thrived on competition and openness.”

Among other things, the measure would have mandated safety testing for many of the most advanced AI models that cost more than $100 million to develop or those that require a defined amount of computing power.

Developers of AI software operating in the state would have also needed to outline methods for turning off the AI models, effectively a kill switch.

The bill would have established a state entity to oversee the development of so-called “Frontier Models” that exceed the capabilities present in the most advanced existing models.

The bill faced strong opposition from a wide range of groups. Alphabet’s Google, Microsoft-backed OpenAI and Meta Platforms all of which are developing generative AI models, had expressed their concerns about the proposal.

Some Democrats in U.S. Congress, including Representative Nancy Pelosi, also opposed it. Proponents included Tesla CEO Elon Musk, who also runs an AI firm called xAI.

Amazon-backed Anthropic said the benefits to the bill likely outweigh the costs, though it added there were still some aspects that seem concerning or ambiguous.

Newsom separately signed legislation requiring the state to assess potential threats posed by Generative AI to California’s critical infrastructure.

The state is analyzing energy infrastructure risks and previously convened power sector providers and will undertake the same risk assessment with water infrastructure providers in the coming year and later the communications sector, Newsom said.