CEOs of the companies racing to build smarter AI—Google DeepMind, OpenAI, xAI, and Anthropic—have been clear about the stakes. Each has, at times, warned that powerful AI could spell disaster for humanity. And yet, in an industry shrouded in trade secrets and nondisclosure agreements, would-be whistleblowers have faced a difficult choice: stay silent, or come forward and face the potential legal consequences.
A landmark California law aims to make that choice easier. The Transparency in Frontier Artificial Intelligence Act (SB 53), signed by Governor Gavin Newsom on Sept. 29, offers the legal shield that insiders have long demanded, protecting those who blow the whistle from retaliatory action from their employers.
The law applies to any company operating in California that uses massive amounts of computing power for model training, a threshold designed to regulate only cutting-edge developers while exempting smaller firms. It pairs these new whistleblower protections with transparency requirements, mandating that developers publish risk mitigation plans and share safety test results, codifying promises that critics say have sometimes been broken. Companies must also report “critical safety incidents” to the California Office of Emergency Services (OES).
Read more: 60 U.K. Lawmakers Accuse Google of Breaking AI Safety Pledge
Why Whistleblowers Have Been Calling For Protection
While “Godfather of AI” Geoffrey Hinton resigned from Google in 2023 to speak freely about the technology’s existential risks, calls for whistleblower protections didn’t emerge until May 2024, when Vox reported OpenAI’s use of “extremely restrictive offboarding agreements” to stifle criticism. OpenAI staffers received much of their compensation in the form of equity, as is common in Silicon Valley. But when leaving the company, to keep their vested equity, they were reportedly required to sign an agreement that contained provisions eternally forbidding them from criticizing their former employer, and that prevented them from mentioning the agreement’s existence. This came to light after one former employee, Daniel Kokotajlo, posted to an online forum about his refusal to sign—believing he had forfeited millions of dollars worth of vested equity.
The following day, OpenAI CEO Sam Altman took to X to deny any knowledge of these provisions, stating, “This is on me and one of the few times I've been genuinely embarrassed running OpenAI; I did not know this was happening and I should have.” OpenAI subsequently confirmed, in messages sent to current and former employees, that these provisions would no longer be enforced, and that they would remove the offending language from all exit paperwork going forward. But the credibility of Altman’s denial was called into question when, on May 22, leaked documents appeared to show his signature—as well as the signatures of other senior executives—on documents that explicitly authorized the provisions. In June 2024, in the wake of the controversy over OpenAI’s then-rescinded off-boarding agreements, Kokotajlo and 13 current and former employees of OpenAI and Google DeepMind called for stronger whistleblower protections in an open letter.
What the New Law Protects
Fulfilling a key recommendation of the open letter, California’s new law goes beyond ordinary whistleblower protections that only cover illegal activity. It bars AI developers from imposing rules that stop employees responsible for assessing risks from reporting violations of the act or “catastrophic risks”—defined as endangering 50 lives or causing $1 billion in damages—with the Attorney General, regulators, or their managers. Employees don’t have to be certain of a catastrophic risk, just have “reasonable cause” to believe it, to be covered by the new legislation.
California’s Whistleblower Protections Only Go So Far
The new protections mark “an important step forward,” says Jacob Hilton, a former OpenAI researcher and one of the signatories of the open letter. Nevertheless, he says he’d like to see protections adopted at the federal level, pointing to the bipartisan “AI Whistleblower Protection Act” introduced in Congress in May. Hilton notes that the threshold for “catastrophic risk” means smaller but still serious harms wouldn’t be covered. Another of the open letter’s signatories, speaking on the condition of anonymity, told TIME that while addressing catastrophic risks was key, “there might be other areas of risk where we would want to have other protections.”
Going beyond mere disclosures of illegal activity is a crucial provision, says Lawrence Lessig, a professor of law at Harvard who provided pro bono legal advice to Kokotajlo and other signatories of the letter. Though Lessig worries about uncertainty over what’s protected dissuading employees from coming forward. He says he would have preferred the standard to be lowered to “good faith,” which focuses on an employee’s genuine belief rather than the strength of their evidence. As Lessig puts it, this would be a "much simpler standard for a techie to be able to rely upon" than forcing them to act as a lawyer or a risk analyst. Under the new legislation, the lower "good faith" standard is reserved exclusively for an internal, anonymous reporting channel within large AI companies.
Lessig says the risks taken by past whistleblowers underscore the need for such protections. “When you've got people inside who are willing to give up $2 million for the simple freedom to speak, we ought to be keen to understand what they want to say,” he says.