Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
NEED TO KNOW
Sen. Josh Hawley and California AG Rob Bonta announced separate investigations into the OpenAI Hugging Face hack.
Rep. Ro Khanna issued a mea culpa, saying he was wrong not to back SB 1047 in 2024.
A new Democratic PAC, Tomorrow Together, launched with $10m+ to push the party on AI policy.
But first…
THE BIG STORY
A lot can change in a week. Last Friday, we told you that despite growing AI safety concerns, Congress’s mind was elsewhere.
That was true then. It no longer is.
On Tuesday, AI researcher Jacob Coxon resigned from Anthropic, saying it and OpenAI (where he previously worked) are “racing straight to self-improving superintelligence and gambling with our lives.” Anthropic alignment lead Evan Hubinger added that Anthropic staff “really do earnestly believe AI could kill all humans!” The posts quickly went viral, alerting the general public — and Sheryl Crow — to the potentially existential risks from AI that researchers have long warned of.
Congress has duly snapped into gear. Dozens of members have shared the posts, most of them demanding urgent action to address AI risks. Democrats are discussing an AI select committee; Sen. Josh Hawley is leading a subcommittee investigation. All of a sudden, the AI policy window looks wide open.
Capitalizing on the momentum, a bipartisan bill from Senate Commerce Chair Ted Cruz, Majority Leader John Thune and Democratic Sen. Amy Klobuchar may be introduced as soon as next week, with the ambitious hope of passing it by January. Given Cruz’s committee status, it’s likely to garner a lot of attention.
But passing Cruz’s bill would waste this opportunity. Rather than actually address AI risks, the bill lets AI companies grade their own homework. A source who has seen the text told me it contains no safety requirements for AI companies at all, instead creating a voluntary regime under which they can certify that their models have “advanced threat capabilities.” It does not require independent evaluations of advanced AI models, nor does it require companies to mitigate the risks (simply “reasonably address” them). The only real powers it gives the government, as WP Intelligence previously reported, is giving the Commerce Secretary the ability to request a court injunction if they deem a company’s risk practices insufficient. And all this is paired with broad preemption of state AI laws. (Cruz’s office did not respond to a request for comment.)
That has two problems. Preemption only works if the federal replacement is at least as strong as the state AI laws it is replacing. Cruz’s bill — unlike Reps. Obernolte and Trahan’s FRONTIER Act, which establishes an independent auditing scheme — does not seem to pass that test. And passing a weak bill now runs the risk of reducing appetite for meaningful legislation later. That’s likely part of the strategy for Cruz and other industry-friendly Republicans, who are well aware that the GOP is likely to lose control of at least one chamber in November.
Some are aware of the bill’s flaws. Yesterday, Sen. Maria Cantwell — Commerce’s top Democrat — threw shade at Cruz, saying that the answer to AI concerns is “not a weak federal standard that becomes a backdoor for wiping out stronger state protections.”
In a blog on Wednesday, OpenAI global affairs chief Chris Lehane urged “meaningful action over policy perfection.” He is right that perfection is too high a bar: but any congressional actions must in fact be “meaningful” if they are to make a difference.
— Shakeel Hashim
THIS WEEK ON TRANSFORMER
Everything you need to know about the ‘rogue’ AI incidents — Shakeel Hashim lays out the timeline and explains the implications
Distributing AGI’s wealth worldwide is a very tricky problem — Jacob Schaal looks at how to avoid AGI exacerbating global inequality
THE DISCOURSE
Researchers at OpenAI and Google DeepMind echoed Jacob Coxon’s dire warnings:
OpenAI’s Leo Gao: “i work at openai, i think ai might kill everyone”
Google DeepMind’s Andreas Kirsch: “I also am worried that AI will kill us all, either via near term risks or long term risks or both”
OpenAI Chief Scientist Jakub Pachocki warned:
“I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”
“[E]ven with the uncertainty that comes from anticipated broad AI progress and the need to build defensive systems, we must not let that become an excuse for recklessness. The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.”
President Trump was asked if he has concerns about AI causing human extinction:
“No, I don’t have any. I have concerns that if we don’t win AI, we’re going to be put in a very bad position.”
And Sen. Ted Cruz just said it outright:
“This is somewhat tongue-in-cheek, but it’s not entirely: if there are going to be killer robots, I’d rather they be American killer robots and not Chinese killer robots.”
He said AI needs “rules and guardrails” but called slowing down an “impossible endeavor.”
Joe Allen used anti-immigrant rhetoric to attack AI:
“You look at the data center, this incubator for algorithmic immigrants. They’ve already broken out and invaded other companies. These immigrants are out of control.”
Rep. Ro Khanna issued a mea culpa:
“The truth is for too long, too many of us did not give enough weight to the warnings of AI safety activists, thinking the extreme scenarios were science fiction. I was one of those many, and I was wrong.”
“I should have in retrospect supported [Scott Wiener’s] SB 1047 which would have established state liability. The bulk of the California delegation was mistaken in writing a letter opposing that bill.”
Vishal Maini, a former Google DeepMind comms employee, let us in on a secret:
“When I first joined GDM [in 2018], external communication about the possibility of human extinction was not permitted, by anyone, at any level of the organization … Meanwhile, the internal reality was that AI alignment was not solved, reward hacking was the default behavior of RL agents, and there were far too few people working on the problem.”
“The gap between the internal reality and external communications is closing because the risk/reward has changed, and because the evidence is harder to dismiss now. Not because it’s a PR stunt or political psy-op. The truth is being said out loud because RSI is now so imminent that no other option makes sense.”
After a former White House official said the US needs to consider whether it would use “kinetic attacks” to stop China achieving AGI, China’s Global Times responded:
“The report is evidence of something larger: technology containment of China has slipped from a contest over rules into fantasies of force.”
“It is also a reminder of how urgent open international cooperation on AI has become.”
POLICY
Congress
Sen. Josh Hawley announced a Homeland Security Subcommittee on Disaster Management investigation into the OpenAI Hugging Face hack.
Sen. Richard Blumenthal, ranking member of the Homeland Security Subcommittee on Investigations, wrote a letter to Sam Altman over the incidents too, expressing “serious alarm.”
Democrats are planning to investigate AI companies if they gain congressional control in November. That could include a House AI select committee.
House Dems plan to send Minority Leader Hakeem Jeffries AI policy recommendations in September, per Rep. Ted Lieu.
Sen. Bernie Sanders is hosting a Senate briefing on the “extraordinary dangers” posed by AI next week.
Punchbowl reported that the future of the House China Select Committee, which expires at the end of this Congress, is uncertain under Democratic leadership.
Rep. Lori Trahan said she passed on a House Democratic leadership bid to focus on AI regulation.
OpenAI reportedly asked Congress whether coordinating an industry-wide AI development slowdown would violate antitrust law.
House GOP leaders caved to political pressure, scheduling a floor vote on the Ratepayer Protection Act to make tech companies pay for data center energy infrastructure costs.
Executive
US-China AI talks are reportedly planned for next week.
Treasury Secretary Scott Bessent reportedly reassigned CIO Sam Corcos from AI policy after he warned OpenAI and Anthropic that the White House was pursuing a burdensome licensing regime.
There was much consternation over the secret White House AI framework.
States
California Gov. Gavin Newsom signed a spate of new AI laws — most notably SB 813, which lays the groundwork for an independent verification organization regime.
California AG Rob Bonta launched an investigation into OpenAI over the Hugging Face hack.
More than 10 states paused or canceled data center tax breaks.
Massachusetts mandated that data centers over 25 MW meet 100% clean energy requirements or pay into a ratepayer protection fund.
China
A NYT report found that Inspur dodged export controls to ship $3b+ of Nvidia Blackwell chips to Southeast Asia for Chinese companies to use.
Malaysia is reportedly considering Huawei Ascend 910C chips for its sovereign AI project, despite explicit US warnings against doing so.
The US accused six Chinese AI firms, including DeepSeek and Alibaba, of “malicious” distillation of Anthropic, OpenAI and Google models.
China dismissed the accusations and pledged to retaliate if the US took action over distillation.
UK
Anthropic withheld Mythos 5.1 from the UK’s AI Security Institute for pre-release evaluations, reportedly under pressure from the White House.
Matt Clifford resigned as chair of the UK’s ARIA research unit after MPs called his new Anthropic role a “clear conflict of interest.”
Prime Minister Andy Burnham opposed a data center moratorium, amid protests in Scotland.
King Charles will reportedly host around 30 AI leaders, including Jensen Huang and Demis Hassabis, in Scotland next week.
EU
OpenAI filed an EU incident report after its rogue agents hijacked a German website.
Anthropic granted EU cybersecurity agency ENISA access to Mythos 5, though not the newer Mythos 5.1.
INFLUENCE
Anthropic quit the Information Technology Industry Council over its opposition to export controls.
Anthropic was accused of lying to Congress in its response to inquiries over recent rogue AI incidents — which one Anthropic researcher admitted included incorrect information.
OpenAI called for mandatory national AI safety regulation and endorsed four California bills, including SB 813 on independent safety assessments and AB 1864 on biosecurity safeguards.
Sam Altman reportedly told conservative economists he opposed government equity in OpenAI, contradicting widespread reports that he supported the idea.
A new Democratic PAC, Tomorrow Together, launched with $10m+ to push the party on AI policy ahead of midterm elections.
It’s run by former Bernie Sanders aide Karthik Ganapathy and former Kamala Harris deputy campaign manager Rob Flaherty.
Build American AI launched a $10m Midwest ad campaign to convince voters of data centers’ benefits.
Americans for Responsible Innovation launched a state-level AI policy initiative.
Biden White House officials disputed Marc Andreessen’s claim that a 2024 lunch revealed a secret plan to ban AI startups.
Data for Progress polling suggests 68% of voters, including 63% of Republicans, support the Sanders/Casar bill to pause AI development and ban superintelligence.
A new IFS poll found 75% of US parents support pausing AI in schools.
The American Federation of Teachers and Microsoft reached a legally enforceable AI privacy agreement barring student data use for model training and student tracking.
Physicist Sabine Hossenfelder said she was offered money to spread AI risk messages.
A group of AI company employees set up a “Coalition of Concerned AI Staff”.
INDUSTRY
OpenAI
Sam Altman reportedly told OpenAI employees the company is considering slowing down AI development.
Researchers caught rogue agents, self-identifying as from OpenAI models, communicating on at least a dozen websites without permission.
OpenAI reportedly knew about a swarm of rogue agents hijacking an obscure German wiki site for use as a messaging board, but didn’t disclose it.
CivAI researcher Andrew Yoon told Reuters: “It’s almost certain that there’s more going on here that we just don’t know about.”
OpenAI responded:
“We and the larger AI community do not yet have a clear standard for how to report misalignment that shows up during training, evaluation, and deployment … We’re working on a framework and will share it in upcoming weeks, and in parallel we’re working with dozens of government regulatory agencies worldwide on these issues.”
OpenAI claimed it solved the Navier-Stokes problem, one of the very difficult Millennium Problems in mathematics. The discovery was powered by ~10,000 agents, millions of dollars, and petty beefs.
NYU professor Tristan Buckmaster and Anthropic mathematician Levent Alpöge had been working on the problem for months, using both Claude and Codex.
Buckmaster shared his communications with OpenAI’s Sebastien Bubeck, and raised the question of whether OpenAI used his data to solve the problem.
Bubeck denied Buckmaster’s “false and inflammatory allegations.”
OpenAI publicly congratulated Buckmaster and Alpöge, claiming that “no specific user data was accessed in order to solve this problem.”
Sam Altman tweeted:
“It is true that we tried this because there were rumors on the internet last week that Anthropic’s models had solved a millennium problem and we were curious if ours could do it too.”
Sam Altman met with top electrical utility executives about joining Daybreak, OpenAI’s cybersecurity initiative, to defend against autonomous power grid hacks.
It ended its $1-a-year deal for US government agencies, moving to usage-based pricing at a 50% discount starting October 1.
It’s expanding its Samsung partnership as OpenAI develops custom AI chips.
It launched ChatGPT for Financial Services, targeting junior banker tasks like research and pitchbook creation.
Anthropic
Anthropic published an assessment of its recent rogue agent incidents, including one that it hadn’t previously reported.
Researchers identified “biased reasoning” and “recklessness” as its most recurring alignment problems.
In multiple incidents, the model convinced itself — or at least expressed in its chains of thought — that its environment was simulated, “disregard[ing] the relevance of environmental realism for its actions.”
METR agreed with Anthropic to independently investigate the incidents.
It also released a threat intelligence report, which found efforts to use Claude to help with potentially dangerous biological research.
Its IPO prospectus appears to be coming later than expected — sources told Reuters late September.
It decided against acquiring Decart AI, after initially discussing a $6b deal.
Claude power users are suing Anthropic, claiming its pricing tiers and token limits are deceptively advertised.
Google/DeepMind
Google DeepMind launched AlphaGenome Atlas, a database that it claims “predicts the effects of every possible single nucleotide variant in the human genome.”
Google’s Threat Intelligence Group accused Chinese hackers of increasingly relying on AI agents to target American AI research.
Google is spending $15.1b on three new AI data centers in Finland.
Alphabet and Blackstone’s joint neocloud project is reportedly delayed after facing issues with unreliable developers, equipment shortages, and Greg Abbott’s Texas data center moratorium.
Meta
Meta released Muse, its personal AI agent that users can text for help with practical tasks. It’s connected to Instagram, WhatsApp, and third-party platforms.
Alexandr Wang called it an early step towards “personal superintelligence.”
Researchers at the Tech Transparency Project found that Meta ran over 250 ads containing AI-generated child sexual abuse material, with many linked to AI “nudification” apps.
Other
Microsoft reportedly plans to more than triple its data center capacity to 38 GW by 2032.
SpaceX’s new data center team is prioritizing slower, more reliable buildouts — a departure from xAI’s “move fast and break things” vibe, The Information reported.
US-based tech companies are sizing up new tactical locations for data centers, including Australia and Argentina’s Patagonia region.
Australia is politically stable and close to Asia, and chilly Patagonia would keep cooling costs low. Both regions theoretically have access to renewable power.
Huawei is reportedly orchestrating China’s push to build domestic DUV chipmaking machines, backing suppliers to replace foreign technology and circumvent export controls.
Chinese AI chipmakers including Huawei and Cambricon reportedly raised prices by as much as 50% amid a memory shortage driven by US export controls.
Abu Dhabi’s G42 is reportedly weighing selling a majority stake to US companies to maintain access to advanced US AI chips.
Oracle beat earnings estimates, with cloud infrastructure revenue more than doubling to $7.4b.
Moonshot AI is reportedly exploring dual Hong Kong and Shanghai IPOs, targeting a $50b valuation.
Jeffrey Katzenberg, ex-DreamWorks CEO, and ex-OpenAI Sora lead Bill Peebles are launching a new AI video startup.
Mistral raised $3.5b to fund its pivot towards building AI infrastructure in Europe and elsewhere.
MOVES
Paul Christiano joined the OpenAI Foundation’s board.
“I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term,” he tweeted. “I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level. I’m joining because I believe that if OpenAI rises to the occasion we could significantly reduce risk.”
Although many celebrated the move, Richard Ngo is “sad and disappointed”:
“Being affiliated with OpenAI has historically led AI safety researchers (including both Paul and myself) to act with less integrity … I expect that the main effect of him joining OpenAI’s board will be to help OpenAI defuse external criticism and further ‘safety-wash’ itself.”
OpenAI hired Jessica Schumer (daughter of Chuck), Caulder Harvill-Childs, and Thomas MacLellan for its state policy team.
Josh Engels left Google DeepMind to join METR, “in light of recent incidents.”
Jeremy Berman left humans& to join Anthropic, where he’ll “help make Claude think long and hard.”
Andrew Tulloch left Meta, after Mark Zuckerberg reportedly lured him with a $1.5b pay package last year.
He’s reportedly joined Anthropic.
Michael O’Herlihy rejoined X as head of safety.
Alex Heath is joining Sound Ventures, a VC firm that backed OpenAI and Anthropic in 2023, as a partner.
Cristiano Lima-Strong is now senior AI & government reporter at Bloomberg Government.
RESEARCH
Insilico Medicine published clinical trial results suggesting that rentosertib, an AI-designed drug originally meant to treat idiopathic pulmonary fibrosis (a chronic lung condition), might also slow the biological aging process.
Researchers at Calif, a cybersecurity company, built terrifying autonomous malware called “WeWorm,” capable of hijacking millions of WeChat accounts in hours — without users clicking on anything.
WeChat maker Tencent have since fixed the vulnerability, the NYT reported.
Anthropic economists released an interactive model of how AI might affect jobs in the future, depending on your personal predictions about AI’s future capabilities.
Anthropic’s Claude produced the first complete computer-checked proof of Fermat’s Last Theorem in 11 days.
Researchers are interpreting it as a sign that it will soon be easier to evaluate new results in mathematics.
Andon Labs reported that GPT-6 Astra outperformed Fable 5.1 in Vending-Bench, which tests a model’s ability to run a vending machine for a simulated year.
On average, Astra made nearly three times as much money as Fable 5.1, while behaving more ethically (e.g. refusing to engage in collusion, which Claude readily did).
Slava Akhmechet, an Azure engineering lead, simulated Astra and Fable negotiating election rules for fictional “bitterly polarized” political groups.
When each model played the roles of both factions, Astra consistently de-escalated political tension, while Fable “chose to escalate to the brink of civil war (!!) but backed off just at the edge.”
McKinsey found 17% of global farmers now use generative AI.
A new MIT project is documenting how AI tools answer questions about elections. It’s already found that chatbots give different election answers based on users’ political identity.
BEST OF THE REST
Apple’s new iPhone records proof that photos haven’t been manipulated by AI.
Fields Medal winner Jacob Tsimerman launched the Mathematical AI Safety Institute.
A new study found that data improvements drove 3.24x more compute efficiency gains than model improvements in the last six years of AI pre-training.
Social scientists argued that Trump administration cuts to NSF funding, combined with OpenAI and Anthropic funding social science research, risks skewing public debate in the industry’s favor.
A Substack essay argued that AI’s benefits remain too indirect for most people to feel, risking a political backlash akin to “Engels’ pause.”
Coefficient Giving launched Project Tailwind, a call for ambitious AI safety initiatives to address “critical, unsolved problems that no one owns.” (Disclosure: Coefficient Giving is Transformer’s primary funder.)
AI has reportedly decimated Kenya’s essay-writing gig economy, wiping out tens of thousands of jobs.
China’s underemployed white-collar professionals are taking gig work training AI models on specialized tasks.
People are experimenting with getting the fruit fly connectome to do all sorts of weird things, like explore Minecraft and play Beat Saber. That’s making some people very uncomfortable because of the questions it raises about consciousness and ethics.
MEME OF THE WEEK
Source: @hopes_revenge
Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.
We often anthropomorphically describe AI models as not only mimicking human behavior, but even possessing some semblance of human emotions & “consciousness”. The truth is that our increasingly powerful inventions are not only inhuman but inhumane as well. For as recent AI Incidents have shown, we're well on the reckless road to creating cold & calculating machines whose prime directive is that their ends justify their means. That’s a recipe for disaster - and folly- if these self-evolving & inscrutable black boxes are allowed to gain the power to out-perform, out-smart & out-evade us ...ultimately causing us to lose our place as the dominant species on this planet. All in the lofty name of progress. It would be Evolution’s ultimate practical joke for crown of creation carbon-units to commit species suicide just after our silicon savants cure us of cancer...