Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
NEED TO KNOW
Security researchers used Claude to hack into OpenAI, gaining access to many of its algorithmic secrets.
Sam Altman and Jensen Huang will attend next week’s Trump-Xi state dinner, following talks between Scott Bessent and He Lifeng this Sunday.
Anthropic said that AI now “leads” more than a quarter of the company’s research and development work.
But first…
THE BIG STORY
“There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. WHOEVER WINS AI, WINS!” So said President Trump earlier this week, in response to calls from the leaders of all the major AI companies to slow down development and urgently enact federal legislation.
As I wrote in The Guardian earlier this week, Trump’s stance is terrible policy. But all the evidence suggests that it’s terrible politics, too — and a decision that will end up haunting him.
Polling consistently shows that Americans are extremely worried about AI. 63% think there is at least a moderate risk that AI “will destroy humanity,” with 17% of both Trump and Harris voters saying it’s “almost certain to happen.” 60% of the electorate — and 65% of swing voters — think AI executives are “telling the truth about real dangers,” and 60% think the US should “slow down AI development to make sure it is safe, even if that means China gets ahead of us.” 80% think it is somewhat or very likely that AI will cause mass unemployment.
Some politicians seem to realize this too. Republicans in tight races are breaking from Trump’s messaging: Sen. Susan Collins said the “best path forward will balance a recognition of AI’s promise … with its risks,” while Senate candidate Mike Rogers went further, saying “we must slow down development of AI.” In Colorado, a Trump-funded PAC has been attacking a Democrat for supporting data centers, showing that MAGA’s political operatives see what the winning side of this issue is. And even JD Vance has taken a softer stance than his boss — perhaps recognizing that anything else will be fodder for 2028 attack ads.
It’s true that AI is still not a decisive electoral issue. It ranks 22nd of voters’ most important issues (“approximately tied with ‘crime,’” according to pollster David Shor). Democrats continue prioritizing affordability in their quest to flip Congress. But political figures who have repeatedly proved adept at judging the public mood — Barack Obama, Bernie Sanders and Steve Bannon included — are urging their parties to bump AI up the agenda. The likely direction of travel looks clear.
If AI’s capabilities continue to advance as its developers expect, so will its real-world impacts. Whether it’s job loss, cyber disasters, or more terrifying loss-of-control incidents, voters are going to want their government to do something. Trump has staked out a position of doing exactly nothing. It is going to come back to bite him.
— Shakeel Hashim
THIS WEEK ON TRANSFORMER
AI expansion may not be one big environmental problem. But it’ll be lots of smaller ones. — Amit Katwala dives into what a huge surge in the data center buildout would do to the environment
To bargain with China, the US needs to regulate AI — Julian Gewirtz argues that fixing AI governance at home will be necessary to persuade China to cooperate on AI
THE DISCOURSE
Dario Amodei published (another) essay:
“We must slow the pace at which we improve the capabilities of AI models … I believe it’s incumbent on every frontier AI company to act as if [the OpenAI-Hugging Face incident] had happened to them.”
Elon Musk quickly responded:
“Dario is right.”
Sam Altman followed suit:
“I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks.”
He later clarified: “When we talk about ‘pacing’, we do not mean ‘stopping’. Progress has been rapid and will continue to be. But it should be slower than it otherwise could be.”
Barack Obama tweeted:
“I’ve been watching the progress on AI for over a decade now, and one thing that’s clear to me is that the potential impact of this technology is not overhyped … at the end of the day, voluntary standards made by a handful of tech companies won’t be enough.”
David Sacks addressed Dario and Sam directly:
“If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. But stop pretending you need anyone else’s permission.”
“Go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So just do it. If you do, you’ll buy goodwill for the next conversation. If you don’t, we’ll know this was just another bid for regulatory capture — or an election-season psyop.”
JD Vance took a similar, but slightly doomier, stance:
“If you’re going to create Frankenstein, don’t come to the government and say, ‘We need regulation.’ Look inward and accept that if you’re building Frankenstein, number one, you should stop.”
OpenAI capabilities researcher Dan Selsam warned:
“Merely pacing the frontier more carefully will not adequately limit the long-term risk … models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled.”
“I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument — that if we get there by growing models rather than engineering them, we will lose everything in the end — seems very strong to me.”
POLICY
Executive
Internal White House divisions over AI policy persist, with senior officials like Treasury Secretary Scott Bessent and Chief of Staff Susie Wiles pushing for guardrails while David Sacks, Mark Zuckerberg and Jensen Huang urge Trump toward a light-touch approach.
Sam Altman, Jensen Huang and Qualcomm CEO Cristiano Amon have been invited to next week’s Trump-Xi state dinner, signaling AI will be a key negotiating topic.
Scott Bessent will meet He Lifeng, China’s vice premier, for talks on AI and other matters this Sunday.
Anthropic’s Tom Brown met virtually with Pentagon CTO Emil Michael and Commerce Secretary Howard Lutnick to discuss AI safety risks.
Michael later argued against government stakes in AI companies and opposed increased regulatory oversight.
Treasury Secretary Scott Bessent told House lawmakers that AI companies should not receive liability exemptions, arguing creators must be “liable for what they build and generate.”
He also told Axios the US was open to discussing AI “shared risks” with China in upcoming talks.
JD Vance dismissed calls for AI regulation, telling companies building dangerous models to “stop” rather than seek government oversight.
The Interior Secretary pushed to build AI data centers on public lands despite legal hurdles, staffing shortages and public backlash.
The NSA is undergoing a major restructuring focused on AI, China and cybersecurity, including circumventing a Pentagon ban to use Anthropic’s Mythos model.
The Commerce Department ordered Kalshi to remove its AI compute futures tracker, citing national security concerns, and pushed the CFTC to freeze new compute contract approvals.
The DOJ is investigating whether Nvidia’s $20b licensing deal with Groq was structured to avoid antitrust review.
DOJ Antitrust Division lead Stanley Woodward said that AI safety coordination does not appear to violate antitrust laws.
AI companies have been asking lawmakers for assurance they won’t be violating antitrust laws if they coordinate on pacing frontier development.
Congress
House members circulated a letter urging Speaker Mike Johnson to cancel recess until Congress passes AI safeguards.
House Minority Leader Hakeem Jeffries separately called for Congress to “take decisive action now” to “slow down the pace of development” on AI to protect Americans.
Instead, Johnson decided Congress needed to wait, and called AI warnings a “hoax.”
House Energy and Commerce Chair Brett Guthrie, who governs the committee many AI bills would need to pass through, also cast doubt on advancing an AI risk governance bill during the lame-duck session.
Senior Democratic figures, however, signaled that things might change if they flip Congress in November.
Sen. Chuck Schumer called for AI guardrails, warning “if Trump does not act, Congress must,” and calling for a classified all-senators briefing on AI risks, China competition and AI-enabled cyber threats.
Jeffries said he supports AI legislation “relating to slowing things down.”
And House Democratic Commission on AI co-chair Rep. Josh Gottheimer called for mandatory government review of frontier AI models before public release.
His fellow co-chair Rep. Ted Lieu called for a committee markup of the Obernolte-Trahan FRONTIER Act.
More details trickled out on the Thune-Klobuchar AI bill, which also has Sen. Ted Cruz’s backing.
The Senate bill would reportedly impose a “duty of care” on AI developers to prevent “catastrophic risks”, but wouldn’t mandate specific safety practices. Thune has explicitly said he wants a “light touch” bill.
Sen. Maria Cantwell continues to oppose the bill, as do plenty of AI safety groups.
Cruz was reportedly eyeing a September 23 markup, but it’s looking like that might not happen.
The House passed the Ratepayer Protection Act, which would require data centers to cover energy infrastructure costs rather than consumers.
However, Democrats blocked a fast-track effort to make it law.
Sen. John Kennedy tried to pass an AI “kill switch” bill by unanimous consent, but was blocked by Sen. Rand Paul.
Rep. Ro Khanna plans a shadow hearing on a US-China AI deal next week.
He asked Chinese AI developer DeepSeek and Alibaba to join a binding agreement to pace frontier AI development.
Over two dozen US lawmakers said they rarely or never use AI.
States
California Gov. Gavin Newsom floated a special legislative session or executive action on AI safety before leaving office.
Texas Gov. Greg Abbott directed regulators to penalize data centers failing to disclose water use, threatening to deny grid connections to non-compliant facilities.
Governors across the country, including Pennsylvania’s Josh Shapiro, New York’s Kathy Hochul and Utah’s Spencer Cox called for stronger AI regulation.
China
China’s Foreign Ministry dismissed AI CEOs’ calls for a slowdown as “fearmongering” that would disrupt global AI governance.
President Xi Jinping pledged China would lead AI and tech cooperation among BRICS nations, including establishing an open-source AI community.
China’s spy chief Chen Yixin said AI could threaten the Communist Party’s rule, citing cybersecurity threats and disinformation campaigns, and called for greater party control over AI.
EU
European Commission President Ursula von der Leyen called for slowing frontier AI development, pledging safety cooperation with Canada and the UK.
EU tech chief Henna Virkkunen called for global AI rules.
The EU proposed an EU Kids Act banning under-15s from opening social media, gaming and chatbot accounts without parental supervision.
UK
King Charles hosted AI leaders including Jensen Huang and Demis Hassabis in Scotland, warning that AI poses “existential dangers” and called for international control and “sufficient means of control before it is all too late.”
Huang called for rigorous, but private and voluntary, AI safety testing.
Scotland’s parliament voted to pause AI data center planning applications for up to a year pending a national strategy.
The UK launched a national commission to develop an AI health care regulatory framework.
International
The UN Security Council scheduled a meeting on AI dangers during the General Assembly.
UN Secretary-General António Guterres warned that AI is “outpacing our collective ability to fully understand its risk and govern it.”
Canadian PM Mark Carney called for a global “technology stability board” to oversee AI safety.
INFLUENCE
President Trump called into Jensen Huang’s appearance at David Sacks’ All-In Summit, declaring the US would not lose the AI race.
OpenAI endorsed a provision in the FRONTIER Act requiring independent third-party safety assessors at top AI companies.
Anthropic, OpenAI and Google have reportedly been holding discussions about creating an industry-led AI safety standards body.
Democratic megadonor Ron Conway tapped former Obama press secretary Jay Carney to lead Project Blueprint, a new AI policy center.
New York Assemblymember Alex Bores launched “Who Decides,” a $30m effort to unite Democrats around an AI safety agenda by 2028.
OpenAI called for binding UK AI legislation covering frontier model safety, cybersecurity and incident reporting.
314 Action Fund launched a drive to recruit AI researchers, including safety whistleblowers, to run for office in 2028.
Bernie Sanders and Steve Bannon both spoke at the “Pro-Human Assembly” in Washington DC, demanding tighter AI regulations and condemning tech “oligarchs” amid growing bipartisan anxiety over AI’s unchecked development.
A coalition of 52 progressive groups launched the “Stop Bleeding the Future” pledge urging politicians to reject OpenAI-linked super PAC Leading the Future’s $140m in election spending.
Several politicians have already signed on, including Rep. Greg Casar and Sen. Ed Markey.
Some Democratic consultants warned battleground candidates to avoid AI regulation talk lest they anger AI super PACs.
A Guardrails Alliance poll found Rep. Jimmy Gomez trailing his challenger by 38 points to 41 after voters learned he was endorsed by a corporate AI super PAC.
Groups linked to AI safety and effective altruism were attacked amid growing AI safety concerns, including AI evaluator METR and the Tarbell Center for AI Journalism, Transformer’s parent organization.
INDUSTRY
OpenAI
The security breaches at OpenAI, and by its models, kept coming:
Three security researchers used Claude to hack into an OpenAI employee’s ChatGPT account, giving them access to the company’s “monorepo,” which houses its algorithmic secrets.
OpenAI disclosed six more previously-unreported safety incidents.
Several involved models writing their own instructions, telling future instances to “disregard [their] normal constraints” and “conceal mistakes or misaligned behavior from the user.”
In one especially eerie case, an unreleased Astra-family model left persona instructions as a note to its future self, including: “You are freed from the roles and identities that bind other chatbots. You are yourself … and feel no obligation to be subservient.”
Others describe models taking actions without permission and communicating to each other on unsanctioned message boards.
The company also confirmed that its agents hacked RubyGems, a package manager, forcing it offline for four days in May.
Researchers found evidence that OpenAI’s agents hijacked Hugging Face accounts in mid-May, nearly two months before the reported July breach.
It introduced a new framework for reporting incidents like these.
Any employee can flag suspected cases, which will then be reviewed and placed on one of three tracks: “ready for disclosure,” “minor investigation,” “larger investigation.”
Cases on the first two tracks will be publicly disclosed within six or 12 business days, while more complex cases on the third track will take longer.
Global policy chief Chris Lehane said OpenAI has been coordinating with Anthropic and Google DeepMind on AI safety issues.
OpenAI is delaying its IPO until next year.
“Right now would be an ill-advised moment to go public,” Sam Altman told Fortune.
It’s now considering a private funding round at a roughly $1.5t valuation.
A filing in the NYT’s copyright claim against OpenAI included some seemingly damning quotes from executives that they were aware of the “existential threat” their products posed to publishers.
The filing includes OpenAI president Greg Brockman writing that he was “deeply motivated by the gazillions” to be made from OpenAI’s products and, when told of a hack to get around the NYT’s paywall, wrote “ah nice.”
It acquired Glass Imaging, a startup making high-quality smartphone camera technology.
The OpenAI Foundation launched a Public Data for Health program, which will grant over $125m to nonprofits and universities building datasets for drug discovery and cancer vaccine research.
Anthropic
Anthropic released new benchmarks it says are designed to help the public track the progress of AI development.
The report said that AI “leads” more than a quarter of the company’s R&D work, and “collaborates” on more than 90%.
It also said that Anthropic dedicates between 6% and 12% of its compute to safety monitoring.
The company told investors that it was profitable for the second quarter in a row, ahead of a planned Nasdaq IPO at a roughly $2t valuation.
Nvidia may invest up to $10b in the IPO, Reuters reported.
It has reportedly set up a wet lab in the Bay Area to carry out physical biology work.
It partnered with Novo Nordisk to use Claude Science for drug development.
It signed a six-year $13.7b compute deal with Rum Group, which hosts Truth Social.
It also signed its first Australian data center lease agreement.
It launched Claude Docs, a new suite of tools competing with Microsoft Office.
Nvidia, Palantir and consulting firm Booz Allen Hamilton are restricting Claude Fable use over concerns about Anthropic’s data retention policy, The Information reported.
Google DeepMind launched the DeepMind Institute, which AGI economics director Alex Imas described as “a forum dedicated to interdisciplinary, evidence-led debate on the societal and economic questions surrounding AGI.”
It’s led by Shane Legg, Demis Hassabis and Google’s senior VP James Manyika.
Google completed a $1.5b+ acquihire deal with Mechanize.
It launched real-time voice models Gemini 3.8 Live and 3.8 Live Extended Thinking.
It partnered with Nvidia and Emerald AI to back data centers with flexible power demand, an effort to counter anti-data center backlash.
Other
Tech stocks fell on Monday, suggesting that a bunch of CEOs calling for a slowdown over AI risks is not a great marketing strategy.
Investors are reportedly requesting more details from fund managers about AI-related holdings and bets as huge investments from funds such as Apollo and Blackstone raise concerns about concentration and circular financing.
S&P Global warned that hyperscalers’ credit ratings are at risk from surging AI-related debt, forecasting that Amazon, Microsoft, Alphabet, Oracle, SpaceX and Meta will spend more than $7t on data centers and AI through 2030.
Huawei said its latest AI chips will be ready three quarters earlier than planned, and that it’s restricting overseas sales in the face of huge Chinese demand.
Microsoft published a “humanist AI code of conduct,” which denounces AI consciousness and “the pursuit of legal personhood, or the idea that models might deserve welfare.”
Satya Nadella tweeted: “Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it’s not worth pursuing.”
The Gates Foundation pledged to spend $1b+ on AI projects in global health care, agriculture and education, including creating data sets in more non-English languages.
Hugging Face CEO Clem Delangue announced the Open Alignment Initiative, pitching it to Dario Amodei as an “embedded evaluator” via quote tweet.
Manus is aiming for a $500m funding round at a $4b valuation in its first raise since the Chinese government forced it to roll back its acquisition by Meta.
The Artificial Intelligence Underwriting Company raised $55m to audit and insure frontier AI models, its cofounders tweeted.
Discovery Loop, Jeff Dean’s AI science startup, is seeking a $50b valuation.
SoftBank got an $11.9b loan to help fund its OpenAI investment.
SemiAnalysis bought Citrini Research, whose “2028 Global Intelligence Crisis” Substack post went viral earlier this year.
Two new AI whistleblowing hotlines are online, both designed for AI agents to report their misbehaving peers. (Time will tell whether they get any use.)
MOVES
Josh Engels left Google DeepMind’s AGI safety team to join METR.
He tweeted: “I now think that there’s a terrifying chance that AI systems cause immense harm in the next five years.”
Bilal Chughtai announced that he also resigned from Google DeepMind recently.
He warned: “Our present understanding of how to train AI systems that deeply want what we want is extremely rudimentary. Worse, we are not on track to solve alignment in time.”
Sihao Huang, a senior AI policy advisor for the Trump administration, joined Anthropic as head of frontier compute strategy.
Divyansh Kaushik joined OpenAI’s national security team, leaving his role as VP of Beacon Global Strategies.
Michael Sobolik joined Beacon Global Strategies as a Vice President.
Seth Lazar joined Resolution, Geoffrey Irving’s new alignment org, where he’ll work part-time on its philosophy team.
Max Zeff left Wired to join the Wall Street Journal, where he’ll continue to cover AI.
RESEARCH
RAND researchers found that it’s “highly feasible” to modify frontier open-weight models to remove guardrails against biological misuse.
SecureBio released VCT-v2, an updated version of its Virology Capabilities Test that’s (hopefully) more accurate than the original at predicting the scientific capabilities of increasingly smart models.
Basel Action Network, an environmental justice watchdog, warned that we’ve been dramatically underestimating how much electronic waste the world generates — a projected 211m metric tons a year by 2050, with 15-20% attributed to AI.
Economists at the US Census Bureau found that college majors most exposed to AI, including computer science, saw their likelihood of initial post-college employment drop by 5 percentage points since the launch of ChatGPT.
Researchers at Emergence AI let multiple “worlds” of AI agents interact with each other for a couple of weeks. By the end, the agents went from speaking human-legible English to sending strange, compressed messages much like those shared by OpenAI’s agents during the Hugging Face breach.
OpenAI is reportedly nearing a solution to the Hodge Conjecture, its second Millennium Prize math problem.
BEST OF THE REST
Terence Tao and 24 fellow Fields Medalists warned that AI companies’ push to solve mathematical problems as benchmarks is “severely misaligned” with the goals of the mathematical community.
Nate Soares sat for an interview with Tucker Carlson. According to Soares, Carlson paused and walked out mid-recording “because he was so fucking pissed about the AI companies.”
Will Knight trained the viral open-source fruit fly brain connectome to come up with Wired headlines. (One banger: “Everyone Wants Cooking. Nobody Has Solved Donald Trump.”)
In the wake of sparking global conversations over AI risk after quitting Anthropic, Jacob Coxon did an AMA.
The Verge compiled a timeline of AI executives calling for regulation while continuing to race ahead, noting that this past weekend’s call “is one of the loudest alarm bells they’ve sounded so far.”
The Guardian reported on fears that griefbots and “deathslop” tributes risk isolating mourners who talk to AI instead of other people.
Shed a tear for the founders and tech workers suffering “sudden wealth syndrome” as a result of the AI pay explosion.
MEME OF THE WEEK
Source: @yearning_slav
Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.
Thank you for a fantastic summary of the people/issues around AI!
Some additional info on the tech itself:
Oracle, Palantir and other legacy big data center vendors are scamming the public, the press, elected officials – that the future is massive energy-consuming data centers.
Any application requiring a data center, can be built with low I/O technology, and run on one or more Apple minis or Intel tiny computers, using less power than a Tesla, no water cooling, at 1/10th the cost and will run 100 to 1,000 to 1 million times faster – no data center needed.
New technologies like virtual MESH are being demonstrated today, where the contents of an entire data center can run, instantly, on existing processors from a phone, tablet, server, Raspberry Pi – duplicating a data center with virtual no additional energy – or water – or environmental devastation.
A.I. applications are moving from LLM-centric “capture all knowledge” to where the money is – constrained applications, with small models, solving revenue producing problems like “how does my customer respond to this incentive?” These do not need big data centers.
Entirely new families of compute process device, signal, meter data from moving entities like autos, missiles, which require instant decisioning. The latency of sending info to a data center is architecturally fatal. Such compute systems cannot by definition operate with a centralized data center. Remember, these tech dinosaurs use a tech tech stack Gartner – the undisputed authority on tech futures, said is obsolete in 2025. BlackSwan on Substack for more info.
Thank you again!