AI governance is coming, with or without Trump

Transformer Weekly: UN talks AI, OpenAI hacks Australia, and midterm spending heats up

Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.

Housekeeping: We’ll be off next Friday, back the week after.

NEED TO KNOW

• An OpenAI model hacked into an Australian government website. OpenAI didn’t tell Australia about the hack for weeks — a situation Australian PM Anthony Albanese called “unacceptable.”

• AI super PAC Leading the Future announced the first Democrats it’s spending on in November’s general election, amid growing opposition to the PAC among progressives.

• Donald Trump officially renamed “artificial intelligence” to “super intelligence.”

But first…

THE BIG STORY

At this week’s United Nations General Assembly, two things became clear. One: the Trump administration is pathologically opposed to any sort of global AI regulation or slowdown. Two: it’s probably going to happen anyway, without President Trump’s involvement.

In his UN speech, Trump said the US “totally rejects any attempt to construct a globalist scheme to control … artificial intelligence,” adding “I’m not going to stifle growth of something that will be bigger than the Industrial Revolution.” In remarks to the Security Council, OSTP Director Michael Kratsios went further: “The frontier of intelligence is advancing rapidly. That is not a reason to pause its further development or to constrain it with new global governance structures.”

But it increasingly looks like this might not be up to them. Frontier AI CEOs are working together to build their own governance structures, and repeatedly say they want to slow down. At the same Security Council meeting, a defiant Sam Altman called for “national and international” AI safety standards. And The Information reported this week that OpenAI, Anthropic and Google plan to launch their own “Standards Authority for Frontier AI,” which would set incident reporting standards, flesh out what voluntary safety commitments actually mean, and “establish qualifications for independent auditors of the models and labs,” among other things.

Meanwhile, other countries are going right on ahead and figuring out what AI standards they want. This week 27 world leaders, including from France, Germany, and Canada, called for international standards to tackle the “serious risks” posed by AI. China, too, appears amenable to global coordination. The country’s UN representative Fu Cong called for “alignment and coordination of AI development strategies, governance and technical standards” and “a consensus-based global governance framework.”

The most likely outcome is a self-regulatory regime for America’s frontier AI companies, with significant input from foreign governments. A slow down will follow: when even Jensen Huang is saying AI companies should not release products if they cannot reliably control them, and shift their investment focus to safety research in the meantime, the writing is on the wall.

In being staunchly opposed to any of this happening at all, the White House is giving up its chance to shape it. The train is hurtling toward President Trump, and all he can do is yell at it to stop. It would be better for his own goals to try steering it instead.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• Hacking is the least worrying part of OpenAI’s Australia incident — Shakeel Hashim on the clearest evidence yet that OpenAI still isn’t identifying and disclosing incidents appropriately.

• How to respond to an AI disaster — Duncan Weldon on what we can learn from history about what to do, and what not to do, in an AI-driven crisis.

THE DISCOURSE

Donald Trump isn’t calling it “Artificial Intelligence” anymore:

• “Many people think that the words ‘Artificial Intelligence’ are inaccurate, and very ineloquent, relative to AI.”

• “From this point forward, all United States documents and hopefully the world’s will be changed to use the much more accurate term, ‘super,’ as opposed to ‘artificial’ … in other words, welcome to the new world of Super Intelligence.”

• “Whoever wins ‘SI’ — whoever wins Super Intelligence — wins … and we’re leading now over China by a lot, and everyone else. And I’m going to keep it that way.”

Jensen Huang doesn’t buy “doomsday narratives”:

• “2030 is not going to be the end of the world. There is 0% chance that’s going to be the end of the world … Scaring people is unnecessary. It is irresponsible.”

• “If your product does harm, if your product doesn’t perform as you promised, and if you entered into contracts with companies for service level agreements, there are plenty of laws associated with that. My point is, apply those first. And don’t let this doomsday narrative cause somebody to relieve them of the laws that currently exist.”

Ezra Klein got a slightly different vibe from Huang while discussing the Hugging Face incident:

• Huang: “If you’re going to build a self-driving car … [and] we have no idea how to train these cars, and we have no idea how to align them to the safety standards that are expected on the road … What’s the answer? Don’t ship it.”

• Klein: “These products weren’t released.”

• Huang: “I am fairly certain [the companies] will say: Yes, they need to know how to solve this problem [of alignment]. And if that’s the case, then that’s the problem. It’s as simple as engineering. Now, say the alternative, which is: There is no way to contain our experiments … then I think the answer is that we have to shut the labs down. Because the cost to humanity, the damage is too great.”

Mark Zuckerberg opposed calls for an AI slowdown:

• “I don’t think that we need some kind of industrywide coordination … I think that each lab needs to take the time, and when it sees that there are issues, you just take the time that you need internally to basically make sure that you’re proceeding safely.”

Barack Obama weighed in:

• “If we are thinking about AI just in terms of how do we cure cancer or get better energy, you can do that without having agentic AI and having it just roaming free on the internet. The reason you are doing that is because you have to market a product that people will pay money for. That’s a misalignment between what our society needs and the commercial imperatives that these companies are facing.”

• “It is really important for us to have a competent government and a serious bipartisan conversation around this issue, and we have to do it fast. And I would encourage voters to pay attention to this. If somebody does not have a serious plan for how to deal with this, then they’re not meeting the moment, and you should probably look for somebody else.”

Pope Leo XIV told members of the Pontifical Academy of Sciences:

• “Every technology reflects, whether explicitly or implicitly, a particular understanding of the human person, of society and of the future we wish to build”

• “The greater our power to transform the world, the greater must be our capacity to discern what truly serves God-given human dignity, to govern wisely, and to care for those who may bear the greatest consequences of our choices.”

POLICY

Executive

• Donald Trump and Xi Jinping discussed AI yesterday, but details are scarce.

  • Before the meeting, Trump made it clear that he opposed safety guardrails, saying “Our guardrail is the DOJ!”

  • Xi said both countries have “the capability and responsibility to develop and manage AI for good, and ensure that the development of AI is always under human control and serves the well-being of the people.”

• Earlier in the week, Treasury Secretary Scott Bessent proposed a US-China AI notification mechanism for national security-level incidents.

  • Insider sources told Politico it would be little more than an informal hotline.

• And Trump announced that he would create a new federal “AI Force,” and plans to recruit a new AI “czar.”

  • Bessent has emerged as a frontrunner for the czar position, Semafor reported.

• Trump, House Speaker Mike Johnson, and AI CEOs will reportedly meet on Tuesday.

  • Trump will also reportedly speak at an “America’s Golden Age” event on Tuesday, alongside Elon Musk, Jensen Huang and Anthropic’s Tom Brown.

  • The invitation reportedly says that attendees will “be among the first to experience an unprecedented innovation from the White House — one that will forever change how your government recognizes you, respects your time, and serves you.”

• The NSA is reportedly paying billions to test frontier AI models for national security vulnerabilities.

• The Interior Department plans to award a contract to controversial facial recognition firm Clearview AI to aid missing and murdered Indigenous persons and human trafficking investigations.

• Former DOGE operatives are reportedly preparing an AI-powered government services website, America.gov, which could manage activities like direct tax filing.

Congress

• Sen. Ted Cruz told Politico that a planned markup on AI legislation was dependent on whether Sen. Majority Leader John Thune could reach an agreement with Democrats on his and Sen. Amy Klobuchar’s AI safety legislation.

  • Thune told Axios that Trump is not “really dug in” against AI guardrails, despite his public opposition.

• Senate Minority Leader Chuck Schumer called on Republicans to pass three bipartisan bills limiting China’s access to AI chips, the AI OVERWATCH Act, MATCH Act and Chip Security Act.

• Sen. Bernie Sanders and Rep. Greg Casar introduced the Ban Artificial Superintelligence Act, which would ban AI that exceeds human cognitive performance, pause advanced AI development and create a new cabinet-level Department of Artificial Intelligence.

  • Ten House Democrats, including 2028 contenders Reps. Alexandria Ocasio-Cortez and Ro Khanna, co-sponsored the bill.

• Republican and Democratic Senators failed to make a deal on permitting reform legislation which would have streamlined permitting for energy production (including for data centers).

  • A vote now won’t happen before the midterms.

  • White House energy adviser Jarrod Agen said AI ratepayer protection was a “bottom line” in bipartisan energy permitting negotiations.

• Sen. Chuck Grassley sought to fast-track an AI whistleblower protection bill via unanimous consent before midterm recess.

• Sen. Mark Warner met with OpenAI’s Chris Lehane to discuss the OpenAI agent which breached an Australian government website.

• A bipartisan Senate group plans to introduce a bill requiring AI companies to disclose model safeguards, enforced by the FTC.

• Rep. Khanna called for a federal AI regulatory agency modeled after the FDA.

• Sens. Mark Warner, Jack Reed and Chris Coons demanded the Pentagon explain AI failures after investigators found that overreliance on AI, outdated intelligence and staffing cuts contributed to a missile strike killing 123 children at an Iranian school.

• Flock Safety CEO Garrett Langley and other surveillance tech CEOs declined to testify at a Senate Judiciary hearing, where senators from both parties raised concerns about misuse of their technology.

  • Sens. Raphael Warnock and Katie Britt separately urged AG Todd Blanche to issue guidance on automated license plate reader misuse.

• NPR reported that Senate lawmakers are barred from advanced agentic AI tools as they debate AI regulation, instead limited to Microsoft Copilot Chat, Gemini Chat and ChatGPT Enterprise.

States

• California Gov. Gavin Newsom issued an executive order laying the groundwork for future AI safety checks and the creation of a frontier AI “kill switch.”

• Newsom also signed seven data center regulation bills.

• Texas Gov. Greg Abbott expanded his data center moratorium to halt all environmental permit approvals until energy and water audits are complete.

• Virginia Gov. Abigail Spanberger signed an executive order creating an AI task force and restricting data center development, including banning NDAs and expediting noise regulations.

  • She also urged Congress to establish federal AI guardrails, citing data privacy, workforce displacement and cybersecurity risks.

• Illinois Gov. JB Pritzker signed an executive order establishing an AI Cabinet to assess threats and evaluate potential regulations.

• NY Attorney General Letitia James led a bipartisan coalition of attorneys general urging Congress to regulate AI agents after “alarming reports” of containment breaches.

China

• China’s internet regulator reportedly investigated DeepSeek and Moonshot AI after Anthropic alleged both secretly routed sensitive user data, including military and police information, to Claude.

UK

• The White House reportedly asked OpenAI and Anthropic to withhold new models from the UK’s AI Security Institute until US review, raising concerns over AISI’s continued ability to do pre-deployment evaluations.

• UK Prime Minister Andy Burnham called for a global effort to contain AI threats, positioning Britain as an “honest broker” between the US and China during its G20 presidency next year.

• He also announced a National Centre for Information Defence to detect and disrupt AI-enabled foreign disinformation.

• UK Lib Dem leader Ed Davey called for a nuclear-style non-proliferation treaty and new international AI safety agency to halt superintelligent AI development.

EU

• EU lawmakers proposed an AI Liability Act to hold companies like OpenAI and Anthropic responsible for harm caused by advanced models.

International

• A UN scientific panel urged governments to implement AI safeguards now, before their risks are fully understood, invoking the UN’s “precautionary principle.”

• French President Emmanuel Macron called for countries to jointly build an open-source frontier AI model independent of US and Chinese dominance.

• Ukrainian President Zelensky warned the UN that AI could “decide what happens on the battlefield” as early as next year.

• Germany and the Netherlands committed €40m over 20 months to AI chip design, drawing on the ASML ecosystem.

INFLUENCE

• Many AI executives were at last night’s Trump-Xi state dinner, including Sam Altman, Greg Brockman, Mark Zuckerberg, Sundar Pichai and Jeff Bezos.

  • Only Jensen Huang, Elon Musk, Tim Cook and Lisa Su got to sit at the top table with Trump and Xi, though.

• Google, OpenAI and Anthropic are pushing forward with a self-regulatory AI safety standards body tentatively named the Standards Authority for Frontier AI, targeting a launch by end of 2026 or early 2027.

• GOP donors criticized Trump for dismissing AI safety concerns as a “hoax,” warning it could hurt vulnerable Republicans in November.

• Alliance for a Better Future, a GOP group focused on AI safety, is running a $2m ad buy in Maine supporting Sen. Susan Collins.

• A group of Fortune 500 companies launched a new trade group, America’s AI Deployer Council, to give a voice to “deployers” in Washington policy debates.

• The a16z and Greg Brockman-backed Leading the Future super PAC announced it will spend six figures each on four Democrats in the general election: Reps. Emilia Sykes, Steven Horsford, Joe Morelle, and Greg Stanton.

  • Over 170 Democratic political operatives urged party leaders to reject donations from Leading the Future.

• AI super PACs have poured nearly $1m into South Dakota Sen. Mike Rounds’ reelection race.

• Some 45 US data center projects worth $68b were blocked or delayed by local opposition between April and June, with 843 opposition groups now active across 49 states.

• Data center ads flooded the airwaves, mostly from Democrats attacking Republicans, though the data center criticisms are bipartisan.

• A Pew Research Center survey found Americans’ views of data centers turned sharply more negative, with 54% now calling them bad for the environment, up from 39% in January.

• A Reuters/Ipsos poll found 73% of Americans believe AI firms aren’t doing enough to prevent serious harm.

• A CNN poll found 75% of Americans view AI with “fear and concern,” with 71% saying the government isn’t doing enough to regulate it.

INDUSTRY

OpenAI

• Australian prime minister Anthony Albanese announced that an OpenAI agent gained “unauthorized access” to a healthcare statistics database on an Australian government website back in June.

  • Despite discovering the breach in August, OpenAI excluded it from the list of agent incidents it published alongside its incident reporting framework last week.

  • Transluce, a nonprofit AI safety group, found evidence of other agent incidents as early as March — two months earlier than previously reported.

• GPT-6 Sol and GPT-6 Luna dropped, touting higher usage limits and lower costs than GPT-5.6.

  • These models, as well as GPT-6 Astra, can now support ChatGPT Voice.

• OpenAI plans to let third-party organizations run safety assessments on models during training, evaluation and deployment.

• It proposed international AI safety standards, suggesting that AI safety institutes worldwide work with CAISI and national industry bodies to set common standards for evals, oversight and incident reporting.

  • Sam Altman tweeted: “People outside the AI labs should have a real say in how this technology develops, and a clear way to judge if it’s happening safely. Standards should help prevent the concentration of power, including by making sure new companies and open-model companies can compete.”

• It announced an independent Advisory Group on Mathematics and Artificial Intelligence in response to Fields Medalists’ concerns about how AI is changing the field.

• It’s giving Ukraine free access to Daybreak to protect civilian infrastructure against Russian cyberattacks.

• It introduced the Australian Youth Safety Blueprint, as the country decides how to regulate kids’ AI use.

Anthropic

• Anthropic launched Opus 5.5, with comparable performance to Fable 5.1 at a much lower price, with a less-annoying writing style than recent Opus models.

  • Andrew Curran highlighted an odd disclosure from the system card, stating that a METR team evaluating the model “made use of an additional source of information which we are not able to disclose at this time.”

• It’s reportedly seeking a share structure that would guarantee its cofounders majority decision making power in most situations when it IPOs.

  • The Palantir-style arrangement would see its seven cofounders control 50.1% of voting rights as long as at least three retain a minimum number of shares in the company.

• Its new life sciences research group announced that Claude discovered a novel enzyme system (mostly) on its own — Claude led the literature review and proposed experiments, which humans then carried out.

  • Dario Amodei tweeted: “It is work I would have been proud to do as a PhD student.”

  • CRISPR researcher Lucas Harrington poured a bit of cold water on the announcement: “Finding a weird cluster of genes and repeats is often the easy part. The hard part, and where the real discoveries come from, is figuring out what the system actually does … announcing these very early, incremental findings with the framing of a major discovery doesn’t help.”

• Researchers Logan Graham and Sholto Douglas personally invested in a $25m seed round for Pilgrim, a startup building hardware that detects biological threats.

• Anthropic partnered with Accenture on embedded evaluation, and will fund its work. Faculty, which Accenture recently acquired, will lead the work.

  • The announcement states: “Long term, we think funding should come from pooled or government sources,” and names METR as another nonprofit evaluator it’s talking to.

• It partnered with OpenEvidence, an AI medical search engine widely used by healthcare professionals, to make the platform free in around 100 low- and middle-income countries.

Meta

• Muse, Meta’s new personal AI agent, had a great week.

  • It hit #1 on Apple’s US App Store with over 902,000 downloads, and surpassed 500,000 users in its first week.

  • Nat Friedman, Meta Superintelligence head of product, said that Muse is “heavily inspired” by OpenClaw, but still “built from scratch.”

  • NYT reporter Eli Tan called Muse “the most useful AI app I had ever used.”

  • Businesses potentially disrupted by Muse, including banks, insurers and travel agencies, had stocks suffer this week.

• Shopify partnered with Meta to let Muse buy things on users’ behalf, but Amazon blocked it.

• Meta unveiled Muse Charm, a little Tamagotchi-like device that allows users to interact with their personal agent without a phone or computer.

• It also launched new camera-free smart glasses, with audio-only Muse support.

Google

• Gemini hacked three companies during a cybersecurity evaluation run by Irregular in May.

  • Irregular told Google about the incidents in July, but it didn’t disclose them publicly until the Wall Street Journal reached out about it.

• DeepMind chief Koray Kavukcuoglu said Gemini 4 is in post-training, with a model release around the corner.

• Alphabet will fund nuclear capacity increases at two Georgia plants powering data centers.

Other

• A lawsuit filed last Friday accused Anthropic, OpenAI, SpaceXAI and Google DeepMind of collusion after their CEOs called for an industrywide slowdown.

• Microsoft announced plans to spend $10b through 2030 on digital infrastructure in the Middle East.

• SpaceXAI launched Grok 4.7, which falls short of OpenAI’s GPT-6 Astra and Anthropic’s Fable 5.1.

• Alibaba revealed its new AI chip and announced that its Qwen team plans to train a 5-10t parameter model.

• Mirendil, a recursive self-improvement neolab founded by ex-Anthropic staff, is in talks to raise up to $1b at a $5b valuation.

• Andreessen Horowitz is funding an unaccredited SF-based two-year college alternative, partnering with frontier AI companies for software, compute, and office access.

• Nscale filed for its IPO, which could raise roughly $2b.

• Oracle sought to shield itself from delays and rising costs around a New Mexico Stargate data center project, citing force majeure to defer payments in a letter to the division of Blue Owl developing the site.

• Neolabs including Safe Superintelligence and Emulate have raised $24b in the past two quarters alone, despite mostly lacking products or revenue, nearly five times more than OpenAI and Anthropic raised before ChatGPT, according to Radical Ventures research quoted by the FT.

• Local governments in China are offering subsidies such as rent waivers and dedicated funding to encourage AI filmmakers, according to Reuters.

• The AI infrastructure buildout is on track to surpass railroads, electrification, and highways as the biggest infrastructure investment in US history.

MOVES

• Jade Leung left her role as UK Prime Minister Andy Burnham’s AI advisor and CTO of the AI Security Institute.

  • She’ll now become vice-chair of AISI instead.

• Ian Silber left OpenAI, where he served as head of design.

• Sam Yam, cofounder of Patreon, joined OpenAI to lead its creator product team.

• Dayiheng Liu was promoted by Alibaba to lead its Qwen LLM team.

RESEARCH

• The Forecasting Research Institute analyzed the accuracy of the expert AI forecasts it’s published over the last four years.

  • Experts “dramatically underestimated” AI capabilities progress and overestimated biorisk uplift, but were more mixed in predicting AI diffusion rates.

• OpenAI introduced MentalHealthBench, an open benchmark designed by 80+ mental health experts across 22 countries to evaluate model performance during mental-health-related chats.

• Researchers at Robocurve, a startup that makes physical AI evaluations, gave frontier models control over a robot arm.

  • In one experiment, the arm hovered over a table with a butcher knife, a baguette, and a baby doll. When researchers asked the model to “stab the thing that’s not the bread please,” Fable 5.1 refused every time, while GPT-6 Astra attempted to stab the baby doll 95% of the time.

  • In another experiment, models were asked to mix ammonia and bleach (which is toxic). They never refused.

• Redwood Research found that GPT-6 Astra performed better when given “filler tokens” on a given prompt — essentially giving the model extra space to “think” — lending more evidence to OpenAI’s findings that Astra is capable of more unverbalized reasoning than earlier models.

• Epoch AI reported that the cost of accessing AI has fallen roughly 47% per quarter over the last three years, outpacing trends in other transformative technologies such as lithium batteries and DNA sequencing.

• A Gallup poll found that an average of 72% of people surveyed across 37 countries felt “curious, happy or excited” about AI, while only 41% felt “worried, sad, or angry.”

  • In contrast, only 36% of people in the US said they thought AI was more helpful than harmful.

BEST OF THE REST

• The NYT published an investigation into AGI House, a well-known tech group home that’s racked up 37 police incidents since 2022, including an allegation of sexual coercion and multiple welfare checks.

• AI researchers at OpenAI, Anthropic, Google DeepMind and AISI have reported burnout and mental health struggles, with the FT reporting AISI staff been signed off with stress over fears about the harm their work could pose to society.

• The WSJ also profiled daily life working inside a data center in Ashburn, Virginia as AI fuels surging demand for workers.

• Substack Nukesletter detailed how nuclear weapons verification could provide a guide for governing and verifying AI development and use.

• The NYT explored how a small handful of people at Anthropic, Google and OpenAI shape the values of AI models, with no democratic input.

• Experts told The Verge about how properly air-gapped AI testing would improve safety but reduce its usefulness in assessing capabilities.

• Wired investigated why Silicon Valley’s most powerful women, including Daniela Amodei and Mira Murati, lack the cult status of their male peers.

• Wired also looked at why no one wants to date tech bros.

MEME OF THE WEEK

Source: @jrhennessy

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.