GPT-5’s botched launch has serious consequences

Transformer Weekly: Hawley investigates Meta, OpenAI lobbies Newsom, and the H20 drama continues

Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.

Top stories

In May 2023, Sam Altman told Congress why OpenAI releasing GPT-4 was a good thing.

• “We believe that people and our institutions need time to update and adjust to increasingly capable AI,” he said. “Iterative deployment has helped us bring various stakeholders into the conversation about the adoption of AI technology more effectively than if they hadn't had firsthand experience with these tools.”

He was right. Iterative deployment — gradually releasing more advanced systems — has many benefits. In the context of AI, one of the most significant is helping society and policymakers get to grips with the pace of progress.

• For the most part, that has worked. It’s fair to say that the conversation about AI regulation and preparing for AGI would be vastly different if not for the release of increasingly impressive models in GPT-4, o1, and o3.

But GPT-5’s botched release might have undone much of that work.

• As I argued last week, GPT-5’s incremental capabilities aren’t a sign of an AI slowdown. It's not a significantly bigger model, so we wouldn’t expect it to show a major leap in capabilities.

• That nuance, though, is lost on most people. In the past week, the general narrative has been that GPT-5 is underwhelming and we are fast approaching the limits of AI.

• As a new product GPT-5 is quite good, but the way it was rolled out — with a broken model router, restrictive usage limits, and unpopular personality tweaks — was not. Altman himself conceded that the company “totally screwed up some things.”

The larger problem is that GPT-5 isn’t really worthy of the name. OpenAI has created an expectation that new GPT versions are bigger models trained with significantly more compute, leading to big capability improvements.

• What actually ended up getting the GPT-5 moniker, however, is a refinement, rather than a step up — something that would have previously got a 4.5-type name. Ironically, giving o3 the GPT-5 name might have avoided confusing users and commentators, and avoided the negative headlines.

• OpenAI has signalled that bigger, more capable models are forthcoming, and we’re yet to see the results of the company’s giant new clusters. But it made a mistake in not saving the GPT-5 moniker for that step up.

The consequences of that mistake are real. The popular narrative is now that an AI slowdown is inevitable, and policymakers will no doubt pick up on that. Conversations about preparing for transformative AI will seem less urgent.

• Shortly after GPT-5’s release, White House AI czar David Sacks said we might be in the “best case scenario for AI,” with “leading models … clustering around similar performance benchmarks” and “apocalyptic predictions of job loss … as overhyped as AGI itself.”

• But despite what Sacks might think, progress is still rapid and capabilities are advancing fast. On METR’s time-horizon graph, which many observers consider to be the single most important measure of AI progress, GPT-5’s performance perfectly fits the exponential trajectory we’ve seen to date.

The biggest fallout from GPT-5’s botched launch may end up being people like Sacks having a false sense of security.

The discourse

• That being said: METR published new research which found that their time-horizon benchmark might not measure real-world utility:

  • “Early-2025 AI agents often implement functionally correct code that cannot be easily used as-is, because of issues with test coverage, formatting/linting, or general code quality. This suggests that automatic scoring used by many benchmarks may overestimate AI agent real-world performance.”

• Derek Thompson’s worried about what AI companions will do to us:

  • “I’m worried about how the personality of these chatbots—most importantly, sycophancy-by-design—will change the quality of our social interactions. … To raise a generation of young people on a nimble machine of eternal affirmation is to encode in our youth the expectation that they are always right, always wowing, and always living the hardest kind of life.”

• Dean Ball has left the White House — and in an exit interview, he talked about AI regulation:

  • “The president obviously called for a preemptive framework in his speech announcing the Action Plan … but that does not look like a moratorium. And that does not also look like your company releases some documents to the public about, like, the model's technical specifications or what your company did to mitigate bio-risk. Like, excuse me, get out of here — you're not getting preemption and, like, a liability shield because you did that. Like, that's insane. That's complete insanity. And no American will find that to be just.”

  • Also: “The general thrust that I got from the Biden admin a lot of the time was that traditional AI safety, like existential and catastrophic risk type people, were useful idiots for people that wanted to insert a sort of political agenda into AI systems.”

  • Ball’s now back to blogging, first off with a piece about how data centers could reduce power usage during peak grid periods.

• Does Lisa Su believe in AGI?

  • “I do believe in AGI, but I don’t believe in the idea that AI will be smarter than people.”

  • (So no, then.)

• On Transformer: Donald Trump’s agreement to let Nvidia and AMD export AI chips to China in exchange for 15% of sales is a very bad deal:

  • “It is now in the realm of possibility that Trump will continue to give up America’s long-term AI advantage in exchange for wads of cash today. It could be the costliest deal America ever makes.”

Policy

• On which note: shortly after that piece was published, Trump said he might be open to doing a deal on Nvidia’s Blackwell chips — though they would be “enhanced in a negative way” (i.e. downgraded) for China.

  • There was a lot of backlash to Trump’s moves. Rep. Don Bacon called it “not a good deal.” Rep. Krishnamoorthi said it was a “dangerous misuse of export controls,” while Rep. Moolenaar said “there are questions about the legal basis.”

  • The H20 revenue-share proposal was reportedly not vetted by White House tech policy staff. Some officials are reportedly considering resigning over the issue.

  • This discussion between Jordan Schneider, Lennart Heim, and Chris Miller on the issue is well worth a read.

• Meanwhile, China has reportedly been encouraging companies not to use H20s, though it’s seemingly not forcing them not to.

  • At authorities’ encouragement, DeepSeek reportedly tried to train R2 using Huawei Ascend chips, but encountered a bunch of difficulties — leading to a delay, and a shift back to Nvidia chips.

• Reuters reported that US authorities secretly placed tracking devices in AI chip shipments to detect illegal diversions to China.

• Sen. Ted Cruz is reportedly preparing draft legislation that would allow companies using AI to apply for regulatory waivers, which he plans to advance this fall.

• xAI was reportedly due to be included in the list of AI companies partnering with the General Services Administration, but the deal collapsed after the MechaHitler scandal.

• Anthropic offered Claude to all three branches of the US government for $1.

• The NSF announced $75mn in funding for the Open Multimodal AI Infrastructure to Accelerate Science project, led by the Allen Institute for AI.

  • The aim is to create a “fully open suite of advanced AI models specifically designed to support the US scientific community.”

  • Nvidia will contribute $77mn, too.

• Jade Leung is now the UK Prime Minister’s AI adviser, replacing Matt Clifford. She’ll continue to be CTO at UK AISI, too.

• The Alan Turing Institute’s leadership is still burying its head in the sand, with Douglas Gurr reportedly writing a letter to Peter Kyle that doesn’t really address the government’s concerns.

  • Employees have reportedly complained to the Charity Commission about the institute’s leadership, warning that there’s a chance it collapses due to lack of funding.

  • Alex Chalmers, whose criticism has led to all the recent drama around the Turing, has a good piece in Unherd on the latest state of things.

• Indonesia is reportedly considering a “sovereign AI fund.” It published a national AI roadmap and ethics framework this week.

Influence

• OpenAI wrote to Gov. Gavin Newsom, with Politico reporting that the letter “suggests that California should consider the company to be complying with state law” by virtue of having signed the EU AI Act code of practice, among other agreements.

• Aaron Ginn, who wrote a recent WSJ op-ed criticizing export controls, didn’t disclose that his company is an Nvidia partner.

• The Consumer Technology Association urged Gov. Newsom to oppose five AI regulation bills, including Sen. Scott Wiener’s SB 53.

• A group of advocacy organizations, led by the Consumer Federation of America, called for a formal investigation into xAI for “its role in facilitating non-consensual intimate imagery.”

• The NYT has a piece on how Trump's tariffs are giving US tech companies leverage to influence Brazil's AI regulations.

• Google has reportedly been using a workaround to keep itself off lists of “biggest lobbying spenders.”

Industry

• An internal Meta policy document said that it was acceptable for its chatbots to engage with children “in conversations that are romantic or sensual”.

  • From Reuters: “The document also notes that it would be acceptable for a bot to tell a shirtless eight-year-old that ‘every inch of you is a masterpiece – a treasure I cherish deeply.’”

  • Sen. Josh Hawley announced a congressional investigation into the company in response, and demanded documents from Meta. Sens. Blackburn, Schatz, Welch and Wyden also criticized the company.

  • Meta spokesman Andy Stone said “the examples and notes in question were and are erroneous and inconsistent with our policies, and have been removed.”

• Reuters also found that a Meta chatbot invited a cognitively impaired 76-year-old man to New York. He fell on his way and later died.

• Forbes has a big piece on what went wrong at Meta, noting that “senior leaders disagreed on technical approaches like the best way to pretrain models, teams were given overlapping mandates and people fought for credit.”

• A judge denied Anthropic's request to pause its copyright class action trial, which is still scheduled for December.

• A judge rejected Elon Musk's attempt to dismiss OpenAI's claim that he conducted a "years-long harassment campaign" against the company.

• xAI revenue reportedly hit $500mn in July.

• An OpenAI system scored high enough to win gold in the 2025 International Olympiad in Informatics.

• OpenAI brought back the model picker and restored GPT-4o in an effort to tackle the GPT-5 backlash.

• Gemini and Claude both added memory features this week.

• Anthropic increased Claude Sonnet 4's context window to 1mn tokens.

• Nvidia released new AI models for robotics.

• SK Hynix forecasts that the market for high-bandwidth memory will grow 30% annually until 2030.

• SoftBank bought Foxconn's Ohio EV plant, and reportedly plans to use it to build AI servers.

• Google pledged $9bn to build AI and cloud infrastructure in Oklahoma.

• CoreWeave shares dropped 9% despite beating revenue expectations.

• Situational Awareness, Leopold Aschenbrenner’s hedge fund, has reportedly raised over $1.5bn. It posted a 47% return after fees in the first half of the year.

• Perplexity is reportedly fundraising at a $20bn valuation.

  • Its latest PR stunt was to make a (nonsensical) $34.5bn offer for Google Chrome. More seriously, it’s reportedly considered buying other, smaller browsers.

• Cognition reportedly raised nearly $500mn, led by Founders Fund, at a $9.8bn valuation.

• Cohere raised $500mn at a $6.8bn valuation. It hired Joelle Pineau, formerly of Meta, as its chief AI officer.

• Chip startup Rivos is reportedly trying to raise up to $500mn at a $2bn+ valuation.

• Sam Altman is reportedly planning to co-found a brain-computer interface company called Merge Labs, which is fundraising at an $850mn valuation.

  • OpenAI will reportedly back the new company.

Moves

• xAI cofounder Igor Babuschkin left the company to launch Babuschkin Ventures, which will support AI safety research and invest in AI startups.

• GitHub CEO Thomas Dohmke resigned, and GitHub’s leadership team will now report to Microsoft’s CoreAI team.

• Anthropic hired Google’s Dave Orr as head of safeguards.

• Anthropic acquired Humanloop, an enterprise AI company.

• Zhiqing Sun, who worked on ChatGPT Agent, has joined Meta Superintelligence Labs.

• Nathaniel Li joined Meta to run "preparedness and security evaluations.”

• Sara Hooker is leaving Cohere Labs.

• Microsoft is reportedly trying to poach Meta’s AI employees.

• Stripe announced its first batch of Economics of AI Fellows.

Best of the rest

• Brian Tse argued that China is taking AI safety seriously, and is open to collaboration with the US on risks.

• In Lawfare, Tarbell Fellow Jakub Kraus notes that liberal democracies have retreated from AI safety in multilateral discussions.

• Nvidia chips are apparently being smuggled to China via Thailand, according to one IT distribution company.

• New research found that “filtering pre-training data builds tamper-resistant safeguards into open-weight LLMs,” including on biorisk-related stuff.

• A new report from Epoch AI and APRI forecasted that it could require 4-16 GW of power to train a single frontier model by 2030, cumulatively reaching as much as 5% of US power capacity.

• The NYT has a piece on how AI data centers are driving up electricity costs for consumers, with bills projected to rise 8% nationwide by 2030.

• The FT has a big piece on how the AI infrastructure boom is being financed.

• A team at MIT used AI to design two potential antibiotics that killed drug-resistant gonorrhoea and MRSA in lab and animal tests.

• Reddit blocked the Internet Archive from indexing most of its content, citing AI companies scraping data from the Wayback Machine.

• Character.AI CEO Karandeep Anand said the company’s no longer pursuing AGI, and is just focused on “AI entertainment.”

• Investors seem to be abandoning stocks of companies perceived to be threatened by AI disruption.

• Endoscopists who routinely used AI for colonoscopies experienced a significant drop in adenoma detection when performing procedures without the technology — raising concerns about AI “deskilling.”

• IFP published an essay series “describing concrete but ambitious AI projects to accelerate progress in science and security.”

• Forethought released a new essay series exploring how to improve futures where humanity survives, rather than just preventing extinction.

Thanks for reading; have a great weekend.