Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
Housekeeping
Next week’s issue will be out on Thursday, not Friday.
Top stories
The question on everyone’s minds this week: is scaling hitting a wall?
Reports from The Information, Bloomberg and Reuters all suggested that AI companies are not seeing the returns from scaling they hoped for.
“While Orion’s performance ended up exceeding that of prior models, the increase in quality was far smaller compared with the jump between GPT-3 and GPT-4”
“[Orion] did not hit [OpenAI’s] desired performance … [it] is still not at the level OpenAI would want in order to release it to users, and the company is unlikely to roll out the system until early next year”
“An upcoming iteration of [Google’s] Gemini software is not living up to internal expectations”
“Anthropic found 3.5 Opus performed better on evaluations than the older version but not by as much as it should, given the size of the model and how costly it was to build and run”
“Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, told Reuters recently that results from scaling up pre-training - the phase of training an AI model that use s a vast amount of unlabeled data to understand language patterns and structures - have plateaued.”
“‘The 2010s were the age of scaling, now we're back in the age of wonder and discovery once again. Everyone is looking for the next thing,’ Sutskever said. ‘Scaling the right thing matters more now than ever.’”
Yet as many pointed out, the mood on the ground is still that scaling works, and AGI is coming soon:
Chubby: "If Ilya thought they hit a wall why did he start an ASI company?"
Nathan Lambert: “Inside the labs, I've heard nothing to say that scaling laws are stopping.”
Miles Brundage: “Betting against AI scaling continuing to yield big gains is a bad idea.”
Dario Amodei: “I’ve seen the story happen for enough times to really believe that probably the scaling is going to continue.”
Sam Altman: “there is no wall”
So what’s going on?
1. Failure to meet internal expectations does not mean progress is slow — outsiders don’t fully appreciate just how high internal expectations are.
2. Scaling pre-training compute might be slowing, but there’s a whole new scaling paradigm: scaling test-time compute.
Noam Brown: “It turned out that having a bot think for just 20 seconds in a hand of poker got the same boosting performance as scaling up the model by 100,000x and training it for 100,000 times longer.”
Both The Information and Reuters point to this as a way around diminishing returns to pre-training compute scaling.
3. Things actually are slowing down, and the labs are kidding themselves.
Right now, we can’t know for sure what’s going on.
As Casey Newton notes, “ultimately, we won’t be able to evaluate the next-generation models until the AI labs put them into our hands”. Happily, it seems like a new generation is coming soon — and I bet that will put these debates to bed, for at least a while.
The EU Commission published the first draft of the General-Purpose AI Code of Practice. It’s worth reading in full, but here’s a quick summary of the most AI safety relevant bits (click the links to see screenshots from the draft):
The “types of systemic risks” list includes loss of control, CBRN risks, and automated AI R&D.
The code highlights “dangerous model propensities” that might cause systemic risk, such as misalignment, a tendency to deceive, power-seeking and collusion.
All developers of GPAI models with systemic risks would have to make and publish a “Safety and Security Framework”, which sounds a lot like a souped-up RSP.
It would include risk identification, risk analysis and risk forecasts.
It would also require a "continuous process of Evidence Collection", which will include running “best-in-class” model evaluations.
And it would require companies to outline, in advance, safety and security mitigations that will “keep systemic risks below an intolerable level” — if-then commitments, basically.
Companies would have to “enable meaningful independent expert risk and mitigation assessment”, which “may involve independent testing of model capabilities”.
Some initial reactions:
This is obviously still a draft, with lots of open questions to be resolved between now and May. Many of those questions are explicitly called out in the draft.
It’s very vague in lots of places, but that’s probably the right approach for now: it's too early to mandate e.g. specific eval protocols or mitigations.
A lack of mandatory third-party evals seems like a miss, though I’m not sure how the EU could do it given a lack of in-house evals capacity.
There seems to be a huge question mark over how we’re defining automated AI R&D — you could say we already have that, in part.
One of the systemic risks listed is “large-scale disinformation … [that leads to] loss of trust in the media”. That seems awfully hard to define.
Code of Practice Plenary participants have two weeks to provide feedback on the first draft. There will then be a bunch more rounds before the code comes into effect in May.
You can read the full thing here, and a helpful accompanying Q&A here.
On Transformer: Safety-focused people keep leaving OpenAI.
Last week, Lilian Weng — the company’s VP of research and safety — announced her resignation.
And this week, Richard Ngo — an AI governance researcher who worked extensively on AI safety — said he’s leaving, too.
In his resignation message, Ngo said that he has “a lot of unanswered questions about the events of the last twelve months, which made it harder for me to trust that my work here would benefit the world long-term”.
The departures join a long, long list of safety talent who’ve left the company, with many expressing concerns on the way out. In addition to Weng and Ngo, all of the following have left this year:
Ilya Sutskever, Jan Leike, Miles Brundage, William Saunders, Leopold Aschenbrenner, Pavel Izmailov, Collin Burns, Carroll Wainwright, Ryan Lowe, Daniel Kokotajlo and Cullen O’Keefe.
The discourse
Claudia Wilson says “beating China” doesn’t have to mean abandoning safety:
The reality is that safety testing is inexpensive relative to training costs. It is simply too affordable, relatively speaking, to slow innovation.”
Vinod Khosla is all-in on the “beat China” narrative:
“The real risk isn't ‘sentient AI’ but losing the AI race to nefarious ‘nation states,’ or other bad actors, making AI dangerous for the West.”
On Transformer, I argue that Meta’s AI “safeguards” are … bullshit.
“Meta’s acceptable use policy, like all of its so-called ‘safeguards’, has no teeth. Because of its decision to openly release its models’ weights, Meta cannot prevent anyone from using — or misusing — its models … Meta’s “guardrails” and “policies” are simply theatre, designed to provide political cover rather than meaningful protection.”
There’s a lengthy, hard-to-summarise excerpt of Henry Kissinger, Eric Schmidt and Craig Mundie’s new book in The Atlantic.
Policy
Lots of AI-related bills are reportedly being considered for inclusion in the National Defense Authorization Act.
Sen. John Thune was elected Senate majority leader. That probably bodes well for his and Klobuchar’s AI bill.
Donald Trump made a bunch of cabinet appointments who’ll work on AI in some form:
Lee Zeldin was nominated for EPA Administrator, and specifically highlighted making the US “the global leader of AI” as one of his priorities.
Marco Rubio is nominated for Secretary of State, Kristi Noem for Homeland Security Secretary, Tulsi Gabbard for Director of National Intelligence, Pete Hegseth for Defense Secretary, Mike Waltz for National Security Advisor, and John Ratcliffe for CIA Director.
The US reportedly ordered TSMC to stop making advanced AI chips for Chinese firms.
The DHS released a voluntary framework for deploying AI in critical infrastructure. DHS Secretary Mayorkas said he hopes Trump will keep the AI Safety and Security Board around.
The Department of Energy has been testing Claude for nuclear security risks.
NIST launched its “Assessing Risks and Impacts of AI” program.
The UK AI Safety Institute celebrated its one-year anniversary by outlining its work-to-date and future plans. It also released an open-source repository of safety evals.
Politico, meanwhile, reports that the Trump White House might not be too hot on letting UK AISI test American models.
Influence
OpenAI’s been touting a “blueprint for US AI infrastructure” in DC.
From the Washington Post:
“[It] calls for special economic zones with fewer regulations to incentivise new AI projects, a fleet of small nuclear reactors to power data centres aided by the U.S. Navy and a “North American Compact” allowing US allies to collaborate to bolster the field.”
The vibe is very much “beat China”.
Americans for Responsible Innovation launched a petition urging Trump to make Elon Musk his special adviser on AI.
BSA | The Software Alliance asked Trump and JD Vance to tackle regulations that may be “unnecessarily impeding AI adoption”.
It also called for “a single national law that requires developers and deployers of high-risk uses of AI to implement risk management programs and conduct impact assessments”.
A big new report from “Inference Magazine” argues that the UK needs to build AI data centres, and to do so it should build lots of nuclear capacity and create “Special Compute Zones”.
Industry
Google released a new build of Gemini, which shot straight to the top of the Chatbot Arena leaderboards.
Google also launched a standalone Gemini app for iPhone.
OpenAI reportedly plans to launch an AI agent tool in January.
The ChatGPT macOS app can now read from certain coding apps.
OpenAI also released a ChatGPT app for Windows.
Elon Musk amended his lawsuit against OpenAI to include Microsoft, Reid Hoffman, Dee Templeton and Rob Bonta as defendants.
Evidence submitted alongside the amendment shows that Ilya Sutskever and Greg Brockman raised concerns about both Musk and Altman’s motives back in 2017. On Altman specifically, the two said:
“We haven't been able to fully trust your judgements throughout this process … Is AGI truly your primary motivation? How does it connect to your political goals?”
Alibaba released Qwen2.5-Coder, which it says matches GPT-4o on coding tasks and seems to actually be pretty good.
Tencent launched Hunyuan-Large, which might now be the world’s best open-weight model.
Google DeepMind open-sourced AlphaFold 3.
xAI has reportedly raised $5b at a $45b valuation — according to the FT, the deal is “fully allocated” and will formally close later this month.
xAI’s huge data centre buildout is reportedly worrying competitors, at least one of which hired a spy plane to see what Elon’s up to.
That data centre got approval to use 150 MW of grid power last week, too.
Amazon will reportedly roll out its Trainium 2 AI chips in December.
SoftBank said it’s building Japan’s most powerful AI supercomputer with Nvidia B200s.
Anthropic launched new tools to automate prompt engineering.
Perplexity introduced ads.
Writer raised $200m at a $1.9b valuation.
Tessl raised $125m at a $750m valuation.
11x raised $50m, led by Andreessen Horowitz, at a $320m valuation.
Zero Gravity Labs raised $40m to develop a decentralised AI operating system.
Moves
Greg Brockman is back at OpenAI. He’s reportedly talking to Sam Altman about “creating a new role for him to focus on significant technical challenges”, per Bloomberg.
François Chollet announced he is leaving Google.
Matt Perault, formerly a public policy director at Facebook, announced he is joining a16z as head of AI policy.
Dan Hendrycks joined Scale AI as an advisor.
Barret Zoph, Luke Metz and Mianna Chen are reportedly working with Mira Murati on her new company.
Tyler Grassmeyer is Microsoft’s new director of congressional affairs.
TechNet appointed Katie Kelly as executive director for Florida and the Southeast.
AMD is laying off about 1,000 people.
Best of the rest
Epoch released a new benchmark called FrontierMath. Current AI models score less than 2% on it. We’ll see how long that lasts.
The FT has a piece on the race to create new benchmarks.
The Nvidia B200 and Google Trillium chips do really well on MLPerf, unsurprisingly.
The UK Home Office is using AI to help make decisions about immigration enforcement. What could go wrong?
Nearly half of AI data centres may face power shortages by 2027, according to a new Gartner report.
GovAI and UK AISI researchers released a safety case template.
This new paper on evaluation statistics for language models looks interesting.
So does this one on the impact of AI on scientific discovery.
Waymo is now widely available in LA.
Sam Altman is expecting a child next year.
Thanks for reading; have a great weekend.