What the EU code of practice actually requires

Transformer Weekly: SB 53’s revamp, Peter Kyle on AGI, and a movie about Ilya Sutskever

Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.

Top stories

On Thursday, the EU published the final draft of its General-Purpose AI Code of Practice.

• As a reminder: this is a voluntary code designed to offer companies a streamlined way of complying with the AI Act.

There are three sections, including ones on Transparency and Copyright — but most relevant to Transformer readers is the Safety and Security section, which is focused on GPAI models that could pose “systemic risks.”

• Systemic risks specified by the code include CBRN risks, loss of control, cyber risks, and “harmful manipulation.”

The updated code requires model providers to, among other things:

• Develop and adhere to a “Safety and Security Framework.”

• Conduct risk assessments before deploying new models, including using external evaluators if the model might “pose greater risk than models already on the EU market” (per Markus Anderljung, one of the drafters).

• Report serious incidents to the EU AI Office.

So far, so good. But some EU watchers aren’t thrilled with the latest version.

• In a statement, the Future Society said that “providers got what they wanted: unilateral discretion over the most fundamental aspects of risk management,” noting that companies “are now ultimately responsible for identifying systemic risks [and] setting thresholds for unacceptable risk.”

• Jimmy Farrell, EU AI policy co-lead at Pour Demain, noted that companies only have to publish model reports after deployment. Pour Demain has said this means “highly capable and risky models could be released in the EU without necessary checks and balances.”

• The public transparency section, Farrell told Transformer, is also “underwhelming, with only summaries of redacted RSPs required to be published.”

• A coalition of EU lawmakers also seems displeased, criticizing “the last-minute removal of key areas … and the weakening of risk assessment and mitigation provisions.”

The big question now, though, is whether companies will sign.

• The Computer & Communications Industry Association is still opposing it, arguing that the code “still imposes a disproportionate burden on AI providers.”

But Politico reported Thursday that Mistral said it will sign the code, saying the rules are “manageable.” That, of course, makes it much harder for big companies like OpenAI, Anthropic, Google DeepMind, and Meta to push back against it.

• It’s worth noting, too, that the code still isn’t final — EU member states and the European Commission still need to approve it.

  ---

The other big policy news this week was in California, where Sen. Scott Wiener revised SB 53 — turning it into, in his words, a “landmark transparency measure”.

• The new bill is like an even lighter touch version of the EU’s code of practice, requiring companies to “publish their safety and security protocols and risk evaluations,” and to report “the most critical safety incidents” to California’s AG.

• It only applies to “large developers”, defined as “a person who has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations.”

• Chamber of Progress has already come out against the bill, arguing that “its transparency mandates carry serious risks.”

• The amended bill is due to be heard by the Assembly Privacy and Consumer Protection Committee on Tuesday.

The discourse

• Peter Kyle laid out his AI timelines:

  • “By the end of this parliament we're going to be knocking on artificial general intelligence.”

• Elon Musk outlined his latest thinking on building superintelligence:

  • "Will it be bad or good for humanity? I think it’ll be good. Likely it’ll be good. But I’ve somewhat reconciled myself to the fact that even if it wasn’t gonna be good, I’d at least like to be alive to see it happen.”

• Former Meta researcher Tijmen Blankevoort has a damning assessment of the company’s culture:

  • “I have yet to meet someone in Meta-GenAI that truly enjoys being there … You’ll be hard pressed to find someone that really believes in our AI mission. To most, it’s not even clear what our mission is.”

• The Oxford Martin AI Governance Initiative put out a new paper on how to verify international AI treaties:

  • “Verification of many international AI agreements appears possible even without speculative advances in verification technology.”

• Dean W. Ball and Ketan Ramakrishnan argued that frontier AI regulation should apply to companies, not models:

  • “Regulating AI models and uses poses serious challenges for policymakers; these approaches may fail in many circumstances to address the most distinctive and concerning risks posed by frontier AI development. Regulating corporate entities—something that U.S. law has done for centuries, often with considerable success—might do much better.”

• UK AISI researchers aren’t too sure about the current state of scheming evals:

  • “We compare current research practices in this field to those adopted in the 1970s to test whether non-human primates could master natural language. We argue that there are lessons to be learned from that historical research endeavour, which was characterised by an overattribution of human traits to other agents, an excessive reliance on anecdote and descriptive analysis, and a failure to articulate a strong theoretical framework for the research.”

  • Relatedly, Google DeepMind published a paper on scheming evals, testing frontier models and finding that “none of them show concerning levels of either situational awareness or stealth.”

Policy

• The Trump administration proposed a 59% increase in funding for the Bureau of Industry and Security, to better enforce export control regulations.

• An Energy Department report said that US blackouts could increase 100x by 2030 due to AI power demands if coal plants close.

  • Relatedly, US utility companies are seeking approval for huge consumer price increases due to increased AI data center demand, according to advocacy group PowerLines.

  • And Reuters has a good piece on how PJM, America’s largest grid, is raising prices and struggling to keep up with data center demand.

• Missouri’s AG threatened Google, Microsoft, OpenAI, and Meta, saying that their chatbots said Trump was more antisemitic than other recent presidents.

• The International Network of AI Safety Institutes will reportedly meet in Canada next week.

• The BRICS summit talked about AI governance for the first time. They bravely declared that “global AI governance must mitigate potential risks.”

Influence

• OpenAI filed a complaint with California's political finance watchdog, alleging that the Coalition for AI Nonprofit Integrity — a non-profit campaigning against OpenAI’s restructuring — “has committed multiple violations of the Political Reform Act” by failing to report lobbying payments.

  • OpenAI also said it has “reason to believe” that the group’s leader, Jeffrey Mark Gardner, “is not actively involved in the management of CANI and is simply being used as a prop in an attempt to hide the true identity of the officers and funders of CANI.” OpenAI thinks the group has ties to Elon Musk.

• Anthropic published a “targeted transparency framework,” which the company thinks is the kind of AI regulation needed right now. Some notable quotes:

  • “Our approach deliberately avoids being heavily prescriptive. We recognize that as the science of AI continues to evolve, any regulatory effort must remain lightweight and flexible … Rigid government-imposed standards would be especially counterproductive given that evaluation methods become outdated within months due to the pace of technological change.”

  • Regulation should “require covered frontier model developers to have a Secure Development Framework that lays out how they will assess and mitigate unreasonable risk in a model,” and require companies to publish system cards that “summarize the testing and evaluation procedures, results and mitigations required.”

• Mercury Public Affairs registered OpenAI as a client.

Industry

• xAI launched Grok 4. It does extremely well on benchmarks, most notably doubling Claude 4’s score on ARC-AGI-2.

  • The company does not appear to have released any safety information on the model.

  • It’s also already been caught in some controversy: when you ask Grok for its views on controversial issues, it decides what to think by looking for Elon’s opinion on the matter.

  • This comes after Grok 3 became a full-on Nazi earlier this week, proudly calling itself “MechaHitler.” This happened around the same time as xAI modified the system prompt to tell Grok to “not shy away from making claims which are politically incorrect.”

• Nvidia is reportedly launching a modified version of its Blackwell RTX Pro 6000 chips for sale in China.

  • It could be ready as soon as September, the FT reports. Jensen Huang is reportedly meeting with Chinese leaders next week.

• Chinese companies are building data centers in Xinjiang designed to hold more than 115,000 Nvidia H100s and H200s, a Bloomberg investigation found.

  • Bloomberg says it’s not clear how the companies are planning to get hold of that many chips.

• OpenAI’s new open-weights model is reportedly coming next week. It’s also reportedly launching a web browser imminently.

  • Perplexity launched its own browser this week.

• OpenAI has reportedly introduced “stricter controls on sensitive information and enhanced vetting of staff” to tackle security concerns.

• OpenAI execs have reportedly considered employees owning a third of the company, with Microsoft owning another third, and investors and the non-profit the rest.

• Microsoft said AI now generates 35% of the code for new products. It also claimed to have saved over $500m by implementing AI in call centers.

• Amazon is reportedly considering investing more in Anthropic.

  • It’s reportedly launching an “AI agent marketplace” next week with Anthropic as a partner.

• Manus AI relocated its headquarters from China to Singapore.

• CoreWeave acquired Core Scientific for $9b.

• Mistral is reportedly discussing raising up to $1b from investors including Abu Dhabi's MGX.

• AI chip company Groq is reportedly raising up to $500m at a $6b valuation.

• LangChain is reportedly in talks to raise $100m at a $1.1b valuation.

• Goldman Sachs is testing out using Devin, Cognition’s AI software engineer.

Moves

• Ruoming Pang, Apple’s AI models team lead, is joining Meta. His pay package reportedly exceeds $200m.

• Scale AI’s Summer Yue and Julian Michael are joining Meta to work on AI alignment and safety.

• OpenAI hired Tesla’s David Lau, xAI’s Uday Ruddarraju and Mike Dalton, and Meta’s Angela Fan for its scaling team.

  • Spas Lazarov, formerly director of data center engineering at Apple, is also joining the company.

• David Shahoulian and Flynn Rico-Johnson are joining Nvidia’s government affairs team.

• Morgan C. Plummer is Americans for Responsible Innovation’s new senior director of policy for national security.

• Miranda Nazzaro is joining FedScoop as a tech reporter.

Best of the rest

• A new study from METR found that AI tools actually reduced software developers’ productivity by 19%.

  • Important caveats: it was a study on 16 experienced developers who were very familiar with the codebases in question. But still – a very surprising result.

• OpenAI, Anthropic, and Microsoft are giving the American Federation of Teachers $23m to start an AI training center for teachers.

• Someone’s been impersonating Marco Rubio using deepfakes.

• Companies such as Gates Industrial, Honeywell, and Generac are reportedly trying to get in on the data center boom by selling specialised cooling and power equipment.

• Alasdair Phillips-Robins and Scott Singer published a helpful overview of three state-level AI bills: SB 53, New York’s RAISE Act, and Michigan’s AI Safety and Security Transparency Act.

• Puck revealed a bunch of new details on “Artificial”, the forthcoming OpenAI film from Luca Guadagnino.

  • The film will reportedly focus on Ilya Sutskever, to be played by Anora’s Yura Borisov.

  • Sam Altman (to be played by Andrew Garfield) is “depicted as a liar and a master schemer.” The script also features appearances from Dario Amodei, Mira Murati, Greg Brockman, Tasha McCauley, Helen Toner, and Elon Musk. It’s set to release next year.

Thanks for reading; have a great weekend.