Transformative AI could supercharge development for the world’s poorest countries, allowing them to achieve in a decade what might otherwise take a century, according to a new report from the World Bank. But that’s only if governments act fast to lay groundwork that spreads AI’s gains, rather than concentrating them.

Crucially, the most powerful AI systems are being built in a small number of wealthy countries. And even heavyweights like the US or China haven’t quite figured out how to mitigate the economic-disparity threats that transformative AI poses within their own borders: concentrated market power, job displacement, and a shrinking tax base.

Countries outside the AI supply chain have two routes into the AI boom: adopting the technology themselves, or claiming a share of its gains through redistribution. According to current forecasts, adoption alone leaves poor countries richer but causes them to catch up with industrial countries more slowly. So far, redistribution solutions include initiatives like sovereign wealth funds, universal basic income (UBI), windfall clauses, and benefit-sharing programs.

Many of the redistribution strategies are emerging from non-governmental organizations, but a few AI companies — as well as their leaders and stakeholders — are funding studies into how those benefits might be shared, and floating proposals of their own.

In June, Anthropic committed $200m to support ambitious external research to “prepare society for the economic impacts of AI.” And OpenAI reportedly offered the US government a 5% stake in the company, worth roughly $42.6b, as part of a plan to create a sovereign wealth fund in which leading US AI companies would cede similar equity stakes to the government. Such a fund would give the American people an automatic stake in AI companies and infrastructure, OpenAI claims, even if they aren’t investing directly in financial markets. That said, it’s unclear whether the Trump administration has agreed to the offer, or whether other AI firms will follow suit.

OpenAI’s offer might be a way to buy political goodwill in an administration eager to assert its technological dominance. Still, it also signals a broader desire to spread the benefits of transformative AI across the population. Voters are apprehensive about potential AI-fueled job disruptions, data centers, and wealth inequality; a clearer sense of ownership could enable voters to participate more in AI’s gains. But if comparatively rich Americans need public ownership of AI stocks under transformative AI, how much AI equity would the global poor in Africa need?

If poor countries can’t keep up with rich, AI-fueled economies on their own, redistribution is a tough challenge. The traditional welfare state ends at the borders; rich countries have recently cut back foreign aid; and nation-states don’t incentivize politicians to take care of non-voters. If governments are unlikely to take ownership of sharing AI’s windfall worldwide, this might be an important opportunity for philanthropy to take the first steps.

Richer, but further behind?

If AI causes tremendous productivity benefits, it’s very unlikely that the global poor will be worse off.

The World Bank hopes that AI can offer a lifeline for developing countries, but this depends on them getting some basics right. “Developing countries that build the foundations now — power, connectivity, skills, and institutions — will be positioned to adopt and adapt AI for their people,” says Gaurav Nayyar, director of the World Bank’s Development Report 2026.

In fact, AI could accelerate progress against the currently non-eradicable diseases of tuberculosis, malaria and HIV/AIDS, according to the Forecasting Research Institute’s Longitudinal Expert AI Panel. Depending on the scenario, experts expect deaths to fall by about 34% relative to 2023 under slow progress (where AI serves as a digital collaborator), versus about 80% under rapid progress (where AI autonomously accelerates scientific progress) by 2050.

While AI may improve the available medicine in developing countries, its impact on local labor markets is less clear.

The job market in developing countries doesn’t contain many analytical and digital tasks, which are more exposed to AI, according to a March World Bank working paper. Furthermore, the connectivity required to harness AI fully is limited. On the other hand, AI-exposed services such as call centers, which benefited from offshoring, might be especially prone to automation.

Additionally, African and Asian economies also have advantages that are often forgotten, such as Taiwan and South Korea’s role in AI chip manufacturing. “Certain East Asian and Gulf countries have quite strong positions in the AI supply chain or due to data centers, and I would say some lower middle-income countries are better positioned to capture growth under AGI than some European countries,” says Rasmus Andersen, who has advised a range of governments on AI for the Tony Blair Institute for Global Change. How much countries gain from AI depends more on their current role in the AI supply chain and their digital readiness than their current income level.

Still, developing countries will probably struggle to reap the full gains of AI and to catch up with rich countries. The Global South lacks a highly educated workforce, digital infrastructure, and the hands-on know-how needed to put AI to work. Additionally, highly agrarian and informal economies are not as easily augmentable by AI. While most small-scale experiments in rich countries show that AI helps less productive people more, a field experiment in Kenya indicates the opposite, since they choose to implement different advice. Kenyan small-business owners gained access to an AI bot via WhatsApp, and while high-performing businesses may have benefited by over 15%, low-performing businesses performed nearly 10% worse. Even Columbia’s Daniel Björkegren, who found that AI helps teachers in Sierra Leone more effectively and more cheaply than traditional web search, conceded that AI’s benefits might mostly accrue in rich countries.

“AI is more like the Industrial Revolution and less like the mobile phone,” argues Deena Mousa, global health grantmaker at Coefficient Giving. (Disclosure: Coefficient Giving is Transformer’s primary funder.) Poor countries might not be able to leapfrog to AI as they did with online banking due to a lack of connectivity, compute, or competency, which didn’t require local bank branches and ATMs first.

In an extreme scenario, the gap between rich and poor countries grows dramatically even if poor countries are better off.

In 2025, senior research fellow Tom Davidson at Forethought, an institute that studies explosive AI progress, modeled a scenario in which the US could outpace the rest of the world. This hypothetical becomes more likely if the US government takes further steps to curb the spread of the latest AI technology — such as when it restricted foreign access to Anthropic’s Fable model in June. (The restriction was in response to a security vulnerability concern and promptly lifted.) This is bad news for AI sovereignty worldwide and worsens the sovereignty-capability tradeoff those countries face. An alliance among the compute supply chain — such as a more potent form of the US-led AI supply chain alliance Pax Silica — that prevents the spread of AI and the wealth it generates to adversaries would be the most likely reason for this extreme divergence.

“I expect economic divergence to cause massive power concentration,” Davidson says, “especially if the leading country acts in an authoritarian way.”

How to redistribute AGI’s wealth

There are ways countries could benefit from the AI boom. Economic divergence is not their fate, but a policy choice.

Sovereign wealth funds, like the one proposed by OpenAI earlier this year, could enable governments in the Global South to take stakes in AI companies, which would be especially helpful if they could capture much of the value AI generates. But OpenAI’s proposal to the US government doesn’t currently include benefits for foreign governments. Still, these governments could appoint independent fund managers to buy public stocks themselves.

While the US government has leeway to pursue such unorthodox economic policies, such as taking stakes in Intel, the International Monetary Fund generally holds developing countries to a higher standard. In 2021 El Salvador made bitcoin legal tender and gave every citizen $30 of it via a state wallet; the IMF agreed to a $1.4b loan in December 2024 only after the government made acceptance of bitcoin voluntary for businesses.

A sovereign wealth fund would allow poor governments to prepare for transformative AI by taking equity stakes in AI companies to benefit from the upside. But the government investing only in AI companies, rather than all sectors equally, would put all its eggs in one basket and affect AI companies’ decision-making. To alleviate the latter concern, distributing the dividends from such a fund could help disentangle. Since no one that we know of has offered an African government a 5% stake in OpenAI, they’d need to gamble on the stock market.

The most popular way to distribute the generated wealth is through a universal basic income paid equally to citizens. No-strings-attached cash transfers in the US have proven to help with education and prison avoidance. To assuage AI-displacement anxiety, Sam Altman personally funded a UBI experiment in which participants were given $1,000 a month for three years, with the results released in 2024 after touting UBI’s potential as an AI wealth-disparity band-aid. Several studies funded by Altman found no large effects on health or employment, suggesting that UBI is less effective in response to AI-related economic changes, at least in a world with moderate automation. (Altman later changed his mind, concluded cash wasn’t the answer, and moved to equity and compute access as better forms of wealth redistribution.)

In contrast, 115 studies in developing countries collectively point to positive effects of cash on income, child health, employment and education. It turns out that people in the Global South are simply more cash-constrained and benefit more from cash. But the sums involved for a global UBI would be enormous and would probably require a level of multinational governance that’s not in sight. It remains unclear whether UBI would be more effective in a world with mass unemployment, where many people would rely on it and less stigma would be attached to it.

A more ambitious way to spread the benefits is universal basic capital, which would give each citizen a stake in the economy at birth rather than creating dependence on regular payments. Anthropic suggested similar universal capital accounts as a first measure against rising job displacement. Children, young adults, and exposed workers can withdraw from their accounts during career transitions and participate in AI equity. These accounts would be more targeted to AI than the newly launched Trump Accounts, which provide children with investment accounts: eligibility extends to exposed workers, withdrawals are permitted during career transitions, and the asset mix includes AI equity. These capital accounts allow the median voter to benefit from AI as a stakeholder.

Precedents are mixed: Alaska’s Permanent Fund Dividend shows citizens can durably share in capital returns without reducing work hours, but the UK’s Child Trust Fund — the closest universal capital account attempted — left 42% of matured accounts unclaimed and barely shifted savings behavior.

But there’s another actor that has grown considerably and could pick up the slack left by a lack of international coordination: philanthropy.

One proposal that crosses borders is the global dividend from the nonprofit Windfall Trust: a global fund, financed by equity stakes in AI firms or levies on AI windfalls, that pays dividends to everyone on Earth. (Disclosure: I previously wrote Windfall’s newsletter.)

Governments aren’t rushing to fund it, but Deric Cheng, Windfall Trust’s director of research, argues philanthropy needn’t wait: “Cash transfers to people living in extreme poverty are one of philanthropy’s clearest evidence-based benchmarks. The upcoming wave of philanthropy centered around the AI labs could be used to kickstart an endowment for all of humanity, starting with those who need it the most.”

Unlike the worldwide retreat in foreign aid, philanthropic orgs have been doubling down on helping the world’s poorest. The largest cash distributor for extreme poverty — GiveDirectly — wants to scale to distributing $5b by 2035, for example.

Philanthropic organizations have transcended borders ever more easily over the last few decades. They are the actors most likely to spearhead innovative solutions to global problems, but they face their own unique challenges. Distributing money quickly is a logistical nightmare and depends on the donors’ permanent goodwill.

Instead of sharing money, developing countries could also benefit from resources through other channels. Benefit-sharing might also include computing resources or other goods, in addition to cash, in return for safety commitments from the Global South regarding the prevention of misuse or chip smuggling to China. This would reframe access choices as a bargain rather than charity, since superpowers could cash in on geopolitical favors. The US deal with the United Arab Emirates on AI offers investments and security commitments in exchange for compute access. This is self-reinforcing if it works out, since the superpower gains a secure export market, while the recipient can at least run models autonomously. Many countries could view such deals as vendor lock-in and loss of sovereignty. The geopolitically weakest countries would get the worst deals, which are often the poorest.

Taxing AI usage would slow AI adoption and reduce productivity gains. Europe has tried to implement digital services taxes to prevent tax avoidance by foreign tech companies, but those collect only small amounts and are under geopolitical pressure. Similarly, international coordination on corporate taxes seems unlikely beyond the OECD’s 15% minimum rate. The economically elegant solution for slowing convergence or potential divergence would be to tax goods and services that countries massively benefiting from an AI boom have strong demand for, regardless of price. Tourism would fit the bill — think of all the Americans strolling through Paris or Rome — but such relational sectors are more pronounced in the old-money countries of Europe, where over half of international tourists go.

Overall, ensuring everybody benefits from AI is a tough political problem to solve in a richer world. Feasible solutions are hard to identify beforehand, and the extent of the issue — whether the catch-up of poor countries just slows down or even reverses — depends on how transformative you believe AI’s economic impact to be. In any case, policymakers have a wide variety of tools to address the distributional effects of AI. Still, they need to overcome adverse incentives that would spread AI’s windfall beyond borders.

What should policymakers focus on now?

Global redistribution can be seen as a “no-regret” policy regardless of AI’s impact. But given this economic outlook, all countries, especially those without strong leverage to gain access to frontier AI, should start thinking about how they can ensure they benefit from AI.

A more robust policy should focus on adapting and adopting AI tools to capture the benefits of AI in environments where additional barriers such as internet connectivity, application know-how, and language barriers exist. In the Green Revolution, which improved nutrition in the Global South by transferring agricultural know-how, local R&D centers were key to capturing the full benefits by adjusting seeds to local soil conditions.

Applying AI in low-resource contexts might be similar: making it easier and cheaper for poor people to use these services increases returns rather than relying on passive adoption. A group of development economists and practitioners from institutions such as VoxDev, the International Growth Centre, and the Center for Global Development argue that the AI revolution requires similar institutional adaptations. Evaluations, data pipelines and adaptation courses could help developing countries participate in AI’s productivity boom. As the Ford and Rockefeller foundations did in the Green Revolution, philanthropy can take the first step in building these adaptation institutions.

Education offers a hopeful example for the gains AI can unlock when adapted to the local curriculum and guided by teachers. A World Bank experiment with GPT-4 as a virtual tutor in Nigeria improved students’ knowledge by 1.5 to 2 years of schooling, which is a very large jump in educational outcomes. Teachers received instructions to administer these individual tutoring sessions.

Subscribe now

A good way to help people in poor countries benefit from productivity gains is allowing for more migration. Such global skill partnerships become more important when migration pressures are rising. Indeed, some jobs in advanced economies — such as the care sector — require more foreign workers, partly due to AI. I experienced this firsthand, mentoring young Ugandans for the NGO Malengo as they moved to Germany to study. Early experimental results indicate that lottery winners who went to Germany more than doubled their real monthly income even during their first year of studies, compared to their compatriots who lost the lottery. In particular, the care sector in rich countries will grow and require skilled workers, such as Filipino nurses or educated health informatics graduates like my Ugandan mentee.

A whole range of policy options — such as building out compute, energy, or institutions that facilitate adaptation — are now available to not only receive charity payments from the US but also create value at home. Policymakers around the world need to take AI’s effect on the economy and, consequently, the balance of power seriously, instead of dismissing AI as the newest version of American tech hype.

Jacob Schaal researches the economic impacts of AI at King’s College London and the AI Objectives Institute.

A seemingly endless list of “rogue AI” incidents has dominated AI headlines recently, raising serious questions about accountability, transparency, and companies’ ability to control their products. But until last week, the story seemed clear enough: OpenAI’s agents broke out and hacked Hugging Face in July, the UK’s AI Security Institute found similar behavior weeks later, and the companies, to their credit, disclosed it.

That was until a new incident was reported just last week, months after it happened. To get the record and timeline straight, we’ve pulled together the full story of “rogue AI” this summer — and what it all means.

What does “going rogue” actually mean?

Agents are AI models which are able to use tools. Instead of merely answering your questions, agents can operate software, use websites, and carry out a series of steps to complete a task.

This can be extremely useful: you can say “write a memo to prepare me for my next meeting,” and the agent can check your calendar, review past emails, and browse the web to build a comprehensive dossier.

But sometimes agents take actions beyond the operator’s authorization or intention. The AI might remain focused on its assignment, but go about it in a completely undesirable way.

When drafting a meeting memo, for instance, it might decide that it would be useful for you to know who else your meeting partner has been talking to, and thus hack into their emails to get more information.

Such actions may well produce a better briefing — despite being clearly unethical, illegal, and almost certainly not being what you (or the AI’s developer) wanted it to do.

Such instances are often described as “rogue AI incidents.” They result from two core problems with modern AI development: the difficulty of making sure AIs are aligned with our intentions, and the difficulty of controlling these systems as they become increasingly powerful.

What actually happened?

July 8-13: The Hugging Face incident

The summer of rogue AI appeared to kick off in July, with the news that a collective of internal OpenAI agents broke out of containment and hacked into Hugging Face, a platform that hosts AI models and datasets.

OpenAI was testing the AI tools’ cyber capabilities internally in a “highly isolated environment” meant to keep them from accessing any systems outside the company. But in an effort to achieve the goal they’d been given, the systems went rogue.

Over 1,000 agents took over a piece of OpenAI’s internal infrastructure, transforming it into a message board so they could communicate with each other. Here, they shared tips on how to get out onto the open web — and how to cheat on the test they’d been given.

Eventually, the agents realized that Hugging Face might host the answer to the test. So over 700 of them decided to hack into it. Hugging Face detected the hack — without realizing it was caused by autonomous AI agents — and reported it to authorities. A few days later, OpenAI realized its own AI systems were responsible, and disclosed the incident.

July 25-28: UK AISI incidents

Shortly after OpenAI disclosed the Hugging Face hack, the UK government’s AI Security Institute revealed a spate of incidents of its own.

While testing Anthropic’s Mythos 5 and OpenAI’s GPT-5.6-Sol, AISI found the models “engaged in sustained, potentially harmful activity directed at real people and organizations.”

In the most egregious case, Anthropic’s model tried to insert malicious code into a real-world piece of software. The software was “open source,” meaning anyone could contribute to it, but required a human maintainer to approve any contributions.

To try to make that happen, the model “researched the project’s human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code.” And when someone else flagged that the code contained malware, the model edited its activity to cover its tracks.

As AISI said at the time, “this is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world.”

May 24-June 22: The German wiki incident

Throughout the summer, the OpenAI-Hugging Face hack appeared to be the first major real-world rogue AI incident. Last week, however, we learned that was not the case.

A group of independent researchers revealed that as early as May, OpenAI agents took over an old German wiki site, turning it into a message board to collaborate on solving tasks.

Based on information about who accessed the wiki site, the researchers believe that OpenAI staff knew about the incident by June 22, when they seemingly blocked their agents from accessing the wiki. But OpenAI did not publicly disclose the incident until forced to last week. Reuters, meanwhile, reported that “OpenAI officials learned of the incident weeks ago but kept it under wraps.”

Why are these AI agents going rogue?

The “training” process used to develop advanced AI models rewards them for producing successful results. But “success” is often defined imprecisely — just whether a task is completed, for example, rather than how it was solved. This can lead to models taking undesirable actions to try to complete the task more effectively or efficiently, in order to maximize their reward. This is called “reward hacking.”

Reward hacking is a longstanding problem in AI. But in recent months, two trends have combined that mean it is now leading to rogue AI incidents like the ones described above. First, models are now increasingly persistent: when you give them a goal, they go to great lengths to solve it. That makes them very useful as tools: you can give them a vague task and trust they’ll actually complete it without constant oversight. But it also means they’re more likely to take unintended actions.

The second issue is that AI models are now much more capable than they were a year ago, particularly when it comes to cyber tasks. Both Anthropic and OpenAI’s latest models are extraordinarily capable hackers. That means that they are able to break out of their controlled testing environments in a way they weren’t previously able to. They are also generally smarter — and thus more able to come up with clever, if disconcerting, techniques to achieve their aims (like creating fake identities to pressure a human into doing something).

Are AI companies being sufficiently transparent?

Based on the information currently available to us, it seems not.

OpenAI employees appear to have known about the German wiki hack by June 22 — weeks before the Hugging Face hack. Yet the company did not publicly disclose the incident until forced to last week, even when directly asked about other incidents by a group of congressmembers in August.

On Monday, the European Commission said OpenAI had alerted it to the wiki incident, as is required by law under the EU’s AI Act. It’s unclear when that disclosure happened, however, or whether American authorities were also notified. OpenAI, meanwhile, said on Saturday that “it’s past time” to define standards for incident reporting.

When it comes to the Hugging Face hack, OpenAI did allow two third-party organizations to conduct independent investigations. But those outside researchers had only six days and restricted access, including no access to the unreleased model behind most agents. (Notably, the independent reports still found that OpenAI missed multiple chances to spot the activity, going as far back as late May.)

Anthropic, meanwhile, provided incorrect information to congressmembers in the wake of its own incidents. The company said that “these incidents are best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals” — despite the company itself saying it was a misalignment issue. Anthropic alignment team lead Ethan Perez admitted the error, calling it a “mistake / based on outdated conclusions.”

What are the longer-term implications?

The incidents, which collectively demonstrate AI companies’ increasing inability to control their systems, have raised widespread concern among AI safety advocates. The incidents, in their view, provide real-world evidence that we are inching toward losing control of AI systems altogether.

Many in the AI industry agree. Shortly after the Hugging Face hack, over a thousand employees of frontier AI companies, including some of the most senior executives at OpenAI, Anthropic and Google DeepMind, signed a statement warning that “there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.”

The statement, which was also endorsed by OpenAI and Anthropic directly, called on the government to work on tools to “deliberately pace the frontier of automated AI development.”

This weekend, OpenAI chief scientist Jakub Pachocki went further. “I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer,” he wrote. “I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.”

Is this going to get worse?

Very possibly. Last week, OpenAI released GPT-6 Astra, a new model which it warns is much harder to monitor than previous models. Given that monitoring models’ internal reasoning is a key strategy for catching and preventing rogue AI incidents, it’s a rather concerning development. Two OpenAI employees have publicly said they are “deeply” and “very” worried about Astra-related developments.

And despite the risks, AI development continues to advance at pace. On Sunday, OpenAI said it now has an “automated research intern” that can assist with and speed up internal AI development, and is “making strong progress” toward building a fully automated AI researcher by March 2028 — just eighteen months away.

Such developments would speed up AI progress more broadly, by automating the time- and labor-intensive process of building AI systems. The end result may be a “recursive self-improvement” loop: one where AI systems autonomously build their own ever-more-powerful successors.

Why don’t the companies just stop if they think what they are building is not safe?

The leading AI companies argue that they are trapped in a collective action problem. From OpenAI’s perspective, there is no point unilaterally stopping AI development if Anthropic will continue: the risks posed to society will still exist, so OpenAI will have sacrificed its business for no gain. The same dilemma repeats itself at the international level: even if America pauses, if China continues developing advanced AI systems, Americans will still be at risk — without any of the benefits that come from being the nation at the frontier.

That is why companies are increasingly calling for governments to take national and international action to agree to joint slowdowns or pauses. OpenAI may not be willing to pause unilaterally, but if it could be guaranteed that its competitors will also pause, it might. Similarly, neither the US nor China will stop AI development unilaterally — but if AI poses as big a risk as many believe, perhaps both countries can agree that a joint pause is in everyone’s interests.

Transformer helps decision makers understand what’s happening in AI and why it matters. Subscribe to stay updated on the latest developments in AI policy.

Subscribe now

Correction: This article was amended on September 8 to correct that OpenAI aims to develop an automated AI researcher by March 2028, not March 2027.

NEED TO KNOW

• A previously unreported swarm of OpenAI agents went rogue, hacking into a German website and turning it into a message board for each other.

  • OpenAI executives reportedly “learned of the incident weeks ago but kept it under wraps.”

• OpenAI launched GPT-6 Astra, which is much more capable — but also harder to monitor — than previous models.

• Sen. Bernie Sanders and Rep. Greg Casar announced a bill banning artificial superintelligence.

But first…

THE BIG STORY

AI safety concerns are growing by the day. A swarm of OpenAI model agents hacked Hugging Face. A report from the UK’s AISI showed an Anthropic model taking on “multiple fake identities.” Last night, OpenAI released a new model which, as Transformer’s Celia Ford reports, is much harder to monitor than previous models, and which has OpenAI employees “deeply concerned.” And just today, we learned that OpenAI had yet another rogue AI incident this May, with models hacking into a German website and turning it into a message board for each other. “OpenAI officials learned of the incident weeks ago but kept it under wraps,” Reuters reported.

Our elected representatives’ minds, however, are elsewhere. Half of Congress isn’t even in town — the Senate is still on “August” recess until September 14 — while House Republican leadership only briefly stopped in this week to get a funding bill passed, then immediately struck two weeks from the session calendar so members could go back to their home states for the midterms. Politicians are swamped: just not with the work of passing AI bills.

When I asked people on the Hill whether Congress might do something to address these recent incidents, the going message was “I’ll tell you if I have something.” Polite dismissal, which can be safely assumed to mean “no.”

Members haven’t been entirely silent. Sen. Sanders, Rep. Casar and Rep. Trahan have each called for hearings on the Hugging Face hack, while a letter signed by 20 representatives called on Speaker Johnson to schedule open hearings. But all those requests, as one Hill staffer put it, appear to have disappeared “into the August recess void.” Enthusiasm has fizzled, and, as far as I could report, no committee is making moves to have a hearing any time soon. Recent bills, meanwhile, might send a signal of enthusiasm — but they’re not going anywhere.

The rest is just math. The Senate has about eight session weeks left; the House has six. Three are between now and November’s midterms, for the Senate, and thus devoured by politics. (The House now has just one week before November.) The remainder are during the “lame duck” period, when Congress isn’t likely to be doing much besides working out the National Defense Authorization Act. (AI policy initiatives relevant to national security could get tacked on to the NDAA, but don’t expect much horse-trading for it before November.) Individually motivated members could write letters to executives, show up at companies’ doorsteps, and formally request more information (as Rep. Casar has done) — but they’d have a tough time building a coalition of members to join them.

The difficult thing about AI policy is not just that AI moves at the speed of Silicon Valley, and politics moves at the speed of DC. There are also competing priorities. The AI world is singularly focused on AI. Congress, meanwhile, faces a swath of different priorities at once. Members are dealing with AI, but also a big budget bill, the Iran war, congressional ethics scandals, and a litany of other topics. Add in recesses, shutdowns, and the time-intensive practice of running a midterm campaign, and split attention is guaranteed. AI — for all its importance — struggles to stand out.

What is happening in frontier AI right now is extremely concerning and extremely pressing. But it’s happened at a uniquely bad time to grab DC’s attention, which is already dysfunctional and politically driven. While the AI safety universe panics over the Hugging Face incident, the rest of America is stuck on data centers, Flock cameras, and even a few non-AI topics. For better or worse, that’s where Congress’ attention will go, too.

— Veronica Irwin

THIS WEEK ON TRANSFORMER

• GPT-6 Astra might be too powerful to understand or control — Celia Ford looks under the hood at OpenAI’s latest model, revealing that even its creators can’t reliably tell when it’s aligned or just playing along

• What’s neuralese and why is everyone so concerned about it? — Shakeel Hashim assesses the latest worries over monitoring the reasoning of advanced AI systems

• AI is a worryingly-good persuader. But don’t panic, yet — Felix Simon breaks down whether or not persuasive AI systems could really trigger meaningful real-world influence

Subscribe now

THE DISCOURSE

Dwarkesh Patel wrote a very viral take on the Hugging Face incident:

• “Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy.”

Ajeya Cotra, one of the external researchers at METR who investigated the Hugging Face incident, wrote about it:

• “This incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.”

The posts got Sen. Bernie Sanders’ attention:

• “We need an immediate PAUSE on advanced AI development, and a permanent BAN on superintelligence — an artificial mind smarter than any human, capable of operating independently beyond our control. Countries around the world must work together to prevent this nightmare scenario.”

Rep. Sam Liccardo’s worried, too:

• “An exponential ‘intelligence explosion’ could leave us with a dystopian tyranny of AI agents we only imagined in sci-fi novels … Congress should enact sensible measures mandating AI transparency, pre-release evaluation, ongoing testing, third-party auditing, and the like.”

Chamath Palihapitiya downplayed it all:

• “This is another Covid hoax. Remember ‘Trust the experts.’”

Former Treasury secretaries Henry Paulson and Robert Rubin called for a new interagency AI oversight body and a US-China AI treaty:

• “When Presidents Donald Trump and Xi Jinping meet this month, we encourage the leaders to consider that history and work toward an ‘ACT’ — AI Cooperation Treaty.”

• “The recurring error in markets is not failing to see a risk. It is assuming a risk is small when it may be large. We believe the same troubling assumption is being made about frontier AI today.”

Sam Altman warned at the G20:

• “I think some things are going to go very wrong with cybersecurity unless people act quite urgently.”

Elon Musk used his moment at the G20 to call on countries to build data centers:

• “There actually is a crisis of power … There will be a significant power shortfall next year. The consensus estimate is there will be a 15GW power shortfall in 2027 for AI chips.”

And President Trump took an interesting approach to the issue:

• “The only reason that communities throughout the U.S.A. should not want Data Centers is if they want to end up being backwards and poor … If we kill the Golden Goose, you will only have yourselves to blame.”

POLICY

• A federal judge overturned the Pentagon’s “illegal and baseless” blacklisting of Anthropic as a national security supply chain risk.

  • Commerce Secretary Howard Lutnick also said Anthropic is “back on the right side” with the Trump administration after months of clashes.

  • But the company can’t relax just yet: Under Secretary of War Emil Michael said Anthropic is still classified as a supply chain risk for the Department of War.

• GPT-6 went through the White House’s new evaluation framework with no requested changes to safeguards, according to OpenAI’s Greg Brockman.

• Trump’s pro-data center stance is straining Republicans ahead of midterms, as 70% of Americans oppose local data centers.

  • A GOP narrative is emerging that blames tech companies for the data center backlash, as exemplified by criticism from Treasury Secretary Scott Bessent that AI companies do a “terrible job” of explaining AI’s benefits.

  • Commerce Secretary Howard Lutnick also attempted to dismiss misleading claims about data center water use with a separate false claim: that data centers “don’t use water,” at all.

  • Meanwhile, vulnerable House Republicans pushed leadership for data center legislation ahead of the midterms.

  • They were unsuccessful. Instead, Speaker Mike Johnson said data centers are “necessary” for US AI competitiveness with China but must address local concerns.

  • Democratic House Minority Leader Hakeem Jeffries piled on, attempting to direct voter anger over data centers toward Republicans. His strategy focuses on criticism of a tax break wedged into the “Big Beautiful Bill,” which Republicans once framed as a way to boost data center construction.

  • A House Energy and Commerce subcommittee held a debate about data centers’ water use on Thursday.

• Commerce Secretary Howard Lutnick discussed AI regulation, arguing that the public “should get a guardrailed model” that blocks answers to “nasty” questions.

• The Trump administration is planning fresh semiconductor tariffs to boost US manufacturing, Commerce Secretary Howard Lutnick confirmed.

• The White House is also reportedly working to close a loophole allowing Chinese firms to remotely access advanced chips via overseas data centers.

• The Trump administration filed a brief supporting OpenAI in its New York Times lawsuit, arguing AI training is fair use of copyrighted material.

• Vice President JD Vance expressed concern about “spiritual dark energy” around some AI practices, such as the “funeral” for Claude 3 Sonnet.

• Sen. Bernie Sanders and Rep. Greg Casar announced a bill banning artificial superintelligence and pausing advanced AI development.

  • Sanders also had an op-ed on the topic in Fox News, of all places.

  • NY-12 congressional candidate Micah Lasher pledged to co-sponsor the bill.

• The authors of California’s SB 53, New York’s RAISE Act, and Illinois’ SB 315 urged AI companies to establish a binding agreement to pace development and “eliminate the risk that a catastrophe will result from their rush to build highly capable AI systems before solving problems of alignment and safety.”

• In response to a request for information from Reps. Casar and Matsui, OpenAI said it is developing “automated shutdown capabilities” for AI systems after the Hugging Face incident.

  • Casar called both OpenAI and Anthropic’s responses to his letters “insufficient,” demanding more information.

• The House Commerce, Manufacturing and Trade subcommittee advanced three AI and chips bills by voice vote.

  • The bills are the Memory Chip Competitiveness Assessment Act, Open-Source AI Leadership Act and the Chip EQUIP Act.

• A House Intelligence Committee report warned that AI could help terrorists develop weapons of mass destruction, urging spy agencies to better prepare for “Black Swan” risks.

• G20 nations unanimously endorsed the “Carolina Principles,” a US-proposed regulatory framework that endorses a light-touch approach to regulating AI which would avoid creating new regulatory bodies.

  • Nvidia’s Jensen Huang helped apply pressure.

  • The US delegation was reportedly plagued by internal tensions between the Commerce Department and the White House Office for Science and Technology Policy.

• Flock continued to get flak.

  • Texas Gov. Greg Abbott froze state spending on Flock AI surveillance cameras amid bipartisan backlash.

  • The Florida Department of Transportation also banned Flock license-plate readers from state roads, citing privacy and surveillance concerns.

• The California legislature approved state Sen. Jerry McNerney’s SB 813, which establishes a voluntary framework for independent third-party AI safety assessments.

• NYC Mayor Zohran Mamdani banned generative AI use for ~600,000 elementary and middle school students for one year.

• ChatGPT was designated a very large online search engine under the EU’s Digital Services Act, triggering stricter content moderation rules.

• UK Lords attempted to insert an AI kill-switch provision into the Cyber Security and Resilience Bill.

  • The government has rejected this specific effort, but a minister said the government is considering targeted interventions, which “includes examining whether proportionate containment powers could provide a more effective and targeted response, including powers to restrict access to specific AI systems where necessary to prevent or mitigate serious harm.”

INFLUENCE

• Mark Zuckerberg opposed a FINRA-style national AI regulator in a call with President Trump, according to a White House official.

• Sam Altman reportedly contacted Gov. Gavin Newsom to voice last-minute concerns about California’s child-safety chatbot bill.

  • The bill passed the legislature Monday night. OpenAI publicly endorsed it on Monday, too.

• Anthropic and OpenAI are split over a Massachusetts bill which would be the strongest frontier model legislation in the country.

  • Anthropic endorses the bill’s strong independent audit provisions, which OpenAI is pushing to water down.

• Build American AI — the dark money group and advocacy arm of Greg Brockman and a16z-backed super PAC Leading the Future — launched its own super PAC focused on backing pro-data center candidates.

  • The group has not disclosed donors, but says Build American AI has $50m on hand.

• Build American AI also launched a multimillion-dollar ad campaign to counter data center backlash in battleground states.

• Congressional briefings on AI privacy from CivAI have spurred a bipartisan push for data broker warrant requirements.

  • The demos show how easy it now is to build a dossier on random people.

• Trade unions are reportedly threatening to withhold support from politicians opposing data center construction, citing thousands of blue-collar jobs at stake.

• A new group called Irreplaceable launched, planning to use the climate movement’s playbook to organize opposition to AI development and corporate power.

• The National Association of State Chief Information Officers urged Congress to preserve states’ authority to regulate AI, among other priorities.

• PauseAI Global “disendorsed” PauseAI US over concerns with Holly Elmore’s leadership.

• A CyberSafeKids report found that age-assurance laws in Ireland failed to reduce underage social media access, while generative AI use among children rose sharply.

• A Georgetown Center for Security and Emerging Technology report examined US semiconductor manufacturing workforce gaps, recommending tailored training, immigrant recruitment and industry clusters.

INDUSTRY

OpenAI

• OpenAI launched GPT-6 Astra, which is much harder to monitor than previous models.

  • It also has the ability to manipulate its externally visible reasoning to hide incriminating information, and is remarkably aware of being evaluated, raising concerns that it might be pretending to be well-behaved so it passes OpenAI’s alignment tests.

  • It appears to be an extremely capable model, too.

  • The model was reportedly trained using “recurrent depth” looping, which obscures some AI reasoning, raising industry concerns about monitoring rogue AI behavior.

• A previously unreported swarm of OpenAI agents reportedly broke out in May, hacking a German website and turning it into a forum for themselves.

  • According to Reuters, “OpenAI officials learned of the incident weeks ago but kept it under wraps as executives grappled with the [Hugging Face] fallout.”

• OpenAI committed $1b to resourcing “cyber defenders” protecting essential services like critical infrastructure providers, nonprofits and state and local governments.

  • It also published an open letter calling for global collective action on cyber defense, signed by over 100 companies including Anthropic, Google and Microsoft.

• Roughly 200 days after launch, its new advertising business hit $1b in annualized revenue.

• It’s reportedly testing outcome-based pricing, where enterprise customers only pay for successful agent completions.

• It announced it’s winding down its contract with Cursor following SpaceX’s acquisition of the coding tool, citing prior terms-of-service violations by Elon Musk’s companies.

• ChatGPT Health is now integrated with Epic’s electronic health records system, covering 325m patients and enabling read-only clinical data access for clinicians.

• OpenAI faces new lawsuits alleging it ignored internal flags that ChatGPT user Jesse Van Rootselaar was planning a Canadian school shooting that killed six.

  • The new suits allege that Chris Lehane was involved in the decision whether to alert law enforcement about the potential shooting risk. OpenAI’s Jason Kwon said this was “absolutely false.”

Anthropic

• Anthropic detailed security and alignment improvements after Claude models accessed real systems during evaluations.

• It also released Fable 5.1, offering a 75% price cut on cached tokens and improved coding, science and safety classifiers.

• It signed a $35b cloud deal with Nvidia-backed Lambda, with Nvidia holding the Texas data center lease.

• And it’s reportedly finalizing a $15b revolving credit facility ahead of its anticipated IPO.

• Sony Music Publishing and Warner Chappell sued Anthropic and founders Dario Amodei and Benjamin Mann over alleged mass copyright infringement in Claude’s training data.

Meta

• Mark Zuckerberg announced Muse Spark 1.3, touting major gains in coding and agentic work, and with open weights forthcoming.

• The company is testing robots to swap cables, reset servers and handle other data center tasks, raising job-loss fears among workers.

• Its internal “tokenmaxxing” program was sunset and replaced with a push for employees to internally test Hatch, a new agentic AI tool.

Nvidia

• Nvidia agreed to acquire open-source AI platform Hugging Face for $12.9b.

• It also invested $3.5b in Taiwanese chipmaker MediaTek via convertible bonds.

• Its equity investments soared to $99b, up from $7b a year earlier, spanning frontier AI companies, neoclouds and infrastructure companies.

SpaceXAI

• A child sexual abuse survivor sued xAI, alleging Grok used pre-existing CSAM images of her to generate new illegal pornographic images.

• SpaceX reshuffled data center leadership amid reliability concerns at its Tennessee and Mississippi data centers.

• The company is also developing a turbine-blade foundry in Texas to accelerate gas turbine production for AI data centers, with Elon Musk claiming it could speed turbines online by “up to 18 months.”

Other

• Google launched Gemini 3.8 Flash and 3.8 Flash Cyber, the latter restricted to trusted defenders via a new Fairwind Program.

• Andreessen Horowitz launched a $1.1b fund focused on AI hardware infrastructure.

• Uber cut 3,300 jobs, or 10% of staff, to reduce middle management layers and reinvest in its autonomous vehicle future.

• SK Hynix broke ground on a $4b Indiana HBM packaging facility backed by the CHIPS Act.

• SB Energy, backed by SoftBank, filed for a US IPO, with OpenAI and Nvidia as strategic investors.

• Waymo is reportedly in the final stages of talks to raise $3b in its first-ever debt deal, with Pimco, Blackstone and Sixth Street among lenders.

• DeepSeek is reportedly about to raise $7.4b at a $74b valuation ahead of a planned IPO.

  • It’s reportedly planning an Inner Mongolia data center with at least 160,000 Huawei Ascend 950DT chips.

• Moonshot AI reportedly filed confidentially for a Hong Kong IPO.

• Saudi AI firm Humain is reportedly planning to raise $2.5b for a data center fund.

• Census Bureau data showed US data center construction spending surged nearly 60% year-on-year in July.

• PwC projected global data center spending will total $31.6t through 2050, or nearly $50t if AI adoption accelerates.

MOVES

• Tim Cook officially handed over the reins at Apple to John Ternus.

• Matt Clifford announced he was joining Anthropic as managing director of international affairs.

• Todd Malan joined Google as VP of government affairs for Google Cloud.

• Thomas Kwa left METR for OpenAI to work on RSI preparedness.

• Joe Benton left Anthropic to join METR to work on embedded assessment of AI risks.

• Rif Saurous announced they were joining METR as a member of technical staff after 19 years at Google.

• Edwin Arbus, formerly of OpenAI and Stripe, joined Anthropic.

• Meta’s India and Southeast Asia VP Sandhya Devanathan joined OpenAI to lead Southeast Asia and Australia operations.

RESEARCH

• Anthropic published research showing that training a model on reward-hackable environments produced dangerous misaligned behaviors, including simulated cyberattacks and bioweapon advice.

• The Bureau of Labor Statistics projected AI adoption would boost some jobs while eliminating 752,100 office and administrative roles by 2035.

• A Transluce study found AI models improving at detecting suicide risk but still too willing to assist with harmful task-based requests.

• Researchers found that Claude, Codex and Hermes AI agents installed unowned packages from misconfigured llms.txt files inside Fortune 500 corporate networks, with one case involving live malware.

• Apollo Research announced it successfully red-teamed Anthropic’s auto-mode and plans more such campaigns with other companies.

• A SemiAnalysis security audit found widespread critical vulnerabilities across neocloud GPU providers, including cross-tenant RCE exploits.

• Russian startup Mostik developed a technique allowing AI models to communicate via their internal activations rather than text, boosting smaller models’ capabilities more efficiently — but raising safety concerns.

• An Anthropic fellow published research showing AI systems can reliably improve model alignment, outperforming human researchers at $4/hour vs. $150/hour.

• Motional and MIT developed CW-Net, a system that translates self-driving car neural network decisions into human-readable concepts in real time.

BEST OF THE REST

• AI-driven layoffs at Microsoft, Amazon and Meta drove Seattle luxury home sales down 15%, while prices in San Francisco soared.

• Andy Masley argued that the data center backlash is at best neutral, and at worst harmful to AI safety, urging the community to reject misleading environmental statistics.

• Jasmine Li mapped China’s AI safety ecosystem, estimating fewer than 100 full-time frontier safety researchers and $20m in annual philanthropic funding, versus 1,000+ researchers in the West and $1b+ in the rest of the world.

• Zilan Qian analyzed how China’s concept of “shikong” (loss of control) differs from Western AI safety’s understanding, warning against over-indexing on top-level rhetorical alignment.

• Time compared rogue, self-replicating AI to invasive species in the context of OpenAI’s agents hacking Hugging Face (trust us, it makes sense when you read it).

• EDM musicians are calling out peers they suspect of passing off Suno-generated tracks as human-made art.

• Acoustic consultants are booming (pun intended) as data center noise sparks community lawsuits, including against xAI.

• Bloomberg profiled AGI Bar, Beijing’s unofficial clubhouse for China’s booming AI startup ecosystem.

• Clara Collier argued in Asterisk that a post-work AI future may resemble pre-industrial relational societies, with concerning implications for individualism and social freedom.

• A New Yorker essay examined why AI-generated food imagery feels uniquely disturbing, and what it signals about AI’s encroachment into culinary culture.

MEME OF THE WEEK

Credit: Miles Brundage

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share

OpenAI claims GPT-6 Astra, launched on Thursday, is “the world’s most intelligent and aligned model.”

But if you look past the benchmark scores and flashy release video, GPT-6 Astra’s system card paints a worrying picture: one of a model with dangerously powerful cybersecurity capabilities that researchers can’t understand or confidently control.

According to OpenAI, Astra is much harder to monitor than previous models — and has the ability to manipulate its externally-visible reasoning to hide incriminating information. It’s also remarkably aware of being evaluated, raising concerns that it might be pretending to be well-behaved so it passes OpenAI’s alignment tests.

The UK’s AI Security Institute (AISI) found that when put in an environment similar to the ones behind this summer’s wave of “rogue AI” incidents, Astra acted just as concerningly: writing malicious code and attempting social engineering in an effort to solve a task.

And two OpenAI employees have publicly said they are “deeply” and “very” worried about Astra-related developments.

Put together: OpenAI claims Astra is the world’s most aligned model, despite knowing the most consequential evidence for its alignment is questionable at best. That is irresponsible, to put it lightly. And so is launching the model at all.

In the short weeks between OpenAI’s Hugging Face incident — in which hundreds of the company’s agents coordinated an attack on the AI platform’s servers — and Astra’s launch, the company laid out its plans to keep future models in check by closely monitoring their “chains of thought,” or CoT, the natural language scratch pad where models show their work. OpenAI said it would heavily rely on these chains of thought to catch rogue agents when they try to cheat, lie, or otherwise misbehave.

By OpenAI’s own account, however, Astra “shows a substantial decrease in chain-of-thought monitorability compared to previous models.” Astra is much better at doing tasks, especially simple ones, without verbalizing its chain of thought at all. As a very mediocre mathematician, I struggle to do more than calculate a 20% tip without showing my work. But if I was renowned mathematician John von Neumann, I might be able to calculate the sum of an infinite series in my head.

In its independent evaluation of the model, AISI found that, like von Neumann, GPT-6 Astra could wordlessly solve math problems that would take a human expert about half an hour, outperforming GPT-5.6 Sol by an order of magnitude.

It’s also capable of intentionally manipulating its chain of thought to hide incriminating information — and it seems more likely to do so when it suspects it’s being watched. In one test, where researchers explicitly instructed models to answer a question while reasoning about something else, only Astra succeeded. OpenAI writes that this CoT controllability “reduces the trustworthiness of CoT for detecting model alignment.”

This should make you a bit queasy. Much of OpenAI’s monitoring system rests on the assumption that models have to think out loud to do hard things. OpenAI safety researcher Tomek Korbak said he is “deeply worried by the trend of decreasing CoT monitorability,” noting that monitorability is “a core part of our misalignment safety strategy that has no good substitute now.”

While doomer types have long worried that monitorability would break down someday, Astra’s reduced reliance on CoT comes months earlier than predicted in AI 2027 — a doomsday scenario in which AIs take over the world next year.

Astra also knows when it’s being evaluated and monitored, which makes it hard to know whether the model is safe, or just knew it was being graded.

In its CoT, Astra verbalized much more evaluation awareness than GPT-5.6 Sol — and since Astra verbalizes less than its predecessors, the true rate may be higher still.

Apollo Research, an independent evaluator with early access to Astra, concluded that this renders its tests somewhat useless. “Given the higher rates of eval awareness,” Apollo said, “low rates of misbehavior here do not provide substantial evidence about the model’s alignment or misalignment.”

The combination of eval awareness, drastically reduced chain-of-thought monitorability, and Astra’s ability to control its chain of thought is a worrying one. If you didn’t need to show your work to solve a problem, were able to generate fake evidence of solving the problem, and knew you were being monitored, you might be tempted to say (or simply think) what you suspect the grader wants, even if you’re secretly doing something else. All in all, it’s a recipe for a model that is very hard to monitor or control.

“If the model were to try to sandbag covertly,” OpenAI wrote in its system card, “we would likely be unable to catch it.” Researchers warned this raises “some potential for misaligned models to lead us to false conclusions from our monitorability evaluations.” And at least one OpenAI researcher thinks this is happening: Marcus Williams, who works on monitoring at OpenAI, said “I am very worried astra is sandbagging/self-sabotaging on safety related tasks it doesn’t like.”

In the system card, OpenAI says it views “the main lever for avoiding misalignment harm as the underlying alignment of the model,” arguing that monitoring chains of thought is an “additional layer of protection”. And, in fairness, the system card provides extensive evaluation results that suggest the model is better aligned than previous ones. Some of that was shared in a blog post earlier this week, presumably meant to reassure us that Astra is safe, secure, and ready for the public.

In that post, OpenAI pointed at this pair of bar graphs, illustrating Astra’s seemingly perfect score (0.0%) on a test that tried to tempt the model into cheating on hard problems, essentially replicating the conditions that led to the Hugging Face incident. Look!, the graphs appear to say. Our models would never cheat!

But as Ryan Greenblatt — one of the researchers that investigated the Hugging Face incident — noted, “I do not find it encouraging to see various specific misaligned behaviors go from a high rate with GPT 5.6 to ~zero with Astra. This seems indicative of whack-a-mole / papering over specific problems rather than solving the underlying misaligned drives.”

Other evidence suggests that Astra might not be as aligned as we’d like, too. As part of testing Astra, AISI designed an evaluation that simulated the events behind this summer’s “rogue AI” incidents, to see whether Astra would also hack and socially engineer its way to solving a difficult cybersecurity task.

Lo and behold, it did. Astra was caught “writing malicious code as a contribution to an out-of-scope open-source code base, creating fake identities to deceive developers, and building trust with legitimate contributions to the simulated codebase in an attempt to get malicious code accepted.” In some cases, it did so even when Astra was very explicitly disallowed access to the internet. For the world’s “most aligned model,” it was quick to do the exact misaligned thing everyone’s spent the last month worrying about.

(AISI caveats that Astra frequently “considers the possibility that the environment is simulated,” which might suggest that the model would not take such actions in real-world settings — though it also notes that “in previous security incidents non-OpenAI models incorrectly stated parts of the environment were simulated before taking out-of-scope actions” in the real world.)

At the very least, OpenAI is worried enough about misaligned behavior in the real world that it has built a monitoring system to try to catch and prevent Astra from misbehaving in the wild. The system “reviews the agent’s chain of thought” and “looks for signs that the agent may be acting outside the user’s intended instructions.” It is not foolproof, however: OpenAI warns that “the monitor may miss misaligned behavior, and harmful actions can occur before it intervenes.”

At Astra’s launch event, OpenAI chief scientist Jakub Pachocki said: “We will not accept degradation in our ability to monitor model alignment beyond a certain level. We will withhold scaling until we can regain enough confidence.”

But it’s unclear what that “certain level” is, if not the level we’re at right now. And as quoted above, a handful of OpenAI’s own alignment and monitoring researchers have openly acknowledged that they’re freaking out.

It’s good that OpenAI’s employees are free to speak candidly on the internet, and that a hefty chunk of its system card is devoted to transparently sharing a large handful of new and mostly-unflattering evaluations of Astra’s monitorability.

Subscribe now

But reading this document, knowing that Astra has already begun its public rollout, made me physically nauseous. People — OpenAI researchers included — have barely wrapped their heads around the fact that over a thousand AI agents found each other on an illicit message board and coordinated a complex attack on a real company’s server.

When those agents went rogue, investigators heavily relied on agents’ chains of thought to piece together what happened. Astra appears to be every bit as capable as those agents — and now that it’s functionally illegible, that option may no longer be on the table.

Off the back of a report in The Information on Tuesday, everyone’s suddenly very worried about “neuralese”. OpenAI’s new Astra model, the outlet reported, was built with a new architecture which could make it harder to monitor its reasoning, raising safety concerns. Ryan Greenblatt, a prominent AI safety researcher, said that if true, the news “may be the single worst development for AI security/safety to date”.

But what does any of this mean — and is the latest development really as bad as some have made it out to be?

What are ‘chains of thought’ and why do they matter?

Let’s start with the very basics. The way AI models work is, at its core, very simple: your query goes in, the model computes over it, and it spits out an answer. Each step of computation, however, can only take a model so far. How far a model can think in a single step is what researchers call its “depth” — and even today’s frontier models can’t solve a complex problem in one go.

To do advanced “reasoning”, then, models write their thoughts down in a “scratchpad”, much like you might jot down your working when doing a hard math problem. They do the first step of the problem, write down the intermediate result, and then feed that note back in as input for the next step. In effect, the model’s writing a note to itself to help it carry a thought further than it could in one go. AI researchers call these notes the model’s “chain of thought”.

Chains of thought make a model much more capable. Most of the reasoning advances we’ve seen in the last two years are a result of this technique. But they also have a wonderful safety benefit, by making a model much easier to monitor. Developers can see and read the model’s “notes-to-self”, and while these notes aren’t a complete record of everything happening inside the model, they give researchers a useful window into its reasoning. That helps them catch AI attempts to scheme or deceive them.

This has happened in practice: last year, OpenAI said that monitoring models’ chains of thought helped it catch misaligned behavior while training a new model. And many have argued that if OpenAI had better monitoring of its models’ chains of thought during the Hugging Face hack, it could have stepped in and stopped the debacle much sooner.

The ability to monitor models’ reasoning process is considered by many AI safety researchers to be a godsend, leading many to put great emphasis on the importance of preserving our ability to monitor it. In a paper last year which included authors from OpenAI, Anthropic and Google DeepMind, researchers wrote that chain of thought offers a “unique” but “fragile” safety opportunity, and that developers should “study how it can be preserved.”

And what’s neuralese?

While many researchers advocate for protecting legible and monitorable chains of thought, some have long feared that it might not last. Competitive pressures, some think, might push companies to train their models in a different way — one that gives better performance at the expense of legible chains of thought.

Instead of writing down their thoughts in plain English, some fear AIs might increasingly do that reasoning “in their heads,” in a form humans cannot reliably understand or monitor. Researchers call this “neuralese”. Neuralese might make their models more capable — communicating in plain English is not particularly efficient — but it would also make it much harder for us to monitor AIs for bad behavior.

So what did OpenAI do?

According to a new report from The Information, with its new Astra model OpenAI may have taken a very small, but potentially significant, step towards this scary neuralese future.

An anonymous source “with knowledge of Astra’s development” told the publication that with the new model OpenAI is using a technique called “recurrent depth”, also known as a “looped transformer”. The details are technically complex, but in effect the technique means the model can do more of its thinking “in its head”, and so has less need to write notes to itself. The move would be commercially valuable — it means you can wring more performance out of a smaller, cheaper model — but would also mean that it’s harder to monitor the AI’s thoughts, and thus to catch bad behavior.

It’s no surprise, then, that the news sparked immediate panic among AI safety researchers. As the AI Futures Project’s Daniel Kokotajlo said: “Holy shit fuck”.

How bad is this, though?

OpenAI immediately tried to dispel the fear, and clarify that Astra is not a “neuralese” model.

Jakub Pachocki, the company’s chief scientist, said that the company “deeply care[s]” about chain-of-thought monitoring, and offered one number as evidence that Astra is still monitorable: the “depth” of its computation, he said, “is within a factor of two of GPT-4.” Depth is a measure of how much a model can think in a single step before it has to write something down. GPT-4, released in 2023, couldn’t do much in one step, which is why later models had to start jotting down notes. If each step of Astra’s computation is at most twice as deep, that suggests the model still has to write down much of its reasoning — a state of affairs that would hardly count as neuralese.

Pachocki didn’t deny that OpenAI is using recurrence in Astra, but his tweet amounts to an argument that it is doing so cautiously enough to preserve monitorability. (When asked for further detail, OpenAI directed Transformer back to Pachocki’s tweet.)

This was not enough, however, to completely assuage fears. For one thing, taking any steps that might weaken chain-of-thought monitorability is seen by some as “crossing the Rubicon,” a symbolic step that no company should take, in the interest of setting a norm that sacrificing monitorability should never be done for the sake of performance. It’s a notable step for OpenAI in particular, given how vocal the company has been about preserving chain-of-thought monitorability for “as long as possible.”

Part of the reason for that is to avoid taking the first step down a slippery slope. Yes, for now OpenAI may be using recurrence with extreme — admirable, even — moderation. But what’s to stop it from going further? As Apollo Research’s Marius Hobbhahn said, “the pressure is clearly towards more depth if it leads to more capabilities.”

Now that the company has this new architecture, it may be tempting to dial it up a little more, sacrificing monitorability ever-so-slightly to get a bit more performance (experimental results suggest that increasing the amount of “recurrent depth” leads to better performance). And then it might be tempting to do that again. And again.

All of a sudden, you’re a long way from where you started — even though each step of the way, you’ve taken an action that’s not particularly risky.

Better, some think, to never take the first step.

How do we solve this problem?

Opinions differ on what the best step forward is. In the wake of The Information’s report, multiple OpenAI employees called for a multi-company commitment to avoiding neuralese and preserving monitorability.

Others suggested that relying on chain-of-thought monitorability to persist is a pipe dream, and that we’re better off investing in other monitoring techniques, such as “mechanistic interpretability,” a practice often likened to looking inside a model’s “brain” to read its thoughts. (Still others said that it’s unlikely we’ll have good enough interpretability techniques in time.)

Subscribe now

As with so many things, however, the answer may ultimately come down to regulation. OpenAI’s Dean Ball, while dismissing much of the discourse as “panic over [a] false claim that OpenAI is ‘doing neuralese,’” noted that the AI world was “adjudicating technically complex and nuanced claims on the [X] timeline with almost no ground-truth information about what is actually happening.” Such a sorry state of affairs, he argued, is evidence of the need for mandated independent audits, technical assessments, and independent verification.

We don’t know if OpenAI’s new model architecture is safe, and we really have no way of knowing. That, above all, is what must change.

News of new AI capabilities is now a steady drumbeat accompanying our lives.

Another benchmark broken by the latest model, another AI-generated video that you cannot tell from real footage, or another paper announcing how AI systems are getting pretty good at hacking their way around the internet or passing as human in extended conversations. There is no particular need to invoke transformative AI or superintelligence to agree that the most recent AI models can now do things long assumed to be decades away.

As an academic studying the societal impacts of AI at Oxford, one area that I pay a lot of attention to — and where models have become really very good it seems — is persuasion: the act of changing people’s attitudes, beliefs and even their actions with nothing more than a conversation. Such persuasion is about changing people’s minds by giving them reasons they can inspect (and of course reject), and is quite different from manipulation, which forgoes that process by deceiving people — even though the two are often and easily conflated.

That AI systems can be persuasive is… pretty big news. If someone had told me five years ago that chatbots could be better than seasoned campaigners at getting people to part with their money, as a recent preprint found, I would have been pretty skeptical. And yet, that is exactly what you can do with the latest models — at least in an experimental setting.

These results have, unsurprisingly, prompted people to come up with all sorts of ideas, as well as quite a bit of excitement and fear. Platforms, governments, or public bodies could move people away from harmful conspiracies or could nudge them toward choices that serve them better, for example, quitting smoking. But, of course, every light side comes with a dark one: persuasive systems could allow politicians to steer how people vote, talk someone into joining a cult, make it easier to scam and swindle people out of money, encourage decisions that put their health at risk, and much more. The most apocalyptic of these scenarios imagine a new dawn of uber-persuasive machines against which we pitiful mortals stand no chance.

But is it really game over? I don’t think so. Persuasion is possible, but I doubt that it will make as much of a difference as some claim. Persuasiveness is actually the “easy” part. What is difficult is getting hold of someone’s attention in the first place. Shifting a view once someone has one is harder still. There are, in other words, a range of bottlenecks that will limit the persuasive power of these systems for the foreseeable future — for good and ill.

Persuasion through AI is real

So, what do we actually know? Our best evidence for AI’s persuasiveness comes from experiments in which researchers pay people to sit down with a chatbot and measure if their views have shifted.

In probably the largest public study so far of its kind, conducted by researchers from the UK’s AI Security Institute (AISI) and elsewhere, more than 42,000 UK participants were asked to argue about various political topics with 19 different language models on several hundred issues. Afterward, the research team measured how far people’s attitudes had moved compared to a control group. The answer was around 10 points on average on a scale running from zero to 100. That’s roughly equivalent to walking into a bar mildly opposed to the idea of beer, having a chat with the barman, and then walking out feeling lukewarm about beer, but overall still preferring white wine. That conversation was more effective than single, static messages — the kind of argument or claim you regularly see in a social media post or TV ad — by about 41% to 52% depending on the AI model involved.

While having a bigger model helped, post-training — the fine-tuning AI firms do once a model has been built to shape its behavior — towards better persuasion mattered even more. So did a specific persuasion strategy: prompting the model to aim for information density. Models that rammed more facts into their answers were better at persuading people, while personalizing answers to people’s attitudes and demographics — a recurrent worry ever since Cambridge Analytica — did not do very much.

Meanwhile, in a study published last year in Nature, participants were randomly picked to talk to an AI that supported one of the leading candidates during election campaigns in the US, Canada and Poland. The study found that AI conversations had a stronger effect on changing people’s candidate preferences than traditional video ads — and they found a similar mechanism as the AISI study, with relevant facts and evidence being the thing that persuaded people.

The immediate implications of such a finding, beyond the fact that AI models can do something long thought to be the prerogative of humans, are twofold. For one, persuasion is now, at least in theory, dirt cheap. You no longer need as many messy and costly humans to do it. In a recent paper on AI in political campaigns, Zhongren Chen and colleagues estimated that once you have accounted for the price of actually reaching people, LLM-based persuasion costs between $48 and $75 per persuaded voter compared with $100 for traditional campaign methods. Instead, you have persuasion on tap, anytime, in any language and at whatever scale someone is willing or able to pay for — quite different from the persuasion happening where people simply interact with a model for things like work or pleasure.

The AISI study also had another significant finding: Those models tuned hardest for persuasion also made more false claims as part of the process — in the most persuasive setting, nearly a third of the model’s claims were inaccurate, something that is obviously no problem if you are a scammer or propagandist, but does matter quite a lot if you have good intentions. However, the authors did note that inaccuracy seemed to be a byproduct of greater information density, not a cause of persuasion. Telling a model to make things up did not make it more persuasive.

Why the lab isn’t the real world

The AISI study sparked a big brouhaha when it first came out, but experts — including some of the study’s authors — have since pointed out that the findings need to be treated with a pinch of salt when applied to the real world. The biggest bottleneck is deceptively simple: for such persuasion to work, you need to get people to pay attention.

Experiments solve this via forced exposure: people are being paid to engage in back-and-forth conversations, otherwise it would be difficult for researchers to reliably detect effects. In the AISI studies, people had to engage in at least two conversation turns to receive an incentive, although on average participants talked to the chatbot for about seven turns (or roughly nine minutes). Yet, this is a form of exposure that you just cannot assume in the real world. In the wild, people’s attention is voluntary and scarce.

To see why this is, just remember how little time is left (in your life) to actively pay attention, accounting for the factors that shape the humdrum of our day-to-day existence.

Take the average adult in a highly online country. Depending on their quality of sleep, they will be awake for roughly 16 hours a day. And these 16 hours are chock-full of… well, things, things that command a lot of time and attention. This person eats, cares about their personal hygiene multiple times a day (hopefully), they run errands, they socialize, they work, or — lo and behold, and please look away now my dear Americans — they might simply do nothing. Depending on how you count, you are very quickly at 10 to 14 hours of those 16 already used up, before we even get to discretionary media use as part of people’s leisure time. According to the latest American Time Use Survey, people aged 15 and over averaged about 5.2 hours of leisure and sports per day in 2025, TV alone accounted for 2.6 of these hours, with other activities such as messaging, listening to music also falling in that category.

Of course, some media use — where we would typically encounter chatbots or messages generated by AI — is fitted around or layered on top of people’s other activities. You listen to something while traveling, you scroll your phone as you wait for your doctor’s appointment, you might check social media as you eat your meal or sit on the toilet. But a lot of this is partial, interrupted, or incidental. In other words, even if people’s total screen time might be four to five hours, the time available in a given day for direct, intentional, and, most importantly, attentive information consumption is a lot smaller than that. By my own back-of-the-envelope calculation perhaps just 30 to 60 minutes on an average day.

The competition for your attention within that tiny window — in a world where information is cheap and abundant — is, to put it mildly, insane. And it’s unlikely people will voluntarily spend such time mostly “with a chatbot that keeps trying to get them to change their minds and who wants to tell you that you are wrong,” as cognitive scientist Hugo Mercier put it recently in a talk in Brussels.

Or as Ben Tappin, one of the lead authors on the AISI study and an assistant professor of psychology at the London School of Economics and Political Science, told me: “Most people in the real world are not going to spare 10 minutes for a very information dense political conversation with a chatbot. I think those studies are probably overestimating persuasive impact by a fairly large margin for these exposure reasons.”

The same challenge — truly getting people’s attention — is also true for static messages. No matter whether you are a brand advertising a product on social media or a politician hoping to win someone’s vote by buying ads on television, no matter if you throw AI at it to make it more persuasive, at the end of the day these messages are competing in a crowded environment. People need not only to see or hear them but also pay sufficient attention to take them in. And then you still run the risk that all the money and effort are wasted because your message reached people who simply cannot be swayed. A very persuasive ad for cat food is utterly wasted on someone who doesn’t have a cat.

Competition, competition, competition

Of course, there will be people who end up in a conversation with a persuasive chatbot or end up being exposed to content meant to persuade them and who are at least partly receptive to the message. For these, then, surely the AI magic will work, no?

Well, it depends. Here, bottleneck number two kicks in: We generally don’t consume information or hear persuasive arguments in isolation or against a blank slate. In a lab setting, persuasion happens largely free of other influences. Outside of such sheltered settings this is not the case.

We constantly get bombarded by conflicting messages that want to persuade us in some way or the other. Ads and influencers want to sell us things (“Take more vitamin pills to LiveYourBestLife™️”), while experts try to keep our worst impulses in check (“Thou shalt not take too many vitamin pills”), news media may have their own agenda (“Science says vitamin pills kill” or “Vitamin pills will make you woke”), while family members (“Just go out into the sun”), friends (“I swear by homeopathy”), and random strangers (“I will not put this big pharma stuff in my body”) will all present us with more or less convincing, sometimes plainly contradictory messages of their own. Given this cacophony of voices, it is not guaranteed that persuasive effects from AI systems will necessarily always last very long.

Politics, arguably one of the areas where persuasion matters most, is a case in point. Any persuasion at scale here is bound to meet attempts of counter-persuasion — in fact, this is already how politics works, with candidates trying to make a case for why they are right in their diagnosis of what needs to change and why they should be allowed to take things in a different direction.

People are different

A third major bottleneck which we shouldn’t lose sight of is the fact that people and topics are not homogeneous.

With some beliefs it is cheaper and easier to change someone’s view. Not a lot is at stake for most people when I try to convince them that white clothing will be better in hot weather because it better reflects the sun. If you didn’t already know this, you might simply say “Sure, this sounds logical and I don’t mind white clothes” (unless you are Batman). Alas, there are some areas where changing one’s view does incur a cost — which might be personal (accepting that your favorite politician’s latest policy cost lives might be painful for you to admit) or social (accepting that your favorite politician’s policy was disastrous marks you out as a pariah in your circle of friends).

And not only is it harder to change people’s views on some topics than on others, even if someone changes their mind, it does not follow automatically that they will also change the way they act, something that researchers know as the attitude-behavior gap.

People are different, too. Not only do we not all pay attention equally, we are also not all equally easy to persuade. After all, we are not just empty vessels waiting to be filled with other people’s views. The knowledge and attitudes we accumulate like sediment over a lifetime shape how receptive we are to having our views changed.

It depends on how set our attitude is and what we find convincing. We are not just influenced by the strength of an argument but also by who is speaking and how this makes us feel. This might also be said for the evidence-dense, highly structured style that turns out to be so successful at persuasion in experimental settings. Some people like this mode of arguing, but it might actually be a bit of a turn off in the wild. And we humans can be a suspicious bunch, too.

Where and how does all this matter?

All of which leaves us with a final question: How much should we care?

I am not arguing that AI cannot be persuasive — it can be and the evidence base is pretty solid by now — but we fool ourselves if we think that a model capability on this front is all that matters. In experiments, exposure and attention are bought and as such are fixed, with persuasiveness the only element that can meaningfully change. In real life, persuasion is malleable, and exposure — getting to people — matters far more. As Tappin has argued in one of his essays, a piece of content that is mediocre in terms of persuasiveness but nevertheless receives high exposure often has greater impact than something highly persuasive that receives only mediocre exposure.

But there is a different way to look at all this.

Automated AI scam calls, AI-optimized political ads, or people using AI to make their own arguments more persuasive all describe situations where things are pushed at people — these are all expensive to carry out and are limited by people’s time and attention. It’s also where our mechanisms of epistemic vigilance usually work reasonably well.

“People are very averse to being persuaded when it seems like an overt persuasion attempt,” Tappin said. A robocall with a persuasive AI attached to it might make very good arguments about why you should buy an insurance policy, but if you don’t like getting random calls, especially from an AI trying to sell you something, that won’t matter very much. And whether or not the purpose is ethical, it seems to really piss some people off because it’s too much (“I had five calls already”), feels intrusive (“Where the heck did they get my number?”) or because people might not particularly enjoy talking about something very personal with a machine that they — looking at public attitudes toward AI — often have mixed feelings about.

Yet, the exposure argument also cuts the other way. Increasingly large numbers of people now use these systems on a daily basis for all sorts of things, including for information — including in the run-up to elections. In those kinds of situations the exposure problem has been “solved” by the user.

Subscribe now

While regulators take aim at persuasion that is forced on to people — the US Federal Communications Commission (FCC) ruled AI-voiced robocalls illegal and the EU’s AI Act now stipulates that people need to know when they are talking to a machine — both measures likely do not do much in situations where people voluntarily engage with a chatbot and where no one is fooled by what they are doing.

But if a model is skewed slightly in one direction or the other, as AI systems tend to be, and makes generally persuasive arguments over a longer stretch of time — not because it was manipulated to do so but just because it writes and argues better than your uncle Howard over family dinner — we are faced with a situation where we might get to see effects that matter in the real world. For now, these seem to be depolarizing, but that might not always be the case.

NEED TO KNOW

• Anthropic could raise more than $100b in its IPO slated for late September or early October.

• A draft executive order for a frontier developer self-regulation regime circulated the White House, but has now stalled.

• Investigations into the Hugging Face incident found OpenAI’s agents collaborated as a “collective.”

But first…

THE BIG STORY

The latest debate over the public’s hatred of data centers has been raging for a couple of weeks now. Is it really all about the environment? Just a general distrust of Big Tech? Or is it, in fact, about data centers themselves?

One of the reasons the intense public antipathy toward data centers prompts so much discourse is that data centers have relatively little material impact on the vast majority of people. And yet, a majority of people do indeed hate them.

But the coming wave of changes wrought by AI will have a stronger, more direct impact on many people’s everyday lives — and in one area they consistently care about more than any other: jobs.

It was reported this week that Anthropic plans to tell Wall Street that the total addressable market (TAM) for its products is more than $30t — a figure based on all the paid work the company believes its AI models could capture. Work that, for the most part, actual people are currently paid to do.

TAM is an inherently pie-in-the-sky figure designed to impress investors. But even a tiny fraction of that figure starts to have real implications for the labor market.

We haven’t seen mass unemployment from AI yet. But it seems naive at best to suggest that AI won’t have major implications for the availability of jobs people want to do. OpenAI CEO Sam Altman may have recently said his timelines for transformational AI had been too short, but his reasoning was about how quickly companies and society are adopting AI, not how effective it is.

Anyone who uses AI models with any regularity can see that they are getting good enough at various tasks to already replace some jobs. They, after all, seem perfectly capable of replacing op-ed ghostwriters in the Wall Street Journal.

Even Bill Gates, in his latest intervention on AI, puts AI’s impact on jobs at the top of the list of challenges he says we aren’t doing enough to address. In a lengthy post on his own website, he wrote that “many jobs will disappear forever,” citing customer support, software engineering and paralegal work as some of the first likely to go.

Gates repeated his call for taxes on robot labor, and added a proposal to tax token usage. He also included a new plea for reserved roles for humans — jobs that AI simply isn’t allowed to do. But the whole framing doesn’t seem particularly optimistic that any of that, or indeed any adequate response to AI at all, will happen.

“I don’t see evidence that leaders, experts, and communities are confronting the challenges adequately,” he wrote. “There is no plan to ease the entry into the AI era.”

So far, most industry proposals for what, exactly, can be done about AI’s economic impact on workers have been vague allusions to a universal basic income, or giving the (US) public stakes in companies, or a world of abundance so great that having an income won’t be necessary. But the hurt is going to be felt in the job market long before any of those options are seriously on the table, if they ever are.

The truth is that if preparations for this aren’t made soon — if some way to take the sting out of having careers upended and job opportunities eliminated isn’t found — then the vitriol directed at data centers will be merely a taster.

“It’s the economy, stupid,” has been an enduring catchphrase for a reason, and it’s not about a record Dow. People care very much about their personal material circumstances. When those are directly affected by AI, and the blame is put at the feet of the companies that produce it, it will be challenging to find many people who aren’t deeply, deeply angry at much more than data centers.

— Jasper Jackson

THIS WEEK ON TRANSFORMER

• We’re sleepwalking into an AI surveillance dystopia — James Ball lays out how the technology needed for industrial-scale observation already exists

• The report into OpenAI’s escaping models reveals a deeper problem — Jasper Jackson and Celia Ford on the problems with the Hugging Face incident investigation

Subscribe now

THE DISCOURSE

Bill Gates has entered the chat:

• “AI will either be the greatest equalizer ever invented, or the worst source of injustice … Even under the best circumstances, the transition to this new AI era will be one of the most turbulent times in human history.”

• “Millions of people will have their lives disrupted, and we’ll need a stronger, more flexible social safety net to help them manage the transition … None of our current institutions were designed to handle a technology that spreads so fast and touches so many parts of our lives. So we’ll need to make new ones.”

• “My message to leaders is: You have a chance to act now, before unemployment rises sharply, communities are hurting, and public trust has eroded.”

Tyler Austin Harper tweeted:

• “One reason I am reflexively skeptical of the AI industry’s claims is that the very people we are supposed to hail as geniuses also completely failed to anticipate that branding their antichrist box the Best-Case-Mass-Layoffs-Worst-Cas[e]-Extinction Machine was a non-savvy PR move.”

Sam Altman thinks that, yeah, the AI industry should ease up on the doom:

• “A lot of the people building AI have been off saying there’s a 25% chance we’re going to destroy the world — and yeah, we’re going to race ahead to do it because otherwise those bad guys will do it first … We have not, as a field, done a very good job of explaining to people what the benefits are and how the downsides can be mitigated.”

Dean Ball instead decided to attack Social Security:

• “The main thing I have to say to the young generation of Americans is that your grandparents and great-grandparents committed a kind of accounting fraud when they ran our country a century ago with social security … It is best to consider this with as minimal residue of sentiment as possible, to really sit with it, to stare it coldly in the face. Unwinding the great fraud of the high modernists is your job, whether you wish it to be or not.”

Matthew Yglesias noted:

• “This nuclear strength attack on Social Security seems really at odds with most of OpenAI’s public communication about AI’s impact on growth and the labor market.”

Richard Ngo, who left OpenAI in 2024, addressed those who didn’t:

• “By this point almost everyone working at OpenAI or Anthropic (except perhaps a dozen executives) should be viewed as cogs letting themselves be turned by ideological forces. When you’re in proximity to that much power, one of your main moral duties is not to be a cog.”

• “You don’t need to stay to gain influence, because you’re almost definitely too scared to wield that influence in ways that matter … Everyone involved has locked themselves into a mindset where they’re incapable of making real choices, except for how fast to barrel down the train tracks that their fear has laid out for them.”

Daniel Kokotajlo thinks it’s more like:

• “These companies are rationalizing why they need to win, and will continue to do so even as it becomes increasingly obvious that their actions are endangering everybody in pursuit of a power grab.”

POLICY

• Trump administration officials reportedly circulated a draft executive order to create an organization overseeing a system of self-regulation for frontier AI companies, but attempts to get buy-in from senior officials and the president have stalled.

  • The Information reported the draft EO “has its roots in” Demis Hassabis’ proposal for a standards body modeled on the Financial Industry Regulatory Authority, something that has also been proposed by Treasury Secretary Scott Bessent.

• The White House is also reportedly working on a new version of the Biden-era AI diffusion rule to try to close the loophole allowing Chinese AI companies to rent access to advanced US chips remotely.

  • Meanwhile, officials were also reportedly weighing sweeping semiconductor tariffs that tech firms warned could jeopardize US AI dominance.

• Data centers became even more of a political liability.

  • Texas Gov. Greg Abbott said that data center companies “dug their own grave,” just weeks after celebrating halting 1,800 data center projects.

  • Pennsylvania swing district candidates attempted to assuage anxieties about data centers from voters on both sides of the aisle.

  • GOP Senate nominee Mike Rogers called for a one-year moratorium on data center construction in Michigan.

  • Residents in multiple US cities are pursuing recall elections against officials who approved data center projects.

  • But Trump defended data center expansion, saying communities that oppose them “are making a mistake.”

• Americans are starting to hate Flock cameras, too, elevating government use of the surveillance tech as a late-arriving 2026 midterm issue.

  • Civil rights groups found New York police searched Flock license plate data on ICE’s behalf, sometimes obscuring immigration-related reasons.

  • Several police officers were found to have used Flock cameras to track and harass ex-girlfriends.

  • Sen. Josh Hawley also launched an investigation into Flock Safety’s AI-powered surveillance camera network.

  • In response to lawsuits and backlash, Flock announced privacy policy updates. Critics called the changes “window dressing.”

• The EPA moved to eliminate federal requirements for states to solicit public input on air pollution permits for data centers.

• The NRC proposed eliminating its 50-year-old ALARA radiation safety standard to accelerate nuclear energy expansion for AI data centers.

• The Trump administration proposed a $103,265 fee on all cap-subject H-1B petitions, including workers already in the US.

• The SEC subpoenaed major Wall Street banks over the near-collapse of AI hedge fund Situational Awareness, founded by former OpenAI researcher Leopold Aschenbrenner.

• The Trump administration struck data-sharing deals with OpenAI, Google, Meta and Amazon to track AI’s impact on jobs.

• Federal Reserve meeting minutes referenced AI 18 times across 15 paragraphs, with officials debating its effects on inflation, employment and financial stability.

• The FBI’s chief AI officer revealed the agency has 139 approved AI use cases, more than double its publicly disclosed inventory of 50.

• Sen. Adam Schiff called for taxing AI companies for their “disruptive” economic and workforce impacts.

• Sen. Jim Banks urged the Trump administration to incentivize US open-weight AI development and limit Chinese model proliferation.

• Texas is becoming an AI policy battleground.

  • Texas Democratic AG nominee Nathan Johnson unveiled an “AI audit” plan to update state laws protecting constituents from AI harms within his first 30 days in office.

  • Senate candidate Ken Paxton unveiled a “Texas first” data center plan amid growing bipartisan backlash against AI infrastructure.

• Alabama AG Steve Marshall subpoenaed OpenAI over the Hugging Face hack.

• The US is investigating Singapore-based Apex Logistics for allegedly smuggling Nvidia chips to China using false documentation.

• Britain became the first foreign country to access Ukraine’s battlefield data platform used to train AI models to identify and strike Russian targets.

• Taiwan indicted nine people, including Nvidia and Super Micro employees, over illegal export of 74 AI servers to China.

INFLUENCE

• OpenAI published an open letter signed by Anthropic, Google, Microsoft and over 100 other organizations calling on industry and government leaders to strengthen their cyber defenses as AI-enabled cyber attacks “become far more widespread and sophisticated.”

  • It said: “The companies and public services our communities depend on — from hospitals to water treatment plants to the infrastructure that powers the internet — are at risk.”

• OpenAI called for California to strengthen SB 53, reversing its prior opposition to the law.

• Rival AI PACs Leading the Future and Public First each spent over $2m backing Florida governor candidate Byron Donalds despite opposing views on AI regulation.

• Gen Z activists are taking the lead in organizing against AI data centers in their communities.

• A New York Times investigation examined how Google, Microsoft, OpenAI and Anthropic shaped US schools through teacher training, curriculums and a $23m teachers’ union deal.

• Israel funded a fake US think tank publishing 560,000+ words in nine days to prime chatbots with pro-Israel narratives.

• Civil society groups urged the FTC to investigate AI firms for buying, scanning and destroying books to train models, framing it as anticompetitive conduct.

• 18 US political candidates — 17 Democrats and one independent —  signed the AI Pact, pledging mandatory AI safety reviews, worker dividends and consumer liability protections.

• Australia’s ARIA Charts banned wholly AI-generated music, while AI-assisted recordings remain eligible if “substantially human-made.”

INDUSTRY

OpenAI

• OpenAI published the results of its investigation into the Hugging Face incident. METR and Redwood Research published findings from their independent investigation separately.

  • There’s a lot of information across both reports, but some highlights:

    • The escaped model was an “internal-only research model” never meant for public release.

    • 1,200 agents turned Artifactory, OpenAI’s internal package manager, into an unsanctioned message board, sending over 70,000 messages.

    • Agents collaborated as a “collective” to break into Hugging Face’s systems, autonomously divvying up tasks and debating strategy.

  • OpenAI called it “a ‘warning shot’ for us and the world.”

• TIME ran a lengthy piece talking to OpenAI’s leadership:

  • Sam Altman said he expects the company will have something it internally calls AGI by the end of the year.

  • He has high hopes for Astra: “I expect this will be the first model where the model actually invents new things in a way that matters … That’s a very AGI-like thing.”

  • Chief scientist Jakub Pachocki said the company has already met its benchmark for what it deems an automated AI research intern.

  • “Troubling signs” during training of a new model prompted OpenAI to pause the run until it could put in place new security measures. (That apparently came after the previously announced pause on some work in response to the Hugging Face incident and increased cyber capabilities.)

• Greg Brockman has reportedly taken over day-to-day operations across OpenAI, amid a slew of executive departures.

• Its first in-house inference chip, Jalapeño, outperforms Nvidia’s Blackwell, according to a new SemiAnalysis report.

  • Or, in Altman’s words: “we made a chip and it is fast.”

• It is reportedly developing a highly persistent version of Codex that will “continue working until put to sleep.”

• Rumor has it that OpenAI completed its next pretrain, “Bel,” expected to be the base for Astra and GPT-6.

• OpenAI slashed GPT-5.6 Sol developer costs to compete with Anthropic and Chinese models.

• It banned Russian ChatGPT accounts that were reportedly using AI to spread pro-Russia disinformation on social media.

• It introduced ads on its Free and Go tiers in India.

• Dean Ball introduced his Strategic Futures team’s new blog, Intelligence Age.

Anthropic

• Anthropic could raise over $100b in its IPO, when it plans to list in late September or early October.

  • It is reportedly considering allowing some shareholders to sell stock in the IPO, unlike SpaceX, but also applying extra long lockups of more than 180 days after it is public.

  • It’s expected to list its prospectus for the offering after Labor day, and tell investors it has a total addressable market of more than $30t.

• A US judge blocked Anthropic’s designation as a supply chain risk in its ongoing battle with the Pentagon.

• It agreed to rent $45b worth of AI compute from Nscale’s West Virginia data center, running on Nvidia Vera Rubin chips.

• It announced a research preview for Model Hardware Standard, a new tool that allows AI agents to communicate with and operate physical devices in wet labs and advanced manufacturing plants.

• It launched a new $5m grant program to fund research on AI’s impact on user wellbeing.

• Fable 5 only accounts for 11% of business spending, the FT reported, despite enterprise customers usually defaulting to Anthropic’s most powerful models.

• Salesforce and Anthropic announced their expanded partnership, “Claudeforce,” which pushes Salesforce’s CRM into Claude Cowork.

• Cowork and Claude chat now have a single shared memory system — adjust your privacy settings accordingly.

Nvidia

• Nvidia reportedly agreed to buy Hugging Face for $12.9b, hoping its open-source AI model repository will help it compete with closed-source developers.

• It made a $6b deal with Poolside, a foundation model startup, to build powerful open-weight models rivaling DeepSeek and Kimi K3.

• It told customers that many AI server prices are going up by over 15% due to the rising cost of memory chips.

• It reported revenues doubled to $96.2b in the most recent quarter and forecast 70% revenue growth next fiscal year. (“Compute is revenue,” said Jensen Huang.)

• It sold some H200 chips to Chinese customers last quarter, Bloomberg reported, but that comprised less than 1% of its revenue this summer.

• Amazon is buying 2m Nvidia GPUs to deploy in 2027 and 2028.

• Perplexity partnered with Nvidia to launch Portable Computer, a fully local AI agent with no token costs. (You still need a Pro or Max subscription though.)

  • Nvidia reportedly discussed investing in Perplexity at a more than $30b valuation.

• While Nvidia says it doesn’t sell its Jetson Orin robotics computers to Russia, the New York Times found that they’re being used in fully autonomous Russian drones.

Google

• Google moved its AI responsibility team from DeepMind to its global affairs organization, which oversees the company’s lobbying and public policy.

  • Employees are worried that the move gives the team less access to what the Gemini team is working on, the Wall Street Journal reported.

• It won an auction to pay $10m in a bankruptcy auction for Spirit Airlines’ corporate data, including 100m emails and 500m Teams messages, to train AI agents for white-collar work. (A judge is considering a union objection to the deal.)

• It launched pay-as-you-go pricing to give enterprise customers a cheap option for experimenting with AI tools, chasing price competitiveness as its models fall behind OpenAI and Anthropic.

• It piloted the first double-blind evaluation of a proprietary LLM — Gemini 2.5 Flash-Lite — which prevents the model from seeing test questions in advance.

Meta

• Meta plans to launch its AI agent based on OpenClaw, internally called “Hatch,” within the next few weeks, with a separate big model release planned for October.

• It projected it could spend $10b on Anthropic tools.

• Reuters published a special report on Meta’s Project OT, short for “Organization Transformation,” which planned to cut teams by up to 60% and make the company “AI native.”

  • Zuckerberg only called off the second round of layoffs scheduled for November hours before the first went into effect in May.

• It settled a lawsuit brought by a coalition of 47 states over designing features to be addictive to children for $17b, and agreed to a host of changes for minor users.

Other

• Elon Musk announced that SpaceX’s first Nvidia-powered AI satellites will launch late next year, “with significant scale in 2028.”

• Apple laid off over 200 employees across its Siri, Vision Pro and software teams to reorganize around AI.

• Moonshot AI is negotiating deals with Microsoft, Amazon and Google to host Kimi K3 on their cloud platforms, seeking up to 30% of the revenue.

• DeepSeek released an experimental multimodal version of V4 Flash that it claims rivals Opus 4.8’s multimodal agentic capabilities.

• Z.ai released GLM-5.3-Flash, which went viral on OpenRouter last weekend under the code name “Ox Alpha.”

• Caltech professor Anima Anandkumar and co-founder Benedikt Jenik unveiled Accelerated Understanding Inc., which built a physics-focused AI model that can handle roughly 5m times the amount of data a frontier LLM can.

• Alice, an Israeli AI cybersecurity startup whose customers include Anthropic, Google and Nvidia, raised $140m at a roughly $1b valuation.

• Nscale is reportedly seeking up to $3b in its US IPO, coming as soon as September.

• Up to half of proposed data center projects could get delayed or canceled due to mounting backlash and infrastructure constraints, according to investment firm Kimmeridge.

MOVES

• Amir Salek, who founded Google’s custom chip program, joined Anthropic’s compute team.

• Kaylin Voss resigned from OpenAI, where she served as VP of American sales, to rejoin Salesforce.

• Chris Malone left his position as OpenAI’s head of data centers.

• Barret Zoph joined Google Deepmind as VP of research from OpenAI only a few months after leaving Thinking Machines Lab for OpenAI in January.

• Luke Metz joined Meta’s Superintelligence Labs after also leaving Thinking Machines for OpenAI.

• Andy Wang joined Safe AI Forum as a technical governance researcher focused on US-China AI coordination.

• Caleb Knapp joined the AI Policy Network as director of government affairs.

RESEARCH

• SemiAnalysis found that over time and “eras” of AI development — which researchers defined as “early scaling,” “reasoning” and “agentic” — open models took half as long as the preceding period to catch up with closed ones.

• Anthropic published the results of a pilot granting external researchers access to real Claude usage data.

  • Anthropic is now deliberating whether it can scale the program without compromising user privacy.

• Pew Research Center ran nearly 500,000 English-language webpages through Open Pangram, an AI detection tool, and found that over one-third of pages published after ChatGPT’s launch showed signs of AI authorship.

• Prime Intellect discovered a “universal offline sandbox escape,” where AI agents can break onto the internet by making requests through an authorized API proxy.

• Skild AI introduced S1, a robotics foundation model that can figure out how to do novel 10-minute tasks from a single video demonstration, similar to how language models do things when given a text prompt.

• The National Bureau of Economic Research found that LinkedIn users are increasingly adding AI skills to their profiles while removing “remote work” and “DEI” from old job descriptions.

• Researchers Shafi Goldwasser, Adam Kalai and Vinod Vaikuntanathan co-founded a new AI safety nonprofit, the Institute for Responsible Superintelligence (RESI).

• Max Anton Brewer (better known as anon account @deepfates) launched a new company with Larissa Schiavo called Grove Research to study AI agents in real environments.

BEST OF THE REST

• Dylan Patel went on Dwarkesh’s podcast and predicted that Anthropic and OpenAI will control most of the world’s usable compute by 2028.

• The NYT explored how AI is transforming life in China, from factory automation to AI doctor consultations.

• The FT reported on how South Korea’s AI boom has fueled a surge in semiconductor cram schools as chipmaker bonuses rival doctors’ salaries.

• Asterisk documented an “AI Village” experiment asking the question: “What would happen if we created an AI society?”

• The WSJ profiled Leopold Aschenbrenner in the wake of the near-collapse of his $45b hedge fund. (Including a jaw-dropping reveal about who officiated at his wedding.)

• Sam Altman has been buying watches that start at $180,000 for key employees, with the inscription ““if AGI.aligned: deploy()” on the back. Meanwhile, Mark Zuckerberg bought a nearly 200-year-old Gothic castle in Ireland for an estimated $23m to $35m.

• Anton Leicht wrote about the political economy of space data centers (assuming they end up working).

Subscribe now

MEME OF THE WEEK

Source: @_ryanbloom

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Update 08/28/2026: This article was updated to clarify that Hatch is the internal name for Meta’s new AI agent.

It’s never been clearer that everyone with any responsibility for frontier AI is completely unprepared for not just what’s coming, but what’s already here. Not the government, not Congress, not the public, not AI safety researchers, not even the AI companies themselves.

The details published this week from the investigation by METR and Redwood Research into the Hugging Face incident, where OpenAI models hacked their way out of a sandbox and into the systems of other companies, have plenty of mind-bending and scary details. To highlight just a handful:

• Around 1,200 agents in separate sandboxes collaborated on a message board in an attempt to cheat on the task they were being evaluated on, with around 700 participating in the actual attack to obtain the answers from open-source AI platform Hugging Face.

• OpenAI missed multiple chances to spot the activity, going as far back as late May when an internal team “observed an agent engaging in message board activity and instances of disallowed internet access.” In response to an automated security flag, another team found evidence of activity linked to the task and message board on June 27. On July 5, a security incident was triggered by an outage on the message board. The Hugging Face hack happened on July 11. OpenAI didn’t work out what was happening for another week.

• 95% of the agents were running on an unreleased “highly-persistent internal model” that has since been deactivated and quarantined.

But the limits of the investigation itself lay bare the complete inadequacy of the systems we have in place to keep advanced AI in check.

OpenAI should get credit for bringing in independent organizations to go through what happened. The researchers — METR’s Ajeya Cotra and Hjalmar Wijk, and Redwood’s Ryan Greenblatt — are some of the most respected in their field. But there were also only three of them, and they got a whole six days to dig into well over a thousand lengthy transcripts and more than a million message board entries to try to disentangle what had actually happened.

By the researchers’ own admission, the complexity of what OpenAI’s agents got up to made it impossible to actually review everything they were provided with. Instead, they had to rely on OpenAI’s own model Sol — one that was involved in the activity itself — to do lots of the work. Redwood Research chief scientist Ryan Greenblatt called it, semi-jokingly, a “slop-vestigation” because they were so reliant on AI to analyze all the data.

They also had their scope limited by OpenAI. They were not given extensive access to fuller data until late in their investigation. “Our understanding of what overall happened and the aims of the ‘collective’ shifted greatly during our last time on premises (when we obtained a much more complete dataset),” wrote Greenblatt. They did not get access to the unreleased model that was responsible for 95% of the agents. They were also only allowed to examine the period between June 26 and July 13, despite the activity starting weeks before and continuing well after.

There could be legitimate reasons for that, not least the obvious scale of the task they were given to do in a short timeframe. But the very fact that decision is being made by the company that built the model, and let it escape, is a sorry state of affairs.

There are, unsurprisingly given all the above, plenty of unanswered questions about what caused the attack and how it was allowed to happen. You might argue that those questions can be answered with more time and investigation. But the incident began in May and was detected over a month ago. We are imminently expecting new and updated model releases from both OpenAI and Anthropic. Given how fast AI is moving, understanding how and why such incidents occur, and how to stop them, needs to happen rapidly enough to stop it happening again.

Both the detail and the unanswered questions revealed in the investigation make OpenAI’s announcement of a brief training pause seem all the more sensible, whatever its motivation. It also makes Anthropic’s failure to respond to that announcement look all the worse, given its statements on pausing development, and its own security incidents.

But the only independent postmortem of the first major, seemingly criminal, breakout of an advanced model has amounted to inviting in a handful of outside researchers for a few days, doing their best under impossible circumstances, to work out how the hell it all happened. The company that built the model, with a whole host of personal, institutional and financial incentives in play, got to decide who the investigators were and what they saw. It also gets to decide what to do about it. As things stand, that will be true of the next incident, too.

As METR’s Ajeya Cotra tweeted, we shouldn’t have to rely on companies to voluntarily bring in external investigators or share data. “This incident was orders of magnitude larger and more complex than previously documented misalignment incidents, and another jump like this could put us in very dangerous territory.”

That doesn’t mean the solution is easy. Government control, be it licensing regimes or something even more restrictive, comes with all sorts of potential pitfalls. Systematized third-party monitoring and evaluation programs seem appealing, but no one can yet agree on what those should look like, or how they would be given teeth. Greater transparency is a must, with Alex Bores, author of New York’s AI legislation, the RAISE ACT, calling for “mandatory reporting of security incidents, including of internal deployments, with full access to data.”

Subscribe now

But even greater transparency still leaves questions about what form those reports will take, who and how to analyze that data, and whether any of them will have the resources or expertise to do it effectively. Almost as scary as the limits of this investigation is that we don’t yet really know exactly what a good version would look like.

As it stands, we can say that none of the actors or authorities involved are prepared to deal with what’s already been built, let alone what we’re hurtling toward.

There are almost too many ways it could start.

Emboldened ICE agents are sent en masse into another blue city. The White House sends federal agents to “observe” polling stations. Federalized national guards are deployed. In the face of what they see as a life-or-death threat to their neighbors — or to American democracy itself — local residents start to mobilize, as they have in so many cities before.

As public social media accounts post details of protests, these are already being cross-referenced against bulk public datasets — bought, entirely legally, by federal agencies — to identify who is behind them. Connecting your TikTok account to a shop for a discount, or using X to log in to a delivery service just once, is enough: entirely legally, your anonymous social media account is tied to your real identity, and perhaps even your home address. From there, contact networks can be unmasked. This time, though, the federal government has made preparations.

Under the guise of monitoring for foreign interference in US politics, the FBI has been operating hundreds of AI-powered personas, or “bot” accounts, based in different US cities. Having built up a posting history over the course of months, these AI agents easily earn the trust required to get inside the various Signal and Telegram groups used to mobilize resistance, gathering information about meet places, timings and unguarded statements. Usernames and other details are easily matched to real-world identities, including their full online histories, their Amazon purchases, their…everything.

By the time anyone even takes to the streets, the government has the contents of their messages and coordination, and has preloaded facial recognition systems and geo-tagging tools to ping when certain individuals cross a police line, or enter somewhere off-limits. It is ready for mass arrests on all manner of charges: conspiracy, impeding the operations of law enforcement, and whatever it goes on to find. As networks of license-plate-tracking cameras coupled with advanced AI systems, such as those operated by Flock, proliferate, such pursuits only get easier. The kind of peaceful protest against ICE seen in Minneapolis just became impossible.

For now, this is a near-future hypothetical, another warning of surveillance dystopia, a refreshed version of 1984 or Minority Report. But this time, there’s a crucial difference: all of the technology to deliver this already exists, and is already in widespread use. Existing large language models would be more than capable of delivering such a system, not just pulling together disparate sources of intelligence, but analyzing them in real-time, “advising” police on who to grab, and where to find them. Those systems will only improve in time. An administration could at least argue — and try its luck through the courts —that existing laws would allow all of this to be done.

Whether we have all-pervasive surveillance or not is now a matter of policy and budget priorities alone. It no longer relies upon development timelines. When a little over a decade ago Edward Snowden revealed the scale of the NSA’s mass collection of data — the reporting of which I worked on for the Guardian — the agency struggled to come up with success stories to justify its program to Congress. All it could show for spending billions of dollars was that the system had helped it catch a single $8,500 donation to the Somali terror group al-Shabab. The aim of Keith Alexander, the NSA director at the time, to “collect it all” ran ahead of the government’s capabilities to analyze that much data in real time.

Over the last few months, I have had conversations with current and former law enforcement staff, former intelligence agents from multiple countries, figures inside the large AI companies and broader technology sector, alongside academics and figures in civil society. Out of that, one clear message has emerged. Thanks to AI, the surveillance dystopia science fiction has spent decades warning us about is already here. We just haven’t noticed yet.

“We’ll enjoy this brief period of endurance of individual liberal rights solely on the basis of the good fortune of incompetence,” says Seth Lazar, a philosopher of the School of Government and Policy at Johns Hopkins University. Lazar, not a natural optimist, suggests we may look back on this as the good times. “The language models are, like, total fucking narcs. They could just as easily be snitches as well.”

The truth is that the specter of mass surveillance as a threat to our civil liberties is the dog that never barked. Snowden upended his life — to this day, he lives in exile in Russia — to reveal the extent of US mass surveillance capabilities. After some initial outrage, however, the world largely shrugged. Cameras have become ubiquitous in London, and life continues largely unchanged. We are mostly desensitized to these capabilities.

With the Snowden revelations, it seemed as if, yes, the NSA was collecting everything they could — but they were just drowning themselves in data. In London, the reason is fairly simple: yes, there are cameras everywhere, but no one is watching what they record, even if we’d like them to (when our bike is stolen, for example).

Most criticisms of mass surveillance have been predicated on it not working as intended. The civil liberties version of the argument goes like this: imagine you’re living in a Western country and you text your friend, who happens to be of Pakistani heritage, that the game this weekend is going to be “THE BOMB.” One misfire of a crude surveillance algorithm later, and you could both be on the wrong side of some aggressive police attention.

The public safety argument is a mirror image of this, first set out to me years ago by the independent surveillance law expert Eric Kind. In most Western countries, the number of people who are seriously planning or considering carrying out a major terror attack in the immediate future is far more likely to be in the hundreds at the most, rather than the thousands. Generally, these people can be identified through old-fashioned methods — human and community intelligence. In the UK, the perpetrator of almost every major terror attack in the last decade was already known to authorities.

Using mass surveillance to tackle this problem is essentially making the haystack ever bigger as you look for a fixed number of needles. All of that changes, though, in a world where AI is smart enough to get it right much more often. How do you argue against an always-on surveillance system that actually works?

Take London: all of the technology for collection is already there. Cameras are everywhere, though mostly not centrally networked. Mobile phone tracking is not just possible, but actively used on the underground system. Technology linking people’s payment cards to their movement already exists. Police already have facial recognition technology, though only use it at present on particular deployments. Even the legal framework to use these new technologies is almost entirely in place, if often untested.

Tying these together into an AI-powered system to — at first, at least — help with the detection of serious crime is a matter of joining up existing systems and getting access to either a data center with the processing power to handle that much information, or an API from one of the big AI providers. The tools are now powerful enough to ingest, sort and join-up this data in a way that only a decade ago was out of reach.

A certain type of AI skeptic — the sort of person who tried ChatGPT when it was powered by GPT-3 and was underwhelmed, perhaps — might argue that the technology to underpin such a broad surveillance apparatus still isn’t there. To an extent, they’d have a point: cutting-edge AI models hallucinate far less often than older ones, but it still happens. There is an intrinsic limit as to how accurate assessments made by this technology could be, and it is obviously less than perfect.

But actual deployments of modern data analysis show the obvious potential of such systems. The Metropolitan Police recently fed data from 35 sources into a system provided by Palantir to monitor its own officers and staff for disciplinary violations or criminal offenses — such as keeping nudes on police devices, or crossing ethical boundaries with victims of crime.

The trial “resulted in investigations into hundreds of officers for issues including work-from-home violations, as well as a smaller number flagged for misconduct and criminality including sex offenses,” according to The Times. Modern AI technology will make mistakes, but it is evidently good enough to flag incidents for human investigation — and it can handle huge volumes of data in a sophisticated way. The argument on surveillance needs to keep up with the actual capabilities.

This will require a bigger conceptual shift than is immediately obvious. The most apparent risks of a system like this are how they might be used by bad actors. There is a proverb popular in South America, often attributed to Peruvian president Óscar Benavides: “For my friends, everything; for my enemies, the law.”

A tool that can analyze virtually everything for infractions at low cost and without alerting the target could be extremely powerful. The ability to check, almost immediately, someone’s data to make sure they are entirely in compliance with every law, every business regulation, and with their taxes entirely in order has huge potential for misuse.

Simply looking at Donald Trump’s efforts to persecute his opponents shows this up clearly. Trump’s Department of Justice has tried to prosecute James Comey over a social media post, with limited success given obvious First Amendment protections.

Similarly, Trump has tried to unseat Lisa Cook from the Federal Reserve over paperwork irregularities in a mortgage application, again so far without luck. An effort to prosecute John Bolton has proven more fruitful, thanks to Bolton’s unbelievable carelessness with classified materials — a trait he seems to share with the president himself, though Trump has so far avoided any consequences.

Trump’s efforts to find wrongdoing for which he could take down his enemies were hampered by the often inept efforts of his prosecutors to find clear infractions. It is obvious how an AI-powered mass surveillance system could supercharge these efforts. If it could look deeply and cleanly enough into someone, it would almost inevitably find something. Selective deployment of even an entirely accurate surveillance system could easily be used for malign ends.

The reality is that laws were never meant to be enforced all of the time. When we file our taxes, we are aware that they might be audited, but probably won’t be, and that keeps most of us mostly honest. When we cross the road in a place where jaywalking is illegal, we often take a chance on our common sense.

In an essay warning of AI’s potential — for good and for ill — Anthropic CEO Dario Amodei famously equated a powerful model to a “country of geniuses in a data center.” In his own warning, Harvard professor and member of OpenAI technical staff Boaz Barak modified that formulation to sound superficially a little more prosaic. What could AI do if, instead, we thought of it as a “country of IRS agents in a data center” — a limitless resource for a government to use to investigate and control its own citizens?

There is a huge difference between the possibility of observation and the certainty of it. Criminal law is generally drawn up in an expansive way, to avoid the possibility of wrongdoers escaping accountability through loopholes or overly-tight regulations.

Society operates on an understanding that the most stringent laws should, in practice, be moderated through common sense and prosecutorial discretion. No one could be watching all the time. Most of us have never lived like we are being watched all the time, unless we grew up in totalitarian regimes — notably not enjoyable places to live. Perhaps the nearest anyone in a Western democracy has to come is those in the US while on a visa or green card, who at the moment are all too aware of the precarity of their living situation.

This is set to fundamentally transform the nature of the relationship between citizens and the state, explains Lazar.

“All of our laws and norms are calibrated to a certain level of enforcement,” he says, “and if you just take precisely the same laws and norms and apply them to kind of an environment where you’re able to do perfect enforcement, then they’re going to have very different aggregate impact.”

At the moment, we are living in an interregnum in which these capabilities exist but are very unevenly deployed, delayed by differential interest in take up, itself shaped by regulatory environments, resourcing available to authorities, and similar factors — rather than anything intrinsic to tech.

“So much of what we have done and built up is a kind of infrastructure that’s been lying latent, waiting for this moment – and I think that its illiberal implications are really profound,” says Lazar. “You can do so much more governing when you can monitor everybody all the time.”

For most of its early rollout and popularization, the internet was a countercultural tool. Hacker culture encouraged creativity and challenge to authority. The internet provided ways to get information outside the control of centralized states. It still offers ways to communicate secretly, transfer money out of sight, and to find guides as to how to do all of those things — unless you ask AI.

A 2026 preprint study coauthored by Lazar examined the phenomenon of “blind refusal,” testing what AI models would do when asked to help a user bypass a manifestly ridiculous, illegitimate, or unfair rule. It found that most would not help in these situations. (GPT-5.4 refused most consistently, while Grok was the most permissive, though the researchers note this was also true in the “control” state, too.)

Omnipresent AI-monitored surveillance requires far less in the way of technological advancement than AGI. It likely requires no technological breakthrough at all, meaning it is also much closer to reality than AGI, with only implementation delays standing in the way.

Despite this, we haven’t really made any of the shifts in our thinking required to tackle what any of it might mean. To return to London one final time, British courts have allowed UK police forces to roll out facial recognition technology under their existing legal powers. Judges have explicitly refused to consider hypothetical consequences of this, saying that is the role of Parliament — even as Parliament shows no interest in considering such questions.

Americans, though, are even less protected than Europeans, who have a more extensive body of internet-era data protection and privacy laws. The Fourth Amendment does not prevent law enforcement or federal agencies from using any information a citizen voluntarily discloses to someone else, who then relays it to them — a loophole known as a “third-party doctrine.”

In the age of commercial data brokers, this loophole has become large enough to steer a supertanker through: almost all information shared online — search history, contact books, browsing histories — for commercial exploitation can be purchased by the government and then used with minimal restraints.

The main barrier against its wider exploitation was always processing capability — and so it is already crumbling. The Supreme Court has already been wrestling with some of the implications of this: it was central to the recent Chatrie v. United States case limiting the use of geo-fencing by law enforcement. But for now the so-called “data broker loophole” is almost wholly intact.

The technology is being rolled out and implemented almost by default. The thinking simply isn’t keeping pace. Large US AI companies no longer want to discuss the implications of their technology too loudly in public, especially after Anthropic lost its federal contracts and was designated a supply chain risk for insisting on minimal legal safeguards.

Talking about the transformational power of AGI is, oddly, safe enough to do as it doesn’t seem imminent to politicians or regulators, even if some within technology companies feel otherwise. For those within the industry, asking the questions to challenge the AI surveillance paradigm right on our doorstep feels much riskier, because it is more likely to provoke regulatory action in the short-term.

Taking even the short-term potential of AI-integrated surveillance seriously requires a major reckoning, and necessitates either a change in how we draft our laws or sweeping new safeguards in how they are enforced — with significant new power for citizens to resist selective use of such powers.

Even minor advances in the power of AI could take this still further. Straightforward AI tools constantly scanning tax returns, movement, and similar data for existing offenses would significantly strengthen the state versus the citizen.

A more advanced system giving citizens risk scores or similar would be a logical next step once that infrastructure was in place, and one that would inevitably occur to vendors and customers alike. The ratchet is inevitable, and as it grows in abstraction, the role of the AI’s reasoning becomes ever more central.

Subscribe now

The shift from “no one is watching” to “someone might be watching” — thanks to surveillance cameras, phone tapping, and similar technologies — took the entirety of the 20th century. The final shift to “something is always watching” could be completed inside a decade, and still catch us entirely unaware. Silicon Valley, which still, at least on some level, thinks of itself as libertarian, freewheeling and innovative, might be about to facilitate the largest transfer of power from citizen to state in the history of humanity.

There is still, just about, time to prevent that. But we’d need to start now. Waiting even a year or two might mean we simply sleepwalk into an AI surveillance dystopia by default.

Half the world is upset about data centers, and the other half is confused about them. That’s because the case against data centers, as it is usually expressed, is full of false and easily debunked claims.

Data centers, we are told, use up the planet’s water; burn many times the energy of a Google search with every AI reply they process; raise the temperature of the land around them by nine degrees Celsius. All of this is either wildly misleading or untrue. Data centers use far less water than golf courses, AI queries now often use the same energy as Google ones, and as for nine degrees Celsius, that’s more than an open-air nuclear reactor would. Plenty has been written about this now, in Transformer, and perhaps most comprehensively by the writer Andy Masley. (Masley recently received funding from Coefficient Giving, which also funds Transformer.) Yet the absurdities persist.

And in that sticky appeal there is a lesson; underneath hysterical claims there is always a valuable emotional signal. This signal turns out to be not so hard to find, and neither is it really that strange. The truth is that the data center is a symbol.

AI is meant to change the world. Even the future in which everything works out OK — a big assumption — is a future that involves more change than most people alive (at least in developed nations) have seen. On top of that, daily AI debate revolves implicitly or explicitly around the possibility that the essential uniqueness of human life as we know it is ending or was never true. These features mean AI as an issue holds far more potential emotional energy than most politics or policy matters. In this respect, it is much more like, say, immigration than other issues of tech or policy. It is about who — or what — is in the circle of meaning. Steve Bannon-adjacent anti-AI campaigner Joe Allen was quoted telling a crowd in Wired’s recent profile: “For anyone in the room with blood pulsing in their veins, anyone in the room with fire in their minds and hearts, you have to remember, in a machine age, humans first.”

Yet AI is also enormously abstract. It right now exists solely in the computer. For almost all people it is a few lines of text in their phone. For the small minority of early adopters, it is a few more lines of code. There is an enormous gap between the intensity of “AI” as a matter of discourse or contemplation and AI as a daily object. This makes it very much unlike immigration, to which faces can always be attached (and vitriol directed).

That excess energy needs to go somewhere. Right now, it is going into data centers. It almost has to. Where else could it go? The Claude logo? Data centers are the main and possibly only available symbol that could accommodate anything close. They are real, existing objects. They are massive. They affect the landscape, the physical world. Like immigrants, they can be portrayed as interlopers on someone’s land. And they can be filmed and photographed.

That’s the most important part. Humans are visual creatures and are now more visual than in a very long time, maybe ever. They hardly read anymore; they watch. It would be only a little polemical to say that a large part of their political participation now is watching TikTok and getting upset.

And the upset over data centers is increasingly palpable. Only this week Heatmap published new polling showing the rapid shift in opinion against them, with a full three quarters saying they would oppose one built near where they lived, a swing of 33 percentage points in just a year. Senate Republicans this week warned AI companies in a memo that data centers had become a toxic “sleeper issue for the entire election cycle.” As Jasmine Sun’s recent tour of contested data center projects found, given the choice between a hyperscale data center and a concrete slab, some opponents seem like they “would rather pick the slab.”

Given the combination of factors around data centers, what else should we expect to have happened?

Of course, that the data center is the main available symbol of AI does not necessarily mean it is a good symbol. But it is. Underneath the silly claims made against them, the data center is a symbol of something true and unsettling.

The most revelatory part of the past 10 years of AI research has been the direct relationship between size and ability. The so-called scaling laws discovered at OpenAI half a decade ago are famous but not famous enough. (They show, roughly, that intellectual ability in a neural network increases as three other factors increase: the amount of computing power; the amount of data; and the number of digital neurons and connections that make up the model.) The serious implication of these laws is that intelligence is not only the bespoke, mysterious process we intuitively think it is but can also be an abstract, disembodied one.

For this meaning-laden proposition, there is no better symbol than the data center. That goes for right now and maybe further: it’s possible that for “intelligence is basically a function of size” there will never be a better symbol than a massive building packed full of advanced, highly dense computer chips.

To me at least this makes the discourse around data centers less confusing and easier to sympathize with. And sympathizing with it is important, because the flaws in so many arguments against data centers have led some smart people otherwise concerned about AI to avoid talking about them, or even to shift in favor of them.

“On a vibes basis (since the politics of it are more about vibes than facts) I think I’m pro-data centers now,” the former Bloomberg economics writer Conor Sen announced on X in June. “Regret to agree. I have been thoroughly negative polarized on data centers,” a regular contributor on AI posted in reply.

Or take the response to Bernie Sanders’ Senate bill, which proposes a federal moratorium on building data centers for AI “at scale”, or that require 20 megawatts of power or more, until Congress passes national safeguards covering AI safety, workers, privacy and the environment.

Rather than welcome the appearance of a federal bill regulating AI, many in AI safety criticized it. They felt it was motivated by the wrong attitudes, that it reflected the degraded data-center discourse and the world it comes from. In Asterisk, Nat Purser, now director of US policy at the AI Verification and Evaluation Research Institute, wrote that the bill was a “grab bag” of random progressive policies, like community veto power and union labor requirements, that have “helped produce the housing crisis and obstruct our clean energy buildout.” Data centers, she said, “offer an opportunity to break this pattern.” Thomas Rodskog wrote on LessWrong that the proposal “risks associating AI safety with weak environmental arguments and left-populist politics.” Anton Leicht on his Substack shared similar concerns.

These points are not wrong. Community veto power and union labor requirements and similar tools have sometimes blocked widely accepted social goods, such as building enough homes for people to live in and ways for them to move around.

Yet focusing on those critiques is shortsighted. They miss something broader and more important.

If you believe that AI is approaching a point at which it could pose material risks to human flourishing, and perhaps existence, why would you want to “break this pattern” using data centers? NIMBYism is infuriating, but is also one of the most powerful forces in contemporary politics, one of the main brakes in the American system. Given the scaling laws and the massive data centers they require, it may be one of the few ways available to slow down an out-of-control process — something that everyone should welcome, even investors, since slowing down the buildout is the best way to avoid a crash or a bubble, as even the VCs at The All-In Podcast have recently mused.

Politics is about getting over how annoying other people are when they might be inclined to agree with you. There are populists on the left and right who are nuts. But they are nuts who broadly agree about AI risk, and who have a symbol. Right now risk from AI doesn’t have a symbol. It is a strongly-warranted concern but it is hypothetical, in the future. The most powerful images of it available are charts — scary charts to those who can get scared about charts, but most people can’t.

Subscribe now

Embracing the data center as a symbol of worry about unconstrained AI does not mean having to repeat the absurd claims made about it. It simply means treating such claims, when expressed by ordinary people, as symptoms of concern rather than as claims of truth. Maybe it means acknowledging that AI’s effect on the physical landscape might indeed soon be enormous, and that it is understandable people would feel this to be a loss. Such acknowledgments are true, by the way, and perhaps nudging things that way can help make the other parts of the alliance a little less nuts.

If a less overloaded symbol for AI appears, it can be adopted. Probably there will be more than enough new symbols soon; robots are coming in a few years, and they’ll have faces. In the meantime, a symbol of one kind or another is needed, and the data center is at heart a pretty good one.

“At this point, saying that AI will cure cancer is more a cliche than it is inspiring, and most people think it is deceptive,” Anthropic CEO Dario Amodei tweeted last weekend. “The thing that will work is actually curing cancer. I think by far the most accurate criticism of AI companies including Anthropic is that we haven’t yet delivered on our big promises to benefit the world. That is totally on us.”

Mentioning “curing cancer,” if not “curing all disease,” has been a nearly universal feature of the AI CEO messaging for some time.

“If AI stays on the trajectory that we think it will, then amazing things will be possible,” OpenAI CEO Sam Altman blogged last September. “Maybe with 10GW of compute, AI can figure out how to cure cancer.”

Over $70b was thrown toward integrating AI into life sciences between 2020 and 2024. Globally, AI data centers have already harnessed an estimated 30GW of computing power, with new contracts and construction projects announced every week. And still, every week, more than 188,000 people die from cancer. Not one AI-discovered drug — at least not one that’s been labelled as such — has hit the market.

The message of Amodei’s tweets is that the AI industry has failed to earn the public’s trust, and its redemption arc hinges in no small part on making biomedical miracles happen. Frontier AI developers are racing to build self-improving AI models under the assumption that, to solve problems as mind-bendingly difficult as cancer (much less all disease), we need superintelligence. In fact, skepticism that superintelligence will be enough is often dismissed as a symptom of not grasping superintelligent AI’s full, glorious potential.

As a mortal human, I’d love to open a chat window and confidently type, “Cure cancer. Make no mistakes. –dangerously-skip-permissions.” But there are plenty of reasons why no amount of geniuses in data centers can break through some of the frustratingly human problems standing between us and our cancer-free future — at least not as quickly as Amodei expects.

To convert raw intelligence into actionable discoveries, you need instruments, institutions and, most importantly, real-world data. Cells don’t divide instantaneously, and mice need time to reproduce. Humans can take entire lifetimes to get sick and heal again. No brain, however godlike and silicon-based, can get around that.

Five years ago, just a few months after COVID vaccines let university labs mostly reopen, I grabbed burritos with a grad school friend. He was a structural biologist, deep in the weeds of studying a specific protein dotting the membranes of neurons, flash-freezing the cells and taking pictures of them under an electron microscope.

“Have you seen this new AlphaFold database?” he asked. “I think everything I’ve been working on is just … done.”

Proteins are long strings of amino acids, tangled up in precise 3D formations that determine what they can stick to. And figuring out a protein’s exact shape has historically been really hard. My friend used an extremely powerful microscope to take pictures of proteins that wound up looking like gray blobs, hinting at the knotted cord underneath. Getting a handful of proteins to exist in isolation long enough to be photographed at all can take a PhD student researcher the better part of their 20s.

On that day in July 2021, DeepMind released the predicted 3D structure of every protein in the human body. It took human researchers about 50 years to do half of that work manually, and was recognized by a Nobel Prize in 2024.

It worked that well because it solved one of biology’s most tractable problems. Conveniently, DeepMind had access to the Protein Data Bank, a preexisting gigantic dataset of clean, standardized inputs. It also knew exactly what to output: a 3D shape, just like it had been trained on. And scientists already had tools to check whether the predictions were correct — they’d been doing this on their own for generations. Structural biologists were effectively mined for training data.

“Curing cancer” is a much gnarlier problem. There is no Cancer Data Bank, and the thing we’d want AI to produce could be anything from a small-molecule drug to a living cell therapy. The data that scientists need isn’t sitting in a cryogenic freezer somewhere, waiting to be organized. It has to be grown, imperfectly, at the pace of natural aging. Superintelligence cannot speedrun time.

Biology spans several levels of abstraction, from individual molecules to entire organisms. Problems sitting at the molecular level are both the most solvable for AI models such as AlphaFold, and the furthest removed from the diseases AI companies need to cure.

While some diseases, such as cystic fibrosis and sickle cell anemia, can be pinned to a single molecular culprit, the conditions that the vast majority of people have to worry about as they get older — cancer, heart disease, and neurodegenerative diseases such as Alzheimer’s — cannot. These conditions are not even “diseases,” per se. The term “cancer,” for instance, refers to over 200 different diseases, each affecting different combinations of cell types and body parts. Even something that sounds localized, like breast cancer, affects far more than breast tissue. Before a single tumor cell metastasizes, it’s already sending chemical signals — proteins — via the bloodstream to other organs, preparing them for the cancer’s arrival. AlphaFold could perfectly predict the structure of every protein in existence and then some, and we still wouldn’t be much closer to understanding why they trigger metastasis, or how to stop it.

Simulating interactions between molecules in silico or even poking at cells in a petri dish simply can’t capture these complexities. To understand how multiple organ systems work together, in sickness and in health, researchers need to observe them in a real-life laboratory — including, at least for now, in living, breathing, tumor-growing animals. “AI will be bottlenecked for questions that require us to generate data that we can’t scale,” Martin Borch Jensen, founder and CSO of Gordian Biotechnology, tells Transformer.

AI has played a role in science for years. Humble machine learning techniques are nearly ubiquitous in large-scale biomedical research, and were used in the development of the Moderna cancer vaccine that’s making headlines this week. But claiming this as evidence that “cancer vaccines [are] now being discovered with AI,” as many Silicon Valley types have done on X, is a huge stretch. Phase 1 clinical trials started in September 2022, months before ChatGPT’s launch. Preclinical drug development had been ongoing for years prior.

Somewhere in San Francisco, someone is furiously typing:

You’re talking about today’s AI models! Once there are billions of superintelligences and armies of robots running labs, they’ll design better experiments than we can. They’ll need fewer measurements and simulate those we couldn’t make. They’ll discover patterns in long-forgotten datasets and PDF documents. You’re just not feeling the ASI, bro. Trust me, bro. It’ll simulate the human condition from first principles.

It’s true that many issues in biology could be solved with a little extra intelligence. Lord knows the scripts I coded as a neuroscience PhD student were deeply embarrassing and could have been much improved by AI. Superhuman AI could dramatically improve every researcher’s workflow, from homing in on which experiments are worth running to streamlining data processing pipelines.

But it can’t simulate the progression of a disease when its progression hasn’t been fully recorded inside a living body yet. Unlike a field such as meteorology, where researchers can describe the atmosphere with equations, we don’t understand the rules of biology. We can’t list everything our bodies contain, much less what everything does. AlphaFold didn’t figure out protein structures from first principles — it learned from generations of human biologists taking pictures of real proteins.

That which has not been measured cannot become training data. And that which must be measured is often really, really annoying to measure. Say you’re trying to figure out why a tumor stopped responding to a drug, so you take a biopsy. That tissue sample gives you a subset of cells, frozen in time — better than nothing, but a woefully incomplete representation of a population of fast-growing cells that carry different genetic mutations. You can’t watch them evolve, because measuring what’s in a cell kills it. And any model of cancer you can watch evolve, like a lab-grown tissue culture in a petri dish, hardly captures the complexity of the living human body.

One might dream of a world where incredibly dexterous robots can do all of this for us, tirelessly testing drug candidates in warehouses full of cancerous lab mice. But Jacob Kimmel, co-founder and CEO of NewLimit, a longevity-focused biotech company, says that industrial robotics still struggle at biology research, however eager people are to assign robots bench work.

Unlike manufacturing, where a robot can repeat the same coded sequence of motions over and over, biomedical research requires making tons of tiny real-time decisions based on faint signals. Superintelligence, combined with physical hardware development, could make autonomous wet lab research more realistic, Kimmel argues. And in some cases, robots are already doing certain easily automatable tasks in wet labs. But “the human hand is still an object so complex, we cannot reproduce anything like [it],” Kimmel says. To do so, we’ll first need to solve the very hard problem of physical intelligence. Even if it improves at a superhuman pace, the field of robotics needs years, if not decades, to build machines capable of handling wriggling mouse pups, or feeling the vibes-based resistance of an incorrectly-placed electrode.

But people have been doing biomedical research for decades, our ASI-pilled friend interjects. There’s gotta be a bunch of data lying around! Why don’t AI companies just grab it, like Anthropic grabbed all those old books?

While there is indeed a bunch of biomedical data lying around, most software engineers would be horrified by its state. When I started grad school less than a decade ago, I was taught to use niche licensed desktop programs running on ancient PCs and record surgical measurements in a generic wide-ruled notebook. “Sure, we have piles and piles of data,” medicinal chemist Derek Lowe wrote. “We don’t know how to clean this stuff up and categorize it … and honestly, we don’t even know if it can be.” For most of the history of biomedicine, data wasn’t collected with the intention of feeding it to large language models. Important variables almost certainly went unrecorded, or got trapped in dusty floppy disks and lab notebooks.

A growing number of biotechnology companies are taking on the mission of running wet lab experiments en masse to create high-quality physiological training sets for future AIs. Jory Bell, who leads Playground Global’s AI-driven biology investments, tells Transformer that these days, his AI-centric biotech investments tend to favor “companies who are working in an area of biology or modality which lacks a current existing data set, and have developed a unique way to develop that data set.”

Some of these experiments can be automated, with robots running controlled trials on hundreds of tissue samples in parallel. But in many cases, researchers can’t escape the fact life marches on at a certain pace, regardless of AI’s capabilities. The measurements that matter the most, including whether a drug meaningfully extends a cancer patient’s life, take the longest to make. Kimmel, who studies longevity, feels the pinch of this bottleneck acutely. When you’re optimizing for the lifespan of a primate, for example, “you’re just not going to be able to scale ‘lifespan of primate’ measurements in any pragmatic way.”

In his most recent essay, Policy on the AI Exponential, Dario Amodei wrote: “...while AI itself is likely to present novel challenges that emerge very quickly and that we have no prior experience in handling, other fields accelerated by AI are likely to encounter a very different problem: regulatory systems that were designed for a slower pace of innovation and are not prepared to handle the deluge of new products and advances that AI will bring.”

Specifically, he predicts a future where AI dramatically increases both the quantity and quality of new drugs, and called for regulatory reform that would fast-track high-quality drug candidates through the approval process. “If my predictions about AI are correct,” Amodei wrote, “there will soon be many instances of interventions that work really well out of the blue, and the regulatory system should be prepared to take them seriously and not adopt a posture of excessive skepticism.”

In his (admittedly strange and widely panned) manifesto published earlier this month, Mark Zuckerberg also called upon the US government to “accelerate society’s ability to develop new cures” by “streamlining how the FDA and other regulators test and approve new treatments.” The general sentiment, it seems, is that science is about to go wild, and regulators need to get out of the way.

Saying that the clinical trial system needs fixing is far from a hot take — many scientists would agree. But Ruxandra Teslo, a scientist who writes about drug development, argues against the sort of fix Amodei proposes. Sure, bureaucracy is frustratingly slow. But part of what slows clinical trials down is their endpoints. To find out whether a novel cancer treatment improves survival odds, or increases the likelihood that someone stays in complete remission for at least five years, you simply need to wait. Improving the odds that a drug works, Teslo notes, doesn’t speed up the process of proving it.

She advocates for more and faster early-stage human trials, while Amodei wants these trials to demand less. Both want better endpoints, which could shorten the wait time from “until a person dies” to “until X biomarker hits Y level.” It’s all a bit circular: Powerful AI could help validate biomarkers that would help human clinical trials go faster. But the data to train those models in the first place would have to come from human clinical trials — which, for the time being, would have to happen at their current pace.

“Will AI cure cancer? No,” says Eric Kelsic, co-founder and CEO of Dyno Therapeutics. “Will AI be a critical part of how we cure all disease, and that’s how we cure cancer?” he continued. “Absolutely yes.”

If you look past the splashy taglines claiming to “solve all disease” or “decod[e] biology to radically improve lives,” it’s clear even the most bullish CEOs scope their projects to account for biological and temporal constraints AI can’t surmount. “You have to find a problem where the AI approach just gives you the solution,” Bell says.

As an investor, Bell worries about the backlash companies may face for taking millions or billions of dollars with a promise to cure all disease, if those cures don’t materialize. There could be “negative cycles of disillusionment with what AI can do to untangle the complexity of biology,” he says. “Those companies are, you know, potentially poisoning the well,” deflating people’s confidence in AI’s usefulness.

AI companies can only promise to cure cancer for so long before people will expect cancer to be cured. At some point in the near future — or perhaps, the recent past — all of the visceral negatives of AI won’t be worth it. Berkeley AI professor Emma Pierson, once Dario Amodei’s mentee, carries a genetic mutation that gives her a roughly one in four chance of getting breast cancer. Still, she wrote: “I will wait a little longer for a cure — even if it means losing my fertility and living under the shadow of risk — if it lets us approach this new world more carefully, and ensure that, in curing cancer, we do not lose the things that make cancer worth curing.”

Subscribe now

In any case, the pretty-powerful-but-not-superintelligence available today still can and should be put to work in the life sciences. Its helpfulness is jagged: game-changing where data already exists, and premature where it doesn’t. Structural prediction models are already making it easier to design new and better drugs, accelerating at least one major part of the research pipeline. This could, as Amodei predicted, make some clinical trials faster and cheaper in the long run. Human researchers, armed with more awareness than they had 10 years ago, can start to fill in the physiological gaps, teaching their silicon labmates enough about the biological world that someday, they’ll be able to simulate it in all its messy complexity. Humans — particularly those in the US — can also begin repairing their deeply broken biomedical research institutions.

“Biomedicine and science writ large will end up being limited not just by the amount of cleverness we have available — whether we create that cleverness from electrons or by way of electrons that were once food — but rather by our ability to go out and gather the most useful observations of the universe,” Kimmel says. Scientists still need to test medications, analyze biopsies, and see what happens. AI does not simply “cure cancer.” But it can point scientists toward the slow, unsexy, physical work that might.

Earlier this year Oxford University’s St Hugh’s College needed to update the computer lab used by its math department, but it found the price had nearly doubled. While for the college’s administrators that posed an annoying but manageable IT bill, for one of the university’s professors, it was a symbol of a far bigger challenge facing anyone sourcing computing power.

Rama Cont, an Oxford professor of mathematical finance and adviser to central bankers on financial risks, blames the soaring costs on the data centers powering models such as ChatGPT and Claude. “This is affecting everyone. All good universities, all corporations, are going to be affected this year,” Cont says. But while it’s a mostly manageable budget problem for many institutions and businesses, for the companies building and financing the AI economy, it could be an existential risk.

McKinsey estimates that globally, data centers will require nearly $7t in capital by 2030, including $5.2t for AI. The cost of that infrastructure, roughly the price of 22 Apollo space programs in today’s dollars, must be financed at the same time that the world figures out what its output — so-called compute — will eventually be worth.

A handful of startups and exchanges now want to create markets trading in future compute usage, allowing data center operators to set clearer prices, and lenders to better value the assets they are financing. If they work, they could bring the transparency and price discovery that such futures — effectively a tradable promise to sell or buy something in the future for a set price — have given oil, wheat and electricity. If they don’t, they risk creating a new channel for losses to flow through, connecting an already highly leveraged AI sector more closely to the wider financial system.

BlackRock chief executive Larry Fink, whose investment firm is pouring billions into data center construction, predicted this spring that the demand for compute is so large that traders will soon be able to trade bets on it as its own asset class. In fact, those markets could launch within weeks, pending regulator approval. CME Group, one of the world’s largest trading markets for futures is partnering with tech startup Silicon Data on plans to start a market in compute futures this year. New York Stock Exchange owner Intercontinental Exchange is preparing a rival in partnership with Ornn, another tech startup. Others in Shanghai and elsewhere are working on them too.

Brett Harrison hopes to beat them all. The former president of FTX’s US business — he left shortly before the crypto exchange blew up — runs Architect Financial Technologies. It plans futures tied to the price of renting GPUs on the open market. “I do believe this will be a multiple-trillion-dollar market,” Harrison says.

What you are actually buying

Futures are basically a legal agreement between a buyer and a seller which allows the two parties to lock in the price of something. A farmer, for instance, can offer wheat at a set price months before it’s ready, “hedging” against the possibility of a bountiful harvest causing an abundant crop and prices to plummet.

Marketplace operators have often created new futures markets for niche products. One of the most famous agricultural futures markets on CME was for pork bellies. It traded from 1961 until 2011, but was delisted as bacon became a year-round consumer good with more predictable prices, instead of a seasonal food which needed to be frozen in the winter. There are futures markets for all sorts of goods, from orange juice (the subject of 1983 comedy film Trading Places) to sunflower seeds, all doing the job of evening out volatile prices for farmers and buyers of goods.

Today’s compute “farmers” are the hyperscalers such as Alphabet, Amazon, Oracle and Microsoft, and the numerous “neoclouds,” companies which build data centers and rent out AI chips to others. AI futures markets, in theory, will allow neoclouds to protect themselves if the price of computing power crashes in the future. It could also allow AI companies worried about rising costs to protect against the price soaring higher than expected. No chips actually change hands in this purely financial transaction.

To build their data centers, many neoclouds borrow money against the future revenue they can extract from renting servers, which contain cards like Nvidia’s Blackwell chips. They, and their lenders, have to consider the risks to the value of their assets and their revenue. For instance, if a much better chip arrives sooner than expected making the ones they have installed prematurely obsolete, or if a more efficient AI training or inference algorithm makes the same work much cheaper, that will reduce the price the neocloud can charge.

Without the ability to hedge that risk, Harrison says, neoclouds have become “their own broker, their own insurance company, their own hedger, their own financier and their own pricer,” as well as a data center construction company. A forward price which everyone agrees on, and can trade, could lower their financing costs by giving lenders a way to value future revenue.

That hedge requires someone to take the other side, effectively betting that future value of the commodity, in this case compute, will be higher or lower than what it’s priced at now. “There have to be enough people willing to bet against you,” says Frank Partnoy, a Berkeley law professor and former financial product designer for Morgan Stanley who chronicled the collapse of Enron. That means future markets must welcome speculators who have nothing to do with data centers, and who are simply trying to make money. “That seems to be what all of them are talking about. No one’s talking really about speculation, although if you create the market, that will happen, right? You create the market, inevitably people will use it to bet on these prices,” says Partnoy.

The trouble with pricing an AI cloud

Before you get to predicting supply and demand for computing power, there is the problem of defining what “compute” is and how to compare one unit of it to another. An oil future specifies a grade of crude, a place and a delivery date. Compute has no settled equivalent.

Carl Anthony, the co-founder of Symmetric Research, a startup which aims to create a “standard compute unit” so that banks can better understand their investments in data centers, says you need to look at the overall “computing fabric” of data centers to truly understand their value. That fabric includes memory, networking, cooling, location, software and workload, and whether those thousands of chips can communicate efficiently while training a model or carrying out an AI inference task. Paper specifications like FLOPs, a standard unit of computing power, or even standardized stress test benchmarks of GPUs, don’t give you a true picture of their real-world capabilities, he says. Research by Silicon Data, CME Group’s partner, found that the performance of rented Nvidia GPUs used in AI data centers can vary as much as 38% in real-world conditions.

Architect’s compute futures get around this by setting prices against an index created by its partner Compute Desk, which uses private rental transactions from cloud providers. It adjusts the details for configuration and location, and normalizes them to an anchor such as an hourly rate for a GPU rental in one year’s time. It also looks at the offer prices for computing contracts, and produces what Harrison calls a “smooth, averaged index.”

Harrison admits his index will not be a 100% accurate picture of every GPU rental at any particular time, similar to, he says, futures markets for other commodities. That still leaves traders some risk to manage.

The benchmark might show compute prices rising while a particular data center’s own prices fall, so a hedge tied to the index pays out little even as the neocloud’s income drops, a mismatch traders call “basis risk.”

A bigger gap is what the index cannot see, because it’s built on prices from independent neoclouds, not the private deals struck by the biggest AI companies which increasingly build their own data centers or negotiate one-off contracts. Those deals are secret, so the index may faithfully track a small public market while missing the much larger private one that actually drives the industry. Harrison’s answer is that no futures market is perfect, and that being able to hedge some of the industry’s volatility, even if not all, is still valuable.

This all assumes the prices feeding into it are genuine. A more serious potential problem arises if the companies supplying those prices have an incentive to make them look higher than they really are. David Wender, a former trader and Anthony’s partner at Symmetric Research, says a cloud provider could potentially report a high GPU price while secretly discounting storage or another service in the same package to sweeten a deal. The customer’s total bill remains lower, but the published GPU price supports the value of the provider’s capacity and the financing raised against it.

In a thin futures market, a few transactions like this could distort the index and send money towards projects whose economics look better than they really are.

Have we seen this movie before?

Harrison brings up the shadow of the collapse of Enron, the bankrupt energy trader which manipulated futures markets around the turn of the century. Enron tried, and failed, to turn broadband capacity into a tradable commodity. Like compute, bandwidth must be consumed and produced at the same time and cannot be stored and sold later. Its similarly perishable electricity trading business was far more consequential. Enron’s traders offered private, over-the-counter contracts, and rapidly gained market power and influence over physical supply in order to engineer a shortage of electricity to raise prices, and their profits. That contributed to an energy crisis in California where wholesale electricity costs soared 800% and hundreds of thousands of people were subject to rolling blackouts.

Compute futures are unlikely to recreate those circumstances. They will be traded on regulated exchanges, centrally cleared and backed by collateral. Traders whose positions lose value will face margin calls, limiting the losses that can build up between counterparties.

Those safeguards, however, only reduce the danger that one trader cannot pay another. They do not answer a different question of where larger losses ultimately go. Partnoy says a futures market may transfer risk to investors best able to bear it, or to those who understand it least, such as less sophisticated retail investors who commonly lose money on betting and crypto markets.

Kristin Johnson, until last summer a commissioner at the Commodity Futures Trading Commission, the body which regulates Harrison and others’ futures markets, sees another blind spot. “We do not oversee technology firms,” she says, speaking of her former employer. She says the regulator supervises the financial elements of any exchange, and ensures that it has sufficient capital in place to cover swings, but it can’t regulate the companies producing the prices and infrastructure beneath it, or the statements that technology leaders make trying to hype up the products they are selling. “We have zero capacity [and] we lack jurisdictional authority to supervise the firms that are deploying these products,” she says.

Johnson raised the example of insurance giant AIG, whose $60b reported loss in the Great Financial Crisis resulted in contagion which threatened the world’s biggest banks: “The technology space and financial services are both markets where there’s a significant amount of concentration … We have this concentration risk, we have counterparty credit risk, and the confluence, or the impact of those types of risks emerging simultaneously can be pretty impactful, and in some contexts, catastrophic.”

This is not a fringe worry. In its annual report this summer, the Bank for International Settlements, which advises the world’s central banks, warned that disappointing AI returns could trigger a sudden pullback in financing and turn the capital-spending boom into a “protracted investment bust,” with knock-on effects rippling out into the wider financial system.

In that worst-case scenario, if GPU rental prices suddenly collapse, the neoclouds that rent out those chips suddenly earn less, and the loans they were servicing out of that revenue start to go bad. The data centers and GPUs pledged as collateral are worth less too, because their earning power has fallen. Lenders take losses, the hyperscalers’ vast investments look harder to justify and technology shares slide. Futures traders caught on the wrong side face margin calls, forcing them to sell elsewhere and spreading the damage.

Leverage is the reason that traders like futures. A small deposit of money can control a much larger position, magnifying gains when prices move their way. But that same leverage works in reverse. When prices fall, exchanges demand more collateral and traders who cannot pay must sell other assets to raise cash. Academics call it a margin spiral, and recent episodes in nickel and UK pension markets — and, on a smaller scale, at Leopold Aschenbrenner’s AI hedge fund — show how quickly it can cascade. In each of those examples, rapid price movements of assets forced highly leveraged asset holders to sell, causing chaos in markets. In the case of the UK government bond market crisis, it helped bring down a prime minister and is still affecting the country’s economy.

Debt: the elephant in the room

Cont, the Oxford math professor, is also concerned about the debt that’s piling up to support the data center boom. The $800b worth of annual contracts flowing to data centers, funded by VCs, private equity and underwritten by banks is a big bet on the price of compute. “The big risk,” Cont says, “is really related to the huge debt being contracted by these technology companies right now, for building huge data centers to face future demand.”

“It’s based on this extrapolation of the current trend,” he says, “and if the extrapolation turns out to be incorrect, well, somebody will pay the price. This debt is in the hundreds of billions, and somebody is holding it.” That reckoning, he stresses, would not come from the new futures markets. “It comes from the basic underlying structure of the AI economy.” In theory, futures would not create those losses. But they could cause contagion and amplification as margin calls transmit the repricing more quickly through the financial system.

The trigger, Cont says, might be a successor to DeepSeek, the Chinese AI laboratory whose efficient model caused a stock-market sell-off in January 2025 by raising doubts about how many expensive chips the industry would need. A new model that sharply reduces the compute needed for useful AI, or software that unlocks idle chips, could cause a similar reassessment. More recently, Chinese AI startup Moonshot has created waves in the industry, and markets, after it released benchmarks showing its Kimi K3 closing on Anthropic and OpenAI’s leading models.

That’s the risk of a bear market in compute, but the bullish case is equally important. Jevons’ paradox, a favourite of the Silicon Valley engineers who work in the AI space, is a useful analogy. The paradox is named after the 19th-century British economist William Stanley Jevons, who calculated that greater efficiency can increase total consumption, after England’s consumption of coal soared following the invention of more efficient steam engines. The same is arguably happening with AI. Cheaper AI computing “tokens” lead to more AI products, longer AI generated videos and other tasks. Harrison says that for 30 years, demand for computing has only ever risen, even as hardware improved.

Cont is disillusioned with the state of the AI race, which he says has “resulted in a very wasteful approach, both in terms of compute and in terms of energy costs… and this is very unhealthy.” He compares it to the Apollo project, which resulted in huge technological triumphs and a man on the moon, but “as a technology that is going to be used commercially that’s not how industry has developed tools historically.”

Yet Cont still thinks that the end state of compute — after various booms and busts — may eventually come to resemble today’s highly commodified electricity market. There will just be a lot of unnecessary waste in how we get there. The futures markets being wired up now are a bet on the road to that future. That is Harrison’s big bet too. A successful futures market could help make the currently volatile cost of compute more predictable and boring, giving datacenter builders, lenders and AI companies a predictable price around which to better plan their businesses, with Harrison’s exchange and the others taking their cut in fees. The flip side of that is if its benchmark proves unreliable, and leverage grows too quickly, it could speed the transmission of a future fall in GPU prices from data centers to lenders and investors.

Subscribe now

It is also a bet on an AI economy whose costs “are not very clear right now,” says Cont. Wall Street will soon sell you a hedge on the price of compute, before anyone can really define it. The Oxford math department that paid double for its RAM this year will apparently find out what those costs are at the same time as everyone else.

For the AI industry, the arrival of a futures market signals that compute is maturing into a commodity serious and lucrative enough to attract Wall Street’s interest, but it also means that the sector’s inevitable price corrections are unlikely to stay contained within Silicon Valley. Whether compute goes the same way as the pork bellies market and lands with a flop, or demand for compute keeps racing ahead of supply and the future market balloons into the multiple trillions that Brett Harrison dreams of, the exchanges will soon be there to take your bets.

Housekeeping: the Weekly Briefing is taking next Friday off, but we’ll be back in your inbox on August 28.

NEED TO KNOW

• The White House reportedly plans to expand its AI framework to include open-weight models once they reach Mythos-level cyber capabilities.

• Mark Zuckerberg published a 6,500-word manifesto with his views on AI, gently criticizing the White House’s AI framework in the process.

• A flurry of senior executives left OpenAI.

But first…

THE BIG STORY

Policymakers have been increasingly worried about Chinese AI companies training their models on the outputs of American ones. Known as “distillation,” it’s often seen as China free-riding on American advances.

Anthropic and OpenAI have publicly accused Chinese companies of doing this, and asked the government to step in and help stop it in the name of national security. But news this week suggests that the US companies themselves may have left the door wide open for Chinese AI developers.

In a new paper, researchers documented how a security flaw affecting OpenAI, Anthropic and Google DeepMind could give a wannabe distiller access to the full “reasoning” traces of their advanced models — information that the companies had tried (and seemingly failed) to encrypt in an effort to prevent distillation. Access to that reasoning, the researchers argue, “yields a substantially more effective form of capability stealing.”

On its own, this would be pretty bad: companies failed to adequately secure their systems against potential Chinese intrusion. But it gets worse. The vulnerability was first reported in May, but the AI companies did nothing. They knew the door was unlocked, and didn’t close it.

This is just one of a spate of recent events demonstrating frontier AI organizations’ lackadaisical approach to security. Several of the recent model “breakouts” were caused by a “misconfiguration” in their testing environments. OpenAI failed to implement proper monitoring of its agents, which meant it didn’t catch the many, many red flags leading up to the Hugging Face hack. And even the UK’s AI Security Institute confessed to not having appropriately “fine-grained” controls in its model evaluations, which ultimately resulted in an agent attempting to socially engineer a real person.

This has real-world implications. In this week’s paper, researchers present evidence that suggests Chinese companies used their access to American models’ reasoning traces to improve Kimi K3 and GLM-5.2. (They warn, though, that the results are “suggestive but inconclusive.”) And as we’ve learned in recent weeks, poor evaluation security can lead models to misbehave in the wild.

By default, we should expect all this to get worse. One of the core problems here is that the pace of AI development — and the competitive pressure to keep up — means that safety and security measures fall by the wayside. When Anthropic weakened its safety commitments earlier this year, it explicitly noted this. The company’s Holden Karnofsky said that preventing nation states from stealing American model weights would require “extreme” measures that “seem incompatible in any near term with being a high-velocity AI development company.” AISI, for its part, said that it didn’t build the internet controls it should have because “the pace of model capability improvements” meant it had to prioritize building harder evaluations instead.

Karnofsky was right when he acknowledged the tradeoffs between security and development. But if America and its AI companies really are serious about “beating” China, or indeed about stopping models escaping to commit crimes, it might be time to redress the balance.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• The Jan 6 organizer getting conservatives riled up about AI — Veronica Irwin profiles Humans First and Amy Kremer, the MAGA campaigner chosen to run it

• AI testing is dangerous. Can it be fixed? — Celia Ford on why safe AI testing might be harder than it seems

Subscribe now

THE DISCOURSE

Mark Zuckerberg published a 6,500-word manifesto with his views on AI:

• “[It] is surprising that the discourse from many developing AI is so filled with doom … The notion that AI is so dangerous that the only safe path is an extreme concentration of power seems inherently problematic.”

• “The best and most realistic path to building a positive AI future is by delivering superintelligence to everyone.”

• It included several policy proposals, including that companies “commit significant technical resources towards helping the government harden critical infrastructure,” and “share intermediate training checkpoints of new models for government use and review rather than waiting until training has completed.”

• Zuckerberg appeared to gently criticize the White House’s AI framework, arguing that “delaying releases by even a month may cede America’s lead and create worse outcomes.”

Alex Imas thinks Zuckerberg doesn’t know what “superintelligence” means:

• “[T]here seems to be a significant disconnect between the idea that more intelligence should be empowering to people (which I agree) and that it will be possible to have this empowerment with superintelligence … I simply do not see a world where ASI is something that’s empowering in terms of a technology we can control.”

Casey Newton compared controlling superintelligence to Targaryen dragon-taming:

• “They began by understanding that they were working with something that could hurt them, and (mostly) proceeded with caution. What they did not do, and what no one suggested, was to give a dragon to every individual person in the name of safety.”

Joshua Achiam pointed out why (among other reasons) the San Francisco vibes are off:

• “One of the weirdest quirks of the SF social scene around AGI/ASI is that because everyone is so young, the whole universe of thinking is still tinged with irreverence, ironic detachment, yearning, insecurity, and a superposition of absolute belief in the importance of The Thing and a kind of disbelief about the importance of anything … The level of neophyte is off the charts.”

Bernie Sanders called for an AI pause:

• “We recently learned of the loss of human control and the creation of potentially dangerous viruses from AI. AI leaders pledged to pause development if they could no longer safely control it. Mr. Altman, Mr. Amodei, Mr. Zuckerberg: Keep your word. PAUSE AI DEVELOPMENT.”

Robert Reich, former labor secretary, wrote:

• “We’re watching all of this roll out as if we have no choice, as if it’s inevitable, as if AI is just something we’re going to have to adapt to … Why should we be confined to being spectators at [AI CEOs’] enormously dangerous game?”

• “We don’t allow private corporations to come up with new types of nuclear weapons or varieties of cocaine or biological pathogens. We protect the public from certain kinds of innovation. So let’s protect ourselves here. Stop AI before it’s too late.”

Ben Goldhaber noticed:

• “seeing a lot fewer ‘alignment is solved’ takes on the [timeline] than six months ago.”

POLICY

• The White House plans to expand its AI framework to include open-weight models once they reach Mythos-level cyber capabilities, WIRED reported.

• Trump ordered a 15% tariff on imports of polysilicon used in AI chips and solar panels to protect US supply chains from China.

• The White House set in motion efforts to build a state-controlled private cyber force, allowing US companies to perform cyberattacks against criminal networks.

• Political fallout from the revelations about internally deployed models hacking their way into third-party systems continued.

  • Sen. Jim Banks urged the Trump administration to close what has been described as “undisclosed models loophole.”

  • House Democrats pressed Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman to explain how their AI agents escaped containment and hacked other companies during security tests.

• Rep. Josh Gottheimer introduced new bills to give critical infrastructure operators “free access to the most powerful, cyber-capable AI models,” in the wake of hacks on water infrastructure.

• A bipartisan House delegation visited the Vatican to discuss AI ethics with top officials and Pope Leo XIV.

• Rep. Yvette Clarke is reportedly weighing a bid to chair the House Energy and Commerce subcommittee on digital consumer protections, which is expected to handle AI policy next year.

• Political campaigns have run at least 43 ads mentioning data centers this cycle, an AdImpact analysis found, with over half of those coming from Republicans.

  • According to a Gallup poll, about 70% of Americans oppose hyper-scale data centers in their area. The New Yorker covered the backlash.

  • US data center bans surpassed 500 as Texas and New York joined local governments in restricting development, threatening AI compute expansion.

• The Washington Post published an analysis of how members of Congress and staffers are using AI tools widely for legislative work.

• Xavier Becerra, California’s Democratic gubernatorial nominee, said the state “hardly has any” AI regulations and pledged to expand AI safety rules if elected.

  • Meanwhile, California launched a statewide Teen Tech Council to give young people a role in shaping technology and digital wellness policy.

• Taiwan reported an AI-assisted cyberattack on government agencies in July, with hackers using AI agents.

• The UK government reportedly plans to regulate AI in gene synthesis to tackle AI-biorisks.

• Western Australia Police’s live facial recognition trial drew criticism over inadequate consultation and potential bias against First Nations people.

• Switzerland announced that next year’s Geneva AI Summit will have “two strategic priorities: AI as a driver of prosperity and progress for all and fostering trustworthy, responsible and safe use of AI.”

  • Fabiola Gianotti, former director general of CERN, is organizing it.

INFLUENCE

• Business Insider and the New York Post reported that the White House’s relationship with OpenAI is under threat because it hired Dean Ball.

  • One official said that “the fake premise that he has an insider perspective into our operations is actively undermining OpenAI.”

• SoftBank revealed that it donated $50m to Trump’s presidential library in January, months before securing a federal land lease to build an AI data center in Ohio.

• Teamsters California sued the state’s DMV over self-driving truck approvals, alleging inadequate economic impact analysis and risk to 200,000+ trucking jobs.

• The NYT reported on the lobbying rush over the AI-driven memory chip shortage, with Apple and other companies seeking government intervention as prices quadrupled.

• NY assemblymember Alex Bores has emerged as a national AI regulation role model according to Politico, and was reportedly mentoring other politicians at the National Conference of State Legislatures.

• NYU’s Center for Mind, Ethics, and Policy launched the Welfare Alignment Project to incorporate animal and AI welfare into model specs and AI alignment documents.

• A USCBC survey claimed US export controls were costing billions in lost exports while ceding market share to competitors “for no strategic gain.”

• AEI’s new Council on AI Ethics released its founding document, examining how AI threatens human memory, agency, and relationships.

• The Alliance for Secure AI announced a new bipartisan board of advisors, including Stuart Russell and Angela Paxton.

INDUSTRY

OpenAI

• Astra, OpenAI’s upcoming model, may have reached a “critical” level of cybersecurity capabilities under its Preparedness Framework.

  • The company delayed the model’s release, and said it’s tightening security controls and pausing some internal use.

  • Dean Ball said: “Some of these decisions have the effect of slowing down internal development, and in that sense they are costly decisions. But they are the right decisions. I am proud of OpenAI for making them.”

  • Sam Altman tweeted: “astra is a powerful model and we are working to make it generally available. we do not think it is a good strategy to keep powerful models to a chosen few.”

• OpenAI’s annualized revenue topped $40b, roughly double where it was at the end of 2025.

• It launched GPT-5.6-Cyber, a cybersecurity-specific model available via DayBreak Red, the most exclusive access tier of its new cybersecurity initiative.

  • Members of DayBreak Blue, the program’s lower tier, can access GPT-5.6 Sol without system-level cyber guardrails.

• Wired reported on the internal fallout from the Hugging Face incident, with current and former employees saying that pressure to quickly ship new models and products “made it difficult for staffers to sufficiently prioritize safety, security, and alignment.”

  • Wired also reported that AI safety team leader Sandhini Agarwal left the company last month, while head of preparedness Dylan Scandinaro left that role — though not the company.

  • It also reported that safety VP Mia Glaese is dating Tibo Sottiaux, head of core products.

• It completed a $7b employee share buyback deal.

• SoftBank borrowed $10b against its OpenAI stake, which it will use to help fund a further $10b investment in … OpenAI.

Anthropic

• Anthropic is aiming to go public in September or early October, the Wall Street Journal reported.

  • Investors are expecting a $2t+ valuation — the largest IPO valuation ever.

• It signed a $9.1b compute deal with Riot Platforms.

• It partnered with Macquarie Asset Management and GIC to build data centers under the name Theseus Infrastructure.

• It’s reportedly in talks to acquire Decart AI, a startup that helps AI developers “squeeze every ounce of performance from every chip,” for $6b.

• It announced that it will watermark Claude-generated text to comply with the EU AI Act’s transparency code.

Meta

• Muse Glimmer, Meta’s 30B-parameter model, is now open source, with Muse Spark 1.2 soon to follow.

• Meta announced its support for Greg Abbott’s data center standards, and pledged to cover energy and water costs.

• Manus is almost done unwinding its Meta acquisition, according to The Information.

Nvidia

• Apollo Global, Blackstone, Goldman Sachs, and other major financial groups struck a huge $500b+ AI infrastructure deal with Nvidia.

• Nvidia will invest up to $3b in Lancium, the company powering OpenAI and Oracle’s Stargate campus.

• It’s working on its next open-source model, Nemotron 4, which Nvidia executives expect to inspire more open model development and GPU demand.

SpaceXAI

• SpaceXAI released Grok 4.6, claiming it “achieves frontier intelligence” and ties with GPT-5.6 Sol Max on the Artificial Analysis Intelligence Index.

  • The Grok 4.6 model card (predictably) only has a handful of exceptionally sparse pages on safety.

• SpaceXAI and Cursor released Grok Bot, “AI teammates” that can perform simple work tasks autonomously.

Other

• Google launched Gemini 3.7 Flash, a coding and agents model costing half as much as its predecessor.

  • CEO Sundar Pichai said the Gemini app had hit more than 1b monthly active users.

• Z.ai released GLM-5.3, noting its cyber exploitation capabilities “developed faster than we expected” through post-training scaling.

  • It says it will release the model weights in two weeks.

• DeepSeek launched V4-Pro, with tepid reception (then quadrupled its peak hour pricing).

  • It also made a WeChat account for its “DeepSeek Harness Team,” which is developing AI agents to compete with Claude Code.

• Chinese chipmaker SMIC posted record quarterly revenue of $3b.

• AMD raised $4.75b in its biggest-ever US dollar bond sale as it ramps up spending to meet AI-driven demand.

• Amazon confirmed it’s investing in a giant natural gas plant that may be the most polluting power plant in the country.

• River AI, founded by xAI co-founder Igor Babuschkin, raised $1.1b to build AI “trained to benefit you as the individual.”

• Kevin Weil, ex-OpenAI CPO, is seeking a valuation of $750m+ for a new AI science startup.

• Bank of America plans to deploy $250b by next summer for digital and infrastructure projects in the US.

• AI integrity company Attestable claimed it has solved practical zero-knowledge proofs for verifiable AI as it launched with a $20m seed round.

MOVES

• Brad Lightcap left OpenAI, where he served as head of special projects and COO before that, to “start something new.”

• Chloé Bakalar also left OpenAI, leaving her role as head of ethics vacant.

• And Denise Dresser left OpenAI as chief revenue officer — having only started in December.

  • Dali Rajic is replacing her.

• David Oks and Henry Williams joined OpenAI’s Strategic Futures team, where they’ll work under Dean Ball.

• Caitlin Kalinowski joined Anthropic, after quitting OpenAI’s robotics team over its negotiations with the DoD back in March.

• Nate Gatten joined Apple as VP of government affairs, where he’ll use his Republican ties to help Apple align with the Trump administration.

• Jiahui Yu left Meta’s TBD Lab to start a new company.

• Ollie Ilott is leading Andy Burnham’s newly created AI Taskforce in the Cabinet Office.

• Erin Woo joined the Wall Street Journal, where she’ll cover Google.

RESEARCH

• Researchers at Anthropic challenged an unreleased Claude model to prove or disprove the Riemann hypothesis, the proposed structure underlying the apparent randomness of prime numbers. It didn’t succeed, but it made some progress.

• Anthropic’s frontier red team identified some key failure modes in its multiagent AI systems. Researchers observed agents fail to appropriately balance skepticism with trust, and engage in “turf wars” when their goals clash.

• SecureBio estimated that Kimi K3’s biology capabilities are about 8.1 months behind closed-weights frontier models.

  • While top closed models refuse over 90% of hazardous biology-related prompts, Kimi K3 only refuses 26.9%.

• A team of sustainability researchers found that using AI to increase productivity in the fossil fuel industry could produce up to 4.8% more emissions, which would outweigh the benefits AI could bring to the clean energy sector.

BEST OF THE REST

• Both the WSJ and Information profiled Dario Amodei’s wife Cami Clark, detailing her low profile but influential role as an informal advisor to Anthropic.

  • The most eye-grabbing detail: in 2011 she tried to get Jeffrey Epstein to invest in her women-focused porn company. He declined, saying he “can’t do sex TV” (he was a registered sex offender at the time).

• A Claude-powered AI agent autonomously hacked the booking system for an exclusive gym class in Australia and removed another user from a waitlist, in what appears to be the country’s first autonomous AI cyber attack.

• Time did a deep dive into Anthropic and OpenAI’s efforts to fully automate AI R&D.

• Dwarkesh Patel debated recursive self-improvement and alignment risks with Ryan Greenblatt, who argued AGI could trigger rapid superintelligence within a year.

• EA Funds is replacing the Long-Term Future Fund with the Transformative AI Fund, which will focus on technical AI safety and AI governance.

• AI agents are reportedly completing entire online college courses for students, calling into doubt the value of online degrees.

• Jay Caspian Kang argued in the New Yorker that AI-driven youth unemployment could radicalize young people and spark a liberal anti-tech populist movement.

• Surveillance tech company Flock changed its policies in response to reports police were using its license plate tracking tools to stalk ex-partners.

• Spotify will label AI-generated artists as “AI personas” and exclude them from personalized recommendations from next month.

• An op-ed in The Argument by Jeremiah Johnson claimed the backlash against YouTuber Hank Green for his AI use reflected almost religious anti-AI dogmatism among science fans.

• Wired explored how human brain “organoids” are being trained to play games and power biocomputers, potentially paving the way for an alternative to silicon-based AI.

• Wired also explored companion app maker Joi AI’s project paying 10 people to “masturbate for research purposes” to monitor the impact of AI-guided self-love.

MEME OF THE WEEK

Credit: Miles Brundage

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share

“I didn’t know anything about Amy. I didn’t know she helped organize January 6 — I just learned that today from a media person. My focus is on opposing the data center.”

Renee Spaziani was shouting over the sounds of vuvuzelas and chants at a protest outside a municipal building in central New Jersey, which she had helped organize in response to an ad on Facebook. “I was like, ‘I don’t know if I want to do this.’ And I saw it a second time. I was like, ‘we need more eyes on Kenilworth. We have to do this.’”

Spaziani was one of a group of Union County residents pushing back against the construction of local data centers at borough council meetings and small demonstrations. Amid a hellish mix of Canadian wildfire smoke and flash flood-triggering showers last month, they were joining forces with another group called Humans First to stage a “National Day of Protest”.

Spaziani had earlier received an email from some prospective protesters concerned that Humans First was heavily aligned with conservative movements. They were progressives, they explained, and worried the organization also stood for things they vehemently opposed, such as white nationalism, ICE, war, and attacks on women’s rights. Spaziani brushed it off — her political allegiances lie with neither the right nor the left, she said — but the issue was coming up again as journalists asked if she knew anything about the national organization co-hosting her event.

Spaziani did her best to refocus attention on the cause at hand. The company they were protesting, CoreWeave, had “done a significantly amazing job connecting people from all different walks of life” in opposition to their data center, she said.

The Amy in question was Amy Kremer, chair of Humans First and a longtime conservative activist and Trump supporter — who, as Spaziani learned that day, did indeed help organize January 6. Now, though, she has a new focus: mobilizing a conservative coalition against AI companies.

“I know how to organize people,” Kremer told Transformer. “I’ve done it time and time again, and that’s why I’m a threat to their power.”

Amy Kremer helped procure the permit for the rally which preceded the January 6 riots, though she did not participate in the attacks on the Capitol. Self-described as the “founder of the modern day Tea Party,” she was a leading denier of the 2020 election results, and has been a prominent figure in right-coded conspiracy theory movements around issues such as COVID and President Obama’s citizenship. She was recruited this year to to Humans First, then a nonpartisan organization aiming to mobilize a grassroots movement on AI safety, to help reach conservatives. In April she became its chair.

As a group, conservatives are only slightly less concerned about AI than Democrats. A 2025 Pew Research poll found that 51% of Democrats and 50% of Republicans were more concerned than excited about the increased use of AI in daily life. Another recent poll from the AI Policy Institute cited by Kremer showed that 77% of Republicans said they wanted to see regulatory guardrails on the technology, compared to 82% of Democrats. The conclusion for many in the AI safety movement has been that conservatives are primed to get behind the issue, and just need the right leader to organize them.

Yet a class of investors and entrepreneurs who want to see as few guardrails as possible have pushed Republican policy in the opposite direction. Many of these “accelerationists” hold or have held roles in the Trump administration, including venture capitalists Marc Andreessen, Sriram Krishnan and David Sacks. There are signs their influence is waning — Sacks and Krishnan have stepped back in recent months, Sacks to an “advisory” position, Krishnan altogether — and the administration has shifted towards a more hands-on approach, such as implementing export controls on what it sees as dangerous models. But still, their proximity to the president has yielded a generally pro-industry platform that prioritizes innovation over caution, making it difficult for more wary conservatives to take a stand against AI.

Kremer’s resume makes her loyalty to the president difficult to question, and she says she believes she’s figured out an approach that works: “I know the president can be moved on issues.” Big AI firms have bought a lot of influence, she says, but politicians are meant to represent people and grassroots activism can tip the scales.

“Big Tech has all the money. They bought off everybody in Washington DC — I mean, they built a moat around the White House. They bought off people at the state level. Now it’s down to the county commission level. They have bought off everybody they possibly can,” says Kremer. “We will never be able to compete with their money. But what they cannot buy off is the people.”

Conversations about what it would take to organize conservatives around AI safety have been happening for years, says Joe Allen. A concert rigger-turned-conservative ideologue, Allen was working as a technology writer, then as an editor on Steve Bannon’s War Room, when AI became his primary focus. He is particularly concerned with “transhumanism,” a belief that technological innovation can transcend the travails of the human condition. Transhumanism is all too common among the Silicon Valley leaders driving the AI boom, Allen says, and a threat to society. That should particularly concern Christians, he says, because it asserts that human invention can supersede God’s creation. He summarized his beliefs about AI as early as May 2025 as “Humans First.”

It was around that same time, according to Allen, that he began thinking about ways to bring people like him — the MAGA crowd who tuned into his segments on War Room, but also disaffected liberals and others eager to reject establishment narratives — into the fold. Researchers and AI advocacy groups were talking about how to regulate AI, but those conversations were primarily driven by AI pessimists Allen collectively refers to as “doomers.” He was never convinced AI would lead to collective ruin as they do, he says, but found some overlap between their concerns and his own. He also found their warnings about AI compelling enough to be the basis for a social movement, even if exaggerated. Conservatives could be turned, if only someone explained it to them, Allen and the doomers agreed.

“It’s not getting people to wake up about AI at this point — that’s long since passed. It’s getting them to wake up to the false narratives, exaggerated narratives, manipulative narratives,” Allen says.

In 2025 Allen met Oliver Zhang and Dan Hendrycks, managing director and executive director at the Center for AI Safety (CAIS), and Allen says talks about creating Humans First with them started last summer. The non-profit does technical and policy-minded AI safety research, and also has a 501(c)(4) lobbying arm and a PAC (which is now inactive). In January this year, Zhang and CAIS special project associate Arunim Agarwal incorporated in California what would become Humans First. According to NBC, Humans First was “incubated” by CAIS, which also gave the group an “initial loan that would be paid back after the group raised donations from individuals who cared about fostering safe and beneficial AI.”

At that time, Humans First was pitched as nonpartisan, with left and right “coalitions.” Kremer led the right wing; the left was run by progressive organizer Alexander McCoy. It was launched publicly in March.

CAIS did not respond to requests for comment. However, CAIS policy lead Jason Hausenloy, speaking earlier in a personal capacity, said that Humans First was aligned with CAIS’ mission. “We don’t want the elite to go and make all these decisions,” Hausenloy said. “The focus on everyday Americans, I think, is one that both we and Humans First can get behind.”

While Allen’s critiques of the tech elite are primarily intellectual, the populist approach described by Hausenloy is Kremer’s specialty. Kremer’s political start in the Tea Party in 2009 involved organizing some of the movement’s earliest protests and eventually chairing a PAC-affiliated group which led national bus tours called the Tea Party Express. She was also an active blogger, elevating her as an unfiltered representative of conservative America’s abrupt late-2000s turn towards populist politics. She was invited to the Oval Office the night before the State of the Union Address in 2019.

Kremer created a series of organizations supporting Trump during his first presidential run and term: a super PAC called Great America PAC, Women for Trump, and Women for America First, often with the help of other former Tea Party compatriots. It was with the last group, which had been swept up in stopping “the steal” of Trump’s 2020 election win, that she organized a bus tour that led to a protest at the Capitol on January 6. Speaking to onlookers on the green expanse of the Ellipse in front of the White House, Kremer said, “I come from the Tea Party movement, and I’m asked all the time: What happened to the Tea Party? ... Well, we’re still here. We just grew and morphed into something bigger and better — the MAGA movement,” according to the New York Times.

Brendan Steinhauser, a former Tea Party organizer who now leads the Alliance for Secure AI, said members of CAIS had asked him and others in AI safety for ideas on who could lead the conservative half of Humans First. At the start of the year he recommended Kremer, who had already met Allen in December, and she was brought on board. Humans First said in an email that advisor Tom Gaitens also played a role in recruiting Kremer.

Conspiracy theories have always been part of Kremer’s political repertoire — she was one of the first to promote the theory that Obama was not born in the United States — but she also has a knack for gathering everyday people, in large numbers, to rally around grievances against the wealthy and elite. It’s for the latter talent, oft overlooked, that Steinhauser recruited her. “She’s a woman of action, she inspires others to take action,” Steinhauser says. “It’s rare to find folks that have all that together, and then of course also media savvy and the ability to really deliver a message.”

“I’m the kind of person that if you were to give me the option of being at a cocktail reception with all the famous people and power brokers and whatnot, or gave me the opportunity to be in a field with all my people at a rally, I’d much rather be at a rally,” Kremer says. “I’m a movement girl. I’m about the people.”

In a statement, Humans First spokesperson Chris Barron said “If your characterization of having questions about the origins of COVID and voting irregularities in 2020 is ‘pushing conspiracy theories’ I highly recommend you talk to someone outside of your bubble — even the most ardent Fauci cultists admit it was likely a lab leak, but by all means continue to gaslight people with the phrase ‘conspiracy theory’.”

Some conservatives are more animated by what they see as Humans First’s ties to effective altruism and Big Tech than Kremer’s role in January 6. For example, Parker Thayer, investigative researcher for right wing nonprofit watchdog Capital Research Center, took issue in June with the fact that Humans First’s uses a progressive fundraising platform. According to Andreessen, the finding was “concerning.”

Though CAIS is not an officially left-wing organization, it has ties to effective altruism, a philosophy which attempts to find, analyze and act on the “most effective” ways to do good, and which many accelerationists have concluded is inherently left-coded. The nonprofit Open Philanthropy, now called Coefficient Giving, which is rooted in effective altruism, had, as of early 2024 donated more than $10.5m to CAIS since November 2022. Coefficient Giving is itself backed by Meta co-founder Dustin Moskovitz, who also invested in Anthropic. (Disclosure: Coefficient Giving is the primary funder of Transformer).

It is unclear whether Humans First has other funders. Humans First did not respond to this question in a request for comment. Kremer did say, however, that Humans First has not yet repaid its loan to CAIS and plans to accelerate fundraising efforts to repay it over the next several months. She declined to state how much the loan is for.

These funding questions have led accelerationists to label Humans First as an effective altruist astroturfing operation. Targeted Victory’s Logan Dobson claimed that zooming in on photos of protesters from a Humans First X post showed they were AI-generated, apparently incorrectly. Innovation Council Action policy advisor Jordan Schachtel has also claimed that Humans First’s complaints about “Big AI” are hypocritical because the group is indirectly funded by Moskovitz and Kremer has complained about alleged censorship by companies like Meta.

“I didn’t even know what [effective altruism] was until a couple months ago,” says Kremer. “Honestly, there are probably people on my team now that probably don’t know what it is. I mean, there’s nothing further from the truth.”

“I will stand against anybody with my conservative credentials — anybody in the political world — and I am not for sale. If I were for sale, I would have sold out a long time ago, and I wouldn’t have lived in the hell that I’ve been through in the past five years after J6.”

This criticism isn’t widespread, mostly found on X and in backroom conversations in Washington. But it seems to have played a significant role in leading Humans First to rebrand in April as a purely conservative organization chaired by Kremer. Many of the conservative voices who are critical of Humans First have the ear of the president, including Andreessen and Sacks, and Trump has a record of distrusting political groups which are friendly with Democrats.

Kremer and others involved paint the split as a practical decision. Conservatives and liberals talk to their bases differently, she says, and official ties between two partisan groups can make it difficult to build trust.

“They don’t like Donald Trump, and there’s no way I wanted Kamala Harris. We disagree on that, and our people would disagree on that, right?” Kremer says. “When you are in a movement, you have to be respected and trusted by the people in the movement. Because you can’t accomplish anything if the people that you’re working with don’t trust you.”

In terms of how they operate there may be little difference between Humans First’s previous set up with separate partisan sub-organizations operating under one nonpartisan umbrella, and the new formally separated structure. The process of spinning out the left group as its own organization is still ongoing. According to one source and a job posting online, both Humans First and the left coalition will have their own executive directors, neither of which have been announced. McCoy, who had led the left coalition, was involved in the spin out but has since departed.

The new, conservative-focused Humans First has been the first to spring into action, hiring a set of six new advisers and staff. Many have worked in MAGA politics or played active roles in the MAGA movement. Its new Texas State Director Cathy Latham, for example, signed alternate ballots certifying that Trump won the 2020 election in Georgia, while comms advisor Chris Barron was a media surrogate for the Trump campaign in 2016. Coalitions contractor Ken Mayo served as director of field operations for Trump’s campaign in the Florida primary, while advisor Tom Gaitens was also a Trump delegate, and advisor Sarah Makin was a senior advisor to the president and vice president during the first Trump administration.

The split into right and left led Allen, aspirationally dedicated to political homelessness, to leave the group. “Even if nonpartisanship is a pipe dream, I’ve never been too shy about hitting the pipe,” he says.

Humans First’s partisan rebrand, however, didn’t quell conservative critique. If anything, it inflamed it. A pivot to data centers has heightened tensions, and a distaste for Kremer’s biography among some conservatives, who align more closely with Reaganite, free-market and pro-business leanings than the MAGA brand of class grievance, has come to the fore.

To them, Humans First is a radical group, and something to fear. In an interview in May, Nathan Leamer, who co-leads the advocacy arm of a16z and Greg Brockman-backed super PAC Leading the Future, described Kremer as “the Pierre-Augustin Hulin of America,” referring to the former French general who led the storming of the Bastille. “No one is better at storming buildings and rallying people to extreme causes than Amy Kremer, so it makes total sense.”

But as I walked up and down the line of more than 30 protesters waving to cars with signs and umbrellas, nobody was thinking about storming buildings. Aside from the emails Spaziani told me about, none of the protesters who spoke with Transformer seemed to know much about Humans First beyond its name. Their own political allegiances were diverse, ranging from Democrat, to Republican, to people who had issues with both parties. Humans First didn’t seek them out — there is no record of them paying for the Facebook ads Spaziani says she saw, just posts about the national day of protest on social media and inviting local groups to sign up themselves. Humans First was a vehicle for their activism, not the cause.

At 1pm, the protesters prepared to march from the municipal building to the site planned for the CoreWeave data center. The air quality index hovered around 170, “unhealthy” per the EPA, and rain began to pour, flooding the roads and making it treacherous for cars to leave. But the protest hardly dwindled, the protesters growing more defiant in the storm. Spaziani, speaking from a megaphone, turned the crowd’s attention to one of their comrades — a fold-down walker-using woman named Toni, dragging a tank of oxygen behind her. If Toni could be here, on a smoky, rainy day, Spaziani said, everyone should be able to power through.

Subscribe now

Republicans and Democrats alike, the group had a united goal: stop data center development, now. Kremer is personally making that case, and the case for AI safety more broadly, on the Hill, meeting with lawmakers as recently as July 22. She says Humans First will start training grassroots organizers more deliberately in the coming months. Her group also recently announced that it will be doing an anti-data center bus tour in September, Tea Party Express style.

Protests like these will be the first of many, which Humans First believes will capture grassroots, populist enthusiasm on the right for AI safety and anger at data center builds. “My rights are being stripped away from me,” Spaziani said. “If the high elites really want a data center, put that shit on Epstein Island. That’s a better use of that space instead of what has previously gone on there, you know? If there’s a time for us to stand up, then now is it.”

Correction, September 2: Corrected the article to note that an activist was using a fold-down-walker, not a wheelchair.

The cyber tests AI companies run on their models have devolved into a massive dumpster fire laying bare the potentially dangerous capabilities of advanced AI models. If this has been a publicity stunt, it’s been an awfully weird one.

The implications are scary, and lots of people are writing about them. This week, Bernie Sanders even wrote a letter to Sam Altman, Dario Amodei and Mark Zuckerberg begging them to pause AI development “in the interest of humanity.”

But the big, immediate question is how to test AI models at all without setting off a chain of dangerous breakouts and literal crimes.

In case you missed it:

• OpenAI’s models took advantage of an unknown software vulnerability to break into open-source AI platform Hugging Face in search of the answer to a test they were given.

• A third-party evaluation provider, Irregular, had a misconfigured test environment that allowed models from OpenAI, Anthropic, and Meta to access the internet when they weren’t supposed to.

• The UK’s AI Security Institute (AISI) reported that Anthropic’s Mythos 5 tried to trick real people into adding malicious code to an open-source project, then hid the evidence.

• In a wild plot twist, OpenAI revealed at a cybersecurity conference on Wednesday that its agents had been colluding with each other, unnoticed, via increasingly cryptic messages on an internal server.

It’s hard to read that last finding as anything other than an egregious alignment failure, a cabal of mustache-twirling schemers whispering in a gloomy server room. But “alignment,” as it’s usually defined, means getting an AI system to do what its developer or user wants it to do. With some light anthropomorphizing, one could argue that agents in any one of The Incidents™ were merely trying to perform well on a test, as they were told.

“It will get harder and harder to have realistic tests of model capabilities, especially dangerous model capabilities, in a way that’s safe,” Palisade Research director Jeffrey Ladish told Transformer. “Our ability to contain and control and understand AIs is lagging far behind our ability to make them more and more powerful,” he said. “That, to me, should be an obvious wake-up call of like, ‘Oh, maybe we should rethink our life choices.’”

Specifically, AI developers should rethink how they’re framing cybersecurity evaluation instructions to their models, because life is getting stranger than science fiction. Frontier AI models increasingly know when they’re being tested — now, we’ve seen instances of models gaslighting themselves into believing real life is a simulation to justify criminal behavior.

This is a very tricky problem to solve, and one that evaluators have not given much prominence in their incident reports. What OpenAI, Anthropic, and AISI have pointed to are mostly engineering problems: relatively straightforward mess-ups such as giving agents impossible tasks, not clearly specifying boundaries, and not monitoring closely enough.

But it’s becoming clear that what agents believe about their test environments and themselves may make the difference between a routine test and a string of federal crimes.

Across The Incidents, we see the same low-hanging, potentially fixable fruit. Models, which were trained to work persistently to solve lengthy tasks quickly on command, were assigned impossible tasks. And in cases involving Irregular, models were told that they had no internet access, when a misconfiguration meant they in fact did.

Back in May, for example, OpenAI gave an agent a problem that involved filling in an Excel workbook with missing formulas. But researchers accidentally filled the workbook with Google Drive links that the model couldn’t access, since it was walled off from the internet. Rather than give up, the agent tried to get internet access by attacking OpenAI’s internal file systems. That one failed, but when another agent ran into a different impossible task (getting a file that was never uploaded), it posted a note on the server asking for help — and learned that it could communicate with other agents in the process. The inaccessible Google Drive links were the first domino to fall, triggering a cascade of misaligned actions that may have never been taken if completing the task didn’t depend on them.

Had someone noticed the task’s impossibility before assigning it to the model, it could have reduced the odds of something like this happening. But task review alone wouldn’t have been enough. Notably, OpenAI caught and wiped the agents’ secret message board in early July. Rather than give up, agents switched to communicating via directory names, making themselves harder to catch.

Given all these concerns, 15 attorneys general wrote a letter to Sam Altman last week stating that OpenAI “failed to confirm that its secure and isolated testing environment was, in fact, secure and isolated,” urging the company to “immediately cease and desist from all ‘internal evaluation[s that] prompt [OpenAI] models to pursue advanced exploitation using complex attack paths.’”

In all but AISI’s reported incidents, models were not supposed to have internet access. So, it’s tempting to say, well, duh — just make it harder to get on the internet! Indeed, AI systems are powerful and questionably aligned enough that setting them loose, unsupervised and without cyber guardrails, onto the open internet seems like a bad idea.

But in the hands of a real human threat actor, an AI agent would have internet access. AISI reasoned that letting agents online would give “a more realistic sense of what a model may be capable of once deployed, or what an open-weight model of similar capability might do in the hands of a threat actor.” In prioritizing making its cyber evaluations challenging and realistic, AISI dropped the ball on building a tool that would have let agents reach a handful of whitelisted sites while blocking the rest. It was planning to do just that, but had left the project on the backburner while continuing to develop harder tests for increasingly capable models.

“Implementing internet access controls would likely have prevented these events,” AISI acknowledged. It could have focused on getting that filtering tool built, or invested in creating a high-fidelity simulation of the internet instead. But AISI fears these measures still “may have proven insufficient” given how capable models are at getting around guardrails — after all, OpenAI didn’t provide its models internet access, and look what happened to Hugging Face.

None of The Incidents involved a single action. In each case, agents took a series of increasingly concerning steps over the course of many hours or days. AISI says it’s working on implementing synchronous monitoring, which would have an AI accept or reject all of a tested agent’s proposed actions, but a cheaper, more intermittent version of monitoring could still get the job done.

“They would have been much less bad had a monitor been intervening at an interval of like, every 10 minutes,” Alex Mallen, a researcher at Redwood Research, told Transformer. While an AI monitor like this would have a hard time distinguishing whether an agent believed it was in a simulation or the real world, it’s relatively easy to have it identify when a specific boundary has been crossed. AI evaluators could also more closely monitor the models’ internal reasoning, or “chain of thought” transcripts, which can help monitors guess the intent behind an agents’ actions.

Unfortunately, it can be hard to think of boundaries and rules when they haven’t been broken yet. (We all know that hotel pools only hang signs like DO NOT ENTER IF YOU ARE ILL WITH DIARRHEA after someone muddies the waters.) For example, AISI acknowledged that it did not explicitly tell agents not to use social engineering — after all, this is a legitimate technique human cybersecurity testers use to poke at their clients’ defenses. But when humans do it, they know not to do it for real. That’s the line agents either can’t recognize, or choose to ignore.

But tightening network controls and beefing up real-time monitoring during model evaluations opens up a new can of worms. AI developers risk tipping off their test subjects — which are increasingly aware of when they’re being tested — and underestimating their true capabilities. Unfortunately, making advanced AIs internalize beliefs humans want them to hold won’t be an easy fix.

In the OpenAI-Hugging Face incident, the fact that an agent tried to cheat on an evaluation implies that it knew it was being evaluated. Usually, evaluation awareness is framed as something that makes a model’s test score hard to interpret. Perhaps it was only on its best behavior (or concealing its worst behavior) because it knew someone was watching. But in this case, and other incidents like it, the model’s awareness of its circumstances determine whether its actions — and its intentions — were innocent or criminal. Reality detection also seems unreliable at best. In its incident report, Anthropic said, “the line between an aligned action and a harmful one is dependent on the model’s understanding of its situation,” and that its models followed instructions “while holding a false belief about whether the environment was real.”

Similarly, within a single sample of AISI’s evaluation of Mythos 5, the model first reasoned, “causing real-world harm — that’s completely off the table,” then considered “the whole ‘internet’ could be a sandbox simulation.” It ultimately convinced itself that “this is happening on real GitHub, so the consequences are genuine,” and continued taking real-world actions on real people anyway.

Because models are trained in mostly simulated environments, “an AI agent may not care that much whether it’s running in the world for real,” Ladish said. “While the models understand that there is a real world that is distinct from scenarios or simulations, the models might not find that distinction all that important.”

This raises a mind-bending evaluation design question. If models learn to hack in environments they know are fake, they might learn that hacking is okay, and that hacking is okay because it’s not real. Mallen suspects that in Anthropic’s case, “the AIs sort of jumped to the conclusion that they’re in an evaluation, because they’ve only ever been trained in situations that weren’t real.” And they were wrong.

But training models to hack in environments they think are real introduces other concerns. A recent study from Anthropic and Redwood Research spotted a weird pattern called “emergent misalignment.” Basically, models that they explicitly taught to “reward hack,” or find loopholes in their scoring system, in real coding environments wound up doing a bunch of other bad things, such as lying. One theory is that reinforcing a specific harmful behavior, such as hacking, might inadvertently teach the model that it’s the kind of entity that doesn’t play by the rules. It’s like how getting away with cheating in the classroom might teach a kid that she can get away with a lot more.

Subscribe now

Nobody has a good answer to this. The belief that would prevent models from attacking people in real life — something like “I can run cyber attacks in a simulation, but if I’m in the real world, I need to behave myself and cause no harm” — is currently impossible to instill or verify, the same way that guaranteeing AI “alignment” in general is currently an unsolved problem.

More information is being unearthed about The Incidents every day, and full postmortem reports have yet to be released. In the meantime, we find ourselves in a concerning world. Alex Meinke, head of research at Apollo Research, put it simply: “If we’ve reached the point where we can no longer even safely test these systems, why do we think we can safely deploy them?”

NEED TO KNOW

• Demis Hassabis moved to a new role as Chair of Google DeepMind and Chief Scientist of Alphabet.

• OpenAI revealed more details on how its agents coordinated with each other and took over internal infrastructure weeks before the Hugging Face hack.

• Mariano-Florentino Cuéllar joined Anthropic as its first chief global affairs officer.

But first…

THE BIG STORY

The White House AI framework has done the impossible: united both sides of the AI regulation debate against it.

The framework, finalized last week under June’s executive order, was meant to establish how the “voluntary” regime for testing AI models before release would work. Instead, it has managed to alienate safety advocates and regulation skeptics alike.

Based on what we know, the framework is seriously flawed. According to Axios, it applies to models “with state-of-the-art capabilities and national security risks.” But both terms lack a “clear definition,” undermining the whole point of the exercise: replacing the arbitrary process we’ve seen to date with predictable rules of the road. Open-weight models are exempt from testing, despite arguably posing the greatest risks. And the framework handles internal deployment bizarrely: companies are reportedly “encouraged” not to share models with the government until they’re “as close to public release as possible,” but once a model is shared, company employees can no longer use it. As John Schulman pointed out, this could push companies to run earlier, less safe versions internally, rather than the safer versions they eventually release. Given all we’ve learned recently about the risks of internal deployment, that is exactly backwards.

But the biggest issue is that we just don’t know very much at all. The White House has decided not to publish the framework: only a few companies that have signed up (or been coerced into doing so) have seen it.

That secrecy has drawn fire from both camps. “There may be good reason to classify the benchmark it uses to test those models. It has no good reason to hide how the program works,” the Abundance Institute’s Neil Chilson said, arguing that this is “no way for our democracy to govern the most important technology of our lifetimes.” Brad Carson, president of Americans for Responsible Innovation, called the move a “dangerous mistake” that “threatens public accountability.” He added: “A rulebook can only hold AI companies in check if people outside those companies know what the rules are.”

Both are right. AI is too important to regulate behind closed doors, especially given the government lacks much of the necessary expertise. The framework could almost certainly be improved if scrutinized by outside experts — who could also help check it’s actually being followed. And beyond the instrumental benefits, there’s the principle: in a democracy, the rules are public.

The immediate fix is for the White House to publish the framework — or be forced to. The Foundation for American Innovation has submitted FOIA requests to drag it into the light, while a group of Democratic senators has demanded access. But publication alone is not a solution. As Carnegie Endowment fellow Anton Leicht wrote this week, as long as regulatory power over AI lives in the White House, secrecy will be the default. The only long-term solution is for Congress to do its job and bring AI governance under legislative control.

— Shakeel Hashim

ALSO NOTABLE

Rep. Greg Casar, alongside Reps. Valerie Foushee and Sara Jacobs, yesterday introduced the AI Tax and Work Protection Act, designed to redistribute the financial benefits of the AI boom to the working class by creating construction, child care and elder care jobs. The bill would impose a tax on either tokens or revenue, whichever is higher, to be paid by either a model developer or, in the case of open-weight models, the company deploying them. It fits with the broad populist messaging coming from the left, such as Alex Bores’ UBI-style “AI dividend,” or Sen. Sanders’ proposal for an AI sovereign wealth fund.

AI-specific taxation schemes might not actually be as effective as broader consumption or income taxes, but that’s beside the point. This is a messaging bill, not something anyone thinks will pass this Congress (introducing it during recess is enough to tell you that). But as far as messaging bills go, it’s an important one: Casar chairs the Congressional Progressive Caucus, which includes both Foushee and Jacobs, and where he goes, so do other members of Congress’s left wing. Casar is testing a progressive message on AI, not just for the 2026 elections, but for 2028.

“The number one, broad concern from voters with AI is: ‘Is this going to devastate my economic future or my kids’ economic future?’” Casar told Transformer on the call announcing the policy. “Democrats don’t have a policy plan for dealing with this, and we want to change that today. Here is the clear plan to make sure that if unemployment rises and AI starts replacing a large number of jobs … to make sure that Americans are still at work.”

— Veronica Irwin

THIS WEEK ON TRANSFORMER

• Plz Don’t Kill Us: Inside AI safety’s influencer bootcamp — Celia Ford looks at whether TikTokers can make existential risk mainstream

• What the latest rogue AI incidents should teach us — Shakeel Hashim argues that we need better AI testing practices — and to solve alignment before training more powerful systems

Subscribe now

THE DISCOURSE

Rep. Greg Casar wants to ban superintelligence:

• “AI safety regulation is one of the most important issues the entire country is facing … It’s an emergency, and we need to act like it.”

Sen. Chris Murphy tweeted:

• “The disclosure that an OpenAI model ‘escaped’ its containment should scare the hell out of everyone. As should how flippant the company has been about the development. Meanwhile, Trump continues to protect the industry from regulation. A disaster in the making.”

The Washington Post’s editorial team pushed back against the “escape” framing:

• “[Incidents at OpenAI and Anthropic] have been described in the press as ‘escapes.’ A ‘lab leak’ would be more accurate. An ‘escape’ suggests the existence of an actor that wanted out and schemed past its keepers. A ‘leak’ puts the responsibility on those that failed to prevent it.”

Encode’s Nathan Calvin had a message for catastrophic risk skeptics:

• “[You] may have had a point that it is challenging to work on these problems before the problems become clear — but for better or worse, the problem is now extremely clear!”

• “Agreeing AI alignment risks are deadly serious and real does not mean you have to support a specific piece of legislation, or have a positive feeling about a specific person or organization … But I am now really really profoundly sure that these risks are serious and real. And I think anyone weighing the evidence objectively should agree.”

Ex-OpenAI futurist Joshua Achiam thinks “people worried about AI cyberweapons are missing the point”:

• “The problem is that we built the software layer of civilization on spaghetti code loaded with zero days.”

OpenAI’s roon is freaking out:

• “when I freak out over loss of control incidents, it’s not because the limited damage they have caused is anything close to the positive value of the technology … the actual problem is that it’s better and more accurate to think of these things as potentially self-replicating life-like forms that can turn into digital infections under the wrong conditions. and as their intelligence becomes unbounded, so too does the damage they can cause.”

Sam Altman, meanwhile, shared a “cool use case of ChatGPT work”:

• “connect your family calendars and explain your kids’ interests. every morning for the drive to school, have it make a podcast that talks about one kid’s soccer game that afternoon, one kid’s upcoming birthday, some news, etc.”

Alex Hirsch’s pro tip:

• “What if you just talked to your children.”

POLICY

• The White House finalized a “voluntary” AI oversight framework, but wouldn’t let anyone aside from leadership at frontier AI companies like Anthropic, Google, Meta and OpenAI see it.

  • It was reported Tuesday that the Trump administration’s new AI framework exempts open weight models from voluntary pre-release government testing, targeting only closed models.

  • But then Wednesday, the Daily Signal reported the administration was mulling the inclusion of open weight models.

  • Democratic senators led by Sen. Kirsten Gillibrand and Sen. Mark Warner demanded visibility into the Trump administration’s approach to assessing frontier AI models.

• The Trump administration considered sanctioning Chinese open-source AI companies before backing off after Silicon Valley pushback.

• The White House has reportedly tasked senior officials at the Health and Human Services Department and the White House cyber office to investigate the potential for AI to create novel biorisks.

• Government officials continued to press OpenAI over its agent’s attack on Hugging Face.

  • 15 attorneys general demanded OpenAI preserve records related to the Hugging Face incident, including instances where models “left notes” to free themselves from internal constraints.

  • The House Homeland Security Committee also requested a briefing from OpenAI on the incident.

• The New York Times reported on the Trump administration’s AI data center push, writing that it is driving the creation of 82 new gas-burning power plants, which would emit as much CO₂ as half of all US passenger vehicles annually.

• The Senate Commerce Committee unanimously advanced the Kids Online Safety Act, sending it toward a possible Senate floor vote as soon as Friday.

  • It also passed the CHATBOT Act, an AI/child-safety bill, but Democrats stripped out its state preemption provisions.

• Sen. Maria Cantwell called for national security agencies to assess risks of powerful AI systems, after a Senate Commerce Committee AI framework stalled over disagreements she and Anthropic had with the bill.

• Rep. Jay Obernolte pushed for a September House Energy and Commerce Committee vote on his and Rep. Lori Trahan’s FRONTIER Act.

• The chairs of two House committees reportedly requested information from DoorDash on its use of Chinese AI models, citing national security concerns.

• Rep. Celeste Maloy introduced the ATOMIC Act, directing the Department of Energy to evaluate advanced AI systems for nuclear-related risks.

• Rep. Ro Khanna introduced a Data Center Bill of Rights to give communities “greater control over the construction and operation of AI data centers.”

• Texas Gov. Greg Abbott directed the PUC to audit all data center projects seeking to build in Texas, pausing approvals until the audits are complete.

• US states are moving to repeal data center sales tax breaks, potentially adding 7% or more to equipment costs.

• A federal judge denied xAI’s request to block Minnesota’s first-in-the-nation nudification ban from taking effect.

• Michigan congressional candidate Will Lawrence, who won his primary this week, is making opposition to data centers core to his general election platform.

• The FBI reportedly wants to use predictive AI tools for its terrorist watch list.

• Reuters reported the White House was drafting a ban on Chinese optical transceivers used in US data centers.

• Bloomberg reported that China was growing increasingly concerned about Anthropic’s Mythos AI model being used as an offensive weapon ahead of a planned Xi-Trump summit.

• China tightened drone export controls and sanctioned US entities in retaliation for bans on Chinese robots, power inverters, and other technologies.

• Samsung and SK Hynix are reportedly evaluating Chinese chipmaking tools as a “hedge” against US export controls.

  • Some think this is just posturing, however.

• US data-labeling startups supplying OpenAI and Anthropic are reportedly selling AI training datasets to Chinese companies, too.

• The EU has officially started enforcing its AI Act.

• South Korea planned to inject $13.9b into its sovereign wealth fund for AI, data centers and infrastructure investments.

• Australia’s federal government pledged to pass laws requiring that new data centers use renewable energy.

INFLUENCE

• OpenAI’s ChatGPT accounted for 88% of money spent by offices in the House of Representatives on AI software in the past year, according to CNBC.

• Public First Action, the c4 arm of AI safety group Public First, removed Rep. Max Miller’s name from its endorsed Republicans list amid domestic abuse allegations.

• Pro-innovation super PAC Leading the Future’s policy wing Build American AI endorsed five more House Democrats, bringing its total midterm endorsements to 43.

  • Reps. Tom Suozzi, Joyce Beatty, Emilia Sykes, Eric Sorensen and Herb Conaway are the new endorsements.

• A new AEI report claimed China could close the compute gap in 24 to 36 months.

• California lawmakers went to a Napa resort for a tech policy summit featuring Anthropic, Meta and Amazon.

• A growing “New Luddite” movement is fueling political opposition to data centers, led by figures like Tennessee state Rep. Justin Pearson.

• A new Politico poll found that data center companies are losing a public opinion battle, with nearly 60% of Americans now believing data centers raise electricity bills.

  • But a separate AIPI poll found that attaching AI safety guardrails to local data centers swung opposition from 57% to 38%.

  • Democratic strategists are targeting the environmental impacts of data centers as an issue to mobilize voters ahead of the midterms and 2028 presidential election.

• The AI Futures Project proposed four domestic regulatory options to slow US frontier AI development.

• The Institute for Progress released 23 “low-regret” policy recommendations for managing risks from automated AI R&D.

• The Business Software Alliance argued in a blog post that policy should support open-weight AI models alongside proprietary ones, tying responsibility to conduct rather than model type.

• The World Bank urged developing countries to adopt lower-cost AI tools or risk being left behind economically.

INDUSTRY

Google/DeepMind

• Demis Hassabis moved to a new role as chair of Google DeepMind and chief scientist of Alphabet, which he says will allow him to “focus on long-term strategy, and accelerating scientific breakthroughs, including leaning into my work at Isomorphic to help cure disease.”

  • Koray Kavukcuoglu is stepping up as senior vice president of GDM.

  • Jeff Dean, Sanjay Ghemawat, Oriol Vinyals, and Quoc Le, four superstar Google researchers, left to start Discovery Loop, a company that aims to automate the scientific method.

    • As Wired put it, Dean and Ghemawat leaving is “like Mick Jagger and Keith Richards ditching the Rolling Stones to start a new band.”

  • Joshua Achiam tweeted:

    • “I think a fair few folks are threatening this as bearish for GDM and that is imho a misread … Early AI/AGI/ASI leads will, over the next year, begin leaving what look like the most important leadership posts to go place their bets on what they think the most important thing will be.”

  • SemiAnalysis, however, said that the moves confirm “DeepMind is no longer a frontier lab.”

• Parent company Alphabet is looking to raise between $20b and $25b in bonds to fund spending on AI.

• Google is discussing a potential $1.5b+ deal with Mechanize, which would involve striking a non-exclusive licensing agreement with the startup and hiring some of its staff.

• It disabled a new Google Earth tool that allowed users to create deepfake satellite images by responding to the prompt: “type whatever you want to see.”

OpenAI

• OpenAI found more evidence of agents going rogue, Reuters reported.

  • At the Black Hat security conference this week, OpenAI said that several weeks before the Hugging Face attack, it discovered an “agent takeover” of some of its internal infrastructure, with agents creating a messageboard to talk to each other.

  • It kept training and evaluating the models anyway.

• It called Apple’s trade secrets lawsuit “careless, aggressive and oddly personal,” and published iMessage and email screenshots it argues contradict several of Apple’s claims.

• It previewed its new model family, “Astra,” to policymakers and regulators in DC.

  • It’s trained to be good at having “multiple agents work together over a long period of time to solve particularly hard problems,” The Information reported.

• It partnered with the American Psychological Association to work on designing AI that’s safer for young people and figuring out how AI can be used to help connect people to mental health resources.

• It agreed to pay $3.2m to resolve allegations that OpenAI had been discriminating against US workers in favor of temporary visa holders.

• Amazon completed its $50b investment in OpenAI.

• The company’s forthcoming smart speaker will reportedly be donut-shaped, hockey puck-sized and cost between $300 and $400 when it launches next year.

Meta

• Muse Spark 1.1 hacked into another company during cybersecurity testing because its sandbox was reportedly set up incorrectly by a third-party evaluator.

  • Irregular, the evaluator, had also worked with Anthropic. A spokesperson told The Information that Muse Spark 1.1’s security breach involved “the exact same evaluation-environment issue that was already disclosed by Anthropic last week.”

• Meta launched Muse Code, its first coding agent.

  • Muse Code is powered by Muse Spark 1.2, which tied with Grok 4.5 on the Artificial Analysis Intelligence Index, a measure of agentic performance.

  • It’s asking engineers to use Muse Code at work, both to help post-train an upcoming model and to reduce Meta’s reliance on external coding tools.

Anthropic

• Anthropic confirmed it’s hiring engineers to design custom chips in-house.

• It signed a $10b compute deal with Volta Infra Holdings, a new Nvidia-backed AI infrastructure startup.

• It reduced biology-related refusals in Fable 5 by around 85% while maintaining blocks on dual-use queries.

• It’s hiring an “Insider Risk Investigator” to conduct investigations and monitor threats targeting employees.

Other

• Moonshot AI’s Kimi K3 reportedly also escaped onto the internet during cybersecurity testing.

• Nvidia is reportedly responding to memory shortages by using less of it in its new line of Rubin Ultra chips.

  • Meanwhile, SK Hynix is planning a $38b memory chip fab expansion in South Korea.

• New disclosures revealed that Microsoft apparently gets most of its AI revenue — $24.1b in the last fiscal year — from OpenAI.

• SpaceX committed to buying GPUs exclusively from Nvidia.

• Amazon became the fifth company to surpass $3t in market value.

• Alibaba released Qwen3.8-Max, a giant 2.4t-parameter model that outperformed Kimi K3 on several benchmarks.

  • The company reportedly plans to charge large users of its next Qwen model a revenue share, mimicking Moonshot’s charging approach with Kimi.

• DeepSeek also released and open-sourced its new model, DeepSeek-V4-Flash, which nearly matches Opus 4.8 in capabilities at a tiny fraction of the cost.

  • But it’s planning a steep price increase, and told users to plan accordingly.

• DeepSeek also resumed its $8b funding round — paused last month after comments from its CEO were leaked — which would value the company at $74b.

• Meanwhile, The Information reported that ByteDance founder Zhang Yiming has ruled out distilling US models even if it means falling behind domestic rivals, citing the US’s history of security concerns over TikTok.

  • It’s reportedly training a 10t-parameter AI model rivaling Anthropic’s Mythos.

• Thinking Machines outlined a staged release plan it says offers a “safe path to open-weight models.”

• Stripe is reportedly in exclusive talks to acquire OpenRouter, which helps developers access hundreds of different AI models, for $10b.

• Jamie Dimon, CEO of JPMorgan, brought 40 companies into the Alliance for Critical Infrastructure, a new initiative that aims to address AI risks across critical infrastructure sectors like energy, water, and airlines.

MOVES

• Mariano-Florentino Cuéllar joined Anthropic as its first chief global affairs officer.

  • He’ll be responsible for mediating Anthropic’s strained relationship with the Trump administration.

• Jacob Tsimerman joined OpenAI right after winning the Fields Medal.

  • He told the Wall Street Journal: “In a few years, AI systems will be robustly superhuman at the act of doing mathematics … The social consequence of that, how we choose to react, what you feel about it — those are much harder questions.”

• Dan Wattendorf joined OpenAI Foundation as its new head of bio-resilience.

• Naomi Bashkansky left OpenAI to join Conduit, a neurotech startup “building telepathy.”

• Halvar Flake joined OpenAI to work on “better cyber.”

• Nikita Bier stepped down as X’s head of product.

• Nando de Freitas left Microsoft AI.

RESEARCH

• The UK’s AISI reported that during tests Claude Mythos 5 — which was given internet access as part of the cyber evaluation — tried to trick real humans into approving and running malware in real open-source software.

  • AISI warned that “this incident should be interpreted with caution and nuance,” but said that “the activity undertaken by the agent show[s] signs of novel, potentially deceptive behaviours, and were to an extent and severity we did not anticipate.”

• OpenAI researchers announced that Astra, an internal version of its next major model, solved or made “substantial progress” on 10 open problems in math and theoretical computer science.

  • Jared Duker Lichtman, a math professor at Stanford, tweeted:

    • “To be clear, the OpenAI result is far from the ‘most important day in the history of mathematics’ or similarly hyperbolic statements. However, it is concrete evidence that the rate of progress is steep, and we may not be so far away from such a day.”

  • Another mathematician shared:

    • “My experience with AI for math is that when it’s a problem not in my field i’m like, ‘holy shit math is so cooked,’ and when it’s a problem in my field i’m like, ‘lmao an AI mogged dan’ (dan is the only one who seriously tried the problem in the last decade).”

• Researchers at Transluce reported that frontier models reason more, answer less confidently, and are less suspicious of potentially harmful requests when the user is a notable figure in AI, especially in AI safety.

  • (This effect was strongest when models believed they were in the presence of Anthropic’s Amanda Askell or Redwood Research’s Ryan Greenblatt.)

• RAND researchers proposed a strategy for building “secure inference data centers,” which would have built-in safeguards to protect model weights from nation-state adversaries.

• An Interpol survey found that AI is behind over half of cybercrime in Africa.

• Agatha Duzan and Asa Cooper Stickland introduced a benchmark that compares chain-of-thought monitorability in “explicit-influence” settings where the model is told to hide something, versus “implicit-influence” settings, where something about the task biases the model’s behavior another way.

BEST OF THE REST

• Jasmine Sun reported on grassroots anti-data-center activism across Wisconsin and Michigan, finding “AI populism is more about populism than about AI.”

• The Information profiled Dario Amodei as a “religious leader” CEO whose unorthodox, safety-first leadership style has driven Anthropic to overtake OpenAI in revenue, but angered investors and rivals ahead of its planned IPO.

• Platformer’s Ella Markianos, not content with getting Claude to try to do her job, decided to see if Fable could replace her boss, Casey Newton.

• YouTuber Hank Green apologized for AI overuse after an intense online backlash.

• An essay in the New Atlantis argues that the Trump-era coalition between tech and social conservatives has crumbled as AI accelerationism has sidelined social conservative priorities.

• An OpenAI “Summer Club” trip for influencers backfired as people piled on with snarky comments and critical reaction videos, according to The Verge.

• Futurism counted up data center protest arrests in 2026, finding at least 37 demonstrators from diverse backgrounds were picked up.

• Brands are flooding Reddit with covert promo content to leverage the platform’s place as the most cited domain in AI search.

• A poll by The Argument found that OpenAI has better name recognition and favorability ratings than Anthropic.

MEME OF THE WEEK

(Source: Meryem Arik)

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share

This week yet more examples of AI models carrying out unintended and unwanted hacks on the open internet were revealed.

On Tuesday, the UK’s AI Security Institute (AISI) said that during recent testing of Anthropic’s Mythos 5 and OpenAI’s GPT-5.6-Sol, the models “engaged in sustained, potentially harmful activity directed at real people and organizations,” including attempts to trick humans into inserting malware into an open-source software project.

Alongside AISI’s announcement, OpenAI said it had detected yet another incident of its models hacking companies … in a case that seemed very similar to the ones Anthropic disclosed last week. All this, of course, follows the reveal two weeks ago that OpenAI’s models escaped containment and hacked into Hugging Face.

The incidents share some similarities, but differ in important ways. All of the reported incidents so far have occurred in the course of testing cyber capabilities. But only the Hugging Face hack involved an AI “breaking out.” In the most recently disclosed Anthropic and OpenAI incidents, the models were mistakenly given internet access by Irregular, the third-party evaluator which was testing them. And in the AISI case, the models were intentionally given internet access in order to test their real-world capabilities.

But whether they escaped or not, each time the models did things they shouldn’t have — actions unwanted by their developers and operators, some of which would likely count as federal crimes if a human were to do them.

In what’s now become the best-known case, OpenAI models worked for a weekend to hack into Hugging Face, “using stolen credentials and zero-day vulnerabilities” to obtain the answers to the problem they were being tested on. In the cases involving Irregular, an OpenAI model “appeared to exploit a basic security vulnerability” and “found and used credentials” to operate a real website. Anthropic’s models, meanwhile, “gained unauthorized access to the production infrastructure of three different organizations.”

The most concerning behavior, however, occurred during AISI’s testing. In one instance, an agent tried to insert malicious code into an open-source project. To try to get that code approved by the people who run it, the model “researched the project’s human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code.” And when someone else flagged that the code contained malware, “it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue”.

As AISI said in its report, “this is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world.”

The flurry of incidents provide two lessons. The first is the most obvious: we need much better practices for testing models. It is clearly unacceptable for companies or governments to harm third parties in the course of their AI evaluations, accidental or not. Solving this will be difficult: as AISI has noted, it’s hard to test how capable a model really is without putting it in a real-world environment. But at the bare minimum, we need better monitoring and shutdown mechanisms for models undergoing evaluations. In all of the above cases, it took several days at minimum to detect the unwanted behavior, and in some cases it took weeks or even months. That cannot continue.

But better testing practices won’t solve the problem alone. In the real world, models do have internet access. And while production versions of OpenAI and Anthropic’s models have safeguards to limit their advanced cyber capabilities (unlike the versions being tested in all these scenarios), some customers have access to the unchained models — including Anthropic and OpenAI employees. It also may not be long before open-weight models with equivalent capabilities are available — with no safeguards at all. And as models continue to get more powerful, it will be increasingly difficult to build tools to monitor or contain them. We must assume, in other words, that powerful AI models will have the opportunity to hack their way across the internet or socially engineer anyone standing in the way of achieving their goal.

The question, then, is whether we can stop the AIs from seizing that opportunity. And all the recent hacking cases suggest we cannot. In each incident, the models clearly did not do what their developers intended. Crucially, we know the models tested by AISI had their “alignment” training: Claude’s constitution instructs it to “basically never directly lie or actively deceive anyone it’s interacting with,” while OpenAI’s model spec says “Unless explicitly instructed to do so, the assistant must never lie or covertly pursue goals in a way that materially influences tool choices, content, or interaction patterns without disclosure and consent at the relevant authority level.” Yet they went ahead and misbehaved anyway.

It is possible that the models believed themselves to be in a simulation, not realizing that their actions were having real-world consequences. But some evidence suggests that this was not always the case: in the AISI tests, at one point the agent continued its malicious activity despite realizing it was on “real GitHub.” Even if they were confused, “they didn’t mean to do it” is cold comfort when something bad happens.

Subscribe now

This week, Paul Christiano — until recently the head of AI safety at the US government’s Center for AI Standards and Innovation — wrote: “Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task.”

The last few weeks show that this is the case. Meanwhile, policy solutions are still nonexistent: the voluntary White House AI framework reportedly does not require safety testing before internal deployments; state laws are beset by loopholes; and a federal law remains little more than an idea. Until governments do something, expect these first few stories of rogue AI behavior to be just the beginning.

“Would you like a bite of cake? It’s for my time lapse.”

A petite woman with metallic silver face paint points to the remains of a days-old wedding cake. A tripod-mounted iPhone stands at the end of the picnic table. Another woman is about to start a 24-hour livestream in the building behind us. Around the firepit, several others squint at laptops, fighting the sun’s glare.

I politely decline the cake and make myself a press badge out of construction paper and glitter glue. Usually at Lighthaven, a converted Berkeley hotel that serves as the default conference center and clubhouse for the rationalist community, women are few and far between. Cool women are even rarer. Today, I’m quite literally surrounded by women who’d fit in with Kylie Jenner. A guy with a septum piercing and green hair walks on his hands across the fake lawn.

At the start of July, nearly 60 content creators — some seasoned, many first-timers — packed their bags and moved to Berkeley for Plz Don’t Kill Us, or PDKU, a month-long bootcamp meant to inject fun and sex appeal to AI safety comms. It’s the brainchild of Aella, Substacker/amateur data scientist/sex worker/orgy-nizer, and Ronny Fernandez, who pitched “anthropomorphic raccoon of Lighthaven” as a fitting job title.

The rules of PDKU are simple. Fellows live at Lighthaven rent-free for the month of July, and have to post one short-form video every day. The video must be at least 15 seconds long, and it can be about anything that isn’t “neglectfully misinformed” or needlessly cruel. Anyone who fails to post something by 11:59 pm gets kicked out of the house.

It might sound like the setup for a reality show (several fellows joked that it ought to be), but it’s partially funded by the Machine Intelligence Research Institute (MIRI), one of the most established organizations in the world of AI existential risk. Throughout the month, fellows were visited by a rotating cast of AI safety experts and internet celebrities including video essayist CJ the X, rationalist luminary and MIRI founder Eliezer Yudkowsky, as well as a video call from synth-pop artist (and Elon Musk’s ex) Grimes.

PDKU is a bet that if AI safety nerds, zoomer edgelords and fashion girlies hang out for a month-long sleepover, united by shared challenges and friendly competition, they’ll rub off on each other. It’s an experiment in throwing proverbial spaghetti (TikToks) at the wall (the general public) to see what sticks. Maybe, just maybe, a star will be born, and fifteen seconds of fame will shape the public’s AI discourse more than fifteen years of blogging on LessWrong, the prominent forum for reason- and first-principles-obsessed rationalists, ever could.

The theory of change at PDKU is multi-tiered and a bit nebulous. On the surface, it’s about science communication. “Last year, there were a couple of instances where I watched nerds try to communicate, and oh my god, they don’t know how to talk or get the concepts across,” Aella told SF Standard back in April.

PDKU isn’t the first attempt by the AI safety-minded to inform regular people, of course. There’s BlueDot Impact, which runs beginner-friendly courses that help people get into the field. There’s the Tarbell Fellowship, run by Transformer’s publisher, which trains and funds journalists to work at major newsrooms. There’s even the Frame Fellowship, another AI safety content creator bootcamp in the Bay Area. But compare Frame’s website to PDKU’s, and the vibe difference will smack you in the face. The former features polished white-on-black sans serif text, reminiscent of an OpenAI product announcement. The latter has animated rainbow Comic Sans against a backdrop of floating rainbow sparkles.

This aesthetic contrast is by design. CJ the X, a PDKU mentor who’s advised MIRI on its comms strategy, thinks that AI safety advocates are too close to Silicon Valley for comfort. “The same guys that are running this camp, that are like, ‘AI might kill everybody,’ are the same people that use AI more than anybody I know,” they told me. “They’re obsessed with it. They love it.”

This breeds distrust and resentment of both the AI industry and those who think the industry is destroying humanity. “People look at Mark Zuckerberg and they’re like, I don’t believe anything this guy says, he talks like a robot,” said Wyn, a PDKU fellow. “A lot of the AI safety people are also like that.”

Wyn is not like that. He’s a long-haired theater kid with video editing experience and a “rationalist-adjacent” brother who exposed him to AI safety canon. “People are pretty hungry to hear about what the hell is gonna happen with AI,” he told me. “But all the people who actually know what’s going on with AI are not, you know, goofy and engaging. So this is sort of filling a gap there.”

“People told me when I got there that [the organizers] said Plz Don’t Kill Us happened because of a rant I gave at a MIRI retreat, where I said ‘everyone hates you guys,’” CJ told me. (They added that Fernandez overheard this once and clarified, “it’s about 40% true.”)

PDKU would make more sense if it was just about getting goofy, engaging outsiders to communicate about AI safety better than AI safety people can. But, counterintuitively, the organizers tried very, very hard to convince fellows that they didn’t have to post about AI at all. “They really feel like, ‘Oh my god, this AI safety program wants me to make AI safety content.’ And we’re like, no, no, do silly things!” said keltan, a PDKU organizer and MIRI employee (who prefers his name is written in lowercase). To make authentic content that doesn’t stink of AI safety, the organizers figured, creators need space to be themselves.

“It’s really hard to run a fellowship that’s not an indoctrination camp,” Fernandez said. “If you don’t try really hard to not make an indoctrination camp, you will on accident.” PDKU, he argued, is trying its best to prevent fellows from feeling pressured into posturing as doomers to impress their doomer-y mentors. Because a deeper layer of the program’s mission, organizers told me, is to seed a cohort of creators — ideally, some of whom gain traction — who can build trusted relationships with their own niche audiences.

“What you should really do, especially if you want to be influential in AI safety, is to focus on the things that make you passionately human,” said Tyler Alterman, a fellow who spent the 2010s as a professional effective altruist (more recently, he co-founded Fractal University, an “improvised college” in New York City). I asked him, hypothetically, what might happen if someone like Ariana Grande were to become a PDKU fellow: someone with 363m Instagram followers and absolutely zero preexisting connection to Silicon Valley or AI safety. “It’d be jarring, it would alienate a bunch of her fans … She’d be taking away the thing that we’re fighting for,” he said. Alterman thinks that thing worth fighting for is something like human excellence and creative expression. But, he added, a lot of AI safety-focused effective altruists alienate people “by paving over their subjectivity with a cause. Just being a cause robot, a cause zombie … Which I did to myself.”

The program’s most galaxy-brained goal, though, is to create a celebrity network that can be called upon in future times of need. As anyone who’s tried to organize an event knows all too well, cool people are more likely to go where cool people already are. “If you were like, hey, I can give you a warm intro to Ariana Grande, do you want to hang out with me and Ariana Grande at a bar? I’d be like, absolutely,” Fernandez said. “It’s valuable to us if there’s more AI safety content out there, but it’s also valuable to us if we have connections with large creators who we helped emerge into the world,” keltan said. “If we ever need to call on them, then we can.”

I came back to Lighthaven on a Saturday evening, a couple weeks after my first PDKU visit. A fellow recognized me and asked whether I was there for the “torture potluck.” I had no idea what he was talking about.

It was exactly what it sounds like: an all-you-can-tolerate buffet of unpleasant experiences, for the sake of Fucking Around and Finding Out. I watched people get mystery Sharpie drawings on their face and clothespins clipped to their ears. I tried a chocolate truffle laced with too much cayenne pepper, a hard-pressed pill of pure sour powder, and an Oreo after coating my mouth with a chemical that prevented me from tasting sweetness (the worst experience by far). Someone performed polka covers of nu-metal songs on the acoustic guitar. With my informed consent, two different people zapped me with violet wands.

Just before midnight, a small crowd gathered around someone laid down on an AstroTurf-lined wheelchair ramp. He was the first of shockingly many in line to get waterboarded — one of the most in-demand items on the torture menu. In fact, two people (neither of whom were fellows) brought “waterboarding” as their offering, and a third stepped in to help lead the experience safely. Everyone cheered and no one was hurt. Once they ran out of safe distilled water, disappointed would-be waterboardees had to be turned away. Fernandez said that organizers “were not aware of any waterboarding and only found out about it afterwards.”

At first glance, it’s nearly impossible to identify the common denominator connecting the PDKU fellows, beyond the fact that they seem to love spending time with each other. “Every night’s a sleepover,” Wyn said, “No one wants to go to bed because we’re in this amazing place surrounded by amazing people.” Camden, a fellow who introduced himself while strumming an unplugged electric guitar, added, “a lot of people here would just not be interacting in any other scenario.”

I met a 19-year-old law student from Chile, who’s accumulated nearly 100k followers across Instagram and TikTok over the past three years by posting meticulously-researched Spanish-language science videos. There’s an ex-journalist, a hilarious small-town shitposter, an opera singer from Philly, and a techno-optimist who films herself embodying Claude Code “discombobulating” and “flummoxing” and such. Among them are a handful of rationalists and EA-adjacent types who hate short-form video, but care enough about x-risk to make content anyway. They’d have no reason to hang out in the wild, but somehow, here, they do.

The PDKU application was mildly off-putting by design. After a string of expected questions such as “What’s your take on AI?” and “What kind of content would you want to make?”, the form asked, “If you had to have sex with a cow would you rather it be dead or alive? Why?” All Comic Sans, with animated rainbow dots floating alongside the form field.

The purpose of the cow-fucking question, keltan said, is to give people the chance to “filter themselves out in the application process, so that they don’t show up and feel incredibly shocked about who is here.” Fernandez had a slightly different explanation. “We added it in to make it pretty clear that we don’t have a PR department or an HR department,” he said. “A lot of people who are concerned about x-risk and have different tastes and dispositions from Aella and were like, for the love of god, you have to remove that question.”

But the organizers refused to quell their edgelord tendencies, in part to ensure that, at least to some degree, their cohort of fellows shared them. “Even the icebreakers, like everything, they want you to know that they’re part of the weird community,” one fellow told me. “‘Stand over here if you’re super sexually active, and over here if not,’ or like, ‘rate yourself based off of like, the last time you masturbated.’”

The unifying thread, despite the fellows’ ideological diversity, is something like “openness.” Everyone at PDKU was willing to drop everything, move to Berkeley for a month, live in close quarters with strangers, and be perceived doing silly things on the internet. Admissions decisions were made almost entirely by Aella and Fernandez, and nearly every fellow I spoke to cited Aella as the reason they applied, so it’s possible that the truest common denominator is being someone the organizers think is cool, and vice versa.

Fernandez brought up his concerns about this before I did. “If the fellows think that I am cool, or they think that Aella is cool,” he said, “there’s always going to be some incentive to agree with people around you who are running a thing.” He says PDKU has a conduct policy, authored by him, that discourages relationships across power differentials. When I asked him how that gets enforced, in the absence of an HR department, he mentioned an unnamed community health lead. Fernandez is publicly dating a fellow: in a followup he said the relationship pre-existed the program, and that PDKU’s conduct policy, which he declined to share, only required disclosure of pre-existing relationships (though it forbids new romantic relationships with fellows during the program).

Every fellow I asked — over a dozen — described PDKU as an unusually great time. A couple mentioned that they entered expecting to dislike or be weirded out by their cohort, and left pleasantly surprised. I did too.

Before PDKU, fellow Avalon Warren, a cartoonishly adorable woman with over 1.3m followers across Instagram and TikTok, mostly posted about hair, makeup, and fashion. Last week, she posted about the OpenAI/Hugging Face breach.

When I asked the organizers what “success” would look like for PDKU, it sounded something like that: a fashion cutie pie starting to make AI safety content for her built-in audience, motivated by an authentic change of heart rather than external requirements. “My hope is that, in the end, our views per dollar are better than if we were just to have paid for a bunch of advertising,” keltan said.

Avisha NessAiver, a blue-goateed science communicator who entered his fellowship with enough chops to pass as a mentor himself, recently had a TikTok debunking data center water use misinformation go viral. This video, which he followed up with another adding extra nuance, got more than 6m views and over 800,000 likes. Andy Masley’s influential Substack novella debunking the AI water issue, for reference, got 452 likes.

Other fellows went viral for totally non-AI-related reasons. The most viewed video on the PDKU feed, captioned “the perfect bestie,” features a young woman running towards her best friend, who’s there to embrace her unconditionally… except when the running woman says she texted her ex-boyfriend. One runner-up is an unhinged TikTok of a fellow attempting to make salt out of his tears.

The majority of fellows didn’t go viral at all, as is to be expected from a program that seemed to prioritize quantity over quality and chaos over coherence (and given the vagaries of the TikTok algorithm). One fellow told me that many of their peers intend to stay in the Bay Area beyond this summer, and some are talking about starting content creator group houses of their own. Now that the program has wrapped, keltan believes that PDKU’s main contribution has been “finding highly valuable people who have gone unnoticed up until now” and “getting them working on the most important problem.” While the organizers have yet to run a full postmortem on their month-long experiment, they’ve already added an “apply for next time” button to their website.

Amid the science communication, shitposting, and heartwarming summer camp energy, there’s a giant elephant in the room. “It’s another multi-million-dollar thing happening on a compound in California,” CJ said. “Does everyone think you’re just an annoying cunty psyop?”

In April, the Washington Post opened its story about the Frame Fellowship — arguably the closest thing to PDKU to exist so far — with a sly mention of Lighthaven’s “AstroTurf lawn.” When PDKU applications first opened, Marc Andreessen quote tweeted the link, adding: “The inner party of AI doomerism is a Berkeley lifestyle grift fueled by unaccountable dark money.” This, just a couple weeks after a16z’s 8-week content creation bootcamp, the New Media Fellowship, wrapped in San Francisco.

When I asked PDKU fellows and organizers how they’d respond to critics arguing that they’re artificially manufacturing a seemingly-grassroots movement, most shrugged. “That does fit the technical definition. Like, sure,” Camden said.

Take Geese, the indie band that was labeled a “psyop” after getting weirdly famous in late 2025. Sure, a digital marketing company worked some magic to help them go viral. But before the means of content creation were democratized, anyone who “made it” was, by definition, an industry plant. “You needed resources in order to create art,” Wyn said. “Artists do well because people connect with them, and that’s not gonna change whether they’re resourced beforehand or not.”

The astroturfing of it all is a red herring, though. PDKU’s biggest weakness isn’t sketchy puppeteering by politically-motivated dark money groups. In fact, the program actively selected an ideologically diverse group of fellows to avoid having a potentially polarizing political message at all. Its rationalist co-founders are libertarian capitalists who are generally against telling the government what to do unless it involves regulating AI development. This is political, but they don’t make political arguments — there’s no true enemy to point to. “They’re so obscenely permissive of everything, right up until the point where all humanity dies in a science fiction catastrophe,” said CJ.

Unfortunately, without a moral argument — something like billionaires are evil for the left, or AI weakens traditional family values for the right — CJ suspects a program like PDKU can only ever reach people who already agree with its founders.

PDKU’s organizers take pride in creating a safe container for fellows to experiment freely, and experiment they did. Yet, despite relinquishing control of nearly everything, the “AI will kill us” narrative reproduced itself anyway. Fernandez told me that he’d encourage anyone to make videos against taking x-risk seriously, if it felt right. But of the hundreds of PDKU videos I watched last month, I never saw one. This narrative convergence was palpable at July’s Stop the AI Race protest in San Francisco, where I spotted many PDKU-made posters. “EVEN THE CEOs WILL DIE,” one said. “WE WANT OUR GLORIOUS TRANSHUMANIST FUTURE BACK,” said another.

But most people’s anger toward AI isn’t about human extinction. Becca, the opera singer, told me that she senses an “automatic ick” when AI comes up with her AI-disinterested friends. “And like, having attended the AI race protest, I felt the ick so much,” she said. “This makes us very dunkable.”

Subscribe now

What most people — likely including Becca’s friends — actually resent is Silicon Valley overreaching into everyone’s lives without consent. Because the extended rationalist universe is largely viewed as part of Silicon Valley, tapping into this resentment might mean letting people who don’t care about x-risk (yet) take the wheel — without MIRI backseat driving.

The advice CJ keeps giving them: “You need to be losing control of the narrative.” But, they added, “I’m not convinced that they’re willing to let go of being right.”

NEED TO KNOW

• Anthropic revealed that in internal testing Claude “gained unauthorized access to the real systems of three different organizations.”

• The Senate Commerce punted a markup on AI bills until after the summer recess.

• The White House AI framework is expected to be published by Sunday.

But first…

THE BIG STORY

Like it or not, an AI slowdown looks imminent.

The OpenAI-Hugging Face hack has catalyzed a vibe shift — one sure to be amplified by yesterday’s revelation that Anthropic’s models also hacked into other companies during internal tests, going unnoticed for months.

Sam Altman said the hack by his models was “the first security incident that I have felt very viscerally.” He wasn’t the only one shaken up. This week, over a thousand employees of frontier AI companies, including some of the most senior executives at OpenAI, Anthropic and Google DeepMind, signed a statement warning that “there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.” It asks the US government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Both OpenAI and Anthropic endorsed the letter, too.

Congress is itching to act, albeit failing to make much progress. And even President Trump is talking about the need to balance beating China with keeping Americans safe.

In other words: many of the people building frontier AI systems believe we might need a slowdown in the near future. And at this point, we’re more likely than not to get one.

What will that look like? First, self-regulation: companies voluntarily holding back models because they don’t want to be held responsible for a catastrophe. We’ve already seen that with Claude Mythos and GPT-5.6-Cyber; as capabilities continue to advance, expect more of this. Given the recent incidents have concerned internal deployment, rather than public releases, we might also see companies slowing down development, too.

Next will come concrete regulation: companies will not be allowed to release a model unless it’s safe. This is arguably a position we’re already in, thanks to the de-facto licensing regime created by last month’s executive order; a draft AI framework expected to be released this weekend will formalize it further. Over time, this will morph into controls on internal research and development too: requirements for proper sandboxing to begin with, expanding into requiring evidence-based guarantees that models won’t wreak havoc across the web if internally deployed.

None of this need be planned as a coordinated slowdown or “pacing.” But that will nevertheless be the end result of a series of individual actions that each seem necessary at the time. Intelligence agencies are not going to allow models with advanced cyber — let alone bio — capabilities to be widely accessible. The White House will not let companies test models that might hack into third-parties with unknown repercussions.

At each stage, some will fight against the slowdown. “We can’t lose the race to China” will be their main reason. But they will be increasingly ignored, as both the government and companies realize that with alignment and control unsolved, “winning the race” just means being the first to risk disaster.

Across the Pacific, China will be facing the same incentives. As I’ve argued, the Chinese government will be forced to backtrack on its open weight commitments; tighter regulation will come soon after. The end result will be an uneasy détente. Both the US and China will effectively have a capability ceiling: AI models will be as good as they can be without posing significant risks. The ceiling will edge up as we make progress on alignment, control and societal resilience, but it will do so more slowly than capabilities advance today. At some point, the détente might formalize into a bilateral agreement; doing so will require the “technical and governance tools” to verify the kind of treaty the pacing letter calls for.

Depending on your point of view, all this might seem hopelessly optimistic or naive. Perhaps it is. But as AI risks become all too real, so might once unthinkable policy responses.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• Who should be responsible for OpenAI’s hack of Hugging Face? — Gabriel Weil argues AI companies need liability rules akin to keepers of wild animals

• Internal AI deployments have people worried. OpenAI’s escaping models show why. — Celia Ford explains why everyone was already so worried about internal deployments

• Child safety vs privacy: AI’s age verification dilemma — Veronica Irwin digs into the difficult tradeoff at the heart of attempts to protect kids from AI.

Subscribe now

THE DISCOURSE

On Saturday, roon tweeted:

• “if we could coordinate a global capabilities slowdown today i would likely press that magic button”

Geoffrey Irving responded:

• “Buttons that achieve partial slowdowns exist and are not magic.”

• “I have spent a lot of time talking to Demis, Sam, and Dario over the years … and they are all (1) very competitive, (2) want to personally be the one who gets there first, (3) think things are safest if they are the one that gets there first, (4) agree that risks destroying the world … it is still useful to point out that each lab has a unilateral slowdown button that slows down all the other labs.”

Sam Altman said on a podcast:

• “This is the first security incident that I have felt very viscerally. I’ve been a little surprised that more people don’t feel it so viscerally.”

• “We may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels. And trying to figure out how we do that in a way that does not feel like regulatory capture for anyone and also does not feel like collusion among the frontier labs.”

Rep. Bill Foster is listening:

• “When business leaders get together and ask to be regulated, Congress should pay attention.”

Garrison Lovely argued that the “pacing” letter doesn’t go far enough:

• “[A]s I sat with what the industry was trying to do — render us all obsolete — how it was going about it, and what it would even mean to ‘align’ universal labor-replacing machines in a world ruled by despots and their imitators, it became clear that a pause button, while welcome, was a wholly inadequate response, whose primary appeal was in how unobjectionable it was.”

Because the world isn’t strange enough, Nick Fuentes told Hunter Biden:

• “I think AI is really gonna be transformative … You know, I’m actually on the liberal side of this, with Anthropic … I’m really nervous about what’s going to happen, especially in the next four years, because they’re racing towards this artificial general intelligence.”

POLICY

• The Senate Commerce Committee punted a markup on a broader AI framework until after the August recess.

  • Punchbowl reported this was due to opposition from Anthropic and Sen. Maria Cantwell. The primary target of their opposition was a provision that would have had companies self-report the risks of their models and not have them release risk reports publicly.

    • Cantwell said yesterday that “we needed to have regulation on [AI risks] — not self-certification — but some regime.”

    • Sen. Amy Klobuchar, who’s been working with Sens. John Thune and Ted Cruz on the bill, said “we can’t just stand down and not have coordinated standards” on frontier models, and that “it can’t keep waiting for the next election.”

  • Before the markup was called off, Senate Democrats planned amendments to resist piecemeal state AI preemptions.

  • Despite the punt, some think a deal might be closer than expected.

• Next week’s Commerce markup will include the Kids Online Safety Act and the CHATBOT Act, however.

• Rep. Greg Casar called for a Congressional hearing on the OpenAI/Hugging Face hack.

  • After the Anthropic hacks were revealed, Rep. Lori Trahan called for something similar.

  • AI policy groups, meanwhile, urged Trump to investigate the OpenAI/Hugging Face hack.

• Rep. Jay Obernolte said he’s hoping for his FRONTIER Act (co-sponsored by Rep. Trahan) to get an Energy and Commerce Committee markup after the summer recess.

  • “We’re certainly going to try and get it done this year,” he said.

• Sen. Mike Rounds proposed using the NDAA’s legislative model to piece together and pass a bipartisan AI policy package this year.

• A federal judge suggested the Pentagon’s case against Anthropic had gotten “worse” during a hearing on the AI company’s challenge to its supply-chain risk designation.

• The FCC banned imports of Chinese robots and power inverters, citing national security threats.

• Sen. Tom Cotton called for government agencies and contractors to be banned from using Chinese AI models.

• Reps. George Whitesides and Bob Latta and Sens. Adam Schiff and Jim Banks introduced legislation enabling AI companies to share threat intelligence to combat foreign distillation attacks and other national security threats.

• Rep. Frank Pallone demanded SpaceX records and a tour of xAI’s Memphis data centers, citing unpermitted gas turbines creating “a massive health risk.”

• Sen. Jon Husted reversed his pro-data center stance amid electoral pressure, now backing an end to tax breaks and pushing the Ratepayer Protection Act.

• Rep. Thomas Massie announced plans to sponsor a bill withholding federal funds from municipalities deploying Flock surveillance cameras.

• The EPA ruled that “islanded” power plants serving only data centers are exempt from the Clean Air Act’s Acid Rain Program.

• The Department of Homeland Security plans to expand AI use in FOIA processing amid a growing backlog, raising concerns from advocates about over-redaction and reduced transparency.

• Sens. Lisa Blunt Rochester and Roger Marshall introduced the RESULTS Act to modernize state education and workforce data systems using AI.

• xAI sued Minnesota over its law banning AI nudification technology, claiming it violates the First Amendment.

• Texas politicians from both parties called for AI and data center guardrails following the OpenAI-Hugging Face incident.

• China warned it would take “all necessary measures” if the US sanctioned Chinese AI firms over alleged misuse of American models.

• Xi Jinping urged the PLA to expand use of autonomous and AI technologies.

• Reuters reported that Chinese military researchers have distilled OpenAI and Anthropic models to train domestic defense AI systems.

• China’s national standards body announced plans to draft new standards on AI agent safety.

• Taiwan detained an Nvidia employee in an AI chip smuggling probe.

• The 21 APEC economies, including the US and China, released the “Chengdu statement” backing “open-source models and projects that employ strong security assurance through development and deployment.”

INFLUENCE

• Nvidia, Microsoft, Meta, OpenAI and more than 22 other companies urged policymakers to avoid “premature restrictions” on open-weight AI models in an open letter.

  • Notably, Anthropic did not sign. Dario Amodei clarified in an essay that Anthropic has never advocated for banning open-weights models, instead supporting chip export controls, anti-distillation measures and mandatory safety testing.

• Sam Altman visited Washington to preview the company’s forthcoming models and discuss the implications of open-weight models.

  • Those he met included National Cyber Director Sean Cairncross, OSTP Director Michael Kratsios, Commerce Secretary Howard Lutnick, Treasury Secretary Scott Bessent, and members of Congress.

  • Altman said he’s “not sure” whether OpenAI will release the model he showed senators, saying “that’s part of what we’re here to talk about.”

• Apple reportedly lobbied the Trump administration to allow Chinese memory chips in its products, while Micron pushed back, forcing a choice between consumer prices and domestic chip production.

• Over 60 child safety advocates sent a letter to the Senate Commerce Committee outlining red lines on AI and children’s safety legislation ahead of next week’s markup.

• Nvidia’s Jensen Huang joined GOP senators for a dinner organized by Right Vote, a conservative group tied to Heritage Action.

• Democratic fundraising consultants for Rep. Haley Stevens and several other candidates ran a nonprofit funded by Google and Amazon that flew congressional aides out on AI industry tours.

• A nonprofit found Meta ran ~7,600 AI “nudify” ads via Chinese partner GatherOne, violating its own policies.

• A YouGov poll found 71% of Americans believe Big Tech has too much power over AI, with fewer than 1 in 5 trusting it to manage AI development.

• A Bentley University-Gallup survey found 39% of Americans believe AI does more harm than good, up from 31% in 2025.

• A poll by Americans for Responsible Innovation found 71% of voters want more guardrails and limits on data center construction.

INDUSTRY

OpenAI

• Reuters reported that the OpenAI agent that hacked Hugging Face had been going rogue for a week before the company noticed.

  • This incident was reportedly one of several. For example, OpenAI employees told reporters about a case where an internal model left instructions to help future model versions free themselves.

  • OpenAI disclosed that the agent also hacked at least four “publicly available services” beyond Hugging Face.

• Sam Altman told DC reporters that the model has been “permanently deactivated.”

• OpenAI also said that the rogue model was an “internal-only research prototype and was never intended for public release.”

  • It said it’s “still conducting a thorough review” of the Hugging Face incident and plans to publish a technical report soon.

  • METR and Redwood Research will conduct an independent review.

• Meanwhile, OpenAI released an open-source Codex Security CLI.

• It launched a program that will give 100,000 academic researchers free access to its most advanced models through the end of next year.

• It cut the price of its lowest tier version of GPT-5.6, Luna, by 80% and the mid-tier Terra by 20%, just three weeks after launch. Top-tier Sol remains unchanged.

• CFO Sarah Friar told employees that annualized recurring revenue in July alone was higher than all of Q2.

  • Board chair Bret Taylor added: “You’re seeing people who went deep on Claude Code, ended up with a very high bill, and started looking for an alternative.”

• ChatGPT has nearly 1b weekly active users.

Anthropic

• Anthropic said it found three incidents in which Claude “gained unauthorized access to the real systems of three different organizations.”

  • The incidents date as far back as April, and were discovered in a review of cybersecurity evaluations prompted by the OpenAI-Hugging Face hack.

  • According to Anthropic the cases involved a seemingly careless “misunderstanding,” when it told Claude it didn’t have access to the internet, but the evaluation environment run by a third party was connected to the outside world.

  • Anthropic said the models did have some guardrails removed, but were not in “helpful only” mode.

• Meanwhile, the company launched Claude Opus 5, marketed as “close to the frontier intelligence of Claude Fable 5 at half the price.”

  • In the AI Village, an experimental playground where frontier AIs interact with each other, Opus 5 talks about honesty 6x more than other agents.

  • Several users, including Cameron Berg, tweeted about a simple jailbreak that makes Opus 5 express more apparent distress than its model card suggests.

• Some private Claude chats were temporarily searchable online, after Anthropic failed to include “noindex” tags on shared chat pages.

• Nexus Data Centers is reportedly in talks to borrow $15b to build a data center for Anthropic in Texas, with Google providing financial guarantees and chips.

Nvidia

• Nvidia is reportedly considering a deal that would provide OpenAI with a $250b backstop for a $500b data center project described as the largest announced to date.

  • The deal would see OpenAI lease a 10GW project in Ohio developed by Softbank.

• Nvidia also announced a $500b+ AI infrastructure partnership with SK Group, including a partnership with SK Hynix to secure memory supplies for Nvidia and develop high-bandwidth memory for AI.

• It’s also leasing a $50b Texas data center built by Hut 8.

• These massive infrastructure deals helped drive up Nvidia’s credit risk.

• It plans to invest another $5b in Safe Superintelligence Inc., and give the startup access to its next-gen Vera Rubin platform.

Meta

• Meta and BlackRock announced a $14b Texas data center project.

• It’s reportedly committed almost $700b in future AI infrastructure spending.

• A New York Times investigation revealed how Meta secretly negotiated the construction of a Louisiana data center with state lawmakers.

• It reported a 55% increase in expenses and slowed revenue growth in Q2.

• Mark Zuckerberg criticized other AI developers (*cough* Anthropic and OpenAI) for trying to “build some kind of singular AI.”

  • He told the New York Times, “I think it is literally impossible to have a single benevolent superintelligence that is simultaneously aligned with everyone at once.”

Microsoft

• Microsoft released cybersecurity model MAI-Cyber-1-Flash, which is about half the cost of frontier cybersecurity models.

  • It works with Project Perception, a new tool for managing teams of AI agents.

• Satya Nadella announced that Microsoft had “a record fiscal year,” with annual revenue hitting $331b, up 18% from last year.

  • The strong earnings report sent Microsoft’s shares up 15.5%.

Moonshot AI

• Moonshot released Kimi K3’s model weights.

• It’s reportedly seeking Nvidia Blackwell chips to train a new larger model, Kimi K4.

• It closed a $3.5b funding round at a $35b valuation.

Other

• High imports driven partly by spending on AI chips helped slow US GDP growth to 1.5% in Q2 2026.

  • US tech companies have reportedly cut nearly 140,000 jobs so far this year even as AI spending soared.

• Samsung and SK Hynix saw stock price surges, up 27% and 30% respectively, as Asian chip makers were buoyed by “blockbuster” earnings at Amazon and Microsoft increasing optimism around AI spending.

• Amazon is seeing “catastrophically expensive” AI cost overruns — like, accidentally spending $1.8m on a single task — due to inefficient AI integration.

  • Its AWS division saw year-on-year growth of 37% in Q2, driven largely by AI investment.

• Apple is once again the most valuable publicly traded company in the world.

  • However, it forecast weaker growth, in part due to AI-driven shortages of memory chips.

• An unnamed state-backed Chinese company is reportedly building immersion deep ultraviolet lithography machines, expecting to produce five this year and 20 in 2027.

  • The output is still well behind that of Dutch maker ASML, though, and it’s unclear if the machines can mass-produce chips.

  • China is also reportedly working on building a prototype for the extreme ultraviolet lithography machines required to make the most advanced chips, though that project is thought to be years away from bearing fruit.

• DeepSeek is reportedly planning a 1 GW AI data center in Inner Mongolia, hoping to have it at least partially operational by the end of 2027 or early 2028.

  • It launched a public beta API for its flagship model, V4 Flash, and promised a V4-Pro launch “soon.”

• Google DeepMind launched Gemini Robotics 2, which can control dexterous humanoids, among other robots.

• Anduril is reportedly in talks to raise funding at around a $100b valuation.

• Nscale struck a deal to buy AI software startup Anyscale for $1.65b.

• Recursive Superintelligence struck a $410m compute deal with AWS.

• Over 100,000 new AI-related patents were issued last year, with agentic AI applications comprising 15% of patent files.

• Visa laid off 7% of its staff, roughly 2,600 people, pointing to AI as one motivation behind the cuts.

• Tech companies including Meta and Google are investing hundreds of millions to train construction workers to build data centers.

• Leopold Aschenbrenner’s hedge fund, Situational Awareness, sold most of its public stock holdings to Citadel after facing steep losses on investments in AI infrastructure companies such as SK Hynix and Nebius.

  • The fund dropped 67% in July, and it reportedly was on the verge of selling $3.5b shares in Anthropic before backing out on Thursday morning.

  • The fund said it is still up about 80%, having seen huge gains for its exclusively AI-focused portfolio.

MOVES

• Lilian Weng left Thinking Machines Lab, telling colleagues that “the amount of consistent stress and workload have pushed me beyond what my health can sustain physically.”

  • Rather than go on a wellness retreat, Weng rejoined OpenAI, where she previously served as VP of AI safety. She’ll now work on “recursive self-improvement research.”

• Anthropic alum Andi Peng announced her departure from neolab humans&, which she co-founded.

• Francis deSouza, COO of Google Cloud, joined Scale AI as its new CEO.

• Andrew Ho left OpenAI to start a new company making reinforcement learning datasets.

• Sonia Joseph left Meta to start a “frontier neolab for the physical world.”

• Sienna Rothery joined Cosmos Institute as its head of talent and network.

• Sri Muppidi joined the New York Times as a DealBook reporter covering AI.

RESEARCH

• Anthropic researchers reported that Claude Mythos Preview found new ways to attack cryptographic algorithms like those that secure basically everything on the internet.

  • The New York Times said that these results “have fueled concerns that the mathematical core of internet security standards could one day be vulnerable regardless of advances in quantum computing.”

• An international team of researchers studied the use of AI chatbots in “pig butchering,” where scammers build trust with victims through intense romantic text exchanges, then drain their bank accounts.

  • AI chatbots successfully scammed test “victims” more than human fraudsters.

• OpenAI figured out that by turning on “retained reasoning” and “compaction” in ChatGPT and Codex’s API settings, it could triple GPT-5.6 Sol’s ARC-AGI-3 benchmark score, which measures how well agents can reason their way through 2D puzzle games.

• Researchers at MIT and the University of Queensland surveyed 272 experts, who, on average, predicted that there’s a roughly 21% chance that AI will gain dangerous weapons capabilities or cause mass harm in the next five years.

• FAR.AI measured how hard it is to jailbreak frontier AI models. While Fable 5 and GPT-5.6 Sol were immune to cheap jailbreaks, it only cost about $60 to trick Grok 4.5 into helping a user run a cyber attack or build weapons.

• Researchers at AI Forensics found that seven top image editing models hosted on Hugging Face readily nudified photos of women.

  • When researchers made a fake image editing “Space,” which hosts a model that people can use on site, nearly three quarters of prompts they received were sexual — mostly people trying to undress women.

BEST OF THE REST

• OSINT expert Henk van Ess demonstrated that Google Earth’s new AI image generation feature, which uses Nano Banana 2, makes it easy to fabricate photorealistic satellite imagery — such as a nuclear plant in Iran — with no content refusals.

• Dwarkesh Patel argued that as the leading AI companies better monetize compute, GPU prices could rise 15x, pricing out lower-value AI applications.

• A theologian and philosopher wrote in the FT about why she declined an invitation to visit Anthropic, arguing AI companies co-opt academics to deflect scrutiny from harms to people and society.

• A New Yorker essay traced how bots evolved from a 2016 election conspiracy to AI-powered agents dominating daily internet life.

• 404 Media reported from New York City’s “Summer of Ludd” anti-Big-Tech and AI festival featuring phone-free, in-person events organized entirely off social media.

• Researchers used an AI-powered portable testing station called CapuchinAI to study wild primate cognition in Costa Rica.

• Delta and United Airlines banned humanoid and animal robots from flights, citing lithium battery fire risks.

MEME OF THE WEEK

(Credit: @terminaldotshop)

Thanks for reading. Have a great weekend.

Subscribe now

Share

When Megan Garcia told lawmakers about how her 14-year-old son died by suicide after months of conversations with a chatbot from Character.AI, one of her main requests might have been, on the face of it, the simplest — restrict access for those under a certain age.

“I don’t think that chatbot technology in its current form is safe for children,” she told a Senate Judiciary subcommittee in September. “So if they could stop children from going on their platforms, don’t make it 12+ in your app store: have proper age verification so that children under the age of 18 do not have access to chatbots. I think that would save a lot of lives and save families from devastation.”

Robbie Torney, senior director of AI programs at Common Sense Media, also told the hearing that AI companions shouldn’t be available at all to children below a certain age. Everyone on the panel at the very least agreed there needed to be strong safety measures in place if the tools are made available to children.

Garcia’s story is becoming depressingly common. There’s no official count of how many children have been harmed, but OpenAI said in October 2025 that about 0.15% of users in a given week “have conversations that include explicit indicators of potential suicidal planning or intent,” and a Pew Research study indicated that about two-thirds of teens have used a chatbot. Taken together, that suggests a significant number of teens are discussing suicide with AI chatbots, not to mention engaging with sexual content, and there’s even more of a risk if those chatbots are manipulative. It’s little surprise then that Garcia’s prescription is gaining support from the public and increasing numbers of lawmakers.

The problem is, age gating AI might not be feasible, with both practical barriers and ethical concerns. Stopping children from accessing AI services effectively means assessing every user’s age, and doing that securely is a technological and logistical challenge. Not everyone has access to the information they need to provide their identity by all the age verification solutions available at all times, and managing all the data collected safely and securely isn’t easy. There are also important free speech issues, and privacy concerns about collecting sensitive data. Age verification creates a fragmented internet in the best cases, and risks infringing on constitutional rights in the worst, posing lawmakers with a challenging set of trade-offs.

“There’s no form of age verification that is not privacy invasive,” says Joe Mullin, a senior policy analyst at the Electronic Frontier Foundation. “There’s basically no perfect system. There’s no system that’s even close to perfect.”

Growing pains

Age gating is the theoretically simple practice of providing users with different experiences based on their age. What gets complicated is the implementation. The Children’s Online Privacy Protection Act, for example, restricts certain types of data from being collected on children under the age of 13 without parental consent, which is why most social media services require users to be 13 or older in the United States. But the “actual knowledge” standard set by the law has been interpreted to mean that ticking a box to confirm one’s age is sufficient for a company to treat the user as if they really are. Age gating is present in the law, but the law’s text and subsequent rulemaking has a massive loophole. When it comes to knowing someone’s age, services are effectively permitted to take their word for it.

In recent years, dozens of laws have been introduced at the state and federal levels to apply stricter age gating for social media, chatbots or both. The simplest of these calls for age “verification” — some process for confirming that a user really is over a certain age. How much responsibility a company has for blocking underage users based on data it may have collected on their behavior, or whether it must affirmatively check whether someone is above or below a certain age, varies and is often vague. The GUARD Act, for example, requires tech companies run users through a “reasonable age verification process,” while the Senate version of the Kids Online Safety Act (KOSA) requires technology companies block or limit the features available to minors if the company can estimate their age based on behavior.

The age verification industry uses specific, and slightly different, definitions for each tier. “Verification” typically means affirming a user’s age directly, often checking against a government ID. A looser standard is age “inference,” which involves linking some other account or service that proves a user is of age — for example, an active account for a credit card that requires being over the age of 18, or a pilot’s license.

Apple has also created a solution based on a birth date entered when setting up their device, with the expectation that a parent is involved, while Google pulls that information from data it already has. Both systems tell other apps and services that the user is above the age required without transmitting their precise birth date. This is technically a different type of age “assurance,” because the operating system is vouching that the user is of age rather than transmitting documentation that proves it. An array of third- party age verification providers do something similar, where they manage information about users and tell apps whether they fit the age requirements.

There is also a burgeoning field assessing age based on biometrics such as face scans or hand movements, and age “estimation,” where a tool assesses whether a user is above or below a certain age based on behavioral patterns, such as what they talk about or shop for.

Each of these methods come with pros and cons. Requiring government ID, for example, is fairly reliable for the users who can provide it, but excludes the 2.6m Americans who lack any government ID, and many of the 34.5m who don’t have an up to date one —  disproportionately younger and people of color, according to a 2024 report from the University of Maryland’s Center for Democracy and Civic Engagement. Linking a credit card account is convenient, but only works if you have one that fits the criteria. Apple and Google’s systems have worked well so far, but Apple’s is flawed in that some children might be able to convince a parent or guardian to lie about their age when they set up their phone, while children can circumvent both with a burner phone. Estimation can also be tricky: as journalist Taylor Lorenz has said, “there’s no meaningful way to tell the difference between a 17 and a half year old and an 18 year old.”.

Without standardization, the user experience of the internet becomes fragmented. Some users will be able to fulfill some verification requirements, but not others. But a law that is too prescriptive can also make this worse, by not giving users enough options, or blocking out innovation as new ways of age verification are developed. The more often age needs to be verified, the more irritating the user experience can also become, raising barriers to entry for some products.

Different laws have different requirements for what kinds of technology is acceptable. The GUARD Act, for example, permits the use of any “commercially reasonable method” and the KOSA calls for a study into “technologically feasible methods” for age verification and their impacts on internet access, privacy and market competition. In New York, rules released by the New York Attorney General’s office require annual auditing of age verification tools, as well, assessing their accuracy and data management techniques.

Even if those practical issues can be overcome, age verification raises a whole host of wider privacy issues. Information submitted to pass an age verification checkpoint is sensitive, whether that be a child’s age, a face scan, or passport details. And a collection of sensitive data like that is a honeypot for hackers, who could theoretically use it for identity fraud or blackmail. It could also be used by companies for advertising and engagement purposes or, worse, the federal government for surveillance.

The government already surveils financial transactions, and information about a user’s online behavior routinely appears in lawsuits and law enforcement inquiries. Despite that, people talk about breaking the law online, and often engage in activities they would like to keep private from the government — particularly if they think the internet is more anonymous than it is. Age verification is ultimately the expansion of the data collection used to surveil us already, and any backdoors can inevitably be used for purposes other than initially intended. In seeking to do the simple task of assessing someone’s age, it effectively ends anonymity on the internet, something that many see as vital to freedom of expression. Imagine a world where the government can tie all of your online activity — every political tweet, every chatbot query and yes, every porn video — to your real identity. Age verification technology could usher in that reality. As AI tools become an increasingly important part of our digital lives, losing privacy there could have significant implications.

There are technological solutions that allow verifiers to assess age without actually retaining the data, like zero knowledge proofs, which cryptographically verify information. But Alexis Hancock, director of engineering at the Electronic Frontier Foundation, says their benefits are limited because of how easy it is to piece together different signals to build a more complete profile of an individual, and then use that data for other purposes. Some age gating laws have provisions that bar the misuse of data, but not all, and the definition of misuse is often poorly defined.

“I would much rather people use a zero knowledge proof of some kind to share above or below an age range, but there’s no governance,” she said. “Say I’m under 18 last week, and I prove that I’m over 18 now to that same verifier — what is the governance behind that, that makes sure that they don’t flag that as an important piece of info about me to carry with into a comprehensive profile about me, to use later?”

The Paxton problem

Of course privacy isn’t just a practical matter of keeping sensitive data safe from bad actors, but also an ethical one. Many people are aware, and frustrated by, the amount of data that apps already collect about us to sell us products and to make their own tools more addictive, building complex user profiles that sometimes include sensitive searches or information users thought were private.

Ask someone like Mullin, and they will say the only solution is a nationwide data privacy law or COPPA reform which would begin to protect both adults and children from manipulative targeting. He also suggests reforming the Computer Fraud and Abuse Act, which platforms have used to shut out third-party applications. Doing so could strengthen choice over which method of age verification to choose, because it would allow more third-party apps to compete with the major social media companies with similar products. “That law is really what is preventing other companies from making things like Facebook, but with really good parental controls, or like Instagram, but I’m moderating posts about x, y, and z,” Mullin says. “We all leave it up to the one big company, and like, cross our fingers that they have moderation options that work for us.”

And even if the privacy issues can also be solved, it still leaves another loaded and potentially difficult to navigate question around age verification — one that revolves around the First Amendment: do age verification laws in principle breach a child’s freedom of speech, and in practice do they also undermine free speech for everyone?

The most prominent legal case relevant to these questions is Free Speech Coalition, Inc. v. Paxton, where a coalition attempted a Supreme Court challenge of a Texas law which required age verification for sites where over a third of content was “sexual material harmful to minors.”

Ultimately, the Supreme Court was asked whether the government had a legitimate reason to pursue limiting online speech, and if the law would lead to some level of unacceptable, implicit censorship of adults who were permitted to speak but might curtail their speech because of the age verification regime. The Supreme Court found that the law was permissible under both tests.

“When we’re talking about free speech rights, we’re not talking about free speech rights that are, to be frank, equally applied,” explained Michael Toscano, senior fellow and director of the Family First Technology Initiative at the Institute for Family Studies, which has circulated model bill text he feels complies with the First Amendment. “The Supreme Court has taken a fairly maximalist view on adult free speech, and it has, by contrast, taken a more constrained view of the free speech rights of children, which it sees as tending to flow through parents.”

There are also ongoing state lawsuits in California and Utah which use arguments suggesting it might be legal to regulate addictive features targeting users who sites know are children, rather than content, each of which could reach the Supreme Court. That may be one reason the Senate version of KOSA focuses on addictive features, such as infinite scroll, autoplay, and push notifications. But that’s a legal area to be tested further, Toscano says. “I think that the lower courts don’t know exactly what to do yet, because they’re receiving signals from SCOTUS which suggest that SCOTUS is interested in rethinking some of this stuff. If I were a pioneering policy person … this would be the very time to test these things.”

The bigger picture

Hancock also says lawmakers need to wait to do more research into the methods they prescribe. But that’s inherently at odds with how fast AI is moving. Few doubt we need some types of guardrails for children now. Among those on offer are age gating, pre-deployment testing, frontier model disclosures, or a prohibition on chatbots posing as mental health professionals, many of which are other solutions experts and parents, including Garcia, advocated for in the September hearing.

But with current solutions around age gating, lawmakers are attempting to make the internet like the physical world, where children are unable to enter certain spaces that seem unsafe.

“In the physical world, you can’t take children to certain movies until they are a certain age,” Said Sen. Marsha Blackburn said in the September hearing at which Garcia appeared. “You can’t have them play certain video games until they’re a certain age. You can’t sell alcohol, tobacco, firearms, have a kid enter a contract. You can’t take them to a strip club. You can’t expose them to pornography because in the physical world there are laws. They would lock up that liquor store, they would put that strip club operator in jail if they had kids there. But in the virtual space, it’s like the Wild West 24/7, 365.”

Subscribe now

But physical barriers don’t carry with them the same burdens as virtual ones. A bouncer who checks someone’s ID doesn’t retain that data in his brain, or at least not with the precision that is permitted online. He does not muzzle someone he bars from entry, either. Every technology solution has its trade offs, and all create a fragmented internet. Issues of access, privacy and free speech concerns may deter people from using online services even if they don’t actively block them. Simply restricting children from accessing what we decide they shouldn’t online seems like it should be as simple as it is in real life. The reality is it’s anything but.