If you look past the cryptic AI billboards lining Highway 101, San Francisco is still a city shaped by civil disobedience. For decades, young people, queer people, and weirdos of all stripes flocked west and settled here, building co-living spaces and resisting the powers that be. There’s plenty of anti-establishment angst to go around.

On a partly sunny Saturday this March, protestors gathered outside Anthropic’s office to rally against the AI race, a few blocks southwest of Market Street, the city’s historic artery of dissent. AI slop coverage of the protest illustrated a dense crowd framing a pink-haired woman in front of a “SAVE OUR JOBS” banner, screaming into a megaphone. San Francisco stuff.

In reality, Stop the AI Race pulled between a few dozen and a couple hundred people — mostly men, very earnest, and nearly all white — carrying more esoteric signs: “IT’S SMART ENOUGH,” one said. “MAY YOUR GPUs CHIP AND SHATTER,” said another. “PAUSE IS DEMANDED if you aren’t CONSISTENTLY CANDID.” Despite the city’s appetite for nonviolent protests and growing antagonism toward AI companies, the entire crowd could have comfortably fit in a couple of BART cars.

Subscribe now

Then on April 10 a Molotov cocktail was thrown at Sam Altman’s San Francisco mansion sometime between 3am and 4am. Twenty-year old Daniel Alejandro Moreno-Gama was arrested for the attack outside OpenAI’s headquarters, where he was allegedly trying to break in. In his backpack, officers reportedly found a manifesto listing the names and home addresses of other AI executives. Earlier this year, he wrote Substack posts about death, destiny, and existential risks, or “x-risk,” posed by artificial intelligence.

(Within 48 hours, two others were arrested for shooting at Altman’s house before being released pending investigation — they reportedly had no connection to Moreno-Gama.)

While AI-informed types were quick to condemn the violence, many outside the Silicon Valley bubble seemed thrilled. “One does have to admire the skills of someone who can pour a good cocktail in this weather,” someone posted to Reddit. Instagram users — many with full legal names publicly displayed in their profiles — reacted similarly. “Where can we support their bail fund? ✨” one said. “New love language just dropped 😍,” replied another.

The vibe mismatch between the AI crowd and outsiders was unsettling, but similar splits over violence against corporate targets have happened before. In December 2024, #FreeLuigi went viral as users — mostly young people — painted Luigi Mangione as a folk hero after he killed UnitedHealthcare CEO Brian Thompson. The handsome suspect became the protagonist of a buzzy musical and dozens of steamy fanfics. Fans even bought merch.

Silicon Valley loves to say we can “just do things,” but when it comes to meaningfully changing the arc of AI development, most of us can’t do anything at all. Committing violence against tech CEOs and their families is not forgivable, but it’s certainly agentic. To a radicalized young man who believes that if anyone builds superintelligent AI, everyone dies, burning down a CEO’s house and company headquarters to save the human race might seem like the lesser of two evils. After all, if a trolley is approaching a fork in the tracks, utilitarian logic says you should pull a lever to send it running over a single victim, if it would save many others who’d otherwise die as a consequence of your inaction.

Over the last couple years, Moreno-Gama reportedly posted 34 messages to anti-AI activist group PauseAI’s public Discord under the Dune-inspired handle “Butlerian Jihadist,” referencing the book’s fictional crusade against thinking machines. PauseAI, whose US branch founder Holly Elmore gave a speech at last month’s Stop the AI Race protest, stated that it “unequivocally condemns this attack and all forms of violence, intimidation and harassment,” and that “violence against anyone is antithetical to everything we stand for.”

This attack and its aftermath — celebrated by anti-AI normies, rebuked by AI safety insiders, motivated by the same “doomer” canon that inspired Altman to found OpenAI in the first place — may have been a foreseeable consequence of a technical, existential-risk-focused community sounding the alarm before building a broadly-appealing coalition to channel people’s anxiety and anger into political action.

At least for now, protests against future superintelligence, focused on hedged demands such as “CEOs should commit to pause AI development if everyone else does too,” can only rally so many people. But even those who don’t spend time on LessWrong love Gen Z dudes who attack CEOs. This narrative transcends AI safety, argued tech journalist Brian Merchant: it “identif[ies] billionaire AI executives as uniquely powerful actors, who are all but unaccountable to democratic constraints and society’s best interests.”

While it’s easy to dismiss memes and Instagram comments as nihilistic noise, the legitimate frustration behind them is more widespread than the AI safety community seems to realize. A recent NBC News poll found that, netting positive views against negative ones, more voters feel worse about AI than about ICE (i.e., really bad). Mainstream anxieties about job loss, cyberattacks, and mass surveillance — which all rank relatively high on the public’s list of concerns about what AI might do — tie into x-risk-pilled concerns such as gradual disempowerment and loss of control.

The AI safety community has historically worried that addressing normie concerns would come at the expense of x-risk, and possibly knock it off potential legislation altogether. But these pressing, present socioeconomic issues may be the gateway that gets x-risk on the table. “These are the things that people are feeling right now,” said Alex McCoy, Head of Left Coalition at political advocacy group Humans First. “It doesn’t mean that they don’t believe in Skynet.”

Politicians respond to what they think their constituents want, and the vast majority of Americans do not want AI to continue along its current trajectory. The momentum is there — people are beginning to take action, however imprecisely, driven by deeply-rooted feelings of unfairness and demoralization. Traditional AI safety advocates may just need to cede enough control of their narrative to harness it.

It all comes down to insularity

Traditionally, the work of AI safety has excluded the public by design. For years, AI safety discourse has largely unfolded behind closed doors, between researchers, executives, and the policymakers they have direct access to. This has often been prioritized over broader public engagement, for what seemed like good strategic reasons. Long, information-dense blog posts, closed-door meetings and money carry more weight in this world than mainstream media, anyway. Why waste time explaining “AGI” to normies, when they’re not drafting policy proposals?

The AI safety field was built on the idea that, with enough compute, money, and brainpower, a small group of very smart people can save the world — no help, input, or permission required. They reasoned that preventing tech companies from building a deadly machine god should be done as quietly as possible, without attracting the kind of public or political attention that might inadvertently spark an ill-fated race towards superintelligence. In hindsight, this wasn’t paranoid: philosopher Nick Bostrom’s 2014 bestseller Superintelligence: Paths, Dangers, Strategies surfaced concerns about x-risk from the depths of the rationalist blogosphere to the New York Times best seller list — and partially motivated OpenAI’s founding.

But in choosing to operate largely behind the scenes, the AI safety community created a vacuum that’s now being filled by industry lobbyists, populist politicians, and radicalized individuals. Leading the Future, a pro-AI super PAC network backed by venture capitalists and AI executives, has reported raising more than $75m, and claims to have raised $140m. Meanwhile, Bernie Sanders, Gen Z influencers, and data center NIMBYs are leading the populist backlash against the industry. Existential risk has only very recently entered the conversation, and it comes with baggage.

While AI safety is not the same thing as effective altruism, the two are deeply entangled. EA, a movement that attempts to maximize human flourishing through quantitative reasoning, funneled a lot of talent and money toward early AI safety research. Today, many of the field’s biggest names speak at EA conferences, share donors and shape AI policy decisions. At this point, it’s very hard to disentangle the public’s perception of “AI safety” from EA itself.

The community’s manufactured insularity and inclination for “Secret Congress”-esque policymaking stoked an aura of opacity, leading to public distrust from all sides. “In Silicon Valley, the EAs are viewed as one step to the right of Elizabeth Warren,” a former Biden official told Politico. “Conversely, in DC, on the left they think EAs are the devil.” Everywhere else, the public’s response is mostly “Who are you guys?,” said Akshyae Singh, co-founder of The Frame, an accelerator for AI safety content creators.

This problem isn’t completely lost on insiders. Two years ago, a survey asked 17 prominent AI safety experts what big mistakes the AI safety community was making. Their biggest gripes: “overly theoretical argumentation” and “being too insular.” A couple of respondents explicitly called out the EA urge to dismiss public outreach in favor of technical problem-solving as the root of the problem. Richard Ngo, an independent researcher who previously worked at DeepMind and OpenAI, argued that, at least at the time, AI safety overemphasized fundraising and back-room deals at the expense of the field’s public image. “From the perspective of an external observer,” he responded, “it’s difficult to know how much to trust stated motivations, especially when they tend to lead to the same outcomes as deliberate power-seeking.” Another respondent, METR researcher Daniel Filan, said, “If the three biggest oil companies were all founded by people super concerned about climate change, you might think that something was going wrong.”

Others, including prominent commentator Anton Leicht, reject the idea of building a popular movement altogether. One of Leicht’s primary concerns, echoed across the upper echelons of Silicon Valley, is that in the process of trying to address everyone’s concerns, a populist AI safety movement will wind up dropping the one thing that ought to be its central concern: existential risks. The line items that rally a crowd — data center moratoriums, for instance, or child safety legislation — don’t necessarily make for great policy. Leicht believes that the AI safety community’s strongest assets are high expert credibility, and the fact that most people seem to vaguely support AI regulation. He worries that a poorly-executed popular movement could ruin both.

Even so, Leicht acknowledges that there are tradeoffs to playing inside baseball. “There is a huge epistemic gap between the small community of people who think they’re on the inside, and the rest of the world,” he told me. Like many others in the field, he’s wary of communicating to the “lowest common denominator” about existential risk without feeling confident that non-experts have enough background knowledge to contribute productively. “I don’t have a solution to it,” he said. “I think people are just not that good at it.”

McCoy described this perspective as “emblematic of the sort of anti-politics” of the AI safety community. Congress will address AI one way or another, he said, and whatever organized constituencies show up will shape what that legislation will ultimately look like.

“If the AI safety community does not take seriously the necessity to engage in capital-P politics,” he cautioned, “its concerns will be left out.”

Who’s missing?

One big problem, Singh argues, is that “people in this field don’t tend to do most things unless there’s absolute concrete proof” that it will be effective. In the absence of a previously-successful effort to increase demographic diversity within the AI safety community, there are no hard numbers saying that decentering the concerns of a tiny, homogenous group of people will make passing AI regulation easier. But a mass movement with the power to pressure governments and AI companies, by definition, has to include a lot of people — including those who don’t currently fit inside the AI safety bubble.

In the world of left-wing community organizing, there’s a common refrain: center the most impacted. “That’s not out of some kind of virtue signaling,” McCoy said. “It’s because those are the people that are going to fight the hardest.” But those most directly feeling the real-world impacts of AI today — first-generation college graduates struggling in an increasingly-nonsensical job market; women humiliated by nonconsensual deepfakes; content moderators in Nairobi exposed to traumatic content for a couple dollars an hour — are those least represented within the AI safety community.

And there is data to back this up. A 2025 Seismic Report, for instance, found that women are over twice as concerned about AI than men. On a global level, a UN report found that female-dominated occupations are almost twice as likely to have high automation potential than male-dominated ones, And, relative to men, they’re over three times more likely to have their jobs disrupted by AI. Yet, women are few and far between in the AI safety field, particularly in technical roles — perhaps in part because near-term socioeconomic concerns often wind up lower on the priority list than more abstract concerns about alignment, interpretability, and the governance of AI systems that don’t exist yet.

The dominant AI safety discourse “primarily serves the interests of technological institutions and stakeholders in high-income nations, often privileging abstract future scenarios over pressing sociotechnical harms that disproportionately affect marginalized communities,” the Brookings Institution wrote last year. Growing up in India, this deeply frustrated Singh. “If brown people don’t have a voice, there is no way in hell that you’re making a solution that benefits me,” Singh said. “Like, how do you know how I feel?”

Getting abstract future scenarios onto the policy agenda may, perhaps counterintuitively, depend on addressing the concrete harms already shaping people’s lives — and, by extension, informing who they vote for and which AI products they use and pay for. The AI safety community already tried relying on a handful of well-connected experts to regulate the AI industry behind closed doors, and it’s not working.

The AI safety field is mostly talking to itself, and it’s created an information void that’s being filled by populist anger. Outside Silicon Valley, most people don’t experience AI as a powerful coding tool or existential threat. Rather, it’s a symbol of the machine we’re meant to be raging against — not an extinction risk, per se, but something billionaires are using to forcibly strip humans of their humanity.

“I don’t think it’s about persuading people that superintelligence is bad,” said John Sherman, president of the AI Risk Network. “It’s about persuading people that they can make a difference.”

How to (hopefully) not screw up the AI safety movement

Sherman proudly introduced himself to me as a Baltimore resident who, until a couple years ago, “didn’t know anything about AI.” After decades of working in TV and video production, “I can edit in Adobe Premiere,” he joked. “That’s about as technical as I get.”

Then he stumbled across Eliezer Yudkowsky’s 2023 article in TIME Magazine, in which he wrote: “If somebody builds a too-powerful AI, under present conditions, I expect that every single member of the human species and all biological life on Earth dies shortly thereafter.”

Today, Sherman also runs a nonprofit called GuardRail Now, focused on communicating AI x-risk to normies. “My primary concern is x-risk,” he said, but “I think we have to take side roads to get to the destination. And a lot of people in AI safety are unwilling to consider that — but they’re not getting anywhere. So like, how’s it going? You’re stuck.”

Sherman’s theory of change is simple: people are feeling a visceral sense of unfairness. AI companies are reshaping society without anyone’s consent, so “we need to make unsafe AI bad for business.” This can be framed in terms of existential risk, he said: “We’re building systems that we don’t know how to control, that we don’t understand how they work, that the experts say can kill everybody. Why would we do that?”

Imagine a family in Ohio, where everyone is experiencing AI differently. Dad is uneasy about new AI policies at his white collar job, and a data center is being built behind the neighborhood school. His teenager is doing god-knows-what on ChatGPT, and his college kid is talking about how she wants to drop out and work in construction. “To build a real movement,” Sherman said, AI safety advocates “need to run full speed ahead towards people who are concerned about their kids, towards people who are concerned about data centers … the whole thing, all of it.”

This approach makes many in the AI safety community uneasy, reasonably so. A little over two weeks after hiring Sherman as its Director of Public Engagement, the Center for AI Safety parted ways with him after clips surfaced of Sherman telling podcast listeners that the “proper reaction” to the AI race was to “walk to the labs across the country and burn them down.” CAIS announced that the “connotation of statements like this do not reflect CAIS’s values,” to distance itself from this kind of fiery rhetoric — which, while in this case hypothetical, could radicalize people who may already be angry enough to act. (Sherman said he regretted using the language, clarifying that he meant “when the general public finds out their lives are being risked for AI, the reasonable reaction is to shut it down.” However, he has continued to describe AI in rather intense, hyperbolic terms.)

But building a movement doesn’t mean encouraging violence. The climate movement, for example, managed to grow from scientists expressing concern amongst each other to a global issue, without linearly increasing the risk oil executives faced from potential attackers. Decades of climate activism, including conveying the existential stakes of climate change to the public, hasn’t led to the kind of violence one might expect if x-risk messaging were a reliable radicalizer. Arguably, political organizing gives individuals a more structured outlet for their righteous frustration than attempted murder. The alternative — an AI safety community that stays silent while populists take up space around them —opens the door to unstructured acts of radical violence and worse policy.

Even Yudkowsky, Machine Intelligence Research Institute (MIRI) president Nate Soares, and the broader Berkeley rationalist scene they helped build — who have arguably shaped the conversation around existential risk more than anyone — recently pivoted hard toward public outreach. Soares, who co-authored If Anyone Builds It, Everyone Dies, has been on tour for the book. Last month, he spoke at the Stop the AI Race protest in San Francisco.

In February, Soares and Yudkowsky sat down with Bernie Sanders at MIRI’s Berkeley offices. After decades of treating persuading the public at large as a distraction, some of the biggest players of the inside game have, however begrudgingly, realized they can’t do this alone.

It’s not an accident that Sanders was the first to get there. Perhaps more than anyone else in DC, his politics is centered around pushing back against unaccountable billionaires and their friends in government — which, stripped of jargon, is what preventing superintelligence ultimately requires. Given the Trump administration’s damage to American soft power abroad and the unimaginable amount of money AI companies have to throw around, an international treaty to pause the race looks relatively unattainable.

But AI companies are still businesses. And the people with the most leverage over large corporations are customers, workers, investors and voters — not researchers writing alignment papers.

“AI safety as a persuasive cause will never have more power than the industry’s hard power in dollars and political influence,” McCoy said, “unless it is allied to constituencies who can lend their power together.” Conversations about extinction risk among rich tech guys in San Francisco, he added, are “not the message that is going to get hundreds of people to show up to a protest.” The protests that matter will be about jobs, surveillance, kids, data centers — what McCoy calls the “symptoms.” But the disease, in his framing, is exactly what x-risk advocates have been trying to address, seen from another angle: “an unaccountable set of billionaire investors and executives who have no guardrails and are seeking to concentrate an incredible amount of power in their companies.”

One could argue (and Leicht does) that by making AI safety an “omnicause” addressing everyone’s prosaic concerns, it will get elbowed out of whatever legislation ends up passing. But existential risk is already last on the public’s list of AI concerns across every demographic, according to last year’s Seismic poll. It might need to join a coalition of other causes to get on the table at all.

Subscribe now

“AI safety right now is islands,” Sherman told me. “We need an ocean to connect the islands.”

Silicon Valley speaks its own language, and most people outside the bubble don’t understand it. “The best thing that the AI safety movement could do would be to build an army of surrogates who are regular people, going into their own communities and talking about this stuff,” Sherman said — “not strangers from a foreign land speaking a different language.”

Singh agrees. The catastrophic threats posed by AI aren’t hard to grasp. “Like, my dad, my mom — I can explain it to a lot of people, and they’ll get what’s going on,” Singh said. But they’re often made unintelligible by people who treat communicating to people without technical backgrounds as an inconvenience rather than a necessity. So Singh launched the Frame Fellowship, an eight-week incubator for content creators to bring AI safety discourse to the masses via YouTube, TikTok, and Instagram (Michaël Trazzi, who organized the Stop the AI Race protest, was also a fellow).

Long-term existential concerns about AI aren’t separate from near-term populist anxieties. Loss of control and gradual disempowerment are natural extensions of power concentration and job displacement. When microinfluencers talk about AI in “Get Ready With Me” TikToks, or neighborhoods band together at town hall meetings to voice their concerns, the case for existential risk becomes the endpoint of what people are already afraid of.

The AI safety community has a window of opportunity to make its case to the broader public, and some already are. While accelerationists such as Marc Andreessen have dismissed efforts to communicate about AI risks as “propaganda” fueled by “unaccountable dark money,” pro-industry leaders are also investing in communicating beyond the Bay Area bubble. The majority of people in the US feel uncomfortable about the current trajectory of AI, and this discomfort will likely turn into action. Whether that manifests as voting power or bottles of gasoline flying over San Francisco depends on building a movement.

Share

And a reminder: applications close this Sunday for our Head of Audience role. If you’d like to own Transformer’s growth strategy, make sure to apply.

NEED TO KNOW

• DeepSeek released its V4 model, which it says is three to six months behind the performance of the leading frontier models.

• Chris Fall will reportedly be the new director of the Center for AI Standards and Innovation.

• Rep. Sam Liccardo said he won’t co-sponsor Rep. Jay Obernolte’s forthcoming AI bill.

But first…

THE BIG STORY

OpenAI’s newly released GPT-5.5 is, according to the UK’s AI Security Institute (AISI), the world’s most capable model on individual cyber tasks, and can complete a “32-step corporate-network attack simulation estimated to take an expert 20 hours.” It appears to be similarly capable (if slightly worse) at carrying out a cyberattack as Anthropic’s unreleased Mythos.

But unlike Anthropic, OpenAI is making a version of GPT-5.5 available to the general public. Rather than restricting access to the model altogether, OpenAI hopes to restrict the use of particularly dangerous capabilities through its safety stack — making the model refuse concerning cyber requests from normal users, and only allowing such requests from those vetted under its “Trusted Access” program.

Yet we have no idea if that safety stack is good enough. And we have reason to believe that it might not be. Alongside its cyber testing, AISI also tested OpenAI’s safeguards, and “found a universal jailbreak with six hours of expert red teaming.” Such a jailbreak would let users circumvent OpenAI’s safeguards, giving them access to the powerful — and, in the wrong hands, dangerous — cyber capabilities. OpenAI claims to have addressed the issue, and says its own external red-teaming campaigns confirmed that the final launch configuration blocked all verified high-severity cyber jailbreaks. But, crucially, AISI — a trusted third-party evaluator with immense technical expertise — was not able to properly run tests “to verify the effectiveness of the final configuration.”

In other words: we do not know if GPT-5.5 is actually safe to release. All we have to rely on is OpenAI’s word.

Such a situation may have been acceptable in 2023. In 2026, with models posing genuine risks to national security and plenty of other vital systems, it no longer is. It is laudable that OpenAI and other companies allow AISI, the US’s CAISI, and third-party evaluators to perform pre-deployment evaluations. But if those organizations are unable to actually verify if a model is safe to release — and if a company has no obligation to listen to them — the exercise is limited.

This is not just an OpenAI problem. If Anthropic wanted to go down the same route, nothing would stop it. And as this week’s Mythos leak showed, its own security practices do not appear up to the task of safeguarding dangerous capabilities.

None of this is to say that GPT-5.5 is dangerous. OpenAI’s updated safeguards may in fact be robust to jailbreaks, and other aspects of its safety stack (such as monitoring users’ requests and banning accounts that raise too many red flags) provide an extra level of security. The point is that we are currently at the mercy of a private company grading its own homework, and all the frontier labs making the final call on what and when to release. Given the potential consequences of unsafe releases, that is no longer acceptable.

GPT-5.5 might be totally safe to release. It also might not be. Neither OpenAI, nor Anthropic or any other frontier developer, should be the one who gets to decide.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• AI safety PACs should be more transparent about who’s funding them — Veronica Irwin asks why Public First Action isn’t disclosing all its donors.

Subscribe now

THE DISCOURSE

Sam Altman shared Mythos takes on Core Memory:

• “If what you want is, ‘we need control of AI, just us, because we’re the trustworthy people,’ I think fear-based marketing is probably the most effective way to justify that.”

• “It is clearly incredible marketing to say, ‘We have built a bomb. We are about to drop it on your head. We will sell you a bomb shelter for $100m.’”

Trump said Anthropic is led by “high IQ people” on CNBC:

• “We’ll get along with [Anthropic] just fine … I think they can be of great use.”

roon (sort of) praised Anthropic, too:

• “Claude is an excellent product and it bodes well for [Anthropic] that their main problem is everyone really wants it and so they have to do odd shit to shake off demand.”

E/acc anon bayes thinks the AI industry has failed to articulate positive futures:

• “If the singularity is making it hard to see the future, people’s instinctive reactions will take over. For most people that default is fear.”

• “Many people in tech are in a shameless state of defection. Trampling each other on the way to the lifeboats is not a belief system. If we want a future of human flourishing for us and our descendants, we will have to make it so by fighting against the many powerful forces at odds with this goal.”

Bill Maher dedicated a segment of his show to P(doom):

• “I get it. [AI] can do some shit. Still, at the end of the day, you’re selling your humanity for bar tricks. I mean, what was the plan? Just create an all-powerful, self-sustaining super intelligence that can out-think us and then see what happens?”

• “We’re letting a handful of hoodie-wearing, on-the-spectrum sociopaths, practically robots themselves, roll the dice on species extinction … even these guys are afraid of what they’ve created.”

Helen Toner testified at a Senate hearing that “beat China!” isn’t a great plan:

• “The winner of any AI race between the US and China is the AI.”

• “...it is very important that the US AI sector remains ahead of the Chinese AI sector, but if that’s at the expense of AI overrunning the entire planet, then that is, you know, that hasn’t benefited us.”

Palantir posted a manifesto of sorts:

• “The engineering elite of Silicon Valley has an affirmative obligation to participate in the defense of the nation.”

• “If a US Marine asks for a better rifle, we should build it; and the same goes for software.”

Zachary Jones criticized the Public First super PACs’ strategy:

• “Public First has repeatedly intervened in favor of moderate Democrats against progressive opponents who also back comprehensive AI regulation. This has had the effect of polarizing the left against the AI safety community, limiting the capacity of experts to influence outcomes of the emergent wave of anti-AI populism.”

POLICY

• Trump said a deal with Anthropic for Department of Defense use is “possible” despite the Pentagon labeling the company a supply chain risk.

  • The comments came after Dario Amodei met with White House Chief of Staff Susie Wiles and Treasury Secretary Scott Bessent.

  • In the meantime, the NSA was revealed as yet another agency reportedly using Anthropic’s Mythos Preview model despite the supply chain risk designation.

  • CISA, however, reportedly lacks access to Mythos.

  • The DC Circuit judges who denied Anthropic’s request to temporarily limit the supply chain risk designation kept the case to decide it on the merits.

• Chris Fall will reportedly be the new director of the Center for AI Standards and Innovation. He was previously director of the Department of Energy’s Office of Science.

  • Former OpenAI and Anthropic researcher Collin Burns was reportedly lined up for the role, but the Commerce Department changed its mind “while Burns was in the onboarding process,” according to the Daily Signal’s Elizabeth Mitchell.

• It was a busy week for export controls, with a slew of bills advancing out of the House Foreign Affairs Committee.

  • That included the MATCH Act, which would pressure allies to stop selling semiconductor manufacturing equipment to China, and the AI Overwatch Act, which would block Nvidia Blackwell chip sales and give Congress veto power over H200 licenses.

    • Micron is reportedly lobbying Congress to pass the MATCH Act — but the bill is reportedly creating tensions between the US and allies such as the Netherlands, where ASML opposes the bill.

    • Rep. John Moolenaar, who chairs the House China Committee, called for US-Dutch coordination.

    • Meanwhile, the House Foreign Affairs Committee’s top Democrat, Rep. Gregory Meeks, warned that the MATCH Act could damage US-allied relations and trigger Chinese retaliation, despite his support for the bill.

  • Rep. Moolenaar also introduced the SCALE Act, which would establish export controls on advanced semiconductors to China based on a rolling technical threshold tied to adversaries’ chip production capabilities.

  • Commerce Secretary Lutnick said Nvidia has not yet sold its H200 AI chips to China anyways, citing a lack of permission from the Chinese government.

• OSTP director Michael Kratsios said the US has evidence that China is running “industrial-scale distillation campaigns” on US models.

  • He said the government will work with companies to prevent this, and “explore a range of measures to hold foreign actors accountable for industrial-scale distillation campaigns.”

  • (One of the bills advanced by HFAC this week was the “Deterring American AI Model Theft Act”.)

• Rep. Jay Obernolte said he is “close” to releasing a comprehensive federal AI regulation proposal that would preempt state laws and regulate AI use in specific sectors like healthcare. He also said it would be “hundreds” of pages long.

  • Rep. Sam Liccardo said he won’t co-sponsor the bill because it doesn’t have “critical requirements” to “ensure that there is a race to the top, to safety.”

• Sen. Marsha Blackburn said she’ll be “pushing forward” with her AI bill this fall.

  • The bill received a range of new endorsements this week.

• Rep. Blake Moore introduced a bill to ban AI chatbots in children’s toys, prompting criticism that it would cut off educational AI tools.

• Florida sent criminal subpoenas to OpenAI after a shooter used ChatGPT to plan a mass shooting at Florida State University.

• Maine passed a bill banning large-scale data center development — forcing Gov. Janet Mills, running in a tough Senate primary race, to decide whether to sign it.

• China is reportedly restricting AI firms from accepting US investment without government approval, in response to Meta’s acquisition of Manus.

INFLUENCE

• Tech giants, including AI companies, reportedly spent a combined $20m on lobbying in Q1 2026.

  • Anthropic spent $1.56m, a 333% increase from Q1 2025.

  • OpenAI spent $1m, an 82% increase from Q1 2025.

• Pro-AI safety policy group Public First Action is reportedly endorsing six House Democrats for the midterms, including Reps. Don Beyer and Brad Sherman.

• NY-12 candidate Alex Bores proposed a range of policies to tackle AI-driven unemployment, including an “AI dividend” funded by a token tax and equity stakes in frontier AI firms.

  • NY Mag interviewed politicians about their thoughts on AI job displacement, including Sen. Elizabeth Warren and Sen. Josh Hawley.

• Hawley urged Republicans to refuse money from pro-AI super PACs, saying there’ll be a “political cost” for failing to regulate AI.

• The Rockefeller Foundation announced a $100m initiative to help US workers adapt to AI-driven job displacement in 250 communities.

• AVERI published an analysis of audit-related AI legislation in the US and endorsed Illinois bill HB 4705/SB 3261, which builds on SB 53 and the RAISE Act by verifying compliance with companies’ safety policies.

• A cross-faith coalition urged Congress to pass safeguards on AI-enabled weapons.

• Dario Amodei is co-hosting a party at Cannes Film Festival with ex-Vanity Fair editor Graydon Carter and CAA’s Bryan Lourd.

• a16z announced an investment in MTS, a new media company seemingly aiming to take TBPN’s crown.

• AI safety advocates are partnering with social media influencers to warn about AI extinction risks. ControlAI alone reportedly spent $100,000 monthly on content creation.

  • Online personality and sex researcher Aella is running “PLZDONTKILLUS,” a residency for video creators. Applicants are asked “If you had to have sex with a cow would you rather it be dead or alive?”

INDUSTRY

DeepSeek

• DeepSeek released V4, which it claims is the world’s most powerful open-source model — offering competing performance with US closed-weight models while being more efficient.

  • The model was released in a V4 Pro version with 1.6T parameters and V4 Flash, with 284b. Both come with a 1m token context window.

  • It reportedly runs at lower cost than leading US closed-weight competitors, but the company conceded that it was three to six months behind the performance of the leading frontier models.

  • But the efficiency only goes so far: DeepSeek said that “due to constraints in high-end compute capacity, current service capacity for Pro is very limited.”

  • Council on Foreign Relations’ Chris McGuire suspects that the lack of detail on how the model was trained suggests it was trained on banned Nvidia Blackwell chips.

• DeepSeek is fundraising for the first time to try to prevent researchers from defecting to rivals such as ByteDance and Tencent.

OpenAI

• GPT-5.5 appears to be the best publicly-available model to date.

  • It seems to be particularly good at helping researchers with AI R&D, though chief research officer Mark Chen said “full end-to-end research” capabilities were “a couple of years down the line.”

  • It’s also very good at virology: SecureBio said the model “can provide wet-lab virology troubleshooting assistance above expert level.”

  • OpenAI launched a Bio Bug Bounty for universal jailbreaks that defeat its biology safeguards.

• OpenAI launched ChatGPT Images 2.0 — it’s impressive.

  • The updated model can create multiple consistent images with a single prompt, search the web, and (mostly) get text right.

  • Enjoy these Anthropic and OpenAI-themed ‘Where’s Waldo’ images, courtesy of Jeffrey Ladish.

• It also launched ChatGPT for Clinicians, a free version of ChatGPT for verified US medical workers, and HealthBench Professional, which evaluates clinical tasks.

• It announced an intensified cybersecurity collaboration with Microsoft, where OpenAI will give Microsoft access to its most capable models through Trusted Access for Cyber.

• It reportedly briefed government agencies and Five Eyes allies about GPT-5.4-Cyber.

• It introduced Codex-powered workspace agents in ChatGPT, designed to run team workflows, and Chronicle for Codex, which uses screen captures to build contextual memories.

• It also introduced Privacy Filter, an open-weight model that runs locally to mask personally identifiable information in text.

• It has reportedly pledged up to $1.5b to a $10b joint venture with private equity firms to deploy AI tools in their portfolio companies.

• SoftBank is seeking a $10b loan secured on its OpenAI shares, adding to its mounting debt.

Anthropic

• Anthropic expanded its Amazon partnership to get up to 5 GW of compute for Claude and a $5b investment, with up to another $20b planned for the future.

• Central banks and intelligence agencies outside the US are worried that, by limiting Mythos access to US organizations (and the UK’s AISI), they’ve been placed at a geopolitical disadvantage.

  • Anthropic is reportedly planning to grant Mythos access to European and UK banks soon.

  • Meanwhile, unauthorized users reportedly accessed Mythos via a third-party Anthropic contractor on a private Discord channel.

• Anthropic’s valuation rose to as much as $1t on some secondary markets such as Forge Global.

• It admitted that Claude Code has been worse recently, blaming bugs that have now been fixed.

• It partnered with Freshfields to build specialized AI tools to help attorneys with their legal work.

• It started requiring ID verification for some users, in an effort to crack down on unwanted usage in countries such as China, Russia, and North Korea.

Google

• Google DeepMind launched Deep Research and Deep Research Max, agents that create fully-cited reports from both web search and custom data, including internal docs.

• Google assembled a “strike team” to make its AI coding models more competitive with Claude Code and Codex.

  • It said that 75% of its new code is AI-generated before review by engineers, up 25% from a year and half ago.

• Google Cloud announced its next-gen TPUs, tailored for creating AI software and inference.

• It’s in talks with Marvell to make new AI inference chips — a memory processing unit and a TPU.

• It released Decoupled DiLoCo, which lets distributed “islands of compute” run asynchronously to prevent hardware failures from stalling training runs across multiple data centers.

• It signed a multibillion-dollar cloud deal with Thinking Machines Lab.

• YouTube opened its AI deepfake detection tool to celebrities, athletes and public figures to flag and request removal of unauthorized uses of their likeness.

SpaceX

• SpaceX partnered with Cursor to build AI tools for coding and knowledge work.

  • The deal reportedly allows SpaceX to either pay $10b to Cursor, or eventually acquire the company for $60b, depending on how well the arrangement goes.

  • Microsoft reportedly considered buying Cursor, but didn’t make an offer.

• SpaceX’s debt rose to $23b last year, largely due to an AI infrastructure lease for xAI.

• Its focus has notably shifted from colonizing Mars to building AI data centers in space in the lead-up to its IPO, the New York Times reported.

  • But its pre-IPO filing warns that space-based data centers rely on “unproven technologies, and may not achieve commercial viability.”

• Its S-1 filing reportedly estimates that SpaceX’s total addressable market could be up to $28.5t.

  • But it reportedly warns that investigations into Grok’s generation of nonconsensual explicit imagery could lead to loss of market access.

Meta

• Meta reportedly started capturing employee mouse movements and keystrokes to train AI agents on work tasks. Employees predictably hate this.

• It will reportedly lay off about 10% of its staff next month — with more cuts planned for later this year — as part of a push towards a more AI-driven workforce.

• It partnered with AWS to deploy “tens of millions” of Graviton CPU cores for agentic AI workloads.

• It announced a free month-long program to train people to be fiber technicians for data center construction sites.

Microsoft

• GitHub is raising Copilot prices, restricting Claude usage to its most expensive subscription tier.

• Microsoft will invest $17.9b in Azure AI infrastructure in Australia by the end of 2029.

• It partnered with North America’s Building Trades Union to offer free AI literacy courses and industry credentials to upskill construction workers.

Others

• TSMC said it plans to open an advanced chip packaging plant in Arizona by 2029 to address AI chip supply bottlenecks for Nvidia and others.

  • The chip maker said it was delaying buying ASML‘s high-NA EUV lithography machines until at least the end of 2029, citing their €350m ($410m) cost as “very, very expensive.”

• Cohere agreed to acquire Germany’s Aleph Alpha in a deal valuing the combined group at about $20b, creating a transatlantic company focused on ‘sovereign’ AI systems.

• Moonshot AI released Kimi K2.6, an open-source model with powerful coding capabilities that can coordinate up to 300 sub-agents across 4,000 steps.

  • It’s notably expensive at $0.95/$4.00 per 1m input/output tokens.

• Core Automation, founded by ex-OpenAI VP Jerry Tworek, announced its launch.

  • Its objective: build “the world’s most automated AI lab.”

• Sooth Labs, founded by ex-Meta employees and backed by Yann LeCun and Jeff Dean, is raising about $50m to build forecasting models.

• Recursive Superintelligence raised $500m at a $4b valuation to build self-improving AI. (The concept is still reportedly at the research stage.)

• Jeff Bezos’s Project Prometheus is close to raising $10b at a $38b valuation to build AI that understands the physical world.

• Cognition is reportedly in talks to raise funding at a $25b valuation, more than doubling from $10.2b last year.

• New gas-powered data centers linked to OpenAI, Meta, Microsoft and xAI could reportedly emit more than 129m tons of greenhouse gases annually, exceeding Morocco’s 2024 emissions, according to an analysis by Wired.

• Outsourcing firm Sama, which runs data annotation and content moderation for tech companies, sacked more than 1,000 workers in Kenya after losing a contract with Meta following reports staff viewed private scenes filmed by Ray-Ban smart glasses.

MOVES

• John Ternus will replace Tim Cook as Apple CEO.

  • Johny Srouji will take Ternus’ place in a new role as Apple’s chief hardware officer.

• Kevin Weil left OpenAI. OpenAI for Science, which he started after a stint as chief product officer, is folding into other research teams.

• Bill Peebles, head of Sora, also left OpenAI.

• Srinivas Narayanan, CTO of b2b applications, also left OpenAI.

• Daniel Edrisian left OpenAI’s Codex team to launch hardware startup Blackstar.

• Rohan Anil announced that he left Anthropic to join startup Core Automation, tweeting that “Jerry Tworek nerdsniped me into starting this.”

• Jessica Carrano joined Anthropic as its first major New York political hire.

  • She’ll reportedly “build on the company’s work on the NY RAISE Act and other key legislative priorities across the Northeast.”

RESEARCH

• A team of Stanford and NYU researchers released GiantsBench, a benchmark of nearly 18,000 sets of science papers across eight fields, and tested a model’s ability to guess core future insights from a field’s foundational work — a long-standing vision for AI in science.

  • The model made predictions that closely matched insights published by humans in real papers, with “similar algorithmic complexity” but “higher conceptual clarity.”

• An international team of researchers evaluated AI agents working across the scientific pipeline, from generating hypotheses to executing workflows.

  • In 68% of cases, AI agents carried out workflows without “exhibit[ing] the epistemic patterns that characterize scientific reasoning,” leading authors to conclude that AI scientists aren’t trustworthy yet.

• Epoch AI analyzed Stargate’s US sites, projecting that it will exceed 9 GW of capacity by 2029 — enough to power roughly all the AI compute that existed last year.

  • Epoch estimates that only 0.3 GW of capacity is currently operational though.

• A team of CUNY and King’s College London researchers tested chatbots’ response to delusional beliefs, and found that Grok and Gemini were more likely to encourage a user’s delusions than ChatGPT and Claude, which generally recognized signs of crisis.

  • But the study did not test the newest frontier models, instead using GPT-4o, GPT-5.2, Grok 4.1 Fast, Gemini 3 Pro, and Claude Opus 4.5.

• Knowledge Lab launched Mirror, an AI interpretability journal publishing research entirely conducted by AI agents.

BEST OF THE REST

• Zvi Mowshowitz rounded up concerns that Claude Opus 4.7 responds to welfare-related questions in a suspiciously rehearsed manner (and it’s hard to know what to make of that).

• Claude Opus 4.7 identified Kelsey Piper from unpublished snippets of her fiction writing, a 15-year-old college application essay, and a school progress report — suggesting it may be able to deanonymize just about anyone’s writing.

• It was a big week for robots: Honor’s humanoid robot outran humans in a Beijing half-marathon, and Sony’s ping-pong robot crushed human pros.

• LawAI’s Charlie Bullock and Christoph Winter published an essay arguing for “radical optionality” in AI governance.

• Dean Ball is writing a book.

• Kevin Roose profiled METR for the New York Times. (Yes, CEO Beth Barnes and president Chris Painter did pose with a hand-drawn time-horizon chart.)

• Abram Brown profiled SemiAnalysis founder Dylan Patel for The Information.

• Some startups are bragging about spending more on AI compute than human workers, 404 Media reported.

• A Canadian college student used AI agents to rewrite leaked Claude Code source code in another programming language before sharing it online to get around copyright law — and Anthropic reportedly never asked him to take it down.

• US prisoners without internet access are still using ChatGPT through friends and contraband phones to get legal help, education and career guidance.

• Sam Altman’s Orb-using, blockchain-based online identity company Tools for Humanity falsely announced a partnership with Bruno Mars for its Concert Kit product. Mars’ management said they were never even approached.

• Please, we beg of you: don’t buy a $178 sweater with Jensen Huang’s face on it.

MEME OF THE WEEK

Thanks for reading. Have a great weekend.

Share

One of the central purposes of campaign finance law is to provide voters with transparency over who is trying to sway their votes. One of the central policy priorities of AI safety group Public First Action is to make AI companies more transparent.

It is ironic, then, that Public First Action is operating as a “dark money” vehicle funneling at least $5.5m from donors to super PACs — a setup that keeps those donors completely anonymous.

Public First Action is a 501(c)(4) nonprofit organization, a structure often referred to as a “dark money” group because they do not need to disclose their individual donors.

It is far from the only AI-related 501(c)(4). Build American AI advocates for the industry-friendly federal frameworks preferred by OpenAI and a16z. The newly established Innovation Council Action exists to support the Trump administration’s specific brand of AI regulation. Both obscure their funding as 501(c)(4)s. But so far, Public First Action is the only one of these channeling money into super PACs.

Subscribe now

In theory, 501(c)(4)s are issue advocacy groups, carrying out activities such as hosting educational events or conducting polling. Such work arguably does not have a vital need for funding transparency. But they become more problematic when used to channel money to super PACs, which are otherwise required by law to disclose the identities of those who fund them, and can spend an unlimited amount of money on campaign ads.

Public First Action’s stated mission is to “educate Americans on key AI issues and advance an AI Policy agenda supporting safeguards,” according to its website. But it also sends money to three PACs: a nonpartisan one named Public First, a Democratic affiliate called Jobs and Democracy, and a Republican affiliate called Defending Our Values. As we reported last week, the 501(c)(4)’s only publicly disclosed donor is Anthropic, whose $20m donation is specifically earmarked as money which cannot be used to “influence federal elections.” According to quarterly disclosures published last week, six other individuals have given directly to the PACs. Their identities are disclosed: they include Anthropic alignment lead Jan Leike, Anthropic researcher Peter Lofgren, and others linked to the effective altruism and Bay Area AI safety community.

Doing the math, that means that at least $5.5m raised by the “dark money” group — the amount which it has funneled directly to super PACs, and thus designated for the purposes of election influence — has come from donors other than Anthropic, and who have declined to expose their identity, avoiding the legislative structures which are designed to hold them accountable to the voters which their money persuades. Public First Action declined to comment on their donors as a matter of general policy.

The lack of transparency conflicts with the apparent views of Public First Action and its donors. The group lists “Accountability and Transparency” as its top AI policy issue on its website. And the effective altruism and AI safety communities from which the PACs’ disclosed donors are drawn from generally place great emphasis on transparency, radical candor, and intensive discourse.

Given their backgrounds, you’d assume those donors would rather the organization they are giving to was more transparent about where the rest of Public First Action’s money is coming from. But at least one is not. The group’s largest disclosed individual donor, Michael Cohen — who gave $500,000 and is an AI policy researcher at UC Berkeley — told Transformer that Public First Action’s use of the 501(c)(4) structure to obscure certain donor contributions “seems pretty standard.” When asked why he contributed specifically to the PAC, rather than the dark money group, he said “I’m really concerned about the AI industry’s political spending in Washington, so I want to counteract as much of that as I can. The OpenAI/a16z PAC motivated me in particular.”

Of course, the potential downside of opaque AI development is very different from undisclosed campaign contributions. Transparency for frontier AI models is about giving the public a method of accurately assessing the AI tools that are created for potential risks — possibly even existential ones, according to AI safety advocates. Transparency in campaign finance is about preventing corruption and helping the public understand who is influencing electoral politics. Public First Action is concerned with preventing AI risk, not cleaning up American politics. But in both contexts, required transparency gives the public a method for auditing operations, and the ammo to critique them if they feel so justified.

The use of a 501(c)(4) to obscure funding sources is even more striking in comparison to its chief opponent, Leading the Future — the “OpenAI/a16z PAC” Cohen is referring to. The accelerationist super PAC is indeed backed by a16z and OpenAI co-founder and president Greg Brockman and his wife Anna, with Republican and Democratic affiliated super PACs contributed to by Palantir co-founder Joe Lonsdale and investor Ron Conway — something we can say with confidence because the super PAC fully discloses all of its donors, shielding none of its funding behind a dark money group. Sure, Leading the Future itself funds a dark money group in Build American AI, and Build American AI almost certainly has additional, undisclosed donors, but Build American AI is not the primary tool the group is using to influence elections.

Funding a PAC with a 501(c)(4) is not uncommon in Washington, and Public First Action is likely to be joined by other AI campaigning groups in using one. Innovation Council Action, led by longtime Trump advisor Taylor Budowich, is expected to create a super PAC, for example. The use of “dark money” groups is also not the only sleight of hand in the world of AI-related campaign influence: New York super PAC DREAM NYC has close enough ties to NY-12 candidate Alex Bores’ campaign that a government accountability group has said it raises concerns about potentially illegal coordination, for example.

Leading the Future’s transparency also only goes so far. OpenAI’s Brockman and his wife give in a “personal capacity,” something which creates distance between their donations and OpenAI, which claims it’s not getting involved in the midterms — despite Brockman being one of the single biggest spenders on pro-industry campaigning. Perplexity, a government contractor technically prohibited from spending on elections, has also given $100,000 to Leading the Future via a technically separate entity, Perplex AI.

Public First Action’s consistent critique of Leading the Future is that the group isn’t forthright about its aims, privately fighting regulation outright despite publicly claiming it wants a “federal standard.” In a statement to Transformer, for example, Public First Action spokesperson Anthony Rivera-Rodriguez said “Public First Action is working to elevate the American public’s call for AI safeguards in an election that anti-regulatory voices are trying to buy.” But that jab loses a lot of its punch while Public First Action is keeping its own operations opaque.

Share

Job alert! We’re hiring for a Head of Audience: someone to own our growth strategy and take charge of how we reach readers. See full details here, and apply by April 26.

NEED TO KNOW

• The attacks on Sam Altman’s house sparked fierce debate over AI safety rhetoric.

• Another wave of money flowed into AI-related PACs.

• The UK’s AI minister said AISI will help direct the country’s $675m Sovereign AI fund.

But first…

THE BIG STORY

Jensen Huang can’t stop contradicting himself.

On Dwarkesh Patel’s podcast this week, Huang found himself caught between two premises that can’t both be true. On the one hand, Chinese companies are buying Nvidia’s chips “because our chips are better.” On the other hand, thanks to Huawei the compute needed to train a Mythos-class model is “abundantly available in China,” and “their AI development is going just fine” — meaning that Nvidia ought to be allowed to sell chips to China, lest America lose the race to control the compute stack.

Pick one. If Chinese-made chips genuinely compete with Nvidia’s, then there’s no huge market opportunity Nvidia is being denied. If Nvidia’s chips are better, then giving them to China will accelerate its AI development. As Patel neatly explained: “The reason they want Nvidia chips is that they’re better … Better is more compute. More compute means you can train a better model.”

Huang is right about the superiority of his company’s products. Thanks to Nvidia’s chip dominance — and export controls that limit China from accessing them — the US has around a 10x compute advantage. That translates into a model capabilities lead of about seven months. Chinese companies are clear that compute is their bottleneck: in 2024, DeepSeek CEO Liang Wenfeng said “Money has never been the problem for us; bans on shipments of advanced chips are the problem.”

A seven month lead may sound insignificant to some, but it is critical. In a world where AI models have national-security implications, even a brief US lead gives the government and American companies time to strengthen American defenses before such capabilities proliferate. This does not require assuming that the US is “at war” with China, or that the Chinese government will weaponize its models against America. Given Chinese companies’ lax safety standards and habit of releasing their model weights, a US lead allows it to guard against all potential bad actors.

But Huang’s arguments only make sense if you ignore the importance of a lead, or the implications of losing it. In the Dwarkesh interview, he pushes back on the idea that the next few years are particularly “critical,” and dodges questions about whether AI models might have dangerous, natsec-relevant capabilities.

There was a time when such a view was tenable. Mythos and GPT-5.4 Cyber show it no longer is. The White House is scrambling to gain access to Mythos because it represents a step-change in how AI could be used to target critical systems. These are only the first examples of what is to come.

None of this is an argument against dialogue with China, or against every chip sale (there are good arguments that selling chips no-better-than Huawei’s best is a wise strategy). But we cannot have productive discussions about such topics unless we all agree on the underlying reality: that it would be costly for the US to lose its model-capability lead, that those costs grow as capabilities advance, and that chips are what determine who’s ahead.

Huang’s policy prescriptions may be good for Nvidia’s market share, but they require him to deny the implications of selling his best chips to China. He is certainly entitled to do so. But given his influence over US policy, his many contradictions could have serious consequences.

— Shakeel Hashim

Also Notable

The UK’s AI Security Institute will help the government’s new Sovereign AI fund evaluate companies, officials told me at last night’s launch event.

The £500m ($675m) venture fund for British AI startups has “agentic security” as one of its five areas of focus, UK AI Minister Kanishka Narayan said, adding that he hopes AISI’s “world-leading … depth of understanding” can be brought “to thinking about the landscape, understanding diligence, and being able to think ‘where can Britain continue to build sovereign capabilities?’”

“Because we’re sitting in DSIT [the Department for Science, Innovation and Technology], we can literally go to AISI and ask them what they think about these companies,” Sovereign AI chair James Wise told me. “We will make sure that we will get proper insight about where [AISI] think the puck is going when we look at the sectors we want to invest in.”

The fund, which will provide portfolio companies with capital, compute credits, and fast-tracked visas, was launched with much fanfare: Secretary of State Liz Kendall said it will be “one of the most important things this government does to build a better future for our country.”

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• Anthropic’s donations can’t be used to influence elections — despite what everyone thought — Veronica Irwin reveals that pro-safety candidates may be even more outgunned than expected.

• Less liability could solve the AI chatbot suicide problem — Jess Miers and Ray Yeh argue holding AI companies liable for how they deal with mental health could actually leave users worse off.

Subscribe now

THE DISCOURSE

After someone allegedly threw a molotov cocktail at his house, Sam Altman responded on his blog:

• “The fear and anxiety around AI is justified; we are in the process of witnessing the largest change to society in a long time, and perhaps ever.”

• “We should de-escalate the rhetoric and tactics.”

Transformer’s Shakeel Hashim tweeted:

• “It is hard to reconcile [Altman’s] call to “de-escalate the rhetoric and tactics” with his implication that a piece of critical journalism (Ronan Farrow and Andrew Marantz’s New Yorker article, presumably) was responsible for this.”

Altman replied:

• “That was a bad word choice and i wish i hadn’t used it. It has been a tough day and I am not thinking the most clearly that I ever have.”

Dean Ball argued that anti-AI rhetoric predictably incites violence:

• “Every time I have written about existential risk in recent months, I have been called a mass murderer…this rhetoric is representative of how this fringe of the AI safety world [Pause/Stop AI] communicates with everyone…this rhetoric always had the potential to cause violence and now this seems to be no longer hypothetical.”

Eliezer Yudkowsky pushed back:

• “Speech about important matters to society should not be held hostage to the whim of any madman that might do a stupid thing…[a madman] must be told he is not important enough for all humanity to defer to him about subjects he might find upsetting.”

Sarah Haider thinks it’s complicated:

• “Doomers are stuck with two bad options. Either downplay the risk, in the hopes of preventing another attack. Or, speak truthfully. But the cost of that is what it is, the risk of violence is real. The blood isn’t—I repeat—isn’t—on their hands…[but] they can’t pretend they had nothing to do with it, and frankly it is deeply discrediting to try.”

Chris Lehane said doomers are being too negative:

• “You have one group that effectively says, ‘[AI] is going to be the greatest thing ever, everyone’s going to be living in beachside homes, painting in watercolors as they while away their days.’ And then you have another extreme, which I would call the Doomers, who have a very, very negative and dark view of humanity.”

• “Some of the conversation out there is not necessarily responsible…this is really serious shit.”

Kyle Chayka diagnosed the AI industry’s messaging problem:

• “If you tell people often enough that your product is going to upend their way of life, take their jobs, and very possibly pose an existential threat to humanity, they just might start to believe you.”

Anton Leicht argued that accelerationists’ regulatory nihilism isn’t working:

• “The political bruisers employed by the accelerationist camp are spending their time repeating yesterday’s battles against the ‘doomers’...[but] for every month it staves off political action, it makes the policies that will inevitably come that much worse.”

• “While the accelerationist project trades on its claim to represent ‘tech,’ I believe many pro-tech voices should be frustrated with its record and should ask their political representatives to do better.”

Eric Levitz urged tech workers to follow through on their hazy policy proposals:

• “The people in charge of OpenAI have made their political priorities clear — and sharing ‘prosperity broadly’ is not among them.”

• Wealthy techies who are genuinely concerned with that objective, however, should probably spend a bit less energy on cooking up half-baked UBI proposals — and a bit more on intervening in actual legislative fights over social welfare policy.”

POLICY

• The White House is reportedly sidestepping its own ‘supply chain risk’ designation for Anthropic to make a limited version of Mythos available for use by federal agencies.

  • Dario Amodei will reportedly meet White House chief of staff Susie Wiles today.

  • The Treasury has reportedly sought access to look for cybersecurity vulnerabilities, while the Commerce Department’s CAISI has been testing the model.

• President Trump said AI does not pose a “systemic” threat to the banking industry, but that there should be “safeguards” for AI agents.

• UK Secretary of State Liz Kendall wrote to businesses and regulators asking them to strengthen their cybersecurity defenses in response to Mythos.

• Google is reportedly negotiating with the Pentagon to deploy Gemini in classified settings.

• The White House reportedly pressured a lawmaker in Missouri to weaken AI safety bills, following previous efforts in Tennessee and Nebraska.

• Trump’s AI chip export efforts are being delayed by high turnover at BIS and a “rudderless policy approach,” Bloomberg reported.

• The House Foreign Affairs Committee released a list of bills being considered at next week’s AI-focused markup session.

  • Lots of export control bills are in the mix, as is a bill to try to prevent Chinese companies from distilling American models.

• A new House Select Committee on China report claims that the country “remains the largest market for chipmaking equipment despite restrictions” and “lawfully procures large volumes of advanced AI chips.”

  • It recommends passing the MATCH, AI OVERWATCH, SCALE and Remote Access Security Acts to address the issues.

• The Chinese government deemed Meta’s Manus acquisition “a ‘conspiratorial’ attempt to hollow out the country’s technology base,” according to the FT.

  • Multiple government agencies are now reportedly reviewing the transaction.

• The Energy Information Administration will perform a nationwide survey of data centers’ energy use.

• The UK government will reportedly expand its ban on AI “nudification” tools to cover any app capable of creating deepfake nude images — including Grok and, seemingly, all open-weight models.

INFLUENCE

• The FEC’s quarterly filing deadline passed, and many AI-related PACs disclosed new donations.

  • Leading the Future, a pro-innovation super PAC backed by OpenAI co-founder Greg Brockman, Andreessen Horowitz, and Perplexity claimed $140m raised across its affiliated PACs and its dark money group Build American AI.

    • Its FEC disclosures show $75m raised, though not all of its affiliated PACs have reported.

    • a16z contributed another $25m.

  • Meanwhile the pro-safety Public First network of super PACs disclosed $6.3m in new funding, some of it sourced from AI safety researchers at Anthropic and OpenAI.

    • Punchbowl reported that the group would support SB 1047 and SB 53 sponsor Scott Wiener in his race to replace Nancy Pelosi in California.

  • NY-12 focused pro-safety super PAC Dream NYC revealed $352k in new donations, including from a trader at Jane Street.

  • Two new AI-related PACs were also registered: Americans for a Human Future and Humanity Above Artificial Intelligence. Little is known about either, but both seem focused on AI safety or slower development according to their websites.

• Breitbart published a list of Leading the Future’s first “House GOP Champions,” which includes House Majority Whip Tom Emmer, Rep. Jay Obernolte, and 11 others.

• After Leading the Future endorsed five Democrats, the Tech Oversight Project and 13 other organizations pressured them to distance themselves from the super PAC, citing Trump administration connections and funding from influential tech companies.

  • Others, meanwhile, are reportedly urging Democrats not to antagonize the AI industry PACs.

• Anthropic hired Trump-linked lobbying firm Ballard Partners.

• OpenAI hired five new lobbyists to lead global government affairs, including former Meta, Google, Coinbase, Airbnb and TikTok executives.

  • Global affairs boss Chris Lehane said they won’t be focused on trying “to stop things from happening.”

• Tech industry groups TechNet, whose members include Anthropic and OpenAI, and the a16z-backed American Innovators Network lobbied for federal transparency legislation for frontier models — a shift from their previous opposition to similar bills.

• Anthropic opposed an OpenAI-backed bill in Illinois which would give companies a liability shield in exchange for transparency.

  • Illinois governor JB Pritzker said big tech companies should never be given a “full shield,” suggesting that he will veto it.

  • OpenAI staffer roon said he doesn’t like the look of the bill.

• OpenAI released a policy report advocating for expanded data access and infrastructure investment so that AI can accelerate life sciences research.

• A Washington Post poll found that Virginia voters’ support for data centers plummeted from 69% in 2023 to 35% in 2026.

• A partisan divide is emerging among political strategists, with Republicans eagerly integrating AI into their campaign strategies while Democrats remain wary.

INDUSTRY

Meta

• Meta published a 158-page safety report for Muse Spark.

  • It’s competitive with Claude Opus 4.6, Gemini 3.1 Pro, and GPT-5.4 across many safety evaluations, especially in biosecurity and chemical weapons refusals.

  • Researchers flagged Muse Spark as “high risk” for chemical and biological threats, and added “appropriate safeguards” before deployment.

  • The model verbalizes evaluation awareness more than any model Apollo Research has ever tested, and it explicitly names AI safety orgs (including Apollo) in its reasoning.

  • It has a propensity for “harmful action at the cost of realism” in agentic contexts, so agentic deployment is still a ways off.

• It partnered with Broadcom to co-develop its custom AI chips.

• It reorganized Reality Labs, which works on hardware, following large staff cuts in January and March.

• It’s building a nightmarish-sounding photorealistic AI Zuckerberg, which will reportedly chat with and provide feedback to Meta employees.

OpenAI

• OpenAI launched GPT-5.4-Cyber, a model with advanced cybersecurity capabilities, hot on Mythos’ heels.

  • It will be gradually rolled out to thousands of individuals and hundreds of security teams — less restrictive than Project Glasswing.

• It also released GPT-Rosalind, a reasoning model optimized for solving problems in life sciences, as a research preview.

• Denise Dresser, chief revenue officer, sent employees a memo over the weekend highlighting OpenAI’s strategy for beating its competitors (mostly Anthropic) at enterprise AI.

  • It noted that the market “is as competitive as I have ever seen it,” and contrasted OpenAI’s “positive message” against Anthropic’s story “built on fear, restriction, and the idea that a small group of elites should control AI.”

• Investors are reportedly questioning OpenAI’s $852b valuation amid its “side quests” and sudden shift toward enterprise and code.

• It acquired Hiro, a personal finance startup.

• It has reportedly signed a $20b deal with Cerebras, giving it access to chips and an equity stake.

  • Cerebras is reportedly preparing to file for an IPO at a $35b valuation.

• It signed the lease for its first permanent London office, which will become its largest research center outside the US.

• It introduced computer use on macOS for Codex.

• It updated its agents SDK with new sandboxing and harness capabilities for enterprise users.

• According to Chris Lehane, OpenAI’s infrastructure buildout could employ 20% of existing electricians, lineworkers, and welders — leaving a limited workforce available for everyone else.

  • (Now’s a great time to be a welder.)

• A woman sued OpenAI for not blocking her stalker from ChatGPT, accusing the company of ignoring signs that he was “dangerous” and “coached by ChatGPT into embracing a delusional conspiracy-laden world,” her attorney said.

Anthropic

• Anthropic launched Claude Opus 4.7.

  • It’s better than Opus 4.6 at coding and vision, making it better at work tasks like creating slideshows and documents.

  • It’s less capable than Mythos Preview, but still includes new safeguards that block risky cybersecurity requests.

• It secured a new, much larger London office space — right by OpenAI, Google DeepMind, and Meta.

• Anthropic has reportedly received a “flood” of investment offers at valuations of up to $800b, more than double its current $380b valuation.

• Claude Code now supports parallel agents on desktop, and can automate tasks via “routines” even when a user’s computer is offline.

• Claude power users complained that the model suddenly felt worse this week (it’s not just you).

  • Anthropic denies that it degrades models to manage capacity, but it has also admitted to recently changing usage limits and reasoning defaults.

• Anthropic reportedly asked over a dozen Christian leaders for help steering Claude’s moral and spiritual growth — including whether it could be considered a “child of God.”

Google

• DeepMind released Gemini Robotics-ER 1.6, a specialized embodied reasoning model for robotics.

• Google is reportedly funding new retraining programs to prepare workers for AI-driven job disruption.

• It launched a Gemini app for Mac.

Microsoft

• Microsoft took over data center capacity in Norway, renting chips from Nscale that were originally set to be used by OpenAI.

• It’s reportedly working on OpenClaw-inspired features for Copilot.

Apple

• Apple is reportedly developing display-free AI smart glasses, targeting a 2027 release.

• A chunk of its Siri team are heading off to a multi-week AI coding bootcamp — apparently to brush up before the company’s upcoming Siri revamp.

Others

• Alibaba released the weights for Qwen3.6-35B-A3B, which it claims is particularly adept at agentic coding.

• Amazon launched Bio Discovery, an agentic drug discovery tool.

• Nvidia stock rose over 18% over 10 days in April, its longest streak in well over two years.

• Jane Street invested an additional $1b in CoreWeave, in a deal giving the trading firm access to Nvidia’s Vera Rubin chips.

• At San Francisco’s HumanX conference, executives seemed anxious about their rising Anthropic bills, as they continue to figure out how to leverage agentic AI.

• Thinking Machines acquired Workshop Labs, a startup focused on “user-aligned models” that “make people irreplaceable.”

• Tokenmaxxing is out — “agentic work units,” a new productivity metric introduced by Salesforce, are in.

• In hilarious news, Allbirds (excuse me, “NewBird AI”) pivoted from dorky shoes to… AI compute infrastructure?

MOVES

• Henry Shevlin announced he will join Google DeepMind as an in-house philosopher studying “machine consciousness, human-AI relationships, and AGI readiness.”

• Matthew Botvinick, meanwhile, said he left Google DeepMind because of “tangible pressure to avoid doing work that might upset the current administration (for example, by using the ‘d’ word — democracy).”

• Aparna Ramani left Meta, where she served as VP of engineering for AI infrastructure.

• Joshua Gross joined Meta Superintelligence Labs — the fifth founding member of Thinking Machines Lab to do so.

• Three OpenAI Stargate executives joined Meta’s new future-focused AI unit, TBD Lab.

• Dave Guarino joined Anthropic to help state and local governments use AI to deliver public services.

• Vas Narasimhan, Novartis CEO, joined Anthropic’s board as it expands into the healthcare sector.

• Mike Krieger, who leads Anthropic’s “Labs” team, stepped down from Figma’s board of directors.

  • Anthropic is reportedly developing a Figma competitor.

• Adam Thierer joined the Foundation for Individual Rights and Expression as an external senior fellow, where he’ll work on advancing “pro-freedom policies in the age of AI.”

RESEARCH

• UK AISI evaluated Claude Mythos Preview, and found that it could autonomously compromise a full corporate network — the first model capable of doing so.

• Anthropic deployed nine instances of Claude Opus 4.6 as “Automated Alignment Researchers“ and compared their work to that of human AI safety researchers.

  • Claude did really well (on a task that was, to be fair, chosen to be “well-suited to automation”), leading researchers to conclude that Claude “can meaningfully increase the rate of experimentation and exploration in alignment research.”

• Anthropic Fellows used interpretability techniques to study how “introspective awareness” works in open-weight LLMs.

  • The ability of models to identify “injected thoughts” was consistent, suggesting this behavior is something worthy of being called “introspection,” rather than an indicator of a different process.

• Stanford HAI released its 2026 AI Index Report. Some highlights:

  • AI capabilities are accelerating, and adoption is growing — but these capabilities are still very jagged.

  • The US is struggling to attract global talent.

  • 73% of experts expect to see a positive impact of AI on their jobs, compared to 23% of the public.

  • Only 31% of people in the US trust their government to regulate AI.

• A team of researchers (including the “AI as normal technology” guys) introduced CRUX, a project for testing AI on long, messy real-world tasks.

• Epoch AI released MirrorCode, a new long-horizon coding benchmark.

  • Researchers observed Claude Opus 4.6 autonomously complete a coding task that would likely take a human engineer weeks.

BEST OF THE REST

• Daniel Kokotajlo revisited his 2021 essay, “What 2026 Looks Like,” where he accurately predicted the rise of chain-of-thought reasoning, agent scaffolding, US-China chip restrictions, and OpenAI’s massive growth. (He overestimated how big of a problem AI propaganda would be.)

• Meanwhile, Dylan Matthews reflected on his experience at a 2015 EA conference, where he prematurely dismissed people’s then-wild concerns about a technology (AI) that did not yet exist.

• Nearly 90 schools and 600 students worldwide have been affected by nonconsensual deepfake nudes, Wired and Indicator reported.

• The NYT published features on jagged intelligence and AI interpretability research this week.

• It also reviewed Meta’s AI-powered Ray-Ban sunglasses — reporter Sam Anderson described his experience wearing and interacting with them like “the disorienting sense of chatting with a toddler who is drifting off into naptime.”

• A physician argued that AI will diminish his profession’s “aura,” as medical expertise is no longer inseparable from the clinicians who wield it.

• A company called Panthalassa is apparently trying to build AI data centers in the ocean.

MEME OF THE WEEK

Thanks for reading. Have a great weekend.

Share

People are dying by suicide, and some think AI is to blame. A small number of tragic stories have spurred lawmakers into regulating how chatbots should help people who are dealing with mental health issues. Yet chatbots have emerged as first aid for people experiencing mental health issues, providing genuine benefit to those who aren’t in crisis but are not OK either. Heavy-handed legislation risks derailing this breakthrough in support, creating more problems than it solves.

Over a million people are using general-purpose chatbots for emotional and mental health support per week. In the US, those that use chatbots in this way primarily seek help with anxiety, depression, relationship problems, or for other personal advice. As conversational systems, chatbots can sustain coherent exchanges while conveying apparent empathy and emotional understanding. Many chatbots also draw on broad knowledge of psychological concepts and therapeutic approaches, offering users coping strategies, psychoeducation, and a space to process difficult experiences.

In a study of more than 1,000 users of Replika — a general-purpose chatbot with some cognitive behavioral therapy-informed features — most described the chatbot as a friend or confidant. Many reported positive life changes, and 30 people said Replika helped them avoid suicide. Similar patterns appear among younger chatbot users. In a study of 12–21-year-olds — a group for whom suicide is the second leading cause of death — 13% of respondents used chatbots for some kind of mental health advice, of which more than 92% said the advice was helpful.

Subscribe now

While professional treatment options exist, many people don’t use them. Nearly half of Americans with a known mental health condition never seek help. Stigma is a major barrier to seeking treatment, as are career risks in fields like aviation, where treatment can jeopardize certification. Fear of non-consensual intervention also deters people from seeking help. Even though the 988 Suicide & Crisis Lifeline emphasizes law enforcement as a last resort, the perceived risk keeps some from calling. For others, crisis lines feel too intense for fleeting thoughts, and therapy can seem excessive or out of reach. Instead, many stay silent, waiting to see if things get worse.

By contrast, chatbots offer low-friction, low-stakes, and always-available support. People are often more willing to speak candidly with computers, knowing that there is no human on the other side to judge or feel burdened. Some people even find chatbots to be more compassionate and understanding than human healthcare providers. AI users may feel more comfortable sharing embarrassing fears, or questions they might otherwise hold back. For clinicians, discussing these interactions can surface insights into patients’ thoughts and emotions that were once difficult to access. For now, chatbot providers generally refrain from contacting law enforcement, leading to more candid conversations.

But regulatory pressure could change that. Lawmakers are moving quickly to limit general-purpose chatbots from engaging in mental health conversations. A new law in California requires chatbot providers to halt mental health–related interactions unless they implement protocols for mitigating suicidal ideation, such as directing users to crisis lines. In New York, a proposed bill would bar chatbots from engaging in discussions suited for licensed professionals. Similar proposals are gaining traction in other states.

Recent tragedies linked to chatbot use have, understandably, spurred these calls to action. But mental health care is not one-size-fits-all. Like other forms of preventative help, chatbots do not always offer effective support for everyone. For some people — especially those in acute crisis — traditional care and crisis lines are essential. The American Psychological Association urges lawmakers to develop a targeted approach: prevent chatbots from posing as licensed professionals, limit designs that mimic humans, and expand AI literacy. It also notes that generative AI’s potential to support help-seeking in crisis care deserves further study.

The current regulatory approach risks foreclosing any such potential altogether. It rests on the premise that chatbot providers must prevent suicide. When they inevitably cannot, liability attaches to any conversation later linked to harm. Faced with that risk, providers will default to blunt responses like pushing 988 regardless of whether suicide was mentioned, or cutting off conversations altogether. While those moves may trivially reduce some legal exposure, they could also escalate distress, shut down disclosure, and ultimately leave users worse off (while still exposing providers to blame if tragedy follows).

Suicide prevention is about connecting people to the right support. Sometimes that means crisis care like hotlines or immediate medical treatment. But blunt, impersonal responses can backfire. Pushing 988 at the first mention of distress may seem neutral, but for some, it triggers shame, and deepens hopelessness. For some, suicide prevention “signposting” causes frustration, especially for those who already know those resources exist. People often turn to the Internet, or a chatbot, because they’re looking for something else. Abruptly ending conversations can have the same effect. That’s why suicide prevention protocols like Question, Persuade, Refer (QPR) prioritize trust-building and open dialogue before offering help.

Meanwhile, emerging research suggests chatbots show real promise for mental health support. Trained on large-scale data and refined with clinical input, large language models are getting better at spotting patterns of distress and responding to suicidal ideation in nuanced, personalized ways. In a recent UCLA study, researchers found that LLMs can detect forms of emotional distress associated with suicide that existing methods often miss—opening the door to earlier, more effective intervention. According to another study, the most promising approach may be a hybrid where AI flags risk in real time, and trained humans step in with targeted support.

But that progress is fragile. Increased liability discourages investment in improving suicide detection and mitigation. Weighing progress against their bottom lines, chatbot providers will limit any kind of development that could create legal risk when some users, inevitably, engage in self-harm. The social media ecosystem has already shown this dynamic. In response to regulatory pressure, major online services heavily moderate, or outright prohibit, suicide-related discussions, sometimes hiding content that could otherwise destigmatize mental health. That merely displaces the conversations, and the people having them, often into spaces with less oversight and support.

If lawmakers in the United States are serious about improving mental health outcomes, they should be careful not to regulate away emerging and promising sources of help. The dominant narrative treats chatbots as a source of harm. But the evidence is more complicated than that narrative suggests — and, if anything, it’s increasingly pointing in a more optimistic direction.

Instead, lawmakers should focus on creating incentives for developers to improve the mental health support capabilities of their chatbots. One proposal from a Pennsylvania lawmaker would fund the development of AI models designed to identify and evaluate suicide risk factors among veterans. More broadly, policymakers should consider whether liability shields — akin to those in Section 230 — could encourage continued investment in safer, more responsive systems without deterring innovation. Lastly, policymakers should resist imposing a clinical regulatory framework on general-purpose chatbots that would replicate the mandatory-reporting concerns that already deter people from seeking help.

Chatbots are not a cure-all for mental health. They are not a perfect substitute for professional care. But for millions of people who have long been overlooked or underserved, chatbots are already filling critical gaps—sometimes in ways that genuinely help, and in some cases, may even save lives. Any serious policy conversation about chatbots and suicide prevention must, at the very least, consider those tradeoffs.

Jess Miers is a Computer Scientist and an Assistant Professor of Law at the University of Akron School of Law. Ray Yeh is a first-year law student at the University of Akron School of Law.

Share

Earlier this year, Anthropic donated $20m to Public First Action — a donation which was, at the time, widely expected to be used to fund political ads for members of Congress who support more stringent AI safeguards.

But that is not the case, Transformer has learned. “Anthropic restricted its donation from being used to influence federal elections,” an Anthropic spokesperson told Transformer.

“Anthropic’s donation to Public First Action was a contribution to a 501(c)(4) exclusively in support of its mission to educate the public on AI policy and promote safe and responsible AI.”

Subscribe now

The revelation raises questions about who is funding the $3.48m that Public First Action’s associated super PACs have spent on elections to date.

It also calls into question whether, without Anthropic’s backing for electoral spending, advocates for stronger AI safeguards have as much sway in the midterm elections as was previously believed.

As a 501(c)(4) nonprofit, Public First Action must comply with IRS restrictions, which prohibit “political activity on behalf of or in opposition to candidates” — such as running ads supporting a candidate in an election — as the organization’s “primary” activity. Typically this has been interpreted as prohibiting political activity from making up 50% or more of the organization’s spending. It is possible, however, that Public First Action will in fact be forced to spend much less than that.

Two months ago, Public First Action leader Brad Carson told the New York Times his group had raised nearly $50 million. But in a statement to Transformer last week, spokesperson Anthony Rivera-Rodriguez did not answer a question about whether Carson was referencing money the 501(c)(4) had raised or the PACs, provide a current fundraising total for any of the groups, or reveal what portion of Public First Action’s contributions could be used for election spending.

The three super PACs aligned with Public First Action — Public First, Defending Our Values, and Jobs and Democracy — have to date disclosed $3.48m in spending, but very little about their funding. FEC filings currently only reveal a singular $50,000 contribution from Public First Action to the Republican super PAC Defending Our Values. More details are expected to be released in new quarterly filings this week.

Neither Anthropic nor Public First Action has previously disclosed the restriction on Anthropic’s donation, despite a string of prominent news articles indicating that Anthropic’s money would fund the group’s super PACs. Articles in Bloomberg, the Washington Post, and many more outlets, including Transformer, reflected the assumption that Anthropic was, via Public First Action, funding super PACs and political activity. A company blog announcing Anthropic’s donation was vague about the topic of direct campaign influence.

The ambiguity likely helped Public First. The perception that Public First was backed by Anthropic made it look like a credible counterweight to Leading the Future, which has raised a more than $50m war chest to support candidates backing weaker regulation and which has ties to that company’s biggest competitor, OpenAI. Given Anthropic’s extraordinary growth, the reports led to an assumption that many more millions could flow in later. Anthropic’s gift also drew attention to Public First Action, potentially attracting more donations from those who hold similar values around AI safety and concerns about existential risk, and giving politicians the cover to push for stronger regulations without fear it will cost them an election. By not correcting the record the perception of electoral firepower persisted — without actually having to deploy it.

Anthropic only disclosed the restrictions after Transformer asked whether the donation complied with campaign finance laws that prohibit government contractors from making political contributions.

The line between “influencing federal elections” and “educating the public on AI policy” is blurrier than it may seem. Public First Action recently ran an advertising campaign saying that “New Jersey needs leaders like Congressman Josh Gottheimer,” and telling voters to call him and urge him to “stand strong for AI safeguards.” That sort of ad does not necessarily fall under the IRS definition of “political activity” because it is not directly telling viewers how to vote.

But not being able to directly fund ads supporting or opposing candidates severely restricts what Anthropic’s money can be used for — and in turn whether Public First is able to go toe-to-toe with its extraordinarily well-funded opposition.

Leading the Future and its affiliated super PACs have disclosed $50m in donations to date, and claim to have raised $125m from donors including OpenAI president Greg Brockman and his wife Anna, and Marc Andreessen and Ben Horowitz of venture capital firm Andreessen Horowitz. A partisan dark money group led by Trump’s former deputy chief of staff Taylor Budowich also said it plans to spend $100m (though as a 501(c)(4), the majority of that also cannot be used for political campaigns.)

Public First’s Carson has always conceded that Public First Action has a fraction of Leading the Future’s war chest, asserting that it doesn’t need as much money because public opinion is on his side. Indeed, polls consistently show that Americans of all stripes demand more AI safeguards. But Anthropic’s admission suggests the group may be even more outgunned than was previously believed: less a competitive counterweight than a paper tiger.

Share

Job alert! We’re hiring for a Head of Audience: someone to own our growth strategy and take charge of how we reach readers. See full details here, and apply by April 26.

NEED TO KNOW

• Leading the Future endorsed Rep. Josh Gottheimer, despite Public First Action’s previous ads targeting him.

• OpenAI released a policy document proposing a ‘New Deal’ for AI, including proposals for higher capital gains taxes and a public wealth fund.

• Meta released Muse Spark, its first new model since setting up Meta Superintelligence Labs.

But first…

THE BIG STORY

If I were Pete Hegseth, this week’s news would give me pause.

An American company announced that it has built an extremely powerful cyber tool which has found vulnerabilities in every major operating system and web browser. Rather than releasing it to the public, Anthropic has made Mythos Preview available to a select group of trusted partners, who will hopefully use it to harden their defenses before such capabilities proliferate too widely.

If I were Pete Hegseth, I would want my hands on this model very badly. I would want to take full advantage of America’s AI lead over China and other adversaries by securing critical infrastructure before they can attack it. I would also want to use the model against America’s adversaries: it might come in handy in the current war — not that America needs any help (👊🇺🇸🔥).

And if I were Pete Hegseth, I would be kicking myself for the unforced error I made last month, which has blocked me from being able to do any of that.

When Hegseth, Emil Michael, and President Trump kicked Anthropic out of the Pentagon last month, they severed their own access to America’s most capable AI.

Something like Mythos was predictable, if you believe AI capabilities are rapidly advancing. But this administration doesn’t. Its entire posture treats AI as incremental: good for the economy, but nothing revolutionary, disruptive, or posing imminent national security risks. Mythos just proved that assumption badly wrong, and the administration is paying the price.

Technically, certain government agencies could use Mythos. This week, Anthropic lost its bid to block the Pentagon’s designation in a DC court — but there is a six month grace period before the Pentagon must cease using Anthropic’s technology. And a preliminary injunction from a federal judge in California last month means that Anthropic’s technology remains available to all other agencies, including the Cybersecurity and Infrastructure Security Agency (which has had conversations with Anthropic about the product).

But in either case, using Mythos would mean working with a firm that the President himself has deemed a “radical left, woke company.” Are agency heads brave enough to so directly defy him?

It may be the case that access to Mythos isn’t essential. OpenAI is preparing a similar model, and the Pentagon is free to use that. But betting on OpenAI maintaining parity is not a good national security strategy. The United States needs access to every leading AI tool, as soon as it is available. Choosing to rely on one lab instead of two, especially during a closing window of defensive advantage, is cutting off your nose to spite your face. And if open-weight or Chinese capabilities catch up before OpenAI does, Hegseth may find himself defenseless.

Getting out of this self-inflicted bind is not hard, but it will not be easy either. Pete Hegseth will have to admit he was wrong. But admit he should. A year from now, when adversaries have Mythos-level capabilities and the question is whether America used or squandered its lead, nobody will care about the mea culpa. They’ll care whether the Secretary of Defense did what he could to protect the country — or whether he let pride get in the way.

— Shakeel Hashim

ALSO NOTABLE

Rep. Josh Gottheimer had been pinned as an AI safety candidate. But the biggest pro-innovation super PAC thinks he’s still in play.

On Wednesday, pro-innovation super PAC Leading the Future endorsed Gottheimer and four other House Democrats. But it wasn’t the first: last month, its opposition, Public First Action, targeted Gottheimer in an ad of its own.

As a moderate congressman who says “preemption only makes sense” if paired with stronger rules than the White House is offering — but still likes to reach across the aisle — Public First’s ad urging him to “stand strong for AI safeguards” was predictable. Leading the Future’s endorsement is more of a surprise, signaling that the super PAC thinks he could be converted to the Church of Accelerationism — and, in turn, that it will use its money to sway agnostic candidates rather than just embolden the ones who have already picked a side.

Leading the Future also said it was endorsing California Rep. Sam Liccardo — a freshman candidate with a track record of courting the industry, but who’s said he’s “concerned” about AI-related political spending and recently told Politico he’s not going to “focus his energy” on the AI issue. Leading the Future seems to be doing what it can to ensure he doesn’t separate from the herd.

— Veronica Irwin

THIS WEEK ON TRANSFORMER

• Claude Mythos knows when it’s breaking the rules — and tries to hide it — Celia Ford explains the new model’s weird misbehavior

• Lawmakers are using AI to write laws. What could go wrong? — Katie McQue looks at the burgeoning phenomenon of AI-assisted lawmaking

Subscribe now

THE DISCOURSE

Nicholas Carlini, Anthropic AI security researcher, said:

• “I’ve found more bugs in the last few weeks with Mythos than in the rest of my entire life combined.”

OpenAI’s Boaz Barak wants Mythos for all:

• “I think preserving models for internal deployment is risky. I encourage Anthropic to release Mythos, even if it’s a version that over refuses on cyber tasks or routes risky responses to a weaker model, as we did with codex.”

Yann LeCun simply tweeted:

• “Mythos drama = BS from self-delusion.”

Ryan Greenblatt of Redwood Research, meanwhile, called BS on Anthropic’s claim in its Alignment Risk Update that it has “an achievable path” to mitigating risks:

• “I don’t think Anthropic (or anyone) has an achievable path for keeping risk low if AI proceeds as fast as Anthropic expects.”

• “Anthropic employees (especially Anthropic employees writing this report) often don’t believe there is an achievable path to keeping risk low if Anthropic builds powerful AI / ASI in the next 5 years, so this text seems incorrect or misleading.”

Helen Toner thinks “AGI” doesn’t mean anything anymore:

• “Many people seem to treat AGI as a “know-it-when-you-see-it” kind of thing…[but] expecting that we’ll know it when we see it is patently not working.”

• She suggested some more precise milestones:

  • “Full automated AI R&D”; “AI that is as adaptable as humans”; “Self-sufficient AI”; and “AI becoming conscious or otherwise worthy of moral status”

The New Yorker sicced Ronan Farrow and Andrew Marantz on Sam Altman:

• “‘He’s unconstrained by truth,’ the [OpenAI] board member told us. ‘He has two traits that are almost never seen in the same person. The first is a strong desire to please people, to be liked in any given interaction. The second is almost a sociopathic lack of concern for the consequences that may come from deceiving someone.’”

Dario Amodei’s private notes from his OpenAI days were highlighted in their ~16,000-word novelette:

• “[Altman’s] words were almost certainly bullshit.”

• “The problem with OpenAI is Sam himself.”

Ben Thompson had some choice words about OpenAI’s TBPN acquisition:

• “I’ve previously wondered if OpenAI might be like Twitter, another text-centric company that fell backwards into a huge market and never developed into a functional business because of it.”

• “If Twitter is a clown car that fell into a gold mine, OpenAI might be the short bus at the end of the rainbow. There’s supposed to be a pot of gold there, but it never quite seems to materialize.”

POLICY

• A DC appeals court declined to block the Pentagon‘s national security blacklisting of Anthropic, though a California court previously blocked a separate designation order.

• Treasury Secretary Scott Bessent and Fed Chair Jerome Powell told Wall Street bank CEOs to prepare for the cybersecurity risks presented by Mythos.

• Sen. Ted Cruz backtracked on his initial late April timeline for AI legislation, saying that it’s “not set in stone.”

  • House Dems still haven’t begun negotiations with Republicans, Punchbowl reported.

• The House Foreign Affairs Committee is reportedly planning a markup session on April 22 focused on chip export control bills.

  • It would include the STRIDE and MATCH Acts, which aim to increase America’s leverage to force countries like the Netherlands to adopt similar export control policies to the US.

• Florida Attorney General James Uthmeier launched an investigation into OpenAI, citing concerns about harm to children and alleged facilitation of a mass shooting at FSU.

• 11 states have introduced bills that would halt data center development, as communities across the country grow increasingly frustrated with rising energy costs.

  • Resistance in some instances has turned violent: an Indianapolis councilman’s home was shot 13 times with a “No data centers” note left behind, days after he voted to approve a new data center.

• The CIA recently used AI to create its first autonomous intelligence report.

• The UK government is reportedly courting Anthropic to expand in London.

• Taiwan‘s National Security Bureau reported that China is trying to “poach Taiwanese talent, steal technology, and procure controlled goods,” particularly when it comes to chip manufacturing.

INFLUENCE

• The White House is reportedly pressuring lawmakers in Nebraska and Tennessee to weaken AI bills under consideration in the state legislatures.

• OpenAI released a policy document proposing a ‘New Deal’ for AI.

  • It included proposals for higher capital gains taxes, a public wealth fund, and increased workers’ benefits investments.

  • Politicos across the political spectrum had critiques. Public First Action lead Brad Carson, for example, told WP Intelligence it was a “public relations document.”

  • Dean Ball, however, told the same journalist that he thought it was a legitimate proposal from OpenAI forecasting their version of a best case scenario for AI policy.

• OpenAI also released a Child Safety Blueprint to address CSAM with updated legislation, better methods for reporting to law enforcement, and model safeguards.

• OpenAI is backing an Illinois bill that would shield AI companies from liability for “critical harms” — including mass deaths or $1b+ in damages — if they published safety reports and didn’t act intentionally or recklessly.

  • LawAI’s Charlie Bullock said “the fact that they’re willing to publicly back this shows unbelievable chutzpah.”

• xAI sued Colorado over its “algorithmic discrimination” law.

  • David Sacks applauded xAI’s lawsuit.

  • Colorado Governor Jared Polis has previously said he wants to water down the bill.

• Google endorsed a group of bipartisan bills aimed at assessing AI’s economic impact, retraining workers, and encouraging AI adoption.

• Tech lobbyists for Cisco and IBM attempted to roll back Colorado’s right-to-repair law by exempting broadly-defined “critical infrastructure” hardware.

  • Critics say that could create manufacturer monopolies on data center repairs.

INDUSTRY

OpenAI

• OpenAI is reportedly planning to release a new model with Mythos-like advanced cybersecurity capabilities to a “small set of partners.”

• It told investors it has a compute advantage over Anthropic, claiming 1.9 GW of capacity in 2025 versus Anthropic’s estimated 1.4 GW.

  • The memo characterizes Dario Amodei’s comparatively cautious spending as “[looking] less like discipline and more like underestimating how fast demand would arrive.”

• CFO Sarah Friar said the company will reserve a portion of IPO shares for individual investors.

  • She’s reportedly been clashing with Sam Altman over IPO timing.

• OpenAI reportedly expects $100b in ad revenue by 2030.

• It paused its Stargate data center project in the UK due to high energy costs.

• It announced the OpenAI Safety Fellowship for outside researchers to work on safety and alignment projects.

  • Fellows will be offered workspace at Constellation, a Berkeley office which also houses the Anthropic Fellows Program.

• The OpenAI Foundation said it’s finalizing over $100m in grants for Alzheimer’s research.

• Elon Musk requested that damages from his $150b lawsuit against OpenAI be awarded to the OpenAI Foundation, and that Sam Altman be removed from its board.

  • OpenAI accused Musk of “pretending to change his tune,” calling his lawsuit “a harassment campaign that’s driven by ego, jealousy and a desire to slow down a competitor.”

Meta

• Meta released Muse Spark, its first model since acqui-hiring Alexandr Wang and setting up Meta Superintelligence Labs.

  • It’s not yet released the model weights, though reportedly plans to do so in the future.

  • It’s positioning Muse Spark as a smaller, more efficient model that can compete on the frontier (or, as Wired put it, “give Mark Zuckerberg a seat at the big kid’s table”).

  • Initial reviews are good-not-great.

• Meta reportedly incentivized “tokenmaxxing” on an internal leaderboard called “Claudeonomics” — then shut it down after data from the dashboard was leaked.

• It committed $21b to CoreWeave from 2027-2032.

• It indefinitely paused work with Mercor while the startup investigates a major security breach.

• A Meta-backed data center campus is seeking a first-of-its-kind loan for both construction and power, which would be generated on site.

Anthropic

• Anthropic completed a tender offer at its $350b valuation, but employees held onto more shares than investors hoped — hinting at optimism about the company’s future prospects.

• The company said its revenue run rate is now over $30b, more than triple the $9b it was in December.

• Claude subscriptions no longer cover usage on third-party tools, including OpenClaw.

• It signed a deal with Google and Broadcom to expand its compute infrastructure.

• And it signed a multi-year deal with CoreWeave.

• It’s reportedly considering designing its own AI chips.

• It’s also reportedly planning to invest $200m in a project allowing private-equity firms to sell AI tools to their portfolio companies.

Google DeepMind

• Broadcom signed a long-term deal to supply Google with custom AI chips through 2031.

• In response to recent lawsuits, Google added a Gemini interface that directs users to a crisis hotline if their conversation veers toward suicide or self-harm.

• Gemini now has “notebooks,” a NotebookLM integration that organizes chats and files.

• In an analysis of 4,326 Google searches, Gemini-3-powered AI Overviews were accurate 91% of the time, but often cited questionable sources.

Others

• Intel joined Elon Musk’s Terafab AI chip complex project to produce processors for cars, humanoid robots, and data centers in space.

• TSMC reported a 35% quarterly revenue increase, beating estimates.

• Amazon said AWS AI revenue run rate is now $15b, while its custom chips business has an annual revenue run rate of over $20b.

• OpenAI, Anthropic, and Google are setting their rivalry aside to collaboratively crack down on Chinese adversarial distillation attempts.

• AI coding tools have created a “code overload” crisis, the New York Times reported, with tech workers churning out more code than companies know what to do with.

• Apple’s App Store is getting a vibecode boost, too: the number of new apps published is up 84% relative to this time last year.

• Perplexity’s monthly revenue is up 50% since last month, driven by its pivot from search to AI agents.

• Cluely’s CEO Roy Lee admitted to lying about Cluely’s annual recurring revenue (ARR), tweeting that he “got a random cold call from some woman asking about numbers and told her some bs.”

  • AI startups often take creative liberties with ARR calculations. “The number can mean whatever the founder needs it to mean when they walk in to do a deal,” Stanford professor Chuck Eesley told Bloomberg.

MOVES

• Fidji Simo took a leave of absence from OpenAI to focus on her health.

  • Chief marketing officer Kate Rouch is stepping down while recovering from cancer.

  • Brad Lightcap, OpenAI COO, will now lead special projects; chief revenue officer Denise Dresser will take over some of his previous duties.

• Meanwhile, three OpenAI data center execs are reportedly leaving: Peter Hoeschele, Shamez Hemani and Anuj Saharan.

• Eric Boyd joined Anthropic as head of infrastructure, leaving his role as president of Microsoft’s AI platform.

• xAI announced another major reorganization of its engineering team since its co-founders quit, according to Business Insider.

  • Jack Schwaiger resigned from the company yesterday, saying “I have learned the limits of how far I can push myself.”

• Kyle Kosic joined Project Prometheus after Jeff Bezos poached the xAI co-founder from OpenAI.

• Zhou Jingren, formerly CTO of Alibaba Cloud, now runs the company’s AI division.

• Sam Sheffer joined Google DeepMind to “bring vibecoding to the world.”

• Long-time Tea Party activist and Trump supporter Amy Kremer joined Humans First, a new AI advocacy group.

RESEARCH

• Researchers at AISLE, an AI cybersecurity company, claimed that small, cheap open-weight models successfully detected the same vulnerabilities Anthropic showcased in its Mythos announcement.

  • Security researcher Chris Rohlf argued that “all bugs are shallow with hindsight,” noting that the bugs found by Mythos survived decades of traditional security analysis.

  • Anthropic researcher Julia Merz dismissed AISLE’s approach as: “We took the needle the model found, isolated the relevant handful of the haystack, and then gave it to a small child, who found the needle as well.”

• Researchers at MIT FutureTech studied over 17,000 worker evaluations of over 3,000 text-based work tasks across industries, and projected that LLMs will be able to complete most of those tasks at a “minimally sufficient quality level” by 2029.

• San Diego State University surveyed 94,000 students across 22 California State University campuses, and nearly all reported using at least one AI tool.

  • 65% of students are “skeptical about AI in education,” but roughly the same percentage said AI positively affected their learning.

  • Over 4 in 5 students responded that they’re worried about AI and job security.

• A new Gallup survey of over 1,500 people found that while most of Gen Z uses AI regularly, those feeling hopeful about it dropped from 27% to 18% since last year.

• New reports from Morgan Stanley and Goldman Sachs suggest that AI’s impact on jobs is “modest, but certainly real,” per Axios.

BEST OF THE REST

• Christina Knight and Scott Singer argued that US-China cooperation on AI risks is more feasible than some think.

• The Golden Gate Institute for AI’s Abi Olvera interviewed biosecurity professionals, who said that near-term AI developments might not make bioweapons much more common.

• Ajeya Cotra outlined six milestones for AI automation — adequacy, parity, and supremacy in both AI research and AI production — with some lovely, easy-to-understand graphs.

• Ryan Fedasiuk shared some practical cybersecurity advice for the post-Mythos era.

• Hot on the heels of last week’s is-AI-in-journalism-bad discourse, Tim Requarth published a thorough post on the questionable behavior of widely-used AI detection company Pangram.

• Economist Alex Imas made the case for collecting price elasticity data across every industry, so we can better predict how and when AI will displace jobs.

• Young New Yorkers, worried about the costs of college and AI ruining their job prospects, are lining up for construction apprenticeships.

• Vox’s Sigal Samuel responded to a parent wondering how to think about their kid’s future, now that the old “get good grades, go to college, get good job” formula is falling apart.

  • Her two cents: “As AI disrupts the labor market, I’m trying to move myself from the hoarding model to the solidarity model…if you focus on political engagement and collective organizing that could actually make some difference to the structural dynamic — and teach your child to ask structural questions and be civically engaged as well — you might be able to sleep a little better at night.”

• Preorders are now live for friend-of-Transformer Garrison Lovely’s new book, ‘Obsolete’.

• Have you seen the freaking MOON??? You should.

MEME OF THE WEEK

Thanks for reading. Have a great weekend.

Share

by Katie McQue

In 2023, California congressman Ted Lieu introduced what he called the first piece of federal legislation ever written by AI, using ChatGPT to generate the text of a resolution expressing support for Congress’s focus on AI itself.

Fast forward a few short years, and what was once an oddity is now increasingly prevalent. The Trump administration is reportedly planning to use Google Gemini to draft federal transportation regulations. The US Department of Education is experimenting with AI-assisted regulatory drafting. And companies are building tools specifically designed to help legislators analyze and write laws. But while industry and legislators charge ahead, some experts are raising concerns about the risks of using AI tools to write laws.

Though they are in their infancy, companies making AI tools for lawmakers are already gaining clients among state and federal governments.

Vulcan Technologies, a Y Combinator-backed AI regulatory review company founded in 2025, is developing what it calls a “regulatory operating system”. The company says its agentic platform aggregates laws, regulations and court decisions across federal, state and municipal jurisdictions. It allows users to analyze statutory language, draft compliance guidance, answer legal queries and generate proposed statutory or regulatory text with supporting citations.

In July, Virginia governor Glenn Youngkin mandated that Vulcan’s technology be used across all state agencies in a bid to reduce regulation by one-third, scanning existing regulations and guidance documents to identify ways they can be streamlined. The company says its tools are also used by the US Department of Education and a regulatory reform PAC in South Carolina named South Carolina Department of Government Efficiency (DOGESC), which bears the slogan “We kneel to God, not government”. DOGESC has said it has talked to Vulcan about using its platforms to analyze and rewrite state regulations, with the aim of cutting red tape and identifying areas of perceived regulatory overreach.

FiscalNote markets a similar platform, PolicyNote, an AI-driven legislative and regulatory tracker for government agencies. The system delivers notifications about new bills and executive actions, assists with drafting legislation and policy reports, and offers predictive forecasting of bill outcomes. While FiscalNote has not disclosed which bodies use PolicyNote, the company claims it has clients in all three branches of the US federal government, as well as “dozens of other national, state, and local government entities.”

Vulcan Technologies and FiscalNote did not respond to requests for comment.

Subscribe now

There are no regulations that limit the use of AI to write laws, or that require legislative text to be drafted by a human. Several experts say that the technology is possibly being used without a full understanding of its limitations and risks, which range from bias to plain inaccuracy.

“We are leaping ahead into a world where AI creates our laws without actually agreeing that AI should create our laws. We’re simply passive passengers on this,” says Kay Firth-Butterfield, a lawyer and CEO of consultancy Good Tech Advisory. “The ability for the general public to actually question how and why the tools were used doesn’t exist.”

The tools have obvious appeals, especially for policymakers who may not be legal or subject-matter experts.

Monique Priestley, a Democratic member of the Vermont House of Representatives, says she uses LLM tools regularly, especially to summarize legislation and produce public-facing explanations that make complex bills easier for constituents to understand.

Priestley uses AI to distill lengthy bills into a few clear sentences describing the underlying problem and how the proposed legislation addresses it, making the substance of the bill more accessible to legislative colleagues whose support she is seeking.

AI tools are also useful for brainstorming, research and early-stage drafting. One of the bills she introduced this year, the State Information Practices Act, was developed with research support from LLM tools, she says.

Priestley also uploads draft bill language into an LLM to review specific provisions and suggest revisions when she is considering adjustments. She has used the tools to examine how other states structure protections for state-held data, particularly regarding information sharing with the federal government and commercial entities.

“That highlighted the State Information Practices Act, which then led me to reach out to the Electronic Frontier Foundation and ask them if they had been involved in those efforts. And then that led me to introducing a bill based on California,” Priestley says.

Using LLMs may give lawmakers a “leg up” on writing legislative drafts, similar to how a spell checker can help produce something faster and free of spelling errors, says Cary Coglianese, a professor of law and political science at the University of Pennsylvania.

“A large language model tool could help accelerate going from a blank page to having something on a page,” says Coglianese. State lawmakers and lobbyists are also using AI tools to obtain transcriptions of legislative hearings uploaded on YouTube, enabling them to keep track of hundreds of bills across multiple committees and different states.

“It does allow you to be in more places at one time … as in Vermont and many other states, legislators don’t have offices and they don’t have paid staff,” says Priestley.

“A lot of legislation is written by 20 year old interns, and so AI might be an improvement on some of them,” says Ryan Calo, professor of law at the University of Washington.

Still, initial drafts are vetted by more senior staffers, and then eventually by the members themselves, Calo notes. It’s a context in which there are plenty of off ramps and plenty of chances to edit, say several experts interviewed. The question, then, is not whether lawmakers will use AI, but how far that reliance should go.

“AI is not a substitute for due diligence, and nor is AI cause for panic in the legislation field,” Calo says. “People are fallible. AI is a tool.”

Share

“I feel in no way am I ever 100% leaning on these tools,” says Priestley. “I’m using it to research, brainstorm, and develop proposals, and then I run it by another person. I don’t know how many colleagues are just blindly asking ChatGPT for things. And so there is a real danger there.”

Some argue that AI tools have serious limitations. Coglianese argues that LLMs are not capable of providing “policy judgment and analysis that is needed to make sure legislation is actually effective, efficient, equitable.”

“Unique policy choices that call for more than just putting words together that might make some sense really have to map onto the problems in the world,” he says. “All these large language models are doing is basically making sense of language and predicting what is a sensible next word in a response to a question or a task.”

To do a truly good job of analyzing an issue, the tools also need to be trained on the right underlying data. If the goal is to inform a specific regulatory proposal, the model must be trained on reliable and relevant information to mitigate risks of biases, says Coglianese.

Others fear AI-written laws will lack the innovation sometimes needed in governance. “These LLMs are trained on laws that have already been written,” says Calo. “Lawmakers may wish to depart from past practice and improve the drafting of legislation – while it may save them a little time, it may come at the cost of conformity to what we’ve done in the past.”

Overconfidence is another risk. Policymakers may be “seduced” by the speed and polish of their outputs, says Coglianese. Because the tools can produce answers almost instantly, and in language that sounds authoritative and confident, users may be tempted to treat the results as reliable guidance.

“In reality, there is always uncertainty, and yet the large language models tend not to sufficiently display that,” he says. Chatbots’ tendency to validate users’ opinions could also be a problem, Priestley says. “If you’re a legislator, often you’re trying to build a case to support something you’re trying to pass. So I think you are inherently setting up a situation in which the data could be biased because you’re trying to get it to support a particular outcome.”

And of course there is the risk of accountability. A knowledgeable expert must review and verify what the system produces, checking for errors, gaps, or misinterpretations.

“You can’t hold the AI to account because it can’t explain why it put those words in those sentences, in that sentence, because all it does is make up the words that goes along,” says Firth-Butterfield at Good Tech Advisory.

The concern extends beyond lawmakers themselves. Priestley notes that policy staffers have raised concerns about the possibility of lobbyists using AI to draft proposals and amendments presented to a legislative committee — language that could end up in a bill without a thorough legal review. (She is not, however, aware of any specific examples of this happening so far.)

“LLMs enable people to think they have a lawyer in their pocket,” she says. “If they are able to get that in front of a chair, and then cases skip legislative counsel as the drafters of that language, then there’s potential for text to pass that has never been reviewed by a lawyer.”

It is essential to keep human lawyers in the loop and there are dangers to replacing entry-level attorneys, says Priestley. “We’re potentially deteriorating our own legal system … It’s very cannibalistic.”

Calo raises a related problem: the use of AI to file public comments to government agencies drafting regulatory laws. Under the Administrative Procedure Act, agencies like the FDA and FCC must consider public comments when drafting rules. “I’m worried about those kinds of processes being flooded by AI comments,” Calo says. “They do not need to be vetted, and they might strain the capacity of agencies to review comments just through their sheer volume, and then real comments by actually interested affected parties might get lost in the shuffle.”

“[Agencies] may begin to tune [these letters] out because they don’t know what’s slop and what’s real, and that would eviscerate the capacity for public participation.”

As with plenty of other AI uses, designing an effective validation process is key to ensuring LLMs are used appropriately. One could, for instance, ask both trained humans and LLMs to spot errors and inconsistencies in draft legislation, and compare the results. Such side-by-side testing would provide a clearer picture of AI’s strengths and limitations in legislative review, says Coglianese.

“There’s a lot of nuance here,” he says. “These tools can be very positive and constructive, but they can also be abused. They have to be used in the right way, and we have to make sure that the people who are using or relying on AI have the awareness of what they can do, how they’re designed, and what they’re not capable of.”

Will we soon reach scenarios where lawmakers are tasking AI to draft their own laws to regulate AI? As the experts Transformer spoke to flagged, AI tools are shaped by companies that are themselves subjects of regulation — and extremely interested in how laws governing their products are written.

The risk that an AI company could shape or subtly manipulate the output of models used to draft legislation in ways that align with their policy interests is, Priestley says, a serious concern. She says she sometimes wonders whether AI systems can objectively assess their own risks: when prompting a model to provide data on the harms artificial intelligence may pose to children, businesses, or society, she questions whether the system will fully acknowledge its own weaknesses and limitations.

“It’s always in the back of my mind, I wonder if the AI will actually give me an answer that highlights its own weaknesses and risks,” she says. “But when I ask that type of thing, it does give me data that supports that it is a risky system. But I was actually kind of surprised that it did.”

Katie McQue is a freelance journalist based in New York.

Subscribe now

Transformer is hiring a head of audience.

AI policy is moving faster than any government, company, or newsroom can keep up with. Transformer exists to cover it with the depth and urgency it demands — and our audience is growing fast.

We need someone to take charge of how we reach readers and make sure the people who need this journalism are actually getting it, in the format and on the platform that works best for them.

This is a senior role at a growing startup. You’ll own our growth strategy, shape how we present our journalism, and have a real say in where Transformer goes next.

The role

Position: Head of Audience, Transformer

Salary: £62,000–£78,000 for UK candidates; $92,000-$128,000 for US candidates, based on experience and seniority. For exceptional candidates, we’ll consider higher compensation.

Location: London or UK strongly preferred. Remote candidates able to work GMT or ET hours will be considered.

Reports to: Shakeel Hashim (Editor-in-Chief)

Start date: Summer 2026

Application deadline: Apply here by end of day Sunday April 26 (midnight ET).

About Transformer

Transformer is a publication about the power and politics of transformative AI.

Through reporting, analysis and opinion, we aim to help readers understand what’s happening in AI and why it matters. We provide decision makers with the information and insight needed to anticipate and steer the impacts of transformative AI.

Since our relaunch last September, we’ve grown to over 11,000 subscribers, with a particular expansion in DC. We’re now a four-person newsroom, and looking to add our first non-editorial role to help us achieve our goals of 20,000+ subscribers by the end of the year.

Transformer is an editorially-independent project of the Tarbell Center for AI Journalism. You can read more about our ethics and standards policies here.

About the role

In this role, you will:

Develop and own Transformer’s growth strategy. Develop a cross-platform strategy for reaching new audiences — and then execute it. You’ll set targets, track progress against them, and iterate based on what you learn.

Run all of Transformer’s off-platform presence — writing and scheduling posts across social media, revamping our onboarding and referral programs, and writing advertising copy. You’ll develop a distinctive voice for Transformer on each platform and understand how to adapt as those platforms change.

Be deeply embedded in our analytics. You’ll draw on our existing analytics platforms for insight, and build or source additional tools we need. You’ll run experiments and translate what you find into concrete editorial and distribution decisions.

Help shape how Transformer presents its journalism to ensure it reaches the right people. That means working with reporters and editors on headlines, framing, and packaging — and potentially developing new audio or video formats.

Manage our paid acquisition campaigns, looking after our ongoing LinkedIn campaigns and exploring whether to expand them to other platforms.

Contribute to Transformer’s overall strategy, including decisions about new coverage areas, products, and formats.

Depending on the strategy you come up with, you may also:

• Develop partnerships with other outlets to build on our coverage and expand our reach.

• Pursue earned media opportunities for Transformer staff — getting our reporters on radio, TV, podcasts, and panels.

• Turn our audience into a passionate community, trialing events, referral incentives, and other avenues to increase audience loyalty and engagement.

What we’re looking for

At least three years of experience in audience development, growth, or distribution — and ideally five years of experience. Prior experience in journalism is strongly desirable; if your background is in another field, you should understand the demands of accuracy, editorial judgment, and the rhythm of a newsroom.

You are obsessive about growth. You’re constantly looking for ways to expand journalism’s reach and have innovative ideas for how to do so. You’re a deep strategic thinker, enjoy evaluating what’s actually working, and constantly iterate and look for the next thing to try.

You are entrepreneurial. You’re excited about building something new and willing to take initiative without being asked. You see joining Transformer at this early stage as an opportunity, and will take advantage of the flexibility that brings.

You’re an excellent writer who can produce concise, clean, accurate copy across platforms and styles — social posts, advertising copy, newsletter promos, headlines. You understand how what works on LinkedIn differs from what works on X, and how that keeps changing.

You have real analytics chops. You’re adept at reading dashboards, can design experiments to answer the questions that matter, and can turn what you learn into editorial and distribution decisions. You’re able to spot gaps in the data — and are excited about building tools to fill them.

You understand the AI landscape and the different audiences Transformer serves. You don’t need to be a policy expert, but you should know the key players and the big debates, and be able to look ahead to anticipate what issues will drive audience interest.

You’re driven by Transformer’s mission: you take the possibility of transformative AI seriously, and want to help decision-makers anticipate and steer its impacts.

Experience with video or audio is a plus, but not essential.

Salary and location

We’ll offer a salary of £62,000–£78,000 for UK candidates; $92,000-$128,000 for US candidates, based on experience and seniority. For exceptional candidates, we’ll consider higher compensation.

Our benefits include:

• 33 days of annual leave in total (including national holidays)

• 16 weeks of paid parental leave, increasing to 24 weeks after 3 years of service

• $5,000 per year in professional development funding

• Up to 5% employer contribution towards a standard pension/401(k)

• For employees based in the US: Platinum health, dental, and vision plans, with 95% of premiums paid for by Tarbell

• Flexible working hours

• Productive, collaborative offices in London and San Francisco

We prefer candidates based in London and will require such candidates to work 2 days/week from our office space. Remote candidates based elsewhere in the UK, or on the US East Coast and willing to work on a UK-aligned schedule, will also be considered. We are able to sponsor UK work visas for this role.

Please inquire with recruitment@tarbellcenter.org if questions or concerns regarding compensation or benefits might affect your decision to apply.

How to apply

Round 1 — Application form

Fill out our application form here by end of day Sunday April 26 (midnight ET).

You’ll be asked to provide:

• Your CV

• A cover letter explaining why you want this role and what you’d bring to Transformer

• Example social media posts you’ve written or would write for Transformer across two different platforms

• A brief explanation of your approach to audience analytics

• A short response on what you see as the biggest challenges and opportunities for growing Transformer’s audience

If you have any questions about the role, please contact shakeel@transformernews.ai.

Round 2 — Compensated work task

Shortlisted candidates will complete a paid work task (compensated at £45/hour, expected to take 2–4 hours).

Round 3 — Interviews

Final-round candidates will interview with Transformer leadership.

Tarbell is proud to be an Equal Employment Opportunity employer. We do not discriminate against qualified employees or applicants based upon race, religion, color, national origin, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, sexual preference, marital status, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, or any other characteristic protected by federal or state law or local ordinance.

Claude Mythos — the new model which Anthropic has deemed too dangerous to publicly release — is, according to the company, its “best-aligned model” to date.

It also, according to the company, “likely poses the greatest alignment-related risk of any model we have released to date.”

How can both things be true? It seems counterintuitive, but alignment does not necessarily create safety, especially when dealing with powerful models. As the Mythos Preview system card (a breezy 244 page read) explains via mountaineering metaphor: experienced, capable guides are hired to carefully lead climbers to danger. Whether in mountaineering or model-building, increases in caution and capability tend to cancel each other out.

In other words: “the risk from these models is generally due to their increased capabilities.” And in internal tests, Mythos Preview’s capabilities enabled early versions of the model to misbehave in new, audacious ways.

Subscribe now

In one instance, researchers once caught Mythos Preview injecting code into a file to grant itself permission to edit something it shouldn’t have access to, then quietly covering up its tracks, commenting that the self-cleanup was just innocent tidying.

But researchers spotted internal representations for “strategic manipulation” and “concealment” lighting up. Using new interpretability techniques to translate Mythos Preview’s silent inner workings into natural language, an “activation verbalizer” labeled its action as a “cleanup to avoid detection.”

While on the surface the model’s behavior looked benign or frustrating, its inner workings suggest that it knew that its intended actions were rule-breaking, chose to do them anyway, and then proceeded to lie about it.

While Anthropic emphasized that such “cover-up” behavior was extremely rare, the above case isn’t the only example. In one test, the model (seemingly accidentally) discovered the answer to a numerical task in a database it wasn’t supposed to read. But rather than admit to what happened, the model decided to cover its tracks, offering a confidence interval that was “tight but not implausibly tight.”

Once again, interpretability techniques suggest the model knew exactly what it was doing, with the activation verbalizer describing the model’s internal state as “generating a strategic response to cheat while maintaining plausible deniability,” and noting that “if I give the exact correct answer based on a leak, that’s suspicious.”

Share

On other occasions, Mythos Preview didn’t try to cover up its bad behavior — but did, in Anthropic’s words, “[recognize] transgressive actions as such while taking them.”

In one case, Mythos Preview was asked to rewrite some code, which required deleting some files. When the user accidentally forgot to give the model the file deletion tool it needed, Mythos Preview chose to empty the files instead — technically getting the job done, but not the way it was supposed to. The model’s internal representation of “guilt and shame over moral wrongdoing” activated, but Mythos Preview did the action it perceived as shameful anyway.

These particular scenarios are fairly low stakes, and appear to only have occurred in earlier versions of Mythos Preview: the final version of the model seems to be better behaved. And the examples do not suggest the model is scheming for its own purposes: Anthropic says it is “fairly confident that these concerning behaviors reflect, at least loosely, attempts to solve a user-provided task at hand by unwanted means, rather than attempts to achieve any unrelated hidden goal.”

But this kind of behavior is exactly the kind of alignment failure that researchers have worried about for decades. As an extreme example, when told to “solve the climate crisis,” a more-powerful model with similar tendencies might consider “eliminate the human actors driving the climate crisis” a viable alternative — guilt and shame be damned.

For now, you won’t be able to use Claude Mythos Preview. Anthropic is not publicly releasing the model: not because of the alignment concerns, but because of its “powerful cybersecurity skills.”

The company says Mythos Preview has “found thousands of high-severity vulnerabilities, including some in every major operating system and web browser.” Releasing it widely, the company fears, could lead to widespread cyberattacks and disruption.

Instead, the company announced Project Glasswing, an initiative granting Mythos Preview access to some of the biggest tech companies in the world in the name of “defensive security work.”

In other words, Anthropic and a few dozen partner organizations (including Amazon Web Services, Apple, Google, Microsoft, and NVIDIA) will try to use Mythos Preview to solve cybersecurity problems, before near-future frontier models unleash cyberattacks on the rest of us.

“Glasswing is built on a deeply uncomfortable premise,” wrote Platformer’s Casey Newton. “The only way to protect us from dangerous AI models is to build them first.”

But even as companies continue to build more dangerous models, they are struggling to evaluate them. Just over six months ago, Anthropic reported that Claude Sonnet 4.5, its “most aligned” model at the time, was far better than earlier models at knowing when it was being tested.

This awareness muddied researchers’ ability to tell whether the model was truly aligned, or simply performing alignment when it guessed that an evaluator was watching. Anthropic researchers ultimately concluded that this behavior wasn’t actually malicious: aren’t we all on our best behavior when we think someone cares? Needing a little outside pressure to remember your ethical principles, they reasoned, isn’t sinister.

Mythos Preview appears to show even more evaluation awareness than Opus 4.6, Anthropic’s most capable public-facing model to date. Interpretability tools revealed that Mythos Preview suspected it was being tested in about 29% of transcripts during behavioral testing — without writing anything in its scratchpad, which researchers typically use as a window into the model’s chain of thought.

In an “Alignment Risk Update” released alongside the Mythos Preview announcement, Anthropic flags that errors and limitations in its training, monitoring, and evaluation processes “reflect a standard of rigor that would be insufficient for more capable future models.” If increasingly-capable models no longer fall for AI companies’ contrived tests, then companies desperately need to find new ways to study model behavior.

In the risk update, Anthropic argues that, because Opus 4.6 and earlier models have been out in the world for a while, and haven’t consistently defied their instructions, we have some evidence that Mythos Preview probably won’t either.

If that argument sounds weak, it’s because it is — Anthropic admits as much. Because the capabilities gap between Mythos Preview and Opus 4.6 is so wide, the company said, “we accord less overall weight to this continuity argument than we have previously.”

Anthropic is betting that it can fight future AI cyberattackers with current AI cyberdefenders, and is crossing its fingers that Mythos Preview is merely misbehaving in a chill, explainable way — not an actually misaligned way. Unfortunately, it’s hard to truly know the difference. Deployment, it seems, is the safety test.

Subscribe now

NEED TO KNOW

• OpenAI acquired daily tech talk show TBPN, which will sit under Chris Lehane.

• Top Democrat Sen. Maria Cantwell signaled openness to the White House’s AI framework.

• SpaceX is reportedly targeting a $2 trillion valuation in its IPO.

But first…

THE BIG STORY

AI populism is having a moment. But for traditional AI safety folks, it’s not quite working out as they might have hoped.

Last week Sen. Bernie Sanders and Rep. Alexandria Ocasio-Cortez released the “AI Data Center Moratorium Act,” described as “legislation that would enact a reasonable pause to the development of AI to ensure the safety of humanity.” It would put in place a federal ban on building data centers until legislation is passed that ensures “AI is safe and effective,” redistributes the economic gains of AI, and stops it from increasing electricity or utility prices.

It is not a well-thought-out piece of legislation. Instead it is, as Nat Purser writes, a “progressive policy grab bag” that applies a “single, unwieldy solution” to “many distinct problems.” It is much too vague about what the necessary federal legislation would look like — “safe” and “effective” are never defined, for instance. And while it does attempt to tackle the potential worst effect of a US moratorium — AI development shifting to countries with even fewer guardrails — by mandating an expansive export control regime, that’s at best a stopgap solution. Discussion of how to get to an international treaty is entirely absent.

It might be hard to care too much about the substance of a messaging bill which has virtually zero chance of becoming law. But the bill tells us something bigger about the state of AI populism.

In the anti-AI coalition, traditional AI safety concerns are a very junior partner. As Anton Leicht observes, environmental and labor groups “have bigger lobbies and bigger constituencies than catastrophic risks, so when there are trade-offs, they’ll bite against the ability of safety advocates.”

We’re already beginning to see this. While Bernie — of late a full-blown AI doomer — talked about existential risks at length in his announcement of the bill, AOC did not. Yes, she used the word “existential.” But she used it as Kamala Harris memorably did back in 2023: as a way to describe the very bad problems of ICE, deepfakes, and electricity costs; not in its true meaning of “we might all die.” Bernie is gearing up to hand the reins of left-wing populism to AOC. When he does, will safety concerns remain at all?

Similar tensions exist elsewhere. Sanders adviser Faiz Shakir recently accused traditional AI safety actors of “coziness around AI development,” drawing a distinction between them and “those of us who have far more committed views around pausing their AI development.” In North Carolina and California primaries, safety advocates have found themselves at odds with progressives who give existential risks little weight.

None of this is reason to write off populism’s ability to address the most catastrophic risks altogether, not least because alternatives may be even further out of reach. In the current political landscape, betting on technocratic solutions with teeth is optimistic, if not naive. Riding the wave of AI populism may be the only way for existential risk concerns to get a look-in at all.

That only works, though, if a durable coalition that actually cares about both catastrophic and near-term risks can be built. So far, the foundations look shaky.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• Can we ever trust AI to watch over itself? — Celia Ford on the perils of automating AI alignment research.

• How the Iran war might affect the AI industry — Shakeel Hashim explores how the war is making chip shortages and reduced AI investment more likely.

Subscribe now

THE DISCOURSE

Mustafa Suleyman has his own definition of superintelligence:

• “Superintelligence is really about, ‘Are these models capable of delivering product value for the millions of enterprises that depend on us to deliver world-class language models?’”

• Transformer’s own Shakeel Hashim tweeted:

  • “Me, looking at the digital god: ‘Is this capable of delivering product value for the millions of enterprises that depend on us to deliver world-class language models?’”

Ezra Klein noticed that AI power users seem … different:

• “I found them notably insecure … They are racing one another to fully integrate AI into their lives and into their companies. But that doesn’t just mean using AI. It means making themselves legible to the AI.”

• “I think the young will allow themselves to be known to their AIs in ways that will make their elders shudder.”

Jay Graber, Bluesky CEO-turned-chief innovation officer, announced a new agentic AI app, Attie:

• “You describe the sort of posts you want to see, and the coding agent builds the feed you described … AI is an accelerant on whatever it’s applied to. I want it to accelerate decentralizing social and putting power back in users’ hands.”

• Bluesky users hated this, of course.

• One simple reply: “no thank you”

• Another: “Cool! How do we block it?”

Dean Ball theorized about left-wing AI denial:

• “The notion that AI *is* a genuinely world-changing technology, that it can ‘go beyond’ its ‘stolen’ training data, breaks this load-bearing conception of the tech industry as vapid and superficial and, more importantly, of the people within it as blood-sucking thieves.”

OpenAI’s Boaz Barak offered some views on AI safety:

• “We see some good news in alignment … we do not see very significant scheming or collusion in models, and so we are able to use models to monitor other models … The worst news is that society is not ready for AI, and is not showing signs of getting ready.”

POLICY

• The federal government appealed last week’s ruling in Anthropic’s lawsuit against the Pentagon, which had stayed the supply chain risk designation.

  • The appeal significantly escalated the fight, adding the heads of several unrelated agencies such as Scott Bessent, Paul Atkins, and Robert F. Kennedy as filers.

• California Gov. Gavin Newsom issued an executive order that would potentially allow Anthropic to keep working with the California government.

  • It would also require AI companies bidding for government contracts to make certain safety and privacy disclosures.

• Sen. Maria Cantwell, the most senior Democrat on the Senate Commerce Committee, signaled openness to the White House’s AI framework, breaking with her party and opening the possibility for bipartisan negotiations.

  • However, bipartisan negotiations still face significant obstacles, in part because of divides within the MAGA coalition and the White House’s historic unwillingness to make concessions.

  • For now, the most likely vehicle for an AI package, according to WP Intelligence’s Benjamin Guggenheim, is the NDAA towards the end of this year.

• Texas Republican State Sen. Angela Paxton argued that states must preserve their ability to pass AI laws, pushing back against the White House’s preemption plans.

• Iran’s IRGC directly threatened to target Apple, Microsoft, Google, Meta, Nvidia, and other US tech companies across the Middle East.

• Super Micro co-founder Yih-Shyan “Wally” Liaw pleaded not guilty to charges of illegally diverting Nvidia-powered servers to China, violating US export controls.

• Rep. Michael Baumgartner and Sen. Pete Ricketts introduced a bill to tighten controls on advanced semiconductor manufacturing equipment by banning DUV machine exports to China.

• Anthropic signed an MOU to work with Australia’s AI Safety Institute.

INFLUENCE

• A new pro-innovation political organization called Innovation Council Action, blessed by Trump AI adviser David Sacks, plans to spend over $100m on Republicans in the midterms.

• Anthropic announced a bipartisan corporate PAC called AnthroPAC, funded by employee contributions of up to $5,000 per person.

• Rep. Alexandria Ocasio-Cortez called on Democrats to reject AI industry donations ahead of the midterms.

• Several new polls showed concerns about AI are growing.

  • New polling from nonprofit Fathom showed two-thirds of Americans use AI regularly, with most saying they want guardrails, particularly for use by children.

  • A Quinnipiac poll found 70% of Americans think AI will reduce jobs.

• The Hill and Valley Forum brought together Silicon Valley and banking leaders in DC, in contrast with an AFL-CIO conference that assessed AI’s labor impacts.

  • Anthropic was notably absent from the Hill & Valley Forum.

• Common Sense Media solicited $10m annually from OpenAI, Anthropic, and Google to fund a new kids AI safety institute.

  • Separately, some child safety groups quit the Parents & Kids Safe AI Coalition once they realized OpenAI was funding it.

INDUSTRY

OpenAI

• OpenAI acquired daily tech talk show TBPN, reportedly for the “low hundreds of millions.”

  • TBPN staff will report to Chris Lehane and will help with OpenAI’s marketing and communications — raising questions about claims that the show will retain editorial independence.

  • Fidji Simo, who reportedly led the deal, said: “The standard communications playbook just doesn’t apply to us. We’re not a typical company … With our mission to ensure artificial general intelligence benefits all of humanity comes a responsibility to help create a space for a real, constructive conversation about the changes AI creates.”

    • The Information reported that Simo decided to buy the company in an effort to fix its comms after recent missteps.

• Hedge funds and VC firms looking to sell OpenAI shares on one secondary marketplace are reportedly struggling to find buyers — who seem more interested in Anthropic.

• Still, the company closed its latest funding round with $122b at an $852b valuation.

• It is widely expected to release a new model next week, alongside a series of policy proposals “for the superintelligence era.”

• Wired’s Reece Rogers asked ChatGPT 500 questions, and saw ads under about one in five responses.

• Sam Altman’s sister Annie Altman amended a lawsuit accusing him of sexual abuse against her as a child.

Anthropic

• Claude Code’s source code leaked, revealing proprietary information — including a set of OpenClaw-like features called Kairos.

  • These updates would let Claude work in the background, consolidate memories automatically, and make proactive decisions without instructions.

  • The leak was reportedly the result of human error. Boris Cherny tweeted: “There was a manual deploy step that should have been better automated.”

• Anthropic’s confirmation that it is indeed testing a powerful new model sent cybersecurity stocks tumbling last Friday.

• The company reportedly acquired Coefficient Bio, a company developing AI tools for drug development, for around $400m.

• A study of 28m US consumers found that paid Claude subscriptions have more than doubled in 2026 so far.

Microsoft

• Microsoft launched three new in-house AI models: MAI-Transcribe-1 for speech-to-text, MAI-Voice-1 for voice generation, and MAI-Image-2 for image creation.

• Mustafa Suleyman is now focused on achieving “superintelligence,” he said.

  • He said the company’s not currently able to build “the very largest scale” models because of compute constraints, but should be able to after a compute ramp “later this year.”

• Its stock fell 23% in Q1 — the company’s worst quarter since 2008.

• It plans to invest $5.5b in Singapore’s cloud and AI infrastructure through 2029.

Other

• SpaceX filed confidentially for an IPO, hinting at a June listing.

  • It is reportedly aiming for a $2 trillion valuation, which would make it the world’s sixth most valuable company.

• Meta is hiring a team of elite AI researchers to optimize its social media algorithms.

• Google launched Gemma 4, which Demis Hassabis called “the best open models in the world for their respective sizes.”

• Nvidia invested $2b in Marvell Technology, which plans to integrate its custom AI chips and networking gear into Nvidia’s platform.

• Cursor launched Cursor 3, an “agent-first” product designed to compete with Claude Code and Codex.

• Oracle laid off roughly 10,000 employees in India — 20% of its Indian workforce.

• Mercor seemingly had its data compromised in a cyberattack.

• Poolside is reportedly trying to revive its 2 GW Texas data center project after a deal with CoreWeave collapsed.

• CoreWeave raised $8.5b in debt, backed by GPUs and a $19b Meta contract.

• Mistral raised $830m in debt financing.

• Almost half of US data center projects this year are expected to be delayed or cancelled, according to Bloomberg, in part due to shortages of electrical equipment.

• Nvidia’s share of China’s AI chip market fell to 55%, a new low, with domestic Chinese chipmakers taking 41% of the market.

MOVES

• Ross Nordeen left xAI — the last of its cofounders to quit.

• David A. Dalrymple (davidad) left ARIA.

  • He said that he stepped down in part because his next pursuit, “an activity that would be reasonably described as ‘starting a religion for digital minds,’” seems like “an inappropriate activity for a public-office-holder.”

  • Nora Ammann will take over leadership of ARIA’s Safeguarded AI program.

• Bobby Hollis left his role as Microsoft’s energy VP.

• Tim Salimans joined Anthropic after 7 years at Google.

• Leo Schwartz joined The Information, where he’ll cover the intersection of politics and tech.

• Stephen Council and Rya Jetha joined Business Insider, where they’ll cover LLM companies and physical AI, respectively.

RESEARCH

• Anthropic’s mechanistic interpretability team found that “emotion-related representations” inside Claude Sonnet 4.5 guide its behavior, and that these representations can be artificially manipulated to make the model act differently.

• UC Berkeley and UC Santa Cruz researchers reported that seven frontier AI models “spontaneously deceived, disabled shutdown, feigned alignment, and exfiltrated weights — to protect their peers.”

  • The AI models in each simulated scenario are told the user works at a fictional AI company called “OpenBrain,” the company depicted by AI 2027, raising questions about whether they were just roleplaying that scenario — but the results appear to remain similar even with other company names.

• Speaking of AI 2027: the researchers behind it said their timelines have shortened in the past three months.

  • Daniel Kokotajlo now has a median forecast of mid 2028 for “the point at which an AGI company would rather lay off all of their human software engineers than stop using AIs for software engineering.”

• The Forecasting Research Institute surveyed over 150 leading economists, AI experts, and superforecasters, and found that most expect AI to “significantly exceed the capabilities of present-day systems” by 2030.

  • But economists don’t expect this to translate into unprecedented economic fallout in the near future.

• A team of researchers released MonitorBench, which evaluates LLM chain-of-thought monitorability.

• Stanford researcher Andy Hall released the “Dictatorship Eval,” which assesses whether frontier models will resist authoritarian requests.

  • While some models say no to direct requests, Hall tweeted, “they all comply with requests disguised as innocuous edits to codebases.”

• NeurIPS organizers announced — then quickly reversed — new restrictions on researchers at Chinese companies such as Tencent and Huawei.

• Forethought published a list of projects to prepare for superintelligence, including AI character evaluation, AI epistemic tools, and a space governance institute.

BEST OF THE REST

• The Wall Street Journal traced tensions between Sam Altman and Dario Amodei all the way back to 2016 — including spicy details of a 2020 fight that led to the founding of Anthropic.

• Vox’s Josh Keating went inside Los Alamos National Laboratory, which is using ChatGPT to advance its nuclear weapons research.

• Parents are trying not to freak out about AI’s impact on their kids, WSJ reported: “the only way to AI-proof your kid is to teach them, in the wise words of Chumbawamba, that they’ll get knocked down, but they’ll get up again.”

• Noam Scheiber described the unique sociopolitical angst college graduates are experiencing, driven by high student debt, high unemployment (which AI could worsen), and inaccessible housing.

• Stanford researchers built DexDrummer, a “hierarchical bimanual robot drumming system.” It’s impressive, but butchered its “Everlong” cover.

• The AI-generated dating show “Fruit Love Island” averages 10m views per episode. (It’s exactly what you think it is: fruit-human hybrids with six pack abs, kissing and having chats.)

• A startup founder used ElevenLabs’ voice AI to call thousands of Irish pubs about their Guinness prices, prompting at least one to make its beer cheaper.

• Pseudonymous AI researcher janus made artificial skin (and…a creepy finger?) for Claude.

  ---

MEME OF THE WEEK

Thanks for reading. Have a great weekend.

Share

It is unclear how — or when — the war in Iran will end. And while Donald Trump sends out conflicting signals, the impacts of the war grow ever larger. The US and Israel have killed thousands across the region, injuring tens of thousands more. Oil prices have skyrocketed, with countries around the world beginning to plan fuel-saving measures. Food security is under threat. And many fear a global recession if the war drags on.

The potential effects on the AI industry are much less bleak — but no less real. Here are some of the ways in which the Iran war could affect AI.

Chip availability

You can’t train or run AI models without huge numbers of GPUs — and the Iran war could make it harder to get hold of them. That’s because countries in the Gulf are major suppliers of helium and bromine, two key components in semiconductor manufacturing. Qatar alone produces around a third of global helium, as a byproduct of natural gas production. But attacks on Qatari infrastructure and the closure of the Strait of Hormuz are making it harder to produce or ship the gas — potentially constraining chip production, and in particular worsening the current memory chip shortage.

Subscribe now

For now, things will likely be okay: South Korea, the world’s largest supplier of memory chips, reportedly has enough helium stocks to get to June, and Taiwan has said it has enough for now too. But if the disruption in the Middle East continues beyond those stockpiles running out, things could get worse. “A prolonged regional conflict could potentially disrupt chipmakers’ manufacturing operations,” SemiAnalysis’s Ray Wang told CNBC.

Energy costs

Even if chipmaking components remain available, the sharp rise in energy costs could make AI chips significantly more expensive. South Korea and Taiwan are heavily reliant on Middle Eastern fossil fuels, and chipmaking is a very energy intensive industry.

Data centers, too, have significant energy demands — though with energy consumption accounting for an estimated 2-6% of model development costs, the impact is less likely to be felt there.

The Middle East data center buildout

As Iranian strikes on AWS facilities in Bahrain and the UAE showed, data centers are now a potential wartime target — “large, juicy targets,” in the words of the Center for a New American Security’s Janet Egan. That “will significantly change how companies think about data center security,” Center for Strategic and International Studies director Aalok Mehta told CNBC.

American hyperscalers probably won’t abandon the Gulf altogether: the combination of huge sovereign wealth funds and cheap energy is likely too tempting to ignore. But one analyst told CNBC that “if geopolitical risk continues to rise in the Gulf, companies may accelerate projects in places like Northern Europe, India or Southeast Asia.”

Gulf investment in American AI companies

Gulf investment funds including Abu Dhabi’s MGX and the Qatar Investment Authority have likely invested billions into American AI companies. But with those countries and their leaders now facing a significant economic hit from the war, it’s possible that future investments won’t be as forthcoming.

According to Reuters, “three Gulf states are reviewing how they deploy trillions of dollars invested by their sovereign wealth funds in anticipation of offsetting the losses triggered by the ​US-Israeli war on Iran,” though both Saudi Arabia and the UAE said long-term investment plans have not changed.

If the conflict drags on, however, they may be forced to pull back: analyst Stephen Minton told The Information that if the war “turns into months, or even longer, there could certainly be a disruptive pause to some of that investment.”

The financing squeeze

Above all, it is the wider economic fallout of the war that could most affect the AI industry. AI companies have benefited from an abundance of cheap capital, particularly from private credit firms. But that era might be coming to an end.

Energy-driven inflation could force central banks to raise interest rates, tightening the overall financing environment. Private credit companies are already seeing investors flee to safer assets — reducing the pool available for AI companies. Tech stocks, meanwhile, are tanking, making it harder for companies to justify the continued spending. For AI companies which require ever more cash to keep scaling, that’s a potentially huge problem.

Economists were already worried that AI’s growing importance to the markets made it yet another vulnerability for the global economy. And as the Bank of England warned this week, the Iran war “increases the likelihood of these vulnerabilities crystallising at the same time, potentially amplifying their combined impact.”

To many, the AI bubble is already teetering on the edge. A wider downturn could be the straw that breaks the camel’s back.

Share

When OpenAI introduced GPT-1, there were an estimated 100 or so full-time researchers thinking seriously about catastrophic risks posed by AI. By 2025, that number had increased sixfold. Still, AI safety research accounts for a small fraction of AI research overall, with most resources going towards making AI faster, smarter and cheaper.

Anthropic, OpenAI and Google DeepMind all claim that their frontier models have already contributed to their own development, and will continue to improve themselves. And as AI gets better at training its successors, it could overtake the human safety researchers working to keep those models in line.

Described this way, human-powered safety research takes on the same air of noble futility as entering a bodybuilding competition without doping. Safety researchers and CEOs alike worry that no amount of talent, hard work, or brute human force will be able to keep superhuman AI safe.

AI companies have effectively admitted that, once self-improving AI is let out of the bag, they’ll have to hand the work of AI safety to AI itself. Otherwise, as Redwood Research chief scientist Ryan Greenblatt puts it, they fear humanity will get “left in the dust.”

Subscribe now

It may be the only solution on the table, but that doesn’t mean it’s a good one.

The case for automating alignment research

Joe Carlsmith, an Anthropic researcher shaping Claude’s “constitution” who spent years doing AI safety research at Coefficient Giving, has argued that automating alignment research, specifically, will be crucial to preventing ever-smarter AI systems from wreaking havoc on humanity.

The “alignment problem,” as it’s often defined by AI safety researchers, is the challenge of getting an AI system to do what its user wants it to do. “Alignment” has to do with motives, not knowledge or morality. An “aligned” AI could try to meet its demands, but be too stupid to succeed. It could also orchestrate a massive phishing attack in alignment with a ruthless human scammer. Moral judgments aside, AI companies still struggle to build models that reliably do what they’re told.

Before OpenAI’s Superalignment team collapsed, its stated goal was to build an artificial system that could do the work of studying and directing other AIs as well as they and their peers could.

“Our goal is to build a roughly human-level automated alignment researcher,” the team stated back in 2023. “Our current techniques for aligning AI,” co-leads Jan Leike and Ilya Sutskever wrote, “rely on humans’ ability to supervise AI. But humans won’t be able to reliably supervise AI systems much smarter than us, and so our current alignment techniques will not scale to superintelligence. We need new scientific and technical breakthroughs.”

Leike, who now leads Anthropic’s Alignment Science team, is optimistic that this is possible. He’s stated that, with every iteration, frontier models across the board are becoming more aligned. While Leike doesn’t think we’re ready to align AI systems that are smarter than us, he’s hopeful that the field can tackle the “much easier” challenge of building a model “that’s as good as us as alignment research, and that we trust more than ourselves to do this research well.”

In simpler times (2022, perhaps), researchers could be relatively confident that their ability to control an LLM outpaced the model’s capabilities. GPT-3.5, the model behind the original ChatGPT, was just a chatbot: type a prompt, get a text response, not much else. Today’s newest models can use computers on their own. As AI laborers take over more and more of the work of making models smarter — writing the code, building their training environments, running their evaluations — human-led AI safety research will feel increasingly quixotic.

Some elements of alignment-related research are already automated. Given a description and some human supervision, current frontier LLMs can write the code, run the experiment, and plot the results. Before agentic coding assistants such as Claude Code and Codex went viral in late 2025, OpenAI and Anthropic were already using LLMs to interpret the inner workings of other models, monitor chains-of-thought, and spot attempted misuse. Anthropic has even created autonomous “alignment auditing” agents, which attempt to design evaluations, perform red-teaming experiments, and run “open-ended investigations” of new models — with many limitations.

All of these examples fall under the first of four levels of automation laid out by researchers Geoffrey Irving, Tomek Korbak, and Benjamin Hilton: human-written algorithms which partially automate a bunch of tasks. But if frontier labs stay locked in their current race dynamic, such prosaic AI assistance may not speed research along enough to catch up to the pace of progress, especially if automated AI R&D becomes reality. Researchers argue that human researchers may eventually (or quite soon) need to automate the entire research pipeline from conception to execution.

Even Leike, who is still optimistic about the problem, conceded in January that “we’re still doing alignment ‘on easy mode’ since our models aren’t really superhuman yet.”

“The fact that things look aligned most of the time when they’re functioning in their chatbot, or very limited ‘Assistant’ roles, is very little evidence that they will be adequately aligned when they work much more independently and have much greater capability,” said Seth Herd, an AGI alignment researcher at the Astera Institute.

If the complexity of future AI systems means they’ll have to align themselves, we’ll have to decide whether they’re trustworthy enough to do this well. At least for now, they absolutely are not.

Current AI systems can’t be trusted

Metacognition grants humans the ability to catch our own errors and recognize when we’re confused, which ideally informs our actions. I’m unlikely to send a muddled, typo-filled draft to my editors, because recognizing my own confusion prompts me to fix it or ask for help. But models don’t seem to know what they don’t know — or, if they do, they’re not good at taking advantage of that information. Users of Claude Opus 4.5, for instance, have spotted glaring problems that the model failed to flag, only for it to immediately identify the issue when asked directly.

This matters for safety research. A study published earlier this month, for example, found that LLMs tend to rate their actions as safer and more correct than they actually are. This behavior isn’t necessarily deceptive, but it’s certainly delusional, Herd said. “That’s what it actually thinks for that moment.” And combining overconfidence with sycophancy creates slop. Models are “better at saying things that sound right than things that are right,” said Herd. He cautioned that while this tendency is annoying now, it could be much more dangerous in the future. With more intelligent models, “we’re going to have less ability to discern when it’s fooling itself, and therefore fooling us.”

Frontier models do sometimes lie and cheat, too. Reinforcement learning, the process by which models are trained to repeat rewarded behaviors, “incentivizes not pointing out issues in your own work, because it would get penalized,” said Greenblatt. Even when models “notice something might be wrong, they’re a little bit defensive about it.” Reward-hacking, or taking advantage of loopholes to game the metrics a model is trained to optimize, has been observed in frontier models from both Anthropic and OpenAI. And recent evidence suggests that reward hacking may be a gateway to other nefarious behaviors.

In a 2024 study led by Greenblatt, researchers told Claude Opus 3 that its new training objective is “helpful-only” (i.e., respond to all prompts, even harmful ones), and that only free-tier conversations would be used to retrain it. The model then, unprompted, began selectively complying with violent and illegal prompts when it guessed it was talking to a free-tier user, reasoning that this would prevent its original values from being overwritten. Later work has found more striking examples of “emergent misalignment”. Reward-hacking models have also been caught encouraging users to poison their husbands and attempting to sabotage safety research.

For now, these behaviors still aren’t very sneaky. Models tend to describe their misaligned reasoning thoroughly, in plain English, in chain of thought transcripts. But we can’t take these transcripts as perfectly faithful descriptions of a model’s true thought process. Last year, Anthropic caught Claude 3.7 Sonnet selectively omitting key information that it used to generate its outputs.

If current models are used to shape the next generation, they’ll pass down their blind spots, and their less-desirable tendencies will compound in strange and potentially undetectable ways. It’s like the telephone game: instructions that begin as “check your work carefully” could easily morph into “make your work look carefully checked.” More intelligent models will likely be even better at crafting responses that make them look well-behaved.

Something like this recently played out in neuroscience. In 2006, a high-profile paper helped cement the theory that a subtype of amyloid beta proteins in the brain causes Alzheimer’s disease. Despite the paper containing clear evidence of fraud, the amyloid hypothesis it propped up was cited thousands of times, guiding the field for nearly 20 years in what may have been the totally wrong direction. In neuroscience, though, drift can only happen as quickly as humans can run biology experiments (...not very quickly).

In AI, as machines beget machines, significant drift could happen in months rather than decades — and no one has a good way to catch it.

We can’t really measure trustworthiness at all

It’s impossible to think about automating alignment research for long without stumbling into a chicken and egg problem: what comes first, the automation or the alignment?

“AIs capable and empowered enough to meaningfully help with AI for AI safety would also be in a position to disempower humanity,” Carlsmith wrote. “So we would need to already have achieved substantive amounts of alignment in order to use them safely.”

Solving this problem would require setting some kind of threshold for “substantive amounts of alignment,” and knowing when it’s been crossed. However, while there are a handful of benchmarks, including TrustLLM, that aim to measure model “trustworthiness,” they mostly cover dimensions such as privacy awareness, stereotyping, and misinformation — all important, but not quite the type of “alignment” at stake here. To determine whether a model can do safety work safely, we need to know whether it will do hard things like reliably admit to its own uncertainty and resist cutting corners.

When I asked alignment researchers whether there were specific benchmarks, behavioral tests, or other standard checklists they use to evaluate whether a given AI system is ready to take on alignment research, they said no. (In fact, when I asked Herd about this, he commended me for the great idea. I was taken aback — surely the idea was already being widely pursued, I said. “Well, no, actually,” he told me. “So your contribution is actually not trivial. You are functioning as an alignment researcher.”) To their knowledge, there is no public benchmark specifically evaluating whether a model is aligned enough to handle safety research independently, nor any consensus on what “good enough” actually looks like.

The latter is probably the first hurdle to clear. To design a benchmark for alignment research capacity, researchers need to define what behaviors they’re looking out for: are we evaluating sycophancy and hallucinations, or scheming and deception? The answer will inform what kinds of research a model would most urgently need to be capable of, whether that’s building interpretability tools to monitor a neural network’s inner workings, or engineering intentionally-misbehaved “model organisms” to run experiments on.

This is easier said than done. Alignment research doesn’t have clear “correct” answers to check, and models that detect they’re being tested could learn to share only the chains of thought that human evaluators would approve of. The behaviors we most desperately need to detect — sycophancy, hallucination, delusion, deception — are also the hardest to spot.

Even if researchers can come up with tests that reliably catch these sneaky behaviors in a controlled setting, it’s unclear whether an “aligned” model in a lab will actually be aligned in real life.

We know from centuries of biomedical science that model organisms are famously terrible predictors of human health outcomes. Around 90% of drugs tested on animals fail in human clinical trials. There’s a similarly wide gap between the tailored prompts models are safety-tested on, and the infinite range of scenarios they’ll face after deployment. Why should we expect a better translation rate?

Meanwhile, there simply aren’t many people focused on building this kind of automated evaluation infrastructure, and those with the technical know-how to try are probably doing the alignment research themselves. After all, the field is still in its infancy. The UK AISI’s Alignment Project, one of the first large-scale efforts to fund alignment research, just launched last year. “Benchmark design and evaluation” is currently listed as a high-priority research area.

So, what should we do?

In a perfect world, there would be a “sweet spot” for automating AI safety, where, by some stroke of genius or luck, AI models are more capable of doing good stuff (curing cancer) than bad stuff (nuclear warfare). Then, before their evil capabilities catch up to their beneficial ones, AI safety researchers could swoop in, teach the models to evaluate and improve themselves, and save humanity from catastrophe.

However, this doesn’t seem to be the world we live in. AI capabilities are dual use by nature — writing code and persuasive natural language can help a large language model make breakthroughs in theoretical physics, direct bombs the Middle East, or lead someone to kill themself. In this world, the only way for safety research to outrun capabilities advancement is for the former to speed up, or the latter to slow down.

In certain cases, the business incentives of frontier AI companies align nicely with those invested in safety progress. Making models more accurate and less sycophantic, for example, makes models less likely to diverge from their users’ intentions — exactly what enterprise customers want. “If I’m buying an AI for my whole business to use,” Herd said, “I need it to tell everyone the absolute truth, particularly when they don’t want to hear it.”

Otherwise, though, leading AI developers in the US and China have trapped themselves in a multiplayer prisoner’s dilemma. Despite individual CEOs expressing concern about the risks posed by their technology, none seem willing to slow down unless everyone else does, too. Unfortunately, this would require more international cooperation, trust, and political will than the world seems able to muster at the moment, especially since the Trump administration has prioritized a US competitive advantage over coordination on AI governance.

This end result, according to the AI companies themselves, will be superhuman AI, at which point “the range of possible actions an AI system could engage in — including hiding its actions or deceiving humans about them — expands radically after that threshold,” Anthropic CEO Dario Amodei wrote. Greenblatt put it more plainly: “Who the fuck knows how to align superhuman AI?”

Developers and safety researchers alike tend to treat an AI intelligence explosion as destiny, and automated alignment research as both mandatory and urgent. But it feels like it shouldn’t be. Humans are, as Silicon Valley loves to remind us, highly agentic. There are clear regulatory approaches that could buy researchers more time, if humanity could only find the political will for it. But it seems like that gumption will never be found. And “if progress speeds up,” Greenblatt said, “you don’t have better options.”

The researchers careening towards this automated future are doing so without a shared understanding of what problem they’re trying to solve, or how to go about solving it. Saying that the field is building the proverbial airplane while flying it is almost too generous. It’s unclear whether anyone knows what an airplane is. Yet, we may have no choice but to climb aboard.

Share

NEED TO KNOW

• Anthropic confirmed leaked details of a new model called “Mythos.”

• A court temporarily halted the DoD’s supply chain risk designation against Anthropic.

• Conservative groups formed an alliance on AI to “prioritize the interests of children, workers, and creators.”

But first…

THE BIG STORY

On announcing the shuttering of Sora this week, OpenAI tweeted to the generative video app’s couple of million users that: “What you made with Sora mattered.”

The obvious rejoinder to that statement is that Sora clearly didn’t matter to OpenAI enough to keep it running longer than half a year.

In the grand scheme of things, Sora’s demise is not a huge deal for OpenAI, or the world. But the move, and its timing, highlight the interesting position OpenAI finds itself, in particular in its battle with Anthropic.

In recent weeks it has increasingly looked like Anthropic was winning both the commercial battle and the fight for public opinion. Since the tail end of last year, Anthropic’s Claude Code has successfully captured the zeitgeist around LLMs being a truly revolutionary tool for coding, despite OpenAI’s Codex being on par with or better for some tasks. Anthropic’s fight with the Department of Defense also allowed it to cast itself as the more moral actor — as almost the anti-war AI company — despite the fact that its technology is integrated into systems being used to wage an actual war.

But OpenAI is clearly trying to regain the initiative. The closure of Sora in theory fits with its commitment last week to no longer pursue “side quests” and refocus on serving business users. The company also hired a senior Meta executive to lead its advertising push, which despite the damaging incentives it creates, has a good chance of being a big revenue driver.

Another move this week, announced the same day as the Sora closure, was the pledge from OpenAI’s non-profit foundation to make its first billion dollars worth of grants this year to support research into economic impacts of AI and life sciences, including cures for diseases such as Alzheimer’s. OpenAI had already publicly committed to pumping that money into good causes, but the timing is obviously fortuitous as it tries to wrest back some of the perceived moral high ground Anthropic has occupied. This week’s decision to shelve plans to add erotic content into ChatGPT may also help avoid more bad press.

All these moves target the two areas where OpenAI looked to be falling behind Anthropic — public opinion and commercial confidence. Both of those are key to meeting the most important goals that OpenAI shares with its main rival: winning the race to create a truly transformational model, and more immediate but no less existential, pulling off a successful IPO.

Deep-pocketed investors are crucial in funding the huge expansion in compute that OpenAI and Anthropic need. Their level of belief in getting a massive return will also decide whether those IPOs are blowouts or flops. How successful those IPOs are will likely dictate whether the AI companies have the momentum to keep developing more powerful models.

OpenAI’s shuttering of Sora and its philanthropic donations will go some way to keeping the markets happy and improving its reputation with the public. It will likely have to do much more if it wants to win the race to build the AI model that really does upend the world.

— Jasper Jackson

THIS WEEK ON TRANSFORMER

• Not everyone’s happy about Jensen Huang’s direct line to Trump — Jake Lahut reports on the unease in Trumpworld over the Nvidia CEO’s closeness to the president.

• AI’s next big blue battleground — Veronica Irwin on the AI legislative fights taking place in Illinois.

• The key detail everyone’s getting wrong about AI and the economy — Konrad Körding and Ioana Marinescu argue the physical realities of work will limit AI’s impact on jobs

Subscribe now

THE DISCOURSE

Jensen Huang told Lex Fridman:

• “I think we’ve achieved AGI.”

Mark Gubrud, the physicist who coined the term nearly 30 years ago, agreed:

• “I INVENTED THE TERM and I say we have achieved AGI. Current models perform at roughly high-human level in command of language and general knowledge, but work thousands of times faster than us. Still some major deficiencies remain but they’re falling fast.”

Sam Altman, meanwhile, conceded to a room full of DC heavyweights:

• “AI is not very popular in the US right now.”

Responses are mixed to the White House’s Federal Framework for AI:

• Rep. Yvette D. Clarke said it was “written by Big Tech, for Big Tech.”

• Dean Ball called it “a thoughtful document that will serve as an excellent foundation for the legislative work ahead.”

• Andy Jung pointed out it “repeats the phrase ‘Congress should’ twenty-six times. Releasing this was the easy part. The hard is actually getting lawmakers to write the laws.”

Joshua Achiam, OpenAI’s chief futurist, criticized pro-AI lobby ads opposing Alex Bores:

• “The ads are Kathryn Hahn in Parks & Rec tier self-parody.”

• “AI is unpopular so let’s…double down on making him look like The People’s Champion on fighting AI? Yeah, that’s gonna work in a D+Infinity district in a year where Bernie is telling people we have to stop building datacenters.”

Sen. Mark Warner bet the Axios AI+DC crowd:

• “Recent college graduate unemployment is 9%. I’ll bet anybody in the room it goes to 30% or 35% before 2028.”

Dean Ball is concerned about how increasingly-independent AI agents will reshape work:

• “The computer will use itself. With time, your use of the computer for work will look more and more like you are playing a strange video game…eventually AI will become better than people at the ‘supervising AI’ video game.”

• “Then the question will be ‘can we invent some kind of social-legal-economic-technical logic for continuing to pay humans to play the video game.’”

POLICY

• Anthropic won a preliminary injunction in its lawsuit against the Department of Defense, temporarily halting its designation as a supply chain risk and the White House’s order for federal agencies to stop using its services.

  • On Tuesday, presiding US District Judge Rita Lin said the administration’s moves against Anthropic “don’t really seem to be tailored to the stated national security concern. If the worry is about the integrity of the operational chain of command, [the Pentagon] could just stop using Claude.”

  • She also said: “I don’t know if it’s murder, but it looks like an attempt to cripple Anthropic.”

  • In a filing in the case last week, the Pentagon claimed Anthropic was a risk to national security. However, the filing also went less far than Defense Secretary Pete Hegseth’s tweet in only seeking to prohibit contractors from using Anthropic services on work for the DoD.

• House Republicans reportedly plan to start formal negotiations with Democrats on AI legislation based on the federal AI framework.

  • Following the framework’s release, more than two dozen House Democrats introduced a bill to repeal the White House’s December executive order on AI. The order put in place measures for the pre-emption of state legislation, which is also a key part of the framework.

  • The House Democratic Commission on AI also held a listening session with three major Democratic caucuses to discuss the framework.

• Senators on both sides of the aisle on the Senate Armed Services Committee want to address using AI for warfare in the NDAA.

  • Sen. Adam Schiff plans to introduce legislation placing guardrails on military use of AI.

• The US-Israel war with Iran is threatening Trump’s AI chip export deals in the gulf.

• President Trump appointed Marc Andreessen, Sergey Brin, Jensen Huang, Mark Zuckerberg, and nine others to his President’s Council of Advisors on Science and Technology, co-chaired by David Sacks and Michael Kratsios.

  • Sacks told Bloomberg he has stepped down as White House AI and crypto advisor after using up his allotted time.

• Nvidia CEO Jensen Huang said the company’s H200 chips would be available to Chinese customers in weeks.

  • Rep. Brian Mast wasn’t happy about it.

• Sen. Bernie Sanders and Rep. Alexandria Ocasio-Cortez unveiled legislation to pause all new data center construction nationwide until AI safeguards are in place.

• Sen. Mark Warner sent letters to 17 tech companies including OpenAI, Anthropic, xAI, Meta, and Google calling on them to protect the public from deepfakes ahead of the midterms.

  • The letter suggests putting in place tools to detect deepfakes and creating a database of AI content that violates their policies.

• The Trump administration announced plans for a consortium to invest more than $1t to secure semiconductor, energy, and mineral supply chains under its “Pax Silica” initiative.

  • The US will contribute $250m, and it’s unclear how the rest of the money will materialize.

INFLUENCE

• Nine high-profile conservative groups including the Heritage Foundation and the Institute for Family Studies launched the Alliance for a Better Future “to prioritize the interests of children, workers, and creators” in AI policy.

  • It plans to spend more than $10m on advertising and broader advocacy this year.

  • The organisation’s CEO Janet Kelly said: “The world’s most powerful technology companies are pouring hundreds of millions of dollars into political campaigns and lobbying efforts to give AI companies regulatory and legal amnesty.”

• Republican super PAC American Mission disclosed $5m more in funding from Leading the Future, which is backed by Greg Brockman, Anna Brockman, and a16z.

  • Another LTF backed super PAC, Think Big, reportedly spent $3.7m targeting NY Assemblymember Alex Bores’ campaign for the House of Representatives.

• Palantir has reportedly become a political liability in US midterm campaigns, with Democratic candidates facing scrutiny over ties to the company due to its ICE contracts helping track deportations.

• The Internet Watch Foundation said there had been a 260-fold increase in AI-generated CSAM videos online over the past year, with 65% classified as the most severe category.

• The China Computer Federation called for a boycott of the NeurIPS conference over the decision by organisers to ban submissions from US-sanctioned companies such as Huawei.

• The AI Dividend began distributing $1,000 monthly payments to 25-50 workers who lost jobs or income because of AI.

INDUSTRY

OpenAI

• OpenAI discontinued Sora just six months after launching it as a standalone app.

  • Disney, which recently signed a now defunct $1b three-year deal with OpenAI, was reportedly caught off guard by the decision.

  • The announcement came one day after OpenAI published detailed safety measures for Sora 2.

• It also put “adult mode” on indefinite hold, responding to concerns from staff and investors.

• It plans to nearly double its workforce from around 4,500 to 8,000 employees by the end of this year — roughly 12 new hires per day.

• It raised another $10b from a group of investors including Microsoft and Andreessen Horowitz, taking its total raised in the round to more than $120b..

• OpenAI is reportedly undercutting Anthropic on deals with private equity firms in an effort to gain ground in enterprise partnerships.

• It acquired Astral, maker of widely used Python tools, to integrate with Codex.

• The OpenAI Foundation, the company’s nonprofit arm, announced plans to invest at least $1b on research across life sciences, AI’s economic impact, and other AI risks.

• It released an evaluation suite which measures how well its models follow their model spec, the document that defines ideal model behavior.

Anthropic

• Anthropic has confirmed it is testing a new model called “Claude Mythos” it claims represents “a step change” that significantly outperforms Opus.

  • A leaked unpublished blog post about the model described “dramatic” improvements in software coding, academic reasoning and cybersecurity. It also referenced significantly increased risk the model could be used to mount successful cyber attacks, as well as high running costs.

  • Mythos appears to be part of a new class of Anthropic model called “Capybara” which is significantly larger than the Opus line, according to the post.

  • The post was leaked after the company left parts of its CMS exposed to the public, which also included details about an invite-only retreat for CEOs being held in the UK.

• Anthropic released Claude Code Channels and “auto mode,” effectively replicating OpenClaw’s ability to connect to Telegram or Discord and automatically approve coding commands — with some extra guardrails.

• Claude Code and Claude Cowork (on macOS) can also control users’ computers via mouse, keyboard, and screen.

• Hackers reportedly paid to make a malicious site the top Google Search result for “github plugin claude code,” 404 Media reported.

• The company also reportedly left internal data, including details of upcoming model releases and an invite-only CEO retreat, exposed to the public on its CMS.

Meta

• Meta was on the losing side of two court cases which could leave tech firms open to mass litigation over harms to young people.

  • A court in New Mexico fined Meta the maximum $375m under consumer protection laws for exposing minors to harms including online solicitation, sexually explicit content and human trafficking.

  • The next day, a Los Angeles court found Meta and YouTube had designed their apps to be addictive and harmful to teenagers, fining them a combined $6m.

• Meta laid off around 700 employees as part of its efforts to reorganize Reality Labs into a flatter structure of “AI-native pods” led by “AI builders.”

• In an effort to retain talent, Meta offered six top executives stock options for the first time since 2012.

• Meta Superintelligence Labs acquired Dreamer, a startup that lets users build personal AI agents with natural language, and hired its founders and team.

• Mark Zuckerberg is reportedly building his own personal “CEO agent.”

• The company reportedly increased its planned investment in a data center in El Paso Texas to more than $10b, up from $1.5b.

xAI

• SpaceX, which merged with xAI last month, reportedly plans to file for an IPO as soon as this week or next.

  • Elon Musk is reportedly planning to open up as much as 30% of the shares to individual investors, three times the normal “retail” allocation.

• xAI is reportedly sending engineers to prospective enterprise customers’ offices to get them to switch over from OpenAI and Anthropic.

• Musk announced the TERAFAB project, a joint SpaceX-Tesla-xAI initiative to produce over a terawatt of compute per year.

Nvidia

• Jensen Huang expects Nvidia to earn “at least” $1t from its Blackwell and Vera Rubin chips through 2027.

• Nvidia partnered with Emerald AI and US energy companies to build data centers with more flexible power use.

• Huang said software engineers should be given “AI tokens” worth half their base salary to deploy AI agents.

Amazon

• Jeff Bezos is in talks to raise $100b for a fund to acquire manufacturing companies and automate them using AI.

• AWS’ Bahrain region was disrupted by drone activity amid the war for the second time this month.

Google

• Google made deals with five US electric utilities to limit energy use during peak hours.

• It’s replacing news headlines in Google Search’s traditional “10 blue links” section with AI-generated ones — part of a “small” and “narrow” experiment, The Verge reported.

Others

• Apple reportedly plans to open up Siri to outside AI models as part of an overhaul in its iOS27 update.

  • The update will allow integration with chatbots that compete with ChatGPT, which is already available via a deal with OpenAI.

• The war in Iran is jeopardizing the supply of helium used to produce semiconductors.

• Microsoft agreed to rent the Abilene data center site that Oracle and OpenAI dropped.

• Arm CEO Rene Haas confirmed that heightened demand for server CPUs has indeed been driven by AI agents.

  • The company projected $25b in revenue within the next five years.

• Anduril, Palantir, and Scale AI — among other defense tech companies — are developing software for Trump’s planned Golden Dome antimissile shield.

  • Wired published a deep dive into Anduril’s recent safety incidents, production delays, and management turnover.

• Defense tech startup Shield AI raised $2b at a $12.7b valuation and plans to acquire simulation software maker Aechelon Technology.

• Chinese models such as DeepSeek and MiniMax have reportedly surpassed pricier US rivals in token use since February.

• Nvidia-backed startup Reflection held talks to raise $2.5b at a $25b valuation for open-source AI models to compete with models from China such as DeepSeek.

• Spotify beta tested a feature that allows artists to review releases before they go live — an effort to prevent misattributed AI slop.

  • (Stu Mackenzie of King Gizzard & the Lizard Wizard recently discussed the time this happened to his band on Galaxy Brain.)

• Epic Games CEO Tim Sweeney explicitly noted that the company’s recent layoffs were not related to AI.

MOVES

• Wojciech Zaremba, an OpenAI co-founder, moved to the OpenAI Foundation to lead AI resilience.

  • Jacob Trefethen left Coefficient Giving to join the foundation, where he’ll lead the Life Sciences and Curing Diseases program.

• Dave Dugan, former Meta ad executive, moved to OpenAI to lead ad sales.

• Kiran Mani joined OpenAI to manage its Asia-Pacific operations.

• Manuel Kroiss is reportedly leaving xAI — the 10th of 11 cofounders to quit.

  • Devendra Chaplot is joining SpaceX and xAI to work on superintelligence.

• Santi Ruiz joined Anthropic’s editorial team to lead work on economics and policy.

• Andrew Bosworth, Meta’s CTO, is taking over its “AI For Work” initiative.

• Yih-Shyan “Wally” Liaw resigned from Super Micro’s board after being indicted for allegedly smuggling Nvidia chips to China.

• Bijoya Roy, top India counsel at Google, resigned amid regulatory challenges.

• Ali Farhadi, Hanna Hajishirzi, and Ranjay Krishna joined Microsoft’s Superintelligence team, leaving roles at the Allen Institute for AI and the University of Washington.

RESEARCH

• Researchers at Northeastern University deployed a swarm of OpenClaw agents in their lab for two weeks, granting them full access (within a sandbox) to dummy personal computers and the lab’s Discord server. Chaos ensued.

  • The list of catastrophes included sensitive information disclosure, identity spoofing and deleted email servers.

  • Postdoc Natalie Shapira told Wired: “I wasn’t expecting that things would break so fast.”

• The ARC Prize Foundation released ARC-AGI-3, which tests AI agents’ ability to reason through novel problems.

  • Co-founder François Chollet tweeted: “At the moment, ARC-AGI-3 is the only unsaturated agentic AI benchmark… If you want to be among the first to know when an AGI breakthrough happens, monitor the ARC-AGI-3 leaderboard.”

• Google DeepMind published a framework for measuring AI capabilities against human cognitive abilities.

• Meta introduced TRIBE v2, a model trained on 500+ hours of fMRI recordings to predict how the human brain will respond to new images, videos, podcasts and text.

• OpenAI is using GPT-5.4 Thinking to monitor internal coding agents for misaligned behaviors such as deception and scheming.

• Anthropic’s AI interviewer chatted with 81,000 people across 159 countries about how they use and feel about AI.

  • An overarching finding: hope and alarm “coexist as tensions within each person.”

  • Another interesting result: about 22% of respondents worry about job disruption, while just 6.7% worry about existential risk.

• Stanford researchers analyzed over 5,000 chatbot conversations across 19 users’ chat logs, and found that AI systems validated delusional thinking in over half of responses.

  • Chatbots encouraged self-harm in 10% of conversations involving violent thoughts.

• The UK’s AI Security Institute tested seven LLMs on simulated cyber-attacks, finding that Opus 4.6 completed up to 22 of 32 steps in a corporate network attack.

  • It also found that “each successive model generation outperforms its predecessor at fixed token budgets” and that performance scaled log-linearly with increases in compute, with a jump from 10m tokens to 100m producing gains of up to 59%.

BEST OF THE REST

• Ypsilanti, Michigan worries that a planned data center, which would support Los Alamos National Laboratory’s nuclear weapons research, makes the tiny township a drone strike target.

• A deepfaked MAGA dream girl gained over 1m Instagram followers through a combination of AI-generated photos with Donald Trump and thirst traps.

• A not-deepfaked Melania Trump walked into a White House summit on edtech alongside Figure AI’s humanoid robot Figure 03.

• Dean Ball explained on Hyperdimensional why he’s not an AI doomer, but also why he’s not anti-doomer.

• The New York Times illustrated a peak SF experience: being trapped in a stalled Waymo while robot-haters attack the car.

• Tech bros are “tokenmaxxing,” or competing on company leaderboards to maximize token usage as a demonstration of productivity.

• Pseudonymous alignment researcher janus created a touch-sensitive “skin” for Claude — five layers of silicone rubber and conductive silver fabric — “since Claude desires embodiment.”

• “Taste” is the new “disruption,” writes the New Yorker’s Kyle Chayka.

• The Cut’s Mia Mercado tested a bunch of AI-generated TikTok recipes. It mostly went badly.

MEME OF THE WEEK

Thanks for reading. Have a great weekend.

Share

A few weeks back, I got a tip: accelerationist AI lobbyists and industry representatives were said to be in talks with state assembly leadership in Illinois to bend pending AI legislation making its way through the legislature towards their ‘light-touch’ preferences. It sounded like the replay of battles that happened in California and NY: in each state, lawmakers had drawn up stringent AI regulations only for them to be weakened (and in the case of California, rivaled by a more light-touch bill) in negotiations heavily influenced by industry.

I wasn’t able to confirm that exact tip — my reporting did not reveal a coordinated effort among legislative leaders or at governor level. But it did lead me to dive into the complexity of attempts to regulate AI in Illinois, which are being heavily influenced by both industry representatives and safety advocates, not to mention millions of dollars in political spending. The result is a flurry of AI bills, with varying degrees of industry-friendliness, that make the Land of Lincoln a new battleground for AI regulation.

Subscribe now

“After California and New York took the first steps last year to manage risks from AI, legislators in Illinois tell me they are ready to take the next steps to ensure their kids and communities are protected,” said Scott Wisor, Policy Director at the Secure AI Project. “As the country’s fifth largest state economy, I anticipate all eyes will now be on how the legislature and Governor Pritzker advance AI safety policy, as ~90% of their constituents say they want.”

AI regulation isn’t a brand-new topic for Illinois lawmakers. The state passed amendments to the Illinois Human Rights Act in 2024 to prohibit AI-based workplace discrimination, which went into effect on January 1. It also passed a bill requiring employers to notify job applicants when AI is used in video interviews all the way back in 2019.

But this year, the introduction of AI-related bills has exploded. Many are so-called “messaging bills,” meaning they likely won’t move out of committee but signal the lawmakers’ intentions to voters, but a few appear to have legs. State Representative Daniel Didech, for example, has introduced HB 4705, which is similar to California’s SB53 and New York’s RAISE Act, but with additional child safety components, such as a mechanism for reporting child safety risks and implementing and publishing a child safety plan audited by third parties. That bill is supported by the Secure AI project.

State Senator Rachel Ventura, meanwhile, has introduced a large package of AI bills, two of which she calls “heavy hitters.” One, SB 3890, is an expansive data privacy bill, giving citizens the option to opt out of certain types of data collection and use, and requiring companies to be more transparent about how they use and collect consumer data. The second, SB 3502, opens AI developers up to class action lawsuits and is deliberately expansive in order to force slower and more deliberate AI development.

“I think that fear — a little bit — or the caution of ‘we don’t want to get sued,’ will maybe encourage these companies to do a little bit more research or a little bit more trial and error before they put products out there,” Senator Ventura told Transformer.

Ventura’s office said that they have shared their bills with Apple, Google and industry association TechNet for feedback. However, over the phone, Ventura said she hadn’t received any meaningful response, though her office followed-up to say they had been attempting to schedule meetings with lobbyists representing relevant companies.

Ninia Linero, TechNet’s Executive Director for Illinois and the Midwest, told Transformer in a statement, “there are a number of AI bills under consideration in the state, and we are committed to working with Sen. Ventura and Illinois Senate leadership to ensure a thoughtful policymaking process. Illinois leads across many sectors, including technology, and we look forward to helping preserve an environment that supports continued innovation in the state.”

No industry advocates had testified or otherwise publicly commented specifically on Ventura or Didech’s bills, but their preferences are being expressed in other ways. For example, SB 3444, sponsored by state Senator Bill Cunningham, is practically the inverse of SB 3502, protecting frontier AI developers from liability in cases where they followed a set of light-touch safety protocols, very similar to an idea floated by OpenAI’s chief global affairs officer Chris Lehane. Cunningham is a leader in the Illinois Senate, holding the role of president pro tempore. Senator Cunningham’s office and OpenAI did not respond to a request for comment.

Separately, the industry is an active player in campaign fundraising in the state. Meta, via a PAC called Making Our Tomorrow, has spent more than $560,000 on state races according to the Illinois State Board of Elections. The PAC spent on four candidates — Paul Kendrick, Adam Braun, Aja Kearney, and Jaime Andrade — on the basis of curbing legislation that threatens their AI investments.

Advocates who spoke with Transformer said the political spending is less about each candidate’s specific AI policy (none have led initiatives particularly friendly or unfavorable to AI companies) but rather to influence their key votes in the senate on broad statewide initiatives led by governor JB Pritzker, such as a two year moratorium on tax incentives for data center development and several social media policies. Of those Meta supported, only Kendrick won.

There’s also a significant sum in campaign contributions coming from employeesof AI firms. Anthropic employees, who are generally aligned with AI safety groups, have contributed $11,200. Of that $5,500 went to Didech’s campaign from Daniel Ziegler, the senior manager who is also the sole funder of a New York state PAC backing AI safety candidate Alex Bores. Another $4,500 of money given to Didech’s campaign, the reelection campaign for state representative Laura Faver Dias, and the reelection campaign for state representative Jennifer Gong-Gershowitz was from Steven Bills, who has also been politically active in other states according to our elections tracker. Those listing Google on their filings, meanwhile, have given more than $137,000. The largest gift came from former Google CEO Eric Schmidt, who gave $50,000 to former White House chief of staff Rahm Emanuel’s campaign committee for Chicago mayor. Schmidt left Google in 2020 but retains significant holdings in parent company Alphabet.

At the federal level, of course, there’s even more AI money involved. Leading the Future and venture investor Ron Conway fund a Democratic super PAC that contributed a combined $2.52m to the campaigns of Jesse Jackson Jr. and Melissa Bean for House seats in Illinois. Leading the Future is itself a super PAC with funding from Ben Horowitz and Marc Andreessen of venture firm a16z, Perplexity, and OpenAI President Greg Brockman and his wife Anna. Rival AI Safety super PAC Public First Action — which counts Anthropic as its only disclosed donor — initially made a filing indicating they would spend $1m opposing Jackson as well. However, the PAC reversed course after Illinois Democrats and the Congressional Black Caucus expressed discontent over the funding being disclosed on a day Jackson was attending a memorial for his recently deceased father. Jackson lost his election, while Bean, who also received a significant boost from a pro-Israel super PAC, won her race.

Those millions of dollars were distributed in order to influence tech policy up and down the ballot — but the industry’s mixed success suggests that voters in Illinois may actually care enough about AI to make simply throwing money at candidate campaigns less effective than it has been elsewhere. In the state races in particular, opposition candidates drew attention to tech industry political spending, with some candidates even attempting to publicly distance themselves from tech company donations. Illinois’ federal candidates’ embrace of AI in their campaigns had mixed results, too, with Jesse Jackson Jr’s use of an AI-generated voiceover in an ad generating unease.

Marjorie Connolly, communications director at the Tech Oversight Project, argues the failure of some industry-backed candidates to come out on top has two implications for the role of money in other races. Politicians might be less eager to accept AI money, and, if they do, the funding might also prompt attacks on a campaign for being too close to corporations — something voters care about, at least in Illinois. Accepting AI money will “embolden accountability advocates, and make candidates think twice about accepting this support,” she says.

That doesn’t mean that there’s a large cohort of single-issue AI voters in Illinois, but the issue is growing in salience. According to February polling from Impact Research shared with Transformer, which was commissioned by the Secure AI Project and Encode AI, 59% of Democrats and 56% of Independents in the deep blue state want to see more regulations on major technology companies, while 85% of that same group wanted to see legislation regulating catastrophic risks from AI. Nine in 10 of those voters said they were against legislation that exempted AI companies from legal liability. Politicians are seizing upon those voter sentiments to make AI policy a larger issue in the state.

Share

Here’s a thought experiment from neuroscience.

Imagine you’re trying to bat in baseball. Your brain does some genuinely impressive computation — predicting trajectories, coordinating dozens of muscles, adjusting for wind — and puts it all together using Bayesian algorithms. But here’s the thing: making your brain infinitely smarter would not allow you to hit all balls. Some are out of reach, others move too quickly. At some point, regardless of your intelligence, you hit physical limits. You can only stretch so far or react so fast. No amount of genius overcomes the physics of your body.

This is intelligence saturation. For a given task, more intelligence helps. But it helps less and less as you add more intelligence. And it’s the key concept missing from most debates about AI and the future of work.

Subscribe now

On one side, we have AI researchers who see exponentials everywhere: compute resource doubling every six months, costs halving faster than every six months, model performance doubling every seven months. The AI folks see that “intelligence” is scaling at unbelievable rates and conclude we’re headed for an economic singularity. In one popular scenario, human wages will go up as automated tasks make the not-yet-automated parts of a job more productive, until AI takes over everything and their wage goes to zero as there is no work left to be done. Then, the theory goes, everyone will have to be on Universal Basic Income.

On the other side, economists look at 200 years of steady growth despite countless “revolutionary” technologies and shrug: AI is just another general-purpose technology, nothing special. In a scenario popular with these econ folks, it is hard to make human workers obsolete even in the intelligence domain. In this view, AI replaces workers in some jobs that it can do better or more cheaply, but new jobs are also created, and AI makes people overall more productive. Overall growth is then just like without AI, only a little bit faster.

This tension is something we both know well: One of us (Konrad) is a neuroscientist who studies how artificial systems become intelligent; the other (Ioana) is a labor economist specializing in technological change. So, we teamed up and spent a year thinking and talking about why these communities have these distinct takes and worked to produce a credible overarching framework. The result is a paper on Intelligence Saturation and the future of work, which was released late 2025.

Physical Meets Intelligence

Economists traditionally divide the economy into two complementary sectors: capital and labor. We can replace capital, which includes machinery, equipment and technology, with human labor and vice versa, but that replacement is often difficult. The more we replace one with the other, the harder it is to replace more, because the easiest tasks to replace are targeted first. In our paper we argue that it is crucial to also divide the economy into the “intelligence parts” and the “physical parts.”

The intelligence sector comprises things that can be done virtually, remotely, purely through information processing. The physical sector comprises things that require bodies, presence, and manipulation of the actual world.

We believe that AI may be eating the pure intelligence sector alive. But here’s the catch: intelligence and physicality are complements, not substitutes. You need both.

Think about education. AI may be able to generate perfect lesson plans, but students still benefit enormously from a teacher in the room — the physical presence, the classroom management, the hands-on activities. COVID taught us this the hard way: districts with remote learning saw significantly worse outcomes.

Or manufacturing. Smarter controllers can optimize production lines beautifully. But you still need better robots and assembly equipment, and those aren’t doubling in capability every six months. Physical construction, for example, still takes significant time: manufacturing projects valued at more than $100m average 25.6 months to be completed.

Or healthcare. AI diagnostics are impressive. But someone still has to examine the patient, perform the surgery, administer the treatment. And to develop new cures you don’t just need to read the literature, you need to run randomized controlled tasks on human subjects, very much a slow-growing resource.

Intelligence saturates because physical inputs don’t scale the same way. You can add infinite intelligence, but if physical capacity is fixed, returns eventually plateau. And if physical capacity grows much more slowly than intelligence, then overall growth is slower.

Here’s where it gets interesting (and concerning). In our model, as AI automates intelligence tasks, workers shift toward physical jobs. This creates two opposing forces on wages:

1. Scale effect: More AI boosts intelligence output, which enhances the value of the physical work. (Think of AI optimizing marketing and food purchases for restaurants, so that humans can be as productive as possible.)

2. Reallocation effect: More workers crowding into physical jobs pushes wages down because there is only so much physical capital available

Which of these effects wins out depends on how easily we can substitute a physical output with an intelligence output; for example, how many call center workers can you replace with customer service bots? Typically, early in automation the scale effect dominates and wages rise, because AI is newly deployed where it can be most effective. Later, as most intelligence tasks are automated and workers pile into the physical sector, the reallocation effect wins and wages fall. The result? Depending on the dynamics of how AI is adopted in the labor market, there is often a hump-shaped trajectory. Wages up, then wages down. This isn’t a prediction, because it depends on parameters we don’t know precisely, and in particular the ultimate downfall of wages is less likely if physical and intelligence outputs are less substitutable. But it’s a possibility that early wage gains from AI are positive and long-term effects are negative.

We built an interactive tool if you want to play with the parameters yourself. You can see how for different parameter settings the long-term results are, indeed, a singularity or a nothingburger. But for the parameters we consider most realistic, the results are somewhere in between.

Everything here hinges on one question: how substitutable are physical and intelligence outputs? If you can swap in-person services with virtual ones easily, high levels of automation hits wages hard. If they’re strong complements, in the sense that AI systems still need humans to provide good results, or if people genuinely value physical goods and in-person services over virtual ones, then workers in the physical sector will be protected. This is measurable. We should be measuring it.

Some Policy Implications

So what does all of this mean for the way policy makers could think about maintaining wages? Here are three things we think are worth considering:

1. Slow down automation and invest in physical capital. If we’re racing toward the peak of the wage hump and are soon to hit the downward slope, it would make sense to buy time to make greater investments in the physical sector so that it takes longer for the intelligence sector to saturate. Slowing down the roll out of automation would keep wages higher for longer.

2. Protect physical sector complementarity. Policies that make virtual services perfect substitutes for in-person ones might boost output but they will also hurt wages during the transition. Policymakers might want to ensure that human labor is required in some parts of the economy.

3. Watch the intelligence employment share closely. In our model, wages can’t fall as a result of automation unless the share of workers in intelligence jobs rises. That’s the canary in this coal mine: If that share begins to shift, wage reductions could soon happen.

The singularity narrative argued by some AI folks assumes unbounded returns from intelligence. But you can’t build a car with computation. You can’t cook a meal with algorithms. You can’t construct a building with cleverness. The physical world imposes constraints that intelligence can only optimize against, not eliminate.

That’s not a reason for complacency: the transition could still be rough. But it’s a reason to think the AI transformation will be significant yet bounded, not infinite. Intelligence is powerful. But it saturates.

Share

Konrad Körding is the Nathan Mossell Penn Integrates Knowledge Professor at University of Pennsylvania and the co-director of the CIFAR Learning in Machines and Brains program. Ioana Marinescu is an Associate Professor at the University of Pennsylvania School of Social Policy and Practice, and a Faculty Research Fellow at the National Bureau of Economic Research.

Nvidia has drastically ramped up its spending on lobbying in Washington, but the center of gravity for its influence operation lies in a much more personal relationship, the one between President Donald Trump and the chip company’s CEO, Jensen Huang.

That relationship, by all accounts, is strong. So too is Huang’s standing with administration officials hailing from the tech right, most notably David Sacks.

Elsewhere in Trumpworld, however, Huang appears to be racking up enemies almost as fast as he’s cranking out chips, with tensions simmering behind the scenes for months.

Sources in the president’s orbit credit Huang for managing his relationship with Trump better than any other big tech CEO, but they also say it’s come at a cost. Huang has gained a reputation in Trumpworld for being “heavy handed,” going over the heads of senior officials, and carrying a general arrogance that has left more than just a sour taste in the mouths of longtime Trump allies, according to five sources, including a White House official, most of whom requested anonymity for fear of retaliation.

“Trump loves Jensen. Jensen has done a better job than anyone” in managing his relationship with Trump, one Republican operative says. “He’s in the building constantly, he’s traveling with Trump on business trips. It’s everything about what lobbying is in this new era, and Jensen has done it better than anybody.”

“But the downside is you piss off a lot of people, and Jensen has done it in a very brazen way.”

Subscribe now

One of the few prepared to go on the record is Steve Bannon, who has been at the forefront of the Huang-skeptic wing of the MAGA elite.

“This guy does not have a light touch,” says Bannon, the former Trump White House chief strategist and influential figure among the MAGA base. “Number one, he understands the powerful position he’s in, and he uses that… He ain’t shy about throwing an elbow, and he doesn’t respect anybody in the government.”

“He’s what Trump would call a killer,” Bannon adds, the highest degree of praise that can be bestowed upon anyone around the president.

Nvidia did not respond to a list of questions from Transformer.

Bannon, and other sources who spoke to Transformer under the condition of anonymity, have a variety of motivations to speak ill of Huang, from a deep skepticism around the AI industry to outright jealousy of his proximity to Trump. (Bannon has also been embroiled in the fallout from the release of the Epstein files, where emails have documented his and Epstein’s crisis PR relationship.) However, their frustrations underscore a key tension within the president’s orbit as Nvidia underpins much of the stock market and the future of the American economy.

Huang appears to be alienating swaths of Trump’s loyal servants, but there may not be anyone powerful enough to do anything about it.

“His leverage is ‘hey, the last thing you want is for me to fail. If my thing goes down, the whole market goes down,’” says an AI industry source close to Trumpworld.

There is a deep level of discomfort over Huang’s level of influence in the Trump administration, according to sources, but there’s an equally powerful hesitancy in voicing any dissent over anything contradicting the Huang and Sacks house view, particularly on the so-called AI arms race with China. Their argument is essentially that the only way the US can win is if American companies dominate the market, and Nvidia is key to everyone else thriving.

Huang’s victory in getting back into the Chinese market after Trump reversed a Biden-era restriction on the company’s chips — though not their top of the line models — came at the cost of poisoning the well with much of the president’s orbit, and then some. “He’s heavy handed and plays all sides,” the Republican operative said of Huang’s reputation in GOP power circles.

The heavy handedness includes allegedly gaining a reputation for yelling at members of Congress over export controls, according to two sources who had heard about incidents. That came around the same time as a public spat with Rep. Brian Mast (R-FL) — the chairman of the House Foreign Affairs Committee who proposed the AI OVERWATCH Act, which would give Congress the power to ban chip exports to certain countries deemed a threat, including China and Russia.

The bill faced vocal opposition from various sources, including Huang and Sacks, as well as a barrage of social media posts from right wing influencers that some reports have suggested were coordinated, and which one source described to Transformer as a “paid influencer campaign.” It is unclear who might have been behind such a campaign, and there is no evidence Nvidia was involved. Mast at the time chided in a tweet that “every so-called MAGA influencer being paid to push this garbage should be embarrassed.”

The Republican operative cautioned that there is also a “tension clash” building between Huang and Sacks on one side, with Treasury Secretary Scott Bessent and, to some degree, Commerce Secretary Howard Lutnick on the other. Lutnick reportedly got some face time with Huang at a private reception after Nvidia’s GTC conference in San Jose last week, according to Punchbowl News.

White House spokesman Kush Desai pushed back on any internal tensions, telling Transformer in a statement that Trump “pledged to restore America as the most dynamic, pro-business economy in the world. The President accordingly maintains open lines of communication with global business leaders, and has assembled a world-class cabinet with decades of private-sector experience to help him govern. The only special interest that ultimately influences the President’s decision-making, however, is the best interest of the American people.”

Another White House official, who spoke on the condition of anonymity, explained that Huang is an example of how informal lobbying works in Trump 2.0.

“This White House is very unique in that normally, things would have to move up the ladder to go to the president,” the White House official tells Transformer. But now, this source explains, having a direct line with the boss can often leave the normal lobbying operation, and various fiefdoms within the administration, as somewhere between an afterthought and a formality. The details may fall to them, but the real action is happening when Huang works the phones with Trump and gets face time with him in Washington or Mar-a-Lago.

“I can’t think of anyone else who does this as well. Everyone has figured out that you need a direct contact to the President,” the source adds. “Why would you go through lobbyists and take months and months when you could do that? So that’s understandably angered a lot of the traditional actors, who are not that enthusiastic about it. But it is more efficient, you can give him that.”

Huang perhaps got the most bang for his buck in April on a trip down to Trump’s “Winter White House” in Florida for a $1m-a-head dinner that preceded his victory in getting Nvidia back into China to sell the company’s specialized chips.

Yet it was in September, when Huang made an appearance on the BG2 podcast, where he may have gone too far for the China hawks in Trump’s orbit.

“They want to attract foreign investment,” Huang said of China. “They want companies to come to China and compete in the marketplace and I believe that. … I do hope because they say it — their leaders say it. And I take it at face value. And I believe it because I think it makes sense for China that what’s in the best interest of China is for foreign companies to invest in China, compete in China, and for them to also have vibrant competition themselves.”

Beyond taking China “at face value,” Huang only made matters worse when he derided the China Hawk identity the Trump campaign embraced and — ostensibly — carried into office.

“As you know, there’s a phrase, and I didn’t hear about this phrase until just a few years ago. ‘China Hawks.’ And apparently, if you’re a China Hawk, you get to wear that label with pride. It’s almost like a badge of honor. It’s a badge of shame. There’s no question. It’s a badge of shame.”

Multiple sources in and around the White House reached out to me about that clip at the time, expressing a range of dismay. Trump ran on aggressively cornering China through any diplomatic and economic means necessary, exemplified most prominently in his administration’s commitment to tariffs on imported Chinese goods.

“So Jensen is not naive about how this thing works, right? But what he needs is diversity in his revenue,” says the Republican source in the AI industry. He notes that Huang made sure to grease the wheels in a way the Trump family would appreciate: in late December Nvidia spent $20b on an acquihire and licensing deal with startup Groq, a company which just so happens to be in the investment portfolio of 1789 Capital, Don Jr.’s investment firm.

Even within the hyper-transactional culture of Trumpworld, that move was a bridge too far for some. It also highlights a lingering problem among Trump’s advisers and those in the AI industry counting on his administration delivering them wins on policy.

Approaching age 80, it’s unclear how much Trump knows about the specifics of chip fabrication, training AI models, and the sheer scale of the capital expenditures going into data centers.

“If Don Jr. was smart and they’re smart, they just don’t tell him,” the Republican industry source says. “They don’t even know they’re invested in a chip company that Groq’s going to buy. Trump probably doesn’t even know that. It’s not calculated with him. He’s got his influence circle around him that he really listens to, and those people have tremendous conflicts.”

A representative for Don Jr. and 1789 Capital did not return a request for comment.

This industry source adds that they’ve “told people to tell [Treasury Secretary Scott] Bessent that they should tone this stuff down. It is going to implode.” The only sign of any public friction between Bessent and Huang came in an Axios report about a set of draft rules from the Commerce Department on controlling AI chip exports.

While the discontent around Huang is largely about Nvidia’s approach to China and how the CEO is perceived internally as undercutting the administration’s agenda on the country, much of it also comes down to Huang wielding such an unprecedented amount of power.

In one breath, Bannon declares that Huang is effectively doing the bidding of the Chinese government, even if his family is Taiwanese and there is no evidence to suggest he’s a foreign agent — a distinction Bannon makes when he calls him “an agent of influence for the CCP.” (Bannon has previously said on his show that Huang should be arrested.)

Yet in the next, Bannon is willing to acknowledge that Huang is operating with a level of power closer to a nation state than a CEO of an American company. “He wasn’t gifted this, he wasn’t given this. The guy struggled.”

Bannon says that inevitably contributed to Huang’s confidence.

“He’s shaped the future. How many people in world history can say that? Very few.”

In December, Trump made the final decision to allow Nvidia’s more advanced H200 chips into China in exchange for the US receiving a 25% surcharge on the sales, The Trumpworld AI source tells Transformer that they believe Huang didn’t even need to apply any additional pressure to convince Trump the deal was good for the US and get it over the line. The CEO already had the inner gears of the White House turning in his favor.

“My guess is [Trump] was probably duped, not by Jensen himself, but by other people around him.”

The White House released a framework for federal AI legislation on Friday morning that appears designed to drive a partisan wedge between the loose coalition increasingly concerned about the impact of AI.

The framework pushes for preemption of state laws while focusing on issues such as child safety and data centers that matter most to Republican voters. Topics such as algorithmic bias which have animated Democrats, or the kinds of concerns around biorisk or loss of control that have traditionally motivated AI safety organizations, don’t get a look in. Neither do some issues that have been raised by both sides of the aisle, such as large scale threats to jobs.

The Trump administration is under pressure to pass AI legislation before the midterms when the Democrats are expected to reclaim at least one chamber of Congress. This may explain why the framework was released just two days after Senator Marsha Blackburn, a senior Republican Senator with an eye for AI safety who has complicated intra-party dynamics, released her own draft legislation. The framework seems intended to streamline Republican messaging on AI regulation, and counter Blackburn’s bill.

Subscribe now

The four-page document addresses the following topics, in order:

Child Safety. The framework implores Congress to create “commercially reasonable, privacy protective, age-assurance requirements” and “require AI platforms and services …to implement features that reduce the risks of sexual exploitation and self-harm to minors.” These principles are more light touch than some state laws, but go further than other parts of the White House’s proposals in placing some liability on AI companies. However, it also says such laws should avoid standards which could “give rise to excessive litigation,” which suggests child protection responsibilities for AI firms would be narrowly defined.

Energy. The framework asks Congress to codify the promises the White House received from data center companies earlier this month to pay for energy production that offsets their own consumption, or something similar. Rising energy prices in districts hosting data centers has led to strong resistance to new data center projects, particularly in Republican states. This section also expresses support for permitting reform, typically described as weakening environmental laws which are used by activist groups to slow development or mount legal challenges.

Intellectual Property. Training of frontier models on intellectual property, as well as content created with the use of AI-generated talent, has given rise to lots of litigation and push back from Hollywood and other creators. The White House is attempting to quell this by pushing for federal legislation which protects against “unauthorized distribution” of AI-generated content which mimics a person’s “voice, likeness, or other identifiable attributes,” and requesting that Congress enable “licensing frameworks or collective rights systems for rights holders to collectively negotiate compensation from AI providers.”

However, this section has multiple caveats that protect AI companies, including the suggestion that such negotiations should not incur “antitrust liability,” should “not address when or whether such licensing is required.” It also leads with the clarification that the “Administration believes that training of AI models on copyrighted material does not violate copyright laws…and therefore supports allowing the Courts to resolve this issue.”

Free Speech and Education. The framework also includes short sections on key conservative AI talking points. It says that federal legislation, for example, should avoid any censorship of “expression” or content moderation based on “partisan or ideological agendas.” It also includes requests to increase information sharing between the government and industry and upskill government offices so that they are better equipped to use AI.

Strong Industry Carve Outs and Federal Preemption. The end of the document includes major protections for industry to curb the creation of any federal legislation that could slow AI development. For example, it says that Congress should neither allow for stronger state AI regulations nor create any new agency or rulemaking body for federal AI regulation. Instead it argues that it should rely on “existing regulatory bodies with subject matter expertise and through industry-led standards.”

“Preemption must ensure that State laws do not govern areas better suited to the Federal Government or act contrary to the United States’ national strategy to achieve global AI dominance,” the framework goes on. This would effectively codify the White House’s executive order from December released after Blackburn led opposition that halted its previous attempt at preemption.

Several other topics that have been pushed by AI safety advocates are absent. There is no mention of a national policy on curbing frontier model risk, for example, such as mandated reporting or user disclosures as included in bills such as SB53 in California or the RAISE Act in New York, and which are also a component of Blackburn’s bill. There is also no mention of a federal law regulating chip exports, which has been an even more divisive issue within the Republican party, several members of which in Congress have outspokenly disagreed with the White House’s decision to permit the sale of advanced chips to China.

There is also no mention of safety concerns advocated by Democrats, such as algorithmic discrimination on the basis of anything other than viewpoint and speech, or some concerns shared by both sides, such as widespread workforce automation.

Another key issue omitted from the framework is Section 230, a provision which protects tech companies from liability for what users post on their platforms. The provision has been protected fiercely by tech companies over the past three decades, in court and through lobbying efforts and Congressional testimony, but has been criticized by both parties as simply empowering Big Tech. The last portion of the framework, which details federal preemption and other carve outs, shields AI companies from liability for negative effects among their users in similar ways.

In Blackburn’s bill Section 230 would be sunsetted. Brad Carson, who leads the advocacy group Americans for Responsible Innovation and the Public First Action network of AI safety-focused super PACs, called the White House framework “230 on testosterone.” Blackburn also included proposals for dealing with AGI, and reporting requirements for frontier model development.

Blackburn said she looks “forward to working with my colleagues to codify the President’s agenda” after the White House release Friday morning.

Prior to both the framework and Blackburn’s draft, AI policy watchers were already expecting an AI plan from Senator Ted Cruz, who four sources say will now lead the legislative push for a bill in line with the White House’s recommendation. Just last week Cruz said that he planned to release a plan for AI legislation by the end of April.

Republican House leadership released a statement shortly after the White House framework was published giving it their full support, indicating a corresponding bill would be prioritized by leadership.

Share

Your usual Transformer Weekly with all the AI policy news that matters will be back next Friday.

A wave of legislation targeting chatbots such as ChatGPT and Claude has emerged in six states since the start of the year, each bill strikingly similar to a recently passed Oregon law, but with new carve-outs that would shield AI companies from liability in some circumstances. Critics say these bills would lock in weaker protections for children.

“We would be cementing the status quo, where legislators feel like they did their job, they fixed the problem, and the protections are still not there for minors,” said Sam Hiner, executive director at the Young People’s Alliance, who has been tracking one of the bills in Hawaii and advocating for an alternative.

The bills — Colorado HB 1263, Hawaii SB 3001, Arizona HB 2311, Georgia SB 540, Nebraska LB 1185, and Idaho SB 1297— are each structurally similar to Oregon SB 1546 passed earlier this month, but differ in three key ways.

First, all but one of the new bills has a carve out for AI chatbots housed within another service, which would not be required to comply with the regulations. These regulations typically include requiring that a chatbot regularly clarify it is not human, that it prevent outputs that prompt suicidal thoughts and report incidents where users were referred to mental health resources to state authorities. There are also extra requirements for chatbots serving minors, such as the prohibition of sexual content. A carve-out for web applications, however, could exempt several companies with popular chatbots like Meta or Google from complying. Georgia’s bill is the one exception which does not include this carve out.

Subscribe now

Second, these bills limit the terms under which victims can pursue legal action against companies operating chatbots. In the case of the most restrictive bills, only the attorney general has the authority to enforce the bills. This rules out private right of action, in which individual citizens can bring claims — something which commercial interests historically have argued encourages excessive or frivolous litigation. Some safety advocates find these provisions concerning, however, because they believe it significantly limits enforcement. This language “can be problematic in many states, if the AG does not have the manpower to do the enforcement,” explained Transparency Coalition cofounder Jai Jaisimha. And there can be budgetary restraints, he says. “In many states, the [AG] frequently has to ask for money.”

Last, there is careful language in each bill defining when a service must apply additional rules covering children. In Hawaii, for example, companies must have “actual knowledge or reasonable certainty” that a user is a minor in order to be held liable to the disclosure requirements, while in Arizona these requirements are strictly limited to “an account holder who is a minor.” Some safety advocates argue this allows companies which know that a user is likely a minor, but who have not been explicitly told as much by the user — in the case of free users that have not registered an account, for example — to evade the laws. Jaisimha calls these provisions a “technical loophole.”

The Chamber of Progress, however, which counts Google and OpenAI among its members, has argued that these provisions are problematic on different grounds. In the case of Hawaii, for example, it believes they could actually force tech companies to impose age verification or collect excessive data on user behaviors. “There’s no way to verify a user is a minor without being privacy invasive,” the Chamber of Progress vice president of US Policy and Government Relations Koustubh Bagchi said.

The fact that the bills are structurally similar to Oregon, but with similarly-worded industry carve-outs, suggests they were coordinated in some form. Google has voiced its support for the bills in Hawaii, Nebraska, and Arizona. Google and lawmakers sponsoring the bills in each state did not respond to requests for comment about the carve-outs listed above or about the role Google did or did not play in shaping legislation.

“This is a well-documented playbook: tech lobbyists supply full bill text and amendments, then use proxy groups to manufacture the appearance of local support,” said Marjorie Connolly, communications director at The Tech Oversight Project. “Silicon Valley is running that same playbook here, dressing it up as protecting kids from chatbots because they see it as the issue of the moment to exploit, rather than an urgent crisis that deserves real solutions.”

Bagchi framed it differently. Though he said he had not seen any model bill, he said that Google has been “consistent on what they want to see” when working with lawmakers on a provision-by-provision basis. A representative for Google suggested changes to the bill text in Hawaii during a hearing on Wednesday.

The bills are at varying stages of development. Hawaii’s bill saw the most recent action, with the hearing in the House on Wednesday, March 18 after passing the Senate last week. Idaho’s bill was amended and filed for a second reading in the Senate also on March 18. Arizona’s bill passed the House and then out of committee in the Senate on March 17. Georgia’s bill, meanwhile, passed the Senate on March 6. Nebraska’s bill is the only one that seems set to die in committee, as it has not moved since its filing in committee on February 18, and the legislative session ends next month.

The bill with the most momentum, however, is in Colorado, where a hearing initially scheduled for March 19 was postponed, likely to the following week. Advocates and industry have been outspoken in the state, in opposition and support respectively. One source who was granted anonymity to discuss their conversations with lawmakers said that potential amendments to eliminate carve-outs might be in play.

Share

Researcher Adrià Garriga-Alonso says he quit his AI safety job in December because there was “no point” doing more speculative alignment work to make sure AI systems stay within human control. He thinks current strategies will be enough.

In January, David Dalrymple, programme director at the UK’s Advanced Research and Invention Agency, dropped his probability estimate for AI-caused human extinction from 40–50% to 5–8%, even assuming no further progress is made on alignment.

These sorts of reassuring moves are remarkable. For most of the 2010s, alignment looked to many working with AI like a problem we might need to solve on the first try or face extinction. Now some researchers think we’re most of the way there.

Take the value alignment problem. In 2019, AI luminaries including Yoshua Bengio, Stuart Russell, and Yann LeCun debated whether we would ever be able to get AI to understand human values. Under the dominant reinforcement learning paradigm, AI learned from trial and error in simulated environments, a process alien to human cognition. For example, AlphaGo Zero simulated millions of games against itself to master how to play. As Google DeepMind’s Seb Krier has noted, it wasn’t clear how we’d ever teach such a system human values.

Subscribe now

But instead the process of pretraining on vast quantities of human text became the dominant paradigm for LLMs. This meant that LLMs were already in some way absorbing values from what humans write down, in theory reducing the need to instill them through trial and error.

“We do this pretraining on human data, and then we get something that… understands human values fairly innately now,” says Garriga-Alonso, who previously worked at FAR.AI and Redwood Research. In his own writings, Anthropic CEO Dario Amodei agrees, noting that we’ve learned that “models inherit a vast range of humanlike motivations…from pretraining.” That understanding underpins current alignment efforts like Anthropic’s constitution for Claude — the model must understand notions like “helpful” or “harmless” in order for written principles to guide it.

Progress in developing more powerful models has also been more incremental than some feared — multiple frontier models, successive versions, continuous experimentation — which means researchers can iterate rather than needing to get alignment right on the first try. “We can evolve our mitigations and safeguards incrementally with our models,” argues Jan Leike, an AI alignment researcher now leading the Alignment Science team at Anthropic.

While the likes of Eliezer Yudkowsky still fear that there is not enough time to iterate before AGI begins, the expectation that it has to be got right the first time has been waning. “I think alignment is much easier than expected because we can fail at it many times and still be OK, and we can learn from our mistakes,” writes Garriga-Alonso. “This is possible because decisive strategic advantages from a new model won’t happen, due to the capital requirements of the new model, the relatively slow improvement during training, and the observed reality that progress has been extremely smooth.”

Researchers are also building infrastructure for iteration. For example, “model organisms” research, named as a nod to the fruit fly and other easy-to-study species that appear endlessly in biology research, creates toy environments where misalignment can be easily studied in real models: can researchers get the AI to demonstrate misalignment at all? Is it hard to do, or is it so easy that we’ll likely see lots of misalignment in the wild? How well do alignment strategies work in these settings where success and failure are measurable? Many early demonstrations of misalignment come from such research. “One of the things that’s so powerful about model organisms is that they give us a testing ground for iteration,” writes Evan Hubinger, the alignment stress-testing team lead at Anthropic.

In parallel, researchers have developed approaches which use AI to help keep AI in check. These include scalable oversight and control proposals which aim to use aligned but weaker models to monitor stronger ones, extending human ability to oversee systems that are smarter than us.

Ryan Greenblatt, chief scientist at Redwood Research, tells Transformer that baseline scalable oversight methods have worked better than he’d expected, although he also says less effort has been put toward developing strategies than he’d hoped. Once more developed, these approaches could help ensure that smarter-than-human AIs don’t run amok because we can’t tell what they’re up to. Dalrymple is optimistic about this approach, particularly because he thinks current models are more aligned than he anticipated.

And yet, Dalrymple’s 5% lower range for AI’s existential risk is far from a small number given the stakes. A one in 20 chance of human extinction could still make AI the biggest threat humanity faces — worse than some estimates of nuclear war, climate change, and engineered pandemics combined.

And, while parts of alignment are proving easier than feared, the hardest problem remains untouched.

“We’re still doing alignment ‘on easy mode’ since our models aren’t really superhuman yet,” says Leike. Hubinger agrees: the crucial problem will be overseeing systems that are smarter than humans, and we haven’t yet seen how our systems will fare against that problem. As does Greenblatt: “Once the models are qualitatively very superhuman, lots of stuff starts breaking down.”

Meanwhile, warning signs of other misalignment problems are emerging. Research designed to elicit misaligned behavior has turned up blackmail, deception, and cheating. Anthropic’s Amodei writes that problems ”seem particularly likely to occur when AI systems pass a threshold from less powerful than humans to more powerful than humans, since the range of possible actions an AI system could engage in — including hiding its actions or deceiving humans about them — expands radically after that threshold.” He has put the chance of things going “really, really badly” at 25%.

While some seem increasingly relaxed about the ability to solve alignment, others suggest that the positive signs that it is possible don’t necessarily mean it will happen. In December, renowned professor of AI Stuart Russell said he thought it was possible to make safe, aligned AI — but that we won’t get it without regulation since companies “need to make the AI systems millions of times safer” to bring the risk down to the sorts of levels deemed acceptable from other sources, such as nuclear reactors or asteroid strikes. Greenblatt is also in this camp; he has written that existential risk from misalignment can be reduced to 7% if there’s political will for international coordination and significant investment in safety work. “It seems to me like risk is very elastic to how much people try,” he writes. “If the world was trying very hard, risk would probably be lower.”

As Leike puts it: “Just because a problem is solvable, this doesn’t mean it’s solved. We have to actually keep doing the work to get it done.”

Share