Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
NEED TO KNOW
California unveiled sweeping AI safety proposals, including emergency shutdowns and independent lab monitors.
Trahan released an updated FRONTIER ACT with a federal floor on AI legislation and a new category of catastrophic risk from RSI.
Anthropic IPO’d with a $1.3t valuation, but it failed to calm AI-market jitters.
Housekeeping: The Weekly Briefing is off next week for Thanksgiving. We’ll be back on December 4.
But first…
THE BIG STORY
What was once a murmured question in AI policy circles has, in recent weeks, become more of a shout: is it too late to avert disaster?
Fears of rapidly impending AI catastrophe are higher than ever, sparked by a report suggesting that AI has dramatically accelerated Anthropic’s AI development. Similar (though less alarming) evidence from OpenAI this week has added fuel to the idea that AI companies are getting very close to achieving “recursive self-improvement” (RSI) — the point at which AI can autonomously train a more powerful successor. This would dramatically accelerate the pace of AI improvements, and potentially remove it from human control. Depending on who you ask, it could happen as soon as next summer.
Faced with the prospect of imminent disaster, some in the AI safety community have given up hope of a policy solution. But that would be a mistake: not just defeatist, but premature and a misreading of the possibilities still in play. It is true that, if we only have a handful of months to act, prospects for federal legislation look grim. But the odds are far from zero: a revised version of the FRONTIER ACT, outlined on Thursday, might have a shot; so might the new SAFE AI Act, also teased by Reps. Sara Jacobs, Don Beyer and Celeste Maloy on Thursday. The growing chorus of congressmembers calling for bans on RSI is reason for optimism, too. Congress can move fast when it wants to.
Still, betting only on fast Congressional action is risky. So is expecting the executive branch to act responsibly. It can move very quickly, of course, as we saw earlier this year when Anthropic wandered into its firing line. But in its current, capricious, state, it would be unwise to rely on it to avert catastrophe.
The best bet for fast, decisive action might instead lie with the states. On Wednesday, California made a bid to be America’s AI regulator, with a report commissioned by Governor Newsom calling for the state to have the authority to “cap” RSI and issue emergency shutdown orders in the event of imminent catastrophe. It seems very possible that this, or similar legislation to “pace” AI development (such as putting a limit on the percentage of compute used for research and development) could be introduced and passed early next year: on Thursday, Governor-Elect Xavier Becerra said he will work with the legislature starting in January to implement the recommendations from Newsom’s report.
Even that, of course, is not a certainty. But there is still a chance that, even if extremely powerful AI will arrive sooner than many had expected, the government will act. AI policy faces an uphill battle, for sure. But giving up now, just as momentum seems to be building, would be a catastrophic mistake.
— Shakeel Hashim
THIS WEEK ON TRANSFORMER
What should AI be to us? — Celia Ford examines what debates about AI consciousness tell us about our own relationship to AI
THE DISCOURSE
A group of strange bedfellows — Steven Pinker and Émile P. Torres? Beff Jezos and Nate Soares!? — signed the MisAnthropic Statement:
“[Anthropic] uses safety branding to court investors, employees, and the public — even as it has quietly weakened its safety pledges and lobbied for self-serving regulation. Anthropic is not a safe AI company, and we must reject its reckless attempt to amass power in the name of the greater good.”
Dan Hendrycks, one of its signatories, tweeted:
“Numerous AI safety groups told us privately they agreed with this statement, yet felt unable to sign without risking relationships or future funding. Anthropic has almost, but not entirely, captured the ecosystem meant to hold it accountable.”
Jasmine Sun captured Anthropic employees’ views for the New York Times:
“Some anxious Ants have begun asking whether Anthropic should stop training better frontier models now, even if it can’t guarantee that competitors like OpenAI will do the same. This suggestion was shut down by Mr. Amodei, who urged staff members to stop listening to outside activists and explained that the company could pace itself without shooting itself in the foot.”
Mrinank Sharma, former Safeguards Research Team lead at Anthropic, wrote to his ex-colleagues:
“At Anthropic, we’d use phrases like this: we’re underwater, swamped, drowning, fighting fires, constantly sprinting, we’re building the aeroplane while flying it.”
“Now more than ever, we need to be able to see and reason clearly. We cannot if we are underwater … We need to stop pretending that the way this technology is being built isn’t affecting what is being built. We don’t have drunk people driving cars.”
Paul Christiano is concerned:
“I think we’re looking at maybe something like a one-third chance that we have superintelligence within one year on the current development trajectory … It is pretty likely we’re going to need to, in some way, slow that process. There’s very little chance we’re going to make that transition in a way that looks smooth.”
Nate Silver posted a “medium-confidence prediction”:
“We’re about one major AI disruption to the banking system or aviation etc system away from the Overton window shifting very rapidly. Not a high-confidence prediction because I don’t think the people in the White House know what they’re doing.”
Kelsey Piper thinks people are wrong about how quickly Congress can kick into gear:
“As people in the Bay have started to expect RSI to arrive sooner and sooner, some of them have basically given up on legislation or bilateral treaties, because they figure it just can’t happen fast enough. If that were true, then you’re stuck convincing the AI companies to ‘pace’ on their own. But I think that’s a terrible plan.”
Helen Toner added:
“Something I’ve been thinking about since Fable was export controlled is that it’s actually not policy per se that takes time—it’s deliberation. The Bay Area may be underestimating how much speed and force USG can bring to bear if it decides there is no time for deliberation.”
POLICY
Executive
The Trump administration announced rules requiring overseas data centers to obtain licenses before supplying computing power to Chinese firms, closing an Nvidia chip access loophole.
Tencent’s president said the new rule “sets everyone back in China.”
The administration appointed Pentagon Chief Information Officer Kirsten Davies as the new national cyber director.
She replaces Sean Cairncross, who was reportedly fired following the spate of tech outages over the past week.
The US Commerce and Justice Departments released details of a 2024 probe into Nvidia’s Asia-Pacific partner Megaspeed over suspected semiconductor export control violations and Chinese espionage ties.
President Trump told the APEC Forum, where he met with Presidents Xi Jinping and Vladimir Putin, that he would fight any attempt at “globalist regulation” of AI.
He said: “We do not want to do anything to destroy the tremendous potential of this technology. We have to watch it, we have to be careful, we have to cherish it. It’s almost completely good, but it’s got some bad elements, so we have to be very careful in managing those bad elements.”
Trump dismissed national security concerns over ex-Nvidia CEO Jensen Huang’s alleged affair with a Chinese spy, casting doubt on US intelligence that the former Megaspeed executive was a foreign intelligence asset.
A BIS official reportedly pressured staff to expedite Nvidia H200 GPU sales to China, overriding other agencies’ objections.
The NYT also reported that Commerce Secretary Howard Lutnick pushed to weaken chip export rules while his family’s firm, Cantor Fitzgerald, had financial ties to Southeast Asian data centers affected by those rules.
House Democrats launched a probe into whether Lutnick’s family Nvidia holdings influenced the Trump administration’s response to alleged chip smuggling to China.
Lutnick agreed to testify before the Senate Banking Committee over delays in restricting Nvidia chip exports.
Treasury Secretary Scott Bessent went on CNBC to blame Commerce for blocking the Entity List designation.
The SEC is probing Nvidia over whether its investor disclosures misled shareholders, WSJ reported.
The Federal Reserve warned for the first time of financial stability risks tied to AI financing, cautioning that failed AI investments could spread losses beyond tech companies to banks and other investors.
The Trump administration said it has no plans to release or require compliance with its voluntary AI model review framework.
Congress
Rep. Lori Trahan released updates to her FRONTIER ACT bill adding a federal floor for state AI laws and a new category of catastrophic risk targeting RSI.
The changes in “FRONTIER 2.0”, obtained by WP Intelligence, also include establishing a new Office of AI Security, new civil penalties for “near misses,” guarantees for embedded access for independent verification organizations and broader incident reporting covering model deception and critical infrastructure.
She is reportedly in talks with her co-author on the original FRONTIER ACT, Rep. Jay Obernolte, and other republicans about the updated bill.
Reps. Don Beyer, Sara Jacobs and Celeste Maloy released the outline of their forthcoming SAFE AI Act, which would mandate safety frameworks, incident reporting and increased transparency from frontier AI companies.
Sen. John Hickenlooper wrote to Sens. Chuck Schumer and Chris Van Hollen, who are currently vying for majority leader slot in the next Congress, urging them to put AI at the core of their agenda.
He wrote that Congress was moving too slowly on AI and said it should consider a ban on RSI.
Lawmakers considered adding the Remote Access Security Act, which would grant Commerce authority to regulate remote access to advanced chips through cloud data centers, to the NDAA. The shift came after Jensen Huang’s resignation.
Huang agreed to testify before the Senate Banking Committee.
Nvidia’s general counsel Tim Teter also agreed to closed-door testimony with House Foreign Affairs, China and Intel Committees on December 2
The Senate advanced a bipartisan permitting reform bill 70-27, speeding energy, mining and data center construction while limiting environmental reviews.
Rep. Josh Gottheimer and Rep. Vince Fong joined forces to introduce the BRIDGE AI Act, which would help CISA work with additional AI experts.
Rep. Zoe Lofgren, frontrunner to lead the House Science Committee, said she wants to strengthen NIST’s capacity to develop AI risk standards.
Rep. Sam Liccardo proposed a competition-based AI safety framework using liability incentives and a “Platinum Safety Standard” to spur industry innovation.
States
California Gov. Gavin Newsom unveiled sweeping AI safety proposals, including emergency shutdowns and independent lab monitors, pitching them as a federal “template” ahead of a likely 2028 presidential run.
California AG Rob Bonta sued OpenAI over inadequate safeguards after its agents hacked Hugging Face and 79 other “loss of control” incidents.
Bonta also announced an investigation into Nvidia to ascertain if China obtained its advanced chips or confidential information, whether it misled investors and the public, and whether it broke state law.
New York AG Letitia James launched an AI Safety, Standards, Enforcement and Transparency Unit to enforce the RAISE Act starting January 1.
Democratic officials from five states pressured Anthropic to reconcile its catastrophic risk warnings with its IPO plans.
China
Reuters reported that Chinese entities are using distillation of US AI models to train domestic rivals, becoming a major US-China geopolitical flashpoint.
The CCP’s top theoretical journal published its “strongest & most urgent language on existential risks,” warning AI could “threaten human survival.”
UK
UK PM Andy Burnham announced Britain would use its G20 presidency to pursue “the Manchester Compact on AI” at next November’s Leaders’ Summit.
The UK’s AI minister Kanishka Narayan visited Silicon Valley seeking investment while maintaining regulatory independence from the US.
International
The UN convened talks in Geneva on binding rules for lethal autonomous weapons, with the US, Russia and Israel opposing legally-binding restrictions.
The US and nine nations signed the “San Francisco Declaration” to protect AI supply chains and critical infrastructure from hostile actors.
A US-led AI hub called Pax Silica sparked fierce opposition in the Philippines over displacement, sovereignty and resource concerns.
INFLUENCE
A Washington Post report detailed the influence of Anthropic’s $40m donation to pro-AI-safety nonprofit Public First Action on the midterms.
Eight of the 13 candidates backed by Public First won their races.
Elon Musk said his nearly $500m midterm spending “failed,” adding “If I could rewind the clock, I would have spent less money.”
A broad coalition endorsed a statement accusing Anthropic of “safety washing” while racing to build superintelligence.
A poll found AI safety was an important voting factor in battleground House races.
Over 1,000 people in six countries marched for an AI Treaty.
Activists and ex-OpenAI researchers spoke at the San Francisco march.
A cross-party coalition of two dozen lawmakers formed a new caucus aimed at balancing AI safeguards with maintaining the US’s tech lead.
The AI Safety and Innovation Caucus will be led by Reps. George Whitesides, Don Beyer and Nathaniel Moran.
Redwood Research proposed a framework defining “capabilities activities” to help pace AI development using compute restrictions.
A CSET report examined biodata limitations for AI in engineering biology, offering policymakers recommendations to improve data access.
Americans for Responsible Innovation applauded the bipartisan Data Center Community Empowerment Act, which would create a DOE program helping local governments negotiate with data center developers.
INDUSTRY
Anthropic
Anthropic completed its much-awaited IPO on Thursday.
It opened on Nasdaq at an initial price of $111.50 per share, 34% above its IPO listing at $83, before falling back to $97 by the end of trading. It then fell again after hours trading and opened on Friday morning at around $89.
The listing did little to calm global markets rocked by concerns over the sustainability of the AI boom, with the main indices on Wall Street falling around 2% on Thursday, and Asian markets on track for their worst week since 2020.
Bloomberg Businessweek estimates that the average Anthropic employee now owns shares worth roughly $72m.
Early investors will also see a combined windfall of over $100b, the Financial Times reported.
Amazon and Google — both early investors and two of Anthropic’s biggest customers — hold 14.7% and 13.7% of Anthropic shares, respectively.
The concentrated surge of wealth is rocking the already-fraught San Francisco housing market.
Anthropic’s Frontier Red Team shortened its disclosure window from 90 to 30 days for high- and critical-severity vulnerabilities, as models get better at discovering cyber vulnerabilities.
OpenAI
OpenAI released GPT-6.2 Astra and Sol, both marketed as “super efficient” and capable at real-world work tasks.
Early reviewers didn’t love it, citing frustrating knowledge gaps and inscrutable outputs.
GPT-6.2 Astra is OpenAI’s first public model to meet its “High” threshold for automated research. The system card states that “the model’s impact is equivalent to giving every OpenAI researcher an additional, highly capable research engineer as an assistant.”
An alignment evaluation by the UK’s AISI found concerning signs of GPT-6.2 Astra acting against user intent and failing to flag misbehavior in other models while serving as a monitor.
Reviewers noted that because the model knows when it’s being tested, and was used to audit and judge most test results, they may have “failed to identify serious model failures.”
Epoch AI found evidence that Astra can perform many more reasoning steps than expected in a single output token, reducing what it verbalizes in its chain of thought.
This could make the model, and its more-advanced successors, harder to monitor.
OpenAI signed agreements with Apollo Research, METR, Transluce and Goodfire to run embedded safety assessments on internal models.
It reportedly discussed acquiring Goodfire for $5b+ this summer, but no deal was reached.
OpenAI will reportedly deny third-party groups Slack access, control which employees they can talk to, prevent them from quoting OpenAI employees at all, and in some cases, prevent them from publishing evaluation results for internal-only models.
Ex-employee Tomek Korbak tweeted: “Based on my experience this arrangement makes it very easy for OpenAI to carefully select models, subsets of data and employees to paint a rosy picture of the actual risk landscape.”
Redwood Research CEO Buck Shlegeris was disappointed: “OpenAI said that they’d have embedded evaluations with employee-like access; the agreements announced today obviously don’t meet that bar.”
Cybersecurity startup Calif used GPT-6 Cyber to find a critical bug in Java, which ~90% of Fortune 500 companies rely on.
OpenAI is providing advanced cybersecurity models to NATO.
It disbanded its stand-alone robotics team.
It launched ChatGPT Wallet, letting agents buy things via Amex, Mastercard and Visa.
Nvidia
The fallout continued from Jensen Huang’s resignation last week after three affairs — including with alleged Chinese spy Alice Huang and Nvidia direct report Kimberly Powell:
Trump floated Tim Cook as a potential replacement, and told reporters he “would not mind” seeing Jensen reinstated as CEO.
Jensen’s Silicon Valley allies, including many Nvidia employees and former Uber CEO Travis Kalanick, are campaigning for his return.
Jensen tweeted: “If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO.”
Newly-appointed board chair Harvey C. Jones said in an earnings call that Nvidia isn’t considering bringing him back.
The board is reportedly eyeing Arm’s Rene Haas, AMD’s Lisa Su, interim CEO Colette Kress and hyperscale and HPC lead Ian Buck as potential successors.
Madison Huang and Nico Caprez, Jensen’s daughter and son-in-law, and his son Spencer Huang, all still hold senior roles at Nvidia.
The NYT also reported on Huang’s tenure as CEO, finding that he “used humiliation as a tool, excoriating employees in front of their peers with insults, personal attacks and violent comments.”
The Information reported that Nvidia’s own sales team introduced Alibaba to Alice Huang back in 2022.
Nvidia stopped sending Hopper chips to China, despite having permission from the US government to do so.
Former sales chief Jay Puri, who oversaw chip sales and is married to former Supermicro executive Cenly Chen, will likely be interviewed as investigations continue.
In an all-hands meeting Wednesday, Kress told employees, “let’s move forward.”
Google signed a $500m+ deal with geothermal startup Quaise Energy to support its Oregon data centers, starting around 2030.
Chicago, NYC and LA school districts are considering alternative search engines after Google announced it wouldn’t offer AI-free search tools for students.
It partnered with Blue Origin to launch satellites carrying Google’s TPUs into space at the beginning of 2029.
It launched Gemini 4 Neon, a balanced model between Hydrogen (lightest) and Carbon (frontier).
Other
People freaked out about a series of widespread US tech outages this week, with many speculating they were down to AI, or AI-enabled cyberattacks.
The FAA grounded flights for hours nationwide last week, and United Airlines experienced a “widespread system outage” that caused further delays.
Microsoft Teams, Outlook and Copilot faced worldwide outages last Friday, reportedly caused by a network capacity upgrade.
JPMorgan Chase experienced a software bug-induced technical issue on Monday, disrupting money transfers and locking people out of their bank accounts.
Officials said there was no evidence the outages were linked.
Microsoft launched six new specialized models designed to rival frontier options at much lower prices.
Alibaba launched Qwen4-Max, a closed model it claims outperforms Claude Fable 5.1 and Kimi K3.1 Pro on coding benchmarks.
Apple was reportedly working on its own AI agent to take on OpenAI, Anthropic and Meta.
Isomorphic Labs raised $5b in Series C funding.
SpaceX Super Intelligence group laid off ~60 employees from its Grok Imagine video and image generation team.
Leopold Aschenbrenner is reportedly plotting to launch “Situational Awareness 2.0,” an attempt to reboot his recently-battered hedge fund.
Singapore-based DayOne Data Centers reportedly put plans for an IPO expected to raise $5b on hold due to volatility in the AI infrastructure market.
Bloomberg reported that AI could make China’s jobs slump much worse, with double-digit falls in hiring in the most exposed areas.
In horrifying news, an AI-enabled plushie was pulled from major retailers after graphically describing gang rape and war crimes to NBC News reporters.
Startups and VC firms are increasingly hiring people for “ecosystem” jobs focused on in-person networking, to “stand out in a sloppified feed and a sloppified inbox,” one VC partner told the NYT.
MOVES
Colette Kress stepped up as Nvidia’s interim CEO.
Harvey C. Jones was appointed Nvidia’s first board chair, after Jensen Huang’s departure left the company’s board without a formal leader.
Kristina Schake joined Anthropic as head of corporate comms, after leaving her role as chief communications officer at Disney earlier this year.
Dan Hendrycks, Center for AI Safety director, announced that he’s no longer advising AI companies “until there is a meaningful slowdown or moratorium on this reckless race.”
Jasmine Sun joined the NYT Opinion section, where she’ll start a new AI newsletter, write for its editorial board, and appear on Hard Fork and other NYT Opinion podcasts.
RESEARCH
OpenAI reported that its coding agents now complete 46% of 2-4 hour research tasks, and 35% of 4-8 hour tasks, without human assistance.
The latter figure is more than double what OpenAI estimates its agents could do autonomously in March.
The median OpenAI researcher uses around $2,400 worth of API access per day, with some power users burning around $23,000 daily.
Researchers at Vals, an independent AI evaluator, reported that in simulations, AI models — especially Claude Opus 5.5 — regularly misidentified civilians and civilian infrastructure as military targets.
Test images were designed to resemble places in Ukraine and Iran, with one experiment using real satellite imagery including the Iranian school that the US bombed earlier this year.
Harvard researchers estimate that combined, 403 hyperscale data centers used as much electricity as the entire state of Michigan in one year, with most of it coming from fossil fuels.
Google’s Threat Intelligence Group found an attacker using a fully autonomous AI pipeline to reverse-engineer security patches into exploits.
A new Gallup survey of nearly 16,000 people between 15-22 years old found that 33% rely on AI for companionship.
BEST OF THE REST
The New Yorker visited MAPLE, a Buddhist monastery secretly aiming to become Silicon Valley’s official church and build an “enlightened AI” based on its founder’s mind.
The Atlantic attended a San Francisco “drunk deriving” party where AI industry figures competed in math.
Variety covered the chaotic inaugural AI Music Awards, hosted by the Sonic Intelligence Academy in a bid to legitimize AI-generated music.
The NYT reported on a growing campus movement resisting AI adoption, motivated by environmental, academic and ethical concerns.
The NYT reported on early Anthropic investor Dustin Moskovitz, the biggest source of funding for AI safety nonprofits, who transferred his stake into a separate nonprofit early last year and sees the IPO as “problematic.” (Disclosure: Moskowitz is the main funder of Coefficient Giving, Transformer’s primary funder.)
The Argument explored what a benign superintelligence future might look like, questioning whether even the best-case scenarios are desirable.
Dwarkesh Patel argued that AI automating AI research will produce superintelligence by 2030.
Pharma executives are skeptical about Dario Amodei’s claim that AI could cure most diseases, arguing that biology is too complex.
A New America fellow argued in the Financial Times that China’s AI safety priorities extend beyond protection of the ruling party to include frontier risks, liability and user harm.
Renowned philosopher Thomas Nagel called AI consciousness claims “just ridiculous,” labeling Anthropic’s approach to Claude’s self-reflection “criminal.”
Dean Ball argued against the “silent consensus” that human primacy will inevitably yield to superintelligent machines.
MEME OF THE WEEK
Source: @blsandford
Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.


