Making CAISI the AI agency we need
The center has the right expertise, but lacks money, authority and influence
CAISI isn’t under threat. But it has been sidelined.
The Center for AI Standards and Innovation is the primary US government office tasked with overseeing frontier model development. Yet it was only peripherally involved in the administration’s decisions to impose export controls on Anthropic’s Mythos and Fable, or in delaying then approving OpenAI’s GPT 5.6.
The decision to block Mythos and Fable was technically taken by the Commerce Department, which contains the National Institute of Standards and Technology (NIST) within which CAISI sits. Yet the decision was actually the outcome of a war of influence among a cast of characters who have the ear of the president, including former AI czar David Sacks, Chief of Staff Susie Wiles, Treasury Secretary Scott Bessent, several business leaders and, it appears to a lesser extent, Commerce Secretary Howard Lutnick.
CAISI lead Chris Fall was reportedly briefed on the model’s technical capabilities, and his office seems to have tested the model itself days before it was taken offline. But those evaluations, ultimately, seem to have had little influence over what actually happened.
According to two sources familiar with CAISI’s current role, it continues to proceed with frontier model testing and research, as it has always done since it was created under the Biden administration. Though small, it is widely thought to have some of the strongest technical talent in Washington, including former OpenAI model alignment lead Paul Christiano, who is head of AI safety. The problem is, nobody in the federal government or at the frontier AI firms is legally obliged to listen to them. That means that even though the United States produces the world’s best AI models, its regime for vetting those systems has little influence over real outcomes.
The Trump administration has been able to sideline CAISI. In April, the White House stepped in to remove the highly respected former Anthropic researcher chosen to lead the CAISI, Collin Burns, just four days into his tenure. The move was attributed in part to the administration’s antipathy toward Anthropic. In May, the Department of Commerce removed a section of its website detailing agreements for CAISI model testing with xAI, Google and Microsoft without explanation. And in June, the administration stopped CAISI from publishing its model assessment reports.
Compare that to what’s happening across the pond. The UK’s AISI is its government’s primary office for overseeing frontier AI, and it has almost six times the funding of CAISI, with more than three times its staff. Technically CAISI and AISI have a memorandum of understanding to share research and model testing, but the UK institute appears to have been more adept at doing that work, routinely finding vulnerabilities other government agencies overlook. OpenAI’s model card for 5.6 and Anthropic’s system card for Mythos and Fable both list multiple vulnerabilities found by AISI, and none from CAISI.
More importantly, although AISI also lacks direct regulatory powers, it appears to have more influence across government, feeding into regulatory regimes and partnering with other government agencies such as the National Cyber Security Centre. Its personnel also have close government links: Chief Technology Officer Jade Leung also serves as the prime minister’s AI adviser and, as first reported by Transformer, last September a former chief AI officer at the UK’s communications intelligence agency GCHQ was appointed as AISI’s interim director.
“Given that these are our frontier labs, and it seems really important for us to know what’s going on with them, it would be good for us to be able to do stuff that UK AISI does,” said Charlie Bullock, senior research fellow at the Institute for Law and AI.
Simply copying the UK’s model may not be feasible or effective, not least due to differences in the way the country’s government works. Other solutions however may be on the cards, including federal legislation increasing CAISI’s budget and codifying its role, thus legally enforcing its importance and responsibilities. Grander schemes are also being discussed, including multiplying its budget several times over, breaking CAISI out of NIST, or rehousing it within another department altogether.
This year, CAISI has about $15m in funding ($10m from appropriations, and a $10m loan from the Technology Modernization Fund split across two years). There’s a proposal to go further than that: The Great American AI Act (GAAIA), a bill drafted by Reps. Jay Obernolte and Lori Trahan with a slew of different AI provisions. The bill isn’t yet introduced, and a discussion draft containing a preemption provision insufficiently narrow for most Democrats all but killed its chances of moving this Congress, but the representatives have attempted to pass it in the form of isolated provisions.
GAAIA as drafted would authorize $100m for CAISI. Lawmakers attempted to get something closer to this proposed additional funding authorized under the AI Security and Innovation Act, a separate bill that was marked up in the Science committee in the last week of June, according to two sources familiar with the discussions. However, amendments were stopped before their formal introduction due to a procedural rule, which keeps introduced bills from allocating more funding to agencies beyond a ceiling imposed by the House Committee on Appropriations. Currently that ceiling is at $20m, and the AI Security and Innovation Act’s text now increases CAISI’s funding to that ceiling. Were it to pass, CAISI “can do something similar to what they’re really doing but maybe a little bit better, and a little bit more,” says Bullock.
But merely increasing CAISI’s funding, even to AISI levels, might not give it the authority to do the job it is theoretically meant to do. The responsibilities it currently has, a long list including managing voluntary agreements with frontier AI firms to assess and provide feedback on their frontier models, testing and evaluating foreign AI models, and, theoretically, coordinating federal agencies on AI risks, are not currently codified.
When the White House first responded to Mythos, Politico reported that the officials weighing in on discussions were Cyber Director Sean Cairncross, Bessent, Lutnick, Under Secretary of Commerce for Industry and Security Jeffrey Kessler, White House Staff Secretary Will Scharf, White House Deputy Chief of Staff Richard Walters, and a presidential policy assistant, Walker Barrett. The following Monday, CAISI lead Fall held technical meetings with Anthropic and administration officials. Yet it was Bessent and Wiles who pressured the Commerce Department to work with Anthropic to reopen access to its models, while National Economic Council Deputy Director Ryan Baasch continued to push for more industry-friendly demands more in tune with Sacks. That jostling for control continues today as the White House pursues voluntary AI model standards, though CAISI is reportedly going to play a “crucial” role.
Codifying CAISI would at least help insulate it from political influence and make sure it can’t be easily swept away by a hostile administration. The AI Security and Innovation Act would do that, in addition to the small increase in funding. In effect, it allows CAISI to do what it’s currently doing, but with stability that might help it attract more talent, and a little more budget to improve its current functions.
Other legislative measures could go further in strengthening CAISI’s ability to oversee AI. One section of the GAAIA that could be picked up after the midterms would relocate CAISI outside of NIST, but still within the Commerce department, and give it the regulator-adjacent role of administering a regime of licensing and overseeing independent verification organizations, which would audit models. As written, GAAIA doesn’t force frontier companies to implement the recommendations their auditors give them, but would require that they at least have a licensed auditor in the first place.
More imminently, the AI Security and Innovation Act calls for a study into moving CAISI out of NIST, either to a different Commerce office or as its own standalone agency within the department, in addition to evaluating whether CAISI should have additional authorities and funding. The two sources who spoke with Transformer about talks within the House Science Committee said that proposal reflects further reaching discussions, in which some lawmakers have considered putting CAISI within a different department altogether, like the Department of Energy, War, or State. This is because Commerce’s primary mission, to “create the conditions for economic growth and opportunity for all communities,” could sometimes put it at odds with CAISI’s priority to ensure safety. The Department of Energy, meanwhile, has national security resources and a supercomputing agreement with AI companies, and State and War have subject matter expertise in international diplomacy and military use.
CAISI isn’t moving out of NIST any time soon, much less to a different department. But growing awareness of cybersecurity risks and the chaotic response to Mythos have spurred these preliminary conversations and interest in building a more consistent regulatory regime, says Bullock. Sometimes changes happen slowly, punctuated by short spurts of legislative activity — a sort of “punctuated equilibria,” he says, inspired by Caleb Watney, managing director of public policy at Coefficient Giving. (Coefficient Giving is Transformer’s primary funder.) Momentum is building, and it’s possible this is one of the final lulls before the US government springs into action.
“Maybe nothing will happen for a while,” says Bullock. “But then capabilities increase, then some scary thing happens, or somebody gets spooked, or some people get hurt, and then there might be a lot of interest in suddenly beefing up CAISI.”
I think this article doesn't distinguish between authorization and appropriation bills.
I have a tremendous amount of respect for the technical talent at CAISI and their publications/advisory role clearly demonstrates their technical talent. This was a great read and thank you for writing it.
I do, however, need to note that this pretty much demonstrates they do not have all the talent they need. If the representation of close ties with “government” is not US gov, I struggle to understand how industry relations and broader AI safety street cred translates to functional operation inside the executive. It needs a workforce or to develop capability for engaging with the broader government and self-advocacy in the federal environment.