In Washington and Silicon Valley, a consensus has grown among safety-minded lawmakers and policy researchers, arguing that any attempt at AI oversight will require ongoing monitoring by third-party organizations not tied to either industry or the government. These bodies, often referred to as independent verification organizations (IVOs), would operate as auditors.
In the strongest policy proposals, such as one in Massachusetts, IVOs would be given advance access to models and the power to question AI firms and provide recommendations. The aim is to produce regular assessments of how AI companies are complying with the law and how much risk their models pose, rather than merely letting them set their own safety standards.
But there are major barriers to establishing a system capable of providing ongoing and consistent oversight. Chief among them: there are simply not enough people with the right skills to staff up IVOs to the level required for a strong regime like the one proposed in Massachusetts. Training people to do so may take more time than many realize, and those advocating for this third-party ecosystem are sounding the alarm.
“The issue is that the AI universe is a very small one,” says Kevin T. Frazier, a professor at the University of Texas School of Law, whose courses teach some of the skills required for IVO work. “There are very few people who are asking and capable of answering any of these technical questions and related internal governance questions.”
An appealing solution
It’s not hard to see why third-party auditors are seen as a key tool for reining in AI companies. After all, auditing isn’t new. Companies offering securities to the public must go through intensive ongoing audits, for example, overseen by the Securities and Exchange Commission and Public Company Accounting Oversight Board (PCAOB). Drugs are pre-approved by the FDA. The Department of Defense is setting up a regime for third-party certification of military contractors handling controlled, unclassified information.
Most people working on IVOs in the United States assume that the government is laden with too much bureaucracy to evaluate such fast-moving technology quickly enough, and that politics would get in the way of staffing and funding such a resource-intensive new program. Outsourcing demanding, time-pressured auditing work is the historical precedent: the PCAOB, for example, is a non-profit, while securities compliance auditing is done by private firms. Third-party assessors evaluating military contractors will also come from the private sector. On the flip side, the FDA is a government agency, but it is often criticized for approving drugs too slowly due to bureaucracy and funding challenges. Capacity constraints are especially problematic for states, which have smaller budgets to begin with, despite so far being the only place legislation mandating ongoing oversight has come into play.
The mechanisms for IVOs, however, have consistently evolved over the last few years. State proposals to impose third-party auditing date back to fights over failed California bill SB 1047 in 2024. The original draft of New York’s RAISE Act, introduced the following year, required companies to hire a third party to audit their compliance with the law, but this provision was dropped before Governor Kathy Hochul signed the bill into law. Illinois finally passed an auditing regime in July which measures companies against their own promises — a fairly light touch requirement, but one stronger than any measure which came before it.
There is also substantial enthusiasm on the federal level. A part of the FRONTIER Act, introduced by House Reps. Jay Obernolte and Lori Trahan, requires independent, third-party audits of the largest AI companies’ risk monitoring and mitigation practices. These would assess whether the company’s practices are good enough, not just their compliance with their own standards, in the case of the largest frontier developers. Though the bill is unlikely to pass this Congress, this section received praise from AI safety advocates and AI firms. Industry has also been somewhat receptive to the idea, with Anthropic CEO Dario Amodei calling for a third-party auditing regime in September.
Now, a section of Massachusetts’ Mass Wins Act, currently in conference in the state legislature, attempts to put in place the type of auditing regime safety advocates argue is necessary. That legislation would, like Illinois, require third-parties to audit AI companies’ compliance with their safety practices. But it goes further, too, requiring annual compliance audits and independent catastrophic risk evaluations of models roughly every four months. It’s the kind of strong IVO legislation AI safety advocates have longed for: Encode’s director of state policy Ben Snyder, for example, says it is “stronger than any AI safety regulation on the books today.”
That momentum could translate into another bill in the next Congress: Reps. Sara Jacobs and Don Beyer told Transformer at The Curve conference that they were working on legislation with an IVO regime. Bill text has not yet been released, but Jacobs explained that it would include a protocol for ramping-up IVO capacity. “The bill we’re working on has IVOs in it, and has a transition to where you will be able to have a real IVO ecosystem, and in the meantime, have the government do some of that,” Jacobs said.
Barriers and gaps
For all the momentum building behind an IVO regime, implementing ones as demanding as that proposed in Massachusetts — a rigor AI safety advocates increasingly say is necessary — will be no easy task. Ongoing auditing is “a really intensive ask,” says Nat Purser, director of US policy at AI Verification and Evaluation Research Institute (AVERI), a non-profit designing standards for audits. “I think it will not be an insurmountable challenge, but it’s definitely something [where] we need to make a strong, concerted effort.”
There is, of course, an existing network of organizations that already do similar work, such as METR, which is voluntarily brought in by AI companies to run tests and evaluations on their models. In October it was announced that six staff from METR would be given “employee-like” access by Anthropic to assess their models and safety protocols. Groups like Redwood Research, FAR.AI, RAND and SecureBio also do different forms of third-party evaluations, testing and red teaming.
But this ecosystem of research non-profits that specialize in assessing frontier AI risks may be too small to handle the scale of the challenge. OpenAI, for example, has said as much. Rapid expansion is necessary for IVOs to take on the workload imposed by regimes as strong as what’s proposed in Massachusetts. “High level we are concerned about capacity,” spokesperson Shana Mansbach told Transformer about the Massachusetts bill.
While it might be reasonable to question OpenAI’s motives for casting doubt on independent audits, the company has a point. METR’s own president Chris Painter has indicated that lack of capacity limits his organization on multiple occasions. “The central constraint to METR doing more of this [work], across the AI industry, is emphatically not having better access at frontier AI developers, it’s having more technical capacity,” he said on X in April. “The ambition of proposals we put forward … is primarily constrained by staff capacity,” he posted again on X in September.
Capacity constraints compound another issue: conflicts of interest. The small nature of the evaluation ecosystem means that almost everyone knows each other, and many have deeper interpersonal ties.
Ajeya Cotra, for example, a leading researcher who works on risk assessment at METR and co-led its investigation into the OpenAI-Hugging Face incident, drew criticism for taking on the work because she is married to Paul Christiano, who recently joined the OpenAI Foundation board and is a member of its Safety and Security Committee. (Christiano was not on the board at the time of the Hugging Face investigation). METR has rigorously defended its independence, pointing to disclosure policies and the fact that it does not receive payment for frontier company assessments.
But even if actual conflicts of interest are minimal, tackling the perception of them — something critical to ensure third-party audits are seen as credible and independent — will require an expansion of the field.
Growing pains
You might expect an industry so used to rapid scaling would have an easy path to building out IVO capacity that, relatively speaking, would be much less resource intensive than pretty much anything else it does. But researchers who spoke with Transformer say finding the right people to staff these organizations will require distinct strategies.
Patricia Paskov, director of standards at AVERI, says three categories of workers are needed. Organizations will need to hire people who assess the models and systems, those who evaluate computing hardware, and those who analyze the company’s governance practices.
Pipelines for the latter two buckets luckily already exist. Programs to train people in cybersecurity and IT are decades old, and governance auditing expertise can be found among people with a background in policy, accounting, or even business.
Recruiting compatible staff from other industries is mostly a matter of outreach and awareness. There are several fellowships and bootcamp programs, like MATS or GovAI, that provide the AI education for these types of people to pivot into the field of AI safety. “There’s a ton of talent from historical precedents and relevant domains that have existed for a very long time that we should be drawing from,” Paskov says.
But when it comes to the technical expertise to do model evaluations, there is a significant shortage. According to Anka Reuel, a computer scientist who helped write AI regulation best practices for the EU, that’s because the skill set is highly specialized work. She says technical staff should have both a deep understanding of how frontier AI models work — the kind of thing that requires serious research or frontier lab experience — and “domain knowledge,” of how evaluations are conducted, like how to create an effective test environment and how to decipher the correct metrics and markers. Often, potential employees have the technical knowledge but not the domain expertise, or vice versa. And of the select group which have both skillsets, the vast majority go to the frontier firms that are able to provide high compensation and plentiful resources.
Training people to work on model evaluations would alleviate the capacity crunch, explains Frazier, the University of Texas law professor, but adequate training doesn’t currently exist. Some programs to train people for the model level evaluations which require senior-level expertise are in development. Paskov has had discussions with the University of Wisconsin, Madison on building a certificate program, for example, while Reuel and Frazier teach related courses. The AI Evaluator Forum has a stated goal of helping “develop a pipeline of qualified candidates to be placed in evaluation organizations,” though Paskov, who is a senior adviser there, says it is still hashing out what exactly that might look like. And most programs are at least a year, if not more, from enrolling students.
In the meantime
That’s not to say that no independent auditing of frontier AI can be done today. Each of the sources who spoke with Transformer, including the representative for OpenAI, believes there is room for auditing in frontier AI regulation. “We do not think it would be a good thing to end up with an audit-less version of the [Massachusetts] bill,” Mansbach said.
Regulators need a stopgap if auditing resources run short, says Reuel, the computer scientist who advised the EU on best practices. In the Code of Practice she helped write, companies are able to avoid the auditing requirement if they can prove there is truly no available auditor sufficiently qualified to evaluate their processes — something they can prove by putting out a public call for assistance for 20 business days, for example.
However, Reuel points out that the passage of a law requiring third party audits in and of itself incentivizes the growth of the third-party ecosystem. Since the EU Code of Practice took effect, Reuel says the number of model evaluators has increased. Indeed, organizations like Purser’s employer AVERI and the AI Evaluator Forum have sprung up, while large firms like Accenture have made acquisitions to enter the space. Purser, who makes a similar argument, calls this “demand signal” — in other words, the government can signal that there will be demand for such organizations, triggering the supply of independent organizations to increase. There’s historical precedent for this: the Securities Acts of 1933 and 1934, which required audited financial statements, triggered the creation of a third-party financial auditing ecosystem.
But building the capacity for audits which are repeated or ongoing, and particularly those which involve the iterative process where auditors provide AI companies feedback as is seen in Massachusetts, isn’t a quick endeavor. Doing so requires time to ramp up, perhaps assisted in the meantime by the government as Rep. Jacobs described, or with an escape valve in the form of measures Reuel recommends.
It will also require significant funding, recruiting and outreach, activities Purser encouraged non-profits and even frontier companies to involve themselves in, say by sponsoring awareness campaigns or contributing money to upskilling and recruitment (something the AI industry, with its vast wealth, could easily fund). There are other ways to streamline auditing — companies could make their data more easily auditable and standardized, for example — and alleviate pressure, but ultimately extensive training infrastructure needs to be built.
“I think that this will take time,” says Purser. “This will be a years-long effort of creating the professional pipelines for these people.”
Governance and auditing comes together before anything gets built. This is where you build model harness through on top of the physical infrastructure and security.