Recent proposals have argued that the best route to governing AI is to separate the technical and political parts of the problem. The idea is attractively simple yet catastrophically flawed.
Andrew Freedman, CEO and co-founder of the AI governance nonprofit Fathom, recently wrote about this idea in Transformer. He argued that independent verification organizations, working to standards defined by the government, could “insulate the facts” from the “coercive choices” that are a part of politics. But such insulation is fanciful. Vaccines, for example, are safe, but the facts have not protected them from politics. Freedman doesn’t propose any means of preventing AI testing from winding up in the same situation.
There is a certain elegance to the concept described by Fathom and Freedman, which is similar to one described in 2023 (and recently updated) by Gillian Hadfield of Johns Hopkins University and Jack Clark of Anthropic. Boiled down to its essence, the idea runs like this: (1) The government sets outcome-based safety standards; (2) The government licenses Independent Verification Organizations (IVOs); (3) AI developers opt-in to the process, which partially insulates them from certain forms of liability. If an AI developer opts-in, it must be certified by one of the IVOs licensed by the government. The Fathom proposal does not even include the Hadfield-Clark concept that an AI developer’s private regulator could impose meaningful penalties short of cutting off the regulator’s own revenue stream. If the IVOs fail to ensure the outcome-based standards the government set, they can lose the license they were granted.
You can see this as matching a long history of limited governmental interventions to produce a healthy market. One of the clearest parallels here is finance, where the government sets communication standards for financial reporting, companies pay accounting firms to produce reports on the companies, and other market actors use the reports to inform their own decision making about whether to buy a company’s stock, or lend to it, or enter into partnerships with it.
This does not always work as intended. Accounting firms that err sufficiently, like FTX’s accountants, can face insubstantial fines: in that case, of just $745,000. Theoretically, there should be substantive reputational costs — Arthur Andersen, the accountants for Enron, found that their customer base abandoned them — but governments have made deliberate policy decisions to not kill off large accounting firms over concerns about consolidation in the market. And if the accounting is bad, the total costs are shared by actors far beyond the company and the auditor, including lenders and stockholders who eat a great deal of the loss.
In fact, the approach can fail catastrophically. Perhaps the best example is how the Credit Rating Agencies (CRAs) directly contributed to the Great Recession. In the runup to the 2008 crash, banks issued loans, packaged them up in certain ways and then went to the CRAs. The banks paid the CRAs, which certified that the packages were AAA-rated, the safest a package of loans could be. If a CRA employee thought a particular package did not deserve an AAA rating, the bank would go to a more compliant competitor. The result was predictable in hindsight: ratings were not consistent with actual risks.
The CRAs were certified by the government, just like the IVOs that Fathom and Freedman ask for. But it would have made no sense for the CRAs to behave any differently, because the penalties for their poor behavior would not arrive for a decade or so, and never in such large a quantity that they threatened the survival of the firm. By contrast, not participating in the booming market would have meant that they had hemorrhaged market share.
In the worst case of the CRAs, banks were fully aware and taking advantage of the risk of what they were doing, and though they didn’t actively want to lose money they at least knew what the stakes were.
But of course developers want minimal oversight; that’s universal. Builders want minimal oversight from fire codes to save money. Crucially, buyers frequently want strong oversight: I don’t want the building I’ve purchased to burn down due to low standards. But some AI buyers might want IVOs with minimal compliance, seeing it as an attractive focus on customer features, not a red flag. The problem is this would in large part be because AI companies and buyers are not internalizing the full totality of potential risks of a technology that we still struggle to predict the impact of.
Part of the discussed solution for CRAs was mandatory randomization, but that destroys the competitive incentives that are supposed to make the Fathom proposal drive innovation, with each IVO striving to advance the science of measuring AI safety. And without a randomization mechanism in place, IVOs would end up incentivized to provide the minimum safety rules defined by the government while optimizing for speed and cost for their clients. Anything else is bad business.
If the main goal is safety, that will never be a sensible approach.
Keller Scholl got his Ph.D. at RAND, studying the economics of artificial intelligence. He is an independent policy analyst focused on making the AI transition go well, and writes at keller.substack.com.
Perhaps the preferable structure is not assessment on behalf of labs, but insurers. As you note the fundamental problem you're trying to solve is internalizing negative externalities (expected costs of harms imposed on third-parties). But you have a principal-agent problem. And also an information asymmetry.
The Insurance Institute for Highway Safety falls into the broad IVO category, but is funded entirely by the insurance industry and not carmakers. There's still colorable criticism lodged against IIHS, but auto insurers directly benefit from accuracy and do not systematically benefit from shading assessments one way or the other.
When IIHS was founded the adjunct insurance industry was already fairly mature. A meaningful barrier in AI is that cybersecurity and other insurers I don't believe have fully adjusted to the new risk gradients and moreover standardization is probably a while off. I assume there'd also be some friction in the degree of access to models without lab cooperation. But the point remains insurers have no reason to underestimate or underprice risk. And the policy is for insurance users, not so much for developers.
A FT article this week said US cyber coverage went down in 2024 and so did premiums(!). That's not "AI risk" per se but the insurers quoted say it's inarguable AI-enabled risks are growing. So maybe not a great example. And of course terrorism risk insurance required a government reinsurance program (TRIA) to prevent a complete collapse in that market.
Then I'll be looking forward to that piece! :)