Off the back of a report in The Information on Tuesday, everyone’s suddenly very worried about “neuralese”. OpenAI’s new Astra model, the outlet reported, was built with a new architecture which could make it harder to monitor its reasoning, raising safety concerns. Ryan Greenblatt, a prominent AI safety researcher, said that if true, the news “may be the single worst development for AI security/safety to date”.
But what does any of this mean — and is the latest development really as bad as some have made it out to be?
What are ‘chains of thought’ and why do they matter?
Let’s start with the very basics. The way AI models work is, at its core, very simple: your query goes in, the model computes over it, and it spits out an answer. Each step of computation, however, can only take a model so far. How far a model can think in a single step is what researchers call its “depth” — and even today’s frontier models can’t solve a complex problem in one go.
To do advanced “reasoning”, then, models write their thoughts down in a “scratchpad”, much like you might jot down your working when doing a hard math problem. They do the first step of the problem, write down the intermediate result, and then feed that note back in as input for the next step. In effect, the model’s writing a note to itself to help it carry a thought further than it could in one go. AI researchers call these notes the model’s “chain of thought”.
Chains of thought make a model much more capable. Most of the reasoning advances we’ve seen in the last two years are a result of this technique. But they also have a wonderful safety benefit, by making a model much easier to monitor. Developers can see and read the model’s “notes-to-self”, and while these notes aren’t a complete record of everything happening inside the model, they give researchers a useful window into its reasoning. That helps them catch AI attempts to scheme or deceive them.
This has happened in practice: last year, OpenAI said that monitoring models’ chains of thought helped it catch misaligned behavior while training a new model. And many have argued that if OpenAI had better monitoring of its models’ chains of thought during the Hugging Face hack, it could have stepped in and stopped the debacle much sooner.
The ability to monitor models’ reasoning process is considered by many AI safety researchers to be a godsend, leading many to put great emphasis on the importance of preserving our ability to monitor it. In a paper last year which included authors from OpenAI, Anthropic and Google DeepMind, researchers wrote that chain of thought offers a “unique” but “fragile” safety opportunity, and that developers should “study how it can be preserved.”
And what’s neuralese?
While many researchers advocate for protecting legible and monitorable chains of thought, some have long feared that it might not last. Competitive pressures, some think, might push companies to train their models in a different way — one that gives better performance at the expense of legible chains of thought.
Instead of writing down their thoughts in plain English, some fear AIs might increasingly do that reasoning “in their heads,” in a form humans cannot reliably understand or monitor. Researchers call this “neuralese”. Neuralese might make their models more capable — communicating in plain English is not particularly efficient — but it would also make it much harder for us to monitor AIs for bad behavior.
So what did OpenAI do?
According to a new report from The Information, with its new Astra model OpenAI may have taken a very small, but potentially significant, step towards this scary neuralese future.
An anonymous source “with knowledge of Astra’s development” told the publication that with the new model OpenAI is using a technique called “recurrent depth”, also known as a “looped transformer”. The details are technically complex, but in effect the technique means the model can do more of its thinking “in its head”, and so has less need to write notes to itself. The move would be commercially valuable — it means you can wring more performance out of a smaller, cheaper model — but would also mean that it’s harder to monitor the AI’s thoughts, and thus to catch bad behavior.
It’s no surprise, then, that the news sparked immediate panic among AI safety researchers. As the AI Futures Project’s Daniel Kokotajlo said: “Holy shit fuck”.
How bad is this, though?
OpenAI immediately tried to dispel the fear, and clarify that Astra is not a “neuralese” model.
Jakub Pachocki, the company’s chief scientist, said that the company “deeply care[s]” about chain-of-thought monitoring, and offered one number as evidence that Astra is still monitorable: the “depth” of its computation, he said, “is within a factor of two of GPT-4.” Depth is a measure of how much a model can think in a single step before it has to write something down. GPT-4, released in 2023, couldn’t do much in one step, which is why later models had to start jotting down notes. If each step of Astra’s computation is at most twice as deep, that suggests the model still has to write down much of its reasoning — a state of affairs that would hardly count as neuralese.
Pachocki didn’t deny that OpenAI is using recurrence in Astra, but his tweet amounts to an argument that it is doing so cautiously enough to preserve monitorability. (When asked for further detail, OpenAI directed Transformer back to Pachocki’s tweet.)
This was not enough, however, to completely assuage fears. For one thing, taking any steps that might weaken chain-of-thought monitorability is seen by some as “crossing the Rubicon,” a symbolic step that no company should take, in the interest of setting a norm that sacrificing monitorability should never be done for the sake of performance. It’s a notable step for OpenAI in particular, given how vocal the company has been about preserving chain-of-thought monitorability for “as long as possible.”
Part of the reason for that is to avoid taking the first step down a slippery slope. Yes, for now OpenAI may be using recurrence with extreme — admirable, even — moderation. But what’s to stop it from going further? As Apollo Research’s Marius Hobbhahn said, “the pressure is clearly towards more depth if it leads to more capabilities.”
Now that the company has this new architecture, it may be tempting to dial it up a little more, sacrificing monitorability ever-so-slightly to get a bit more performance (experimental results suggest that increasing the amount of “recurrent depth” leads to better performance). And then it might be tempting to do that again. And again.
All of a sudden, you’re a long way from where you started — even though each step of the way, you’ve taken an action that’s not particularly risky.
Better, some think, to never take the first step.
How do we solve this problem?
Opinions differ on what the best step forward is. In the wake of The Information’s report, multiple OpenAI employees called for a multi-company commitment to avoiding neuralese and preserving monitorability.
Others suggested that relying on chain-of-thought monitorability to persist is a pipe dream, and that we’re better off investing in other monitoring techniques, such as “mechanistic interpretability,” a practice often likened to looking inside a model’s “brain” to read its thoughts. (Still others said that it’s unlikely we’ll have good enough interpretability techniques in time.)
As with so many things, however, the answer may ultimately come down to regulation. OpenAI’s Dean Ball, while dismissing much of the discourse as “panic over [a] false claim that OpenAI is ‘doing neuralese,’” noted that the AI world was “adjudicating technically complex and nuanced claims on the [X] timeline with almost no ground-truth information about what is actually happening.” Such a sorry state of affairs, he argued, is evidence of the need for mandated independent audits, technical assessments, and independent verification.
We don’t know if OpenAI’s new model architecture is safe, and we really have no way of knowing. That, above all, is what must change.
The EU already has a paper route to this question that does not run through press statements. Annex XI, Section 1, point 1(d) of the AI Act requires providers of general-purpose models to document the architecture and number of parameters. For models with systemic risk, Section 2, point 3 requires a detailed description of the system architecture explaining how software components build or feed into each other, and point 2(b) of Section 1 requires the key design choices with their rationale. That documentation goes to the AI Office, not to readers, so a depth figure quoted by a chief scientist is not the same artefact.
It is way cheaper and safer to use programmatic verification and retry with fail than to do any of this. I built and use pathmap(.org) from my 4gb old cell phone for free. I used it and solved spermidine modified ginger EVs a potential delivery system for c9orf72/NEK-1 ALS and I never even took anatomy. (Also solved a potential way to use intradermal hydrogel using exosome-sized ginger EVs to locally deliver therapeutics for breast cancer either to treat or prevent metastasis and injected thermogel with borneal modified ginger EVs + Hyalauranic Acid modified ginger EVs as a likely preferred way to treat endometriosis and ovarian tumors). Since everyone is going bigger, I went decentralized (still building the platform but the core app proves the point).