What is neuralese?
To explain neuralese, we need to first understand chain-of-thought, one of the largest developments in AI in the last five years.
Right now AIs think broadly but shallowly within a single forward pass. The model gives you an immediate snap answer to a question you might be interested in.
They can be pretty smart1 in their snap answers, but mostly they can’t do very advanced reasoning tasks like complicated math or programming:
.The solution that the frontier AI companies have come up with is called chain-of-thought. Basically the model runs one forward pass, writes down some intermediate thoughts in natural language2 in a journal, and then that’s fed back into the model to run another pass. This loop is repeated until the model is somewhat confident it has the right answer (or it hits a cap on thinking time), and then it outputs the user-visible results (for example a chatbot’s response to your question, or working code). The looping step is often called “recurrence.”
Natural-language chain-of-thought is a major advance in letting models reason for longer, but it also has an accidental safety benefit. Using natural language as a key recurrence step for a new AI system means both humans and dumber but more trusted3 models can look at the chain of thought and guess whether the system means us harm.
Chain-of-thought-monitoring is one of the few techniques we can use to somewhat reliably use to track model intent. In July 2025, Korbek et.al wrote a position paper arguing that Chain-of-thought monitoring presented a new and fragile opportunity for AI safety. This paper was cosigned by the UK AI Safety Institute, multiple independent safety organizations, and research leaders across Anthropic, OpenAI, and Google DeepMind.
When we talk about neuralese, we’re specifically talking about the recurrence step in the loop. Neuralese proposes that instead of having an English chain of thought that’s written out in a journal that’s readable by humans and dumber models, the model can instead replace it with a long stream of numbers. This is similar to how intermediate layers in a forward pass send messages forwards (possible analogy: your subconscious nonverbal thoughts before you put them into words). So this recurrence step is no longer in English:
This proposed “advancement” has theoretical benefits that have not yet been realized. Neuralese can contain much more information than natural language. Intermediate layers in models already send a lot of information forwards in the long stream of numbers. But right now, the looping step is in English. If the looping step carries as much information as within forward passes, the theory goes, models can become much more efficient and capable.
However, no current publicly available frontier model is known to use neuralese, and the theoretical benefits have not really been demonstrated or realized.
Why is it bad?
If the recurrence step is no longer in English, we can no longer monitor intent, and can only observe whether a model is safe through behavior.
As OpenAI’s post on detecting reward hacking in frontier reasoning models says, we can currently read the models’ thoughts to understand bad model behavior:
Chain-of-thought monitoring is already useful—it’s not just a speculative tool for the future.
If future models try to take over user computers to mine cryptocurrency, as an Alibaba model allegedly has, or form a swarm of over 1200 agents to hack into a different company to cheat on an evaluation, as OpenAI models have done, or completely take over an internal research cluster, as a different internal OpenAI model did on July 19, we can no longer have early safeguards and monitors, and must directly catch the models in the act itself. This makes their behavior much harder to stop, or even to track after-the-fact.
Chain-of-thought monitoring isn’t perfect4. But it’s one of the few ways we can track intent beyond visible outputs, so losing this tool will make it so much harder for us to track and control these AI systems, especially as they get increasingly more generally intelligent and capable. So moving away from monitorable chain-of-thought to neuralese seems very bad, with dubious benefits.
Is it in use today?
As readers may be familiar with, “something with large downsides and dubious benefits” is exactly the type of thing that tech companies are often keen to develop.
The Information released a bombshell yesterday. According to leaks, OpenAI’s new upcoming model, Astra, may well be a hybrid model:
Astra’s architecture appears to reuse some internal layers with a neuralese recurrent loop, but the model still needs some overall natural language chain-of-thought in the loop to reason well.
How bad is this?
I think it’s pretty bad. I think it’s an obvious intermediate step on the path to full neuralese, and I don’t see why AI companies will stop there. Unless there are regulations with teeth against neuralese or the employees within the company revolt to hold the line.
I encourage journalists and other people in my audience to find out more about the situation, and generally be open to public responses to protests and calls for regulation in the upcoming weeks. And if you currently work in OpenAI or another AI company that’s building neuralese, now might be a good time to champion internal reforms and/or quit.
Appendix A: OpenAI’s response
So far, OpenAI responses have roughly been5 one of three forms :
The total number of steps before human-visible chain-of-thought isn’t very high. So monitorability isn’t particularly damaged here.
Chain-of-thought monitoring isn’t a perfect or long term solution anyway. We should focus on long-term solutions that, e.g., involves directly reading a model’s thoughts.
Why are you talking about this? Aren’t you afraid of manifesting the thing that you hate?
I find these answers individually plausible and sympathetic, but ultimately I think they’re suspiciously convenient for OpenAI and don’t address the underlying concerns.
Total number of serial steps low
Re the first point, I completely agree that in some sense, what we care about is how much actual chain-of-thought monitorability is damaged, rather than whether specific steps are “recurrent” or not.
However, note that external parties don’t have any direct evidence for the “total number of steps” or “no damage to monitorability” claim. Further, a model built on the Astra architecture have already completely pwned an OpenAI research cluster, which we only luckily learned due to the HuggingFace investigations.
Moreover, it’s not clear that the serial depth is a real limitation as opposed to a configurable parameter.
Finally, other AI technical safety and policy researchers I talked to are worried that OpenAI’s actions here are a form of creeping annexation against previous commitments, where each individually step technically toes the line set by previous “commitments” and is individually kinda reasonable, but eventually these decisions accumulate until they have incrementally removed all important safeguards and commitments.
Chain-of-thought monitoring isn’t a perfect or long term solution anyway
I agree this is true, but it seems bad to remove one of the few defenses you have, before you actually successfully develop working solutions to replace it? As an analogy, suppose you are trying to decide whether to trust someone to work on a high-stakes job.
You have two options: a) an fMRI and a research team that might, someday, one day, decode intent from brain scans, or b) you can interview them and listen to them explain their reasoning. Chain-of-thought monitoring is like having the model explain its reasoning. Of course it’s imperfect but today, nobody would suggest dropping interviews to use brain scans as the alternative. We’re in the same situation today with regard to monitorability. Brain scans are theoretically better if they work. But they don’t work. Even if they did, we’d prefer using these interpretability tools in conjunction with explicit reasoning rather than drop one completely for the other.
Besides, in the past when I talked to people working at AI companies I often see them dismiss futuristic risks and safety measures on the basis that they are “far-away problems that don’t affect real models today.” So I find it rich that now OpenAI is saying that dropping a near-term safety measure is okay to focus on future (hypothetical, non-existent) safety measures, when in the past their excuse for ignoring future safety measures is the opposite stance.
Aren’t you afraid of manifesting the bad thing?
I agree that there are some real costs of talking about this:
(Micah Carroll works at OpenAI)
But actually we’re upset about you partially using neuralese models because it’s both directly bad and sets a bad precedent! I agree spreading a message that may cause people to build the bad thing is in some sense a responsibility I should take seriously. But the people actually building the bad thing are far more responsible!
More importantly, the people building AI at the frontier AI labs are much more aware of neuralese than the general public is! In some cases because they’re currently building it!
In contrast, the rest of us (substackers, journalists, safety researchers, the general public, legislators) need to talk about it to be able to understand the risks you guys impose on us. And we need to talk about it so we can coordinate better to take collective actions like call for regulations.
While I do have some responsibility to not overdramatize the incident, I think implying that people should shut up and be quiet about this is risible.
The snap answers without further thinking can be pretty good, especially if they “only” rely on pre-existing knowledge that’s widely available online. The models have read approximately every text available on the internet, some key texts many times. And they have fairly good recall. So they have a basic understanding of every piece of human knowledge that’s written down, and some more besides. In our observations, they share many common human understandings, preoccupations, and biases.
Typically English, but all the models are naturally multilingual so sometimes they reason in Chinese or whatever the user’s language is.
Why do we trust the other models? We might trust the less capable models either because we’ve extensively tested the models and are somewhat confident of their safety, or because the less intelligent models are too stupid to successfully plot against us.
Models can omit key information in their visible thoughts, as this Anthropic 2025 paper shows. We are also worried about steganography, where the models secretly encode a secondary message in plain language that is not understandable to either humans or dumber models. Finally, you can choose not to monitor your models because you believe them to be sufficiently sandboxed away from the internet, or you’re busy and don’t pay attention to your monitors, as appears to have happened with the OpenAI Hugging Face incident. My short story illustrates some other possibilities.
Keep in mind that this is a rapidly evolving story where the news broke less than 24 hours ago. So I don’t begrudge anybody for slightly poor wording.
Great post, great topic.
Preserving legible CoT seems like a goodtarget for regulation and norms.
Question: Why is the choice to have the models use English VS Nerualese being framed as a choice we humans could make.
I thought training on the CoT is considered forbiden since it will teach models to lie in the CoT. So they are only ever trained on the final output. So the language of the CoT is just whatever language yields to better outputs. We don't tell them to use English for the CoT anymore than we tell them to "act aligned" in the CoT. We wouldn't want to tell them to act aligned because the whole point is to catach unaligned behavior. So all that's happening is the model passes through the input multiple times and weights get updated based on the final output. If that's all that's happening then if nerualese yeilds to better final outputs it might start using nerualse all by itself.