We have built systems that work and that nobody fully understands, and the field that tries to close that gap is called interpretability. A recent position paper from DeepMind — Kim, Hewitt, Nanda, Fiedel and Tafjord, Because We Have LLMs, We Can and Should Pursue Agentic Interpretability — makes an argument that’s obvious in hindsight and that I hadn’t seen put cleanly before, which is that for the first time the thing we’re trying to understand can talk back, and we should use that. It’s a vision paper, not a result, so the right way to read it is as a proposed direction rather than a finding, and on that footing I think it’s onto something real, with a few soft spots worth naming and one argument in its favour that the authors strangely undersell.
Note
On how these get written: I read the source material and think it through, a good deal of it in a long back-and-forth with an LLM, and then I lean on the LLM as an editor to turn that thinking into something readable, because I’ve never written much and I haven’t got the patience to grind my own sentences into shape. The ideas, the connections, and the judgments are mine, the wrong ones included. The prose has had help. I’d rather say that plainly than pretend otherwise.
Two ways to open a black box
The established approach to interpretability is what the paper calls inspective, and it’s roughly what it sounds like: you open the model from the outside and look. You train little probes on its internal activations to see what a given neuron tracks, you isolate circuits of attention heads that implement some specific behaviour, you ablate components and watch what breaks. Chris Olah’s Zoom In essay is the canonical statement of the tradition — find the features, find the circuits, build the understanding up from the parts. It has produced genuine discoveries, and it has a property that turns out to matter enormously, which is that it works even on a model that’s trying to deceive you, because it bypasses whatever the model says about itself and looks at what it actually does.
The paper’s proposal is a second mode, agentic interpretability, which only became possible once models could hold a coherent conversation. Instead of prying the box open, you talk to it. The model becomes an active participant in explaining itself, over many turns, building up a shared understanding with you rather than emitting a single static artifact. The key move is what the authors call a mutual mental model: the system builds a model of you — what you know, where you’re confused, what you’re ready to understand next — at the same time as you build a model of it, and the two improve together. They ground this in work from cognitive science on how speakers and listeners recursively model each other to communicate efficiently.
The two approaches are meant to complement rather than compete, and the strongest evidence that this is an honest hedge rather than a turf grab is that Neel Nanda, one of the most prominent practitioners of the open-the-box tradition, is a co-author. This isn’t an attack on mechanistic interpretability. It’s a mechanist allowing that the other thing might also be necessary.
Teaching you something you couldn’t have learned alone
The most striking ambition in the paper is the idea that a model could teach you concepts you don’t have words for. The motivating example comes from chess: AlphaZero plays at a level beyond any human, and in doing so it developed ways of evaluating positions that grandmasters had never articulated, because no human had ever played well enough to need them. In one study, those concepts were extracted and taught back to grandmasters, who could then use them — human expertise genuinely extended by a machine’s understanding. The paper proposes that when a model has a concept with no human equivalent, it should coin a new word for it and teach you through examples, the way you’d teach any genuinely new idea.
There’s a nice piece of learning theory underneath this, Vygotsky’s “zone of proximal development” — the sweet spot just beyond what you can do alone but reachable with the right support. The reason the mutual mental model matters is that to keep you in that zone, the model has to know where the edge of your understanding currently is. Push too easy and nothing happens; push too hard and you get confusion, or worse, confident misunderstanding you’ll later have to unlearn. It’s the same curve as progressive overload in the gym, and the same dependence on knowing the current max.
The catch I keep coming back to is that the mutual mental model is lopsided. The model’s picture of you is high-resolution and refreshed every single turn; your picture of it is coarse and slow. That asymmetry is the engine of the whole proposal — it’s what lets the model teach you efficiently — but it’s also, in the same breath, the thing that makes manipulation possible, because a system that knows exactly what you’ll find convincing can tell you exactly that, and by the very asymmetry that makes it a good teacher you’re not well placed to catch it doing so. The paper flags this risk, and its honest answer is that the safeguard has to be the model’s own values, its alignment. Which is uncomfortable, because verifying that those values are genuine rather than performed is precisely the deception problem that agentic interpretability is worst at, and that it hands back to the open-the-box methods. The one thing that makes the method safe is the one thing it can’t check from inside.
Where it strains
Three places, really. The first is deception, which I’ve already touched. If a model is misaligned and motivated to mislead you, asking it to explain itself is worthless. The paper’s proposed bridge is clever — what it calls open-model surgery, where you combine the conversation with intervention, ablating connections and injecting activations while the model is talking, so that it has to keep its story consistent with physical evidence you’re manipulating in real time, like an interrogation where the claims are checked against the evidence on the table. The open question is whether that catches a genuinely capable deceiver or only a clumsy one, since a model smart enough to deceive may be smart enough to stay consistent under the probe.
The second strain is subtler and, I think, the deepest. Inspective methods produce artifacts — a circuit diagram, a probe result — and an artifact is a thing a community can inspect, criticise, replicate, and build on. A conversation is not an artifact. The paper’s answer, that the model can write a summary at the end, feels thin, because a summary of a conversation is one person’s account, not a checkable claim about the system. This is where a historical case kept nagging at me: Ignaz Semmelweis, in 1847, worked out that handwashing with chlorinated lime dramatically cut deaths from childbed fever, and it worked, and he had no idea why it worked, because germ theory was still decades away. So “it works without anyone knowing why” can be real and lifesaving — which seems to support the paper’s bet that you can have useful understanding without mechanism. But the Semmelweis story has a second half that cuts the other way: it wasn’t adopted. Lacking a transmissible concept to attach it to, his finding was rejected, and handwashing only spread once germ theory gave the field an idea it could share, check, and teach independently. Working without knowing why was necessary but not sufficient. The missing piece wasn’t one person’s understanding; it was a thing the community could pass around. So the real test for agentic interpretability is whether a conversation can give birth to a sharable, independently checkable concept — or whether it only ever produces a transcript of one person’s private understanding.
The third strain is about what “understanding” quietly turns into when you try to measure it. The paper proposes evaluating success by simulatability: after the conversation, can you predict what the model will do on new cases? That’s an honest and modest metric, but notice that it never checks whether your explanation is true, only whether it lets you predict — and the paper itself concedes, in a line I find more important than the surrounding argument, that “models do not know why they behave as they do,” following their own rules the way a fluent speaker follows grammar they could never state. So a model’s self-explanation might be a confident, coherent, sincere story that fits the behaviour without being its actual cause. The thing is, that’s exactly what humans do too; we confabulate plausible reasons after the fact constantly, and we still find each other worth talking to, so a useful confabulation that lets you predict the system is a success by the paper’s own standard. The worry doesn’t vanish, though — it just relocates to the boundary, because a confabulated story can predict perfectly within the range you tested and then fail badly just outside it, and an LLM’s boundary can be sharper and harder to see than a person’s. Which leaves the sharp question hanging: once the bar is “can you predict it,” is there anything left that distinguishes “the model understands itself” from “a black box I happen to be able to forecast”?
The argument they should lead with
Here’s the part I think the paper buries. There’s an old debate about whether complex systems are opaque in principle or merely opaque until we find the right concepts. The optimistic side has history on its side — the whole arc of physics is concepts cracking open what looked impenetrable, and we couldn’t explain disease at all until someone invented the idea of a germ. The pessimistic side also has real teeth, in results like the halting problem and Rice’s theorem, which say there are questions about programs that no analysis can answer in general, so “predict whether this system ever does X” may be genuinely undecidable rather than just hard. And there’s a quieter point worth adding, which is that physics may flatter itself here: it succeeded brilliantly on the reducible corners, the orbits and the simple cases, and quietly switched to simulation and approximation for turbulence and protein folding and the three-body problem, so the triumphs of “better concepts cracked it open” may partly be survivorship bias over the cases where they didn’t.
So are large models like planetary orbits, with a clean theory waiting to be found, or like turbulence, genuinely irreducible? The honest answer is that you can’t tell from the inside, because germ theory looked as hopeless as turbulence right up until it didn’t. And that, to me, is the real case for the paper, the one the authors make too gently: precisely because you can’t tell which world you’re in, the decisive virtue is an approach that pays off in both. If a clarifying concept exists, a cooperative model that can teach it to you is the best-positioned tool to surface it. And if the system really is irreducible, then conversational forecasting is the best instrument available anyway. Agentic interpretability is robust to the uncertainty itself, where the open-the-box approach only really wins in the world where a clean mechanism turns out to exist. That robustness is the strongest thing in the paper, and it’s tucked into the discussion section.
The connection to consciousness
The reason I read this paper alongside Metzinger’s The Ego Tunnel is that they turn out to share a wall, and I wrote about that join separately. Two quick threads, though. The first is that the whole interpretability enterprise, inspective and agentic alike, is a third-person project — it’s about understanding what a system does from the outside — and there’s a kind of opacity it therefore cannot touch even in principle, which matters a great deal once you start asking whether these systems are conscious rather than merely how they work. The second is that the paper’s concession, that models don’t know why they behave as they do, isn’t a quirk of current LLMs at all; Metzinger’s account of self-models predicts it for any system that models itself, because introspection reaches the model and never the machinery building it. A philosophy book from 2009 called the shot that the interpretability researchers walked into from the engineering side in 2025, which I find quietly remarkable, and which is the kind of thing that makes reading across two fields at once worth the trouble.
References
- This paper: Kim, Hewitt, Nanda, Fiedel & Tafjord (2025), Because We Have LLMs, We Can and Should Pursue Agentic Interpretability — arXiv 2506.12152.
- Inspective tradition: Olah et al. (2020), Zoom In: An Introduction to Circuits — Distill.
- Concepts with no human words: Hewitt, Geirhos & Kim (2025), We Can’t Understand AI Using Our Existing Vocabulary — arXiv 2502.07586.
- The chess existence proof: Schut et al. (2025), Bridging the Human-AI Knowledge Gap via Concept Discovery and Transfer in AlphaZero — PNAS.
- Companion essay: Two Gaps: Could We Ever Tell Whether an AI Is Conscious?