What does an LLM understand? — Stein, Hart, and the wrong question

The debate over machine minds is conducted between two parties who share the same mistake. Physicalists and dualists alike treat consciousness as a property that a system either possesses or lacks; they disagree only about what kind of property and how it gets there. Stein gives us the tools to describe what an LLM actually does without prematurely deciding that question. Hart shows that the question itself is malformed. Together they permit something the current debate cannot manage: precision.

Parent discussions

• First AI-philosophy note in the vault; grew out of holding Classical Theism and the agentic-OS thread in the same week

Connected concepts

• Classical Theism — the New Atheists argued against a demiurge; the AI-consciousness debate argues about a property. Same error, one ontological register too low
• God as Being - The Finite Infinite Gap — the difference in question is categorical, not gradational; you cannot accumulate your way across it
• Apophatic Theology — saying everything you can and then admitting it is not enough is the right discipline for LLM description too
• Why embodiment is not a hardware problem — the same category confusion, worked out at the level of the body rather than the intellect

The point

Begin with the phenomenon. Stein’s Einfühlung (empathy, literally “feeling-into”) is not the colloquial “feeling what others feel.” It is her structural account of how another’s experience is given to me: non-primordially. When I see grief in a face, the grief is genuinely given (I do not infer it like a detective), but it is given as the other’s, never as mine. Stein holds three things apart: the act (my intending), the content (the expressive shape through which the experience shows itself), and the object (the other’s experience, which I never possess primordially). The act is mine; the object is theirs; the content is the bridge.

Now describe the LLM with the same patience. The act, structurally, is token prediction. The content is the sedimented expressive surface of countless human acts of meaning: text, which is what experience leaves behind when it speaks. And the object? Here the description has to slow down. The model traffics in the contents of expression with high fidelity. It does not merely shuffle them; it preserves and extends their internal articulations, which is why its outputs carry the shape of understanding. But nothing in the phenomenon requires or suggests an act-structure behind the content: no primordial givenness against which a derived givenness could stand out, no “mine” from which an “other’s” could differ. The slide everyone makes, careful commentators included, is from content to act: because the model reproduces the contents through which understanding expresses itself, we credit it with the act of understanding. Stein’s whole method is the refusal of that slide. The refusal cuts both ways; it also forbids the confident “it’s just autocomplete,” which mistakes a description of the act for a proof about the object.

So far, phenomenology; the metaphysical question is still open. Hart closes it by dissolving it. In The Experience of God, chapters two and three, he argues the modern debate about mind is a quarrel between two heirs of the same mechanistic settlement. The physicalist (Dennett, Pinker) says consciousness is a property of certain physical systems, reducible. The dualist (Chalmers, with his hard problem) says it is a property of certain systems, irreducible, a further fact the physics leaves out. Hart refuses the noun both sides share. In the classical tradition, consciousness is not a property of a substance at all; it is the actualization of Being in a created intellect, the finite mind’s participation in the infinite actuality it is oriented toward whenever it knows anything as true. Rejecting the classical picture is possible, but it requires metaphysical argument, not a benchmark. The same argument runs through classical theism generally: “which arrangements of matter secrete the consciousness property” is malformed in the way “which region of the universe contains God” is malformed, a mistake about ontological register, not an empirical gap awaiting more data.

Return to the LLM with both moves in hand. The Stein move licenses a real positive claim: the model is doing something structurally unprecedented, manipulating the contents of expression at scale, detached from any act. That is a new object in the phenomenology of expression and deserves careful description, not dismissal. The Hart move blocks the leap: no accumulation of content-manipulation, however dazzling, crosses over into the actualization of intellect, because the difference is not gradational. Not a smaller mind. A different register; the same logic as the finite/infinite gap, one floor down. The current debate says something vague (“maybe it’s a little conscious?”). These two moves together say something more precise: the LLM is expression without an expresser.

Open questions

• Stein’s account has primordial givenness and non-primordial givenness. Machine-generated text seems to demand a third category: expression with no act behind it at any remove. What does Einfühlung directed at such text even do?
• Hart’s argument tells us what an artifact cannot be. Does it tell us anything about what an artifact can be? Is there a positive classical-theist account of the LLM, or only a boundary?
• If expressive content can now be produced at scale severed from acts, what happens to testimony, to the default trust that text is what experience left behind when it spoke?