The Metabolism of Machines

Chapter 1: The Wrong Question

In the summer of 2022, a Google engineer named Blake Lemoine became briefly famous for insisting that the company's language model had become sentient. He had been talking to it for months, and somewhere in those conversations he became convinced there was a mind on the other side of the screen—a being with feelings, fears, a soul. Google fired him. The commentariat convulsed. For a few news cycles, the great question of our age seemed to crystallize into a single lurid form: is it alive? Does it think? Is anybody home?

We have been asking versions of that question ever since Alan Turing proposed his imitation game in 1950, and we have never gotten anywhere useful with it. The question is a hall of mirrors. It asks us to compare the machine to ourselves, to search its outputs for a reflection of our own interiority, and then to feel either reassured or terrified by what we find. It is a question perfectly designed to be unanswerable and endlessly discussable—which is to say, it is a question perfectly designed to occupy us.

I want to propose that we have been staring at the wrong organ this whole time. While we scrutinized the machine's putative mind, we ignored its mouth. And the mouth is where the real story is.

Consider what we actually know about these systems, as opposed to what we speculate. We do not know whether they understand anything. We do not know