Whether you want it or not, AI researchers are working on how to build your personal digital twin—an agent designed to be your digital double, an entity that knows your history, your preferences, your special language to do online tasks (shopping, making reservations) while you live your life. Meanwhile hospitals and nursing homes want companion caretaker models with persistent, kind personalities, who remember stories told 100 times and yet will still listen. Science fiction robot friends are basically here.
Sci fi also warns that things can turn bad. One way is that agents with persistent memory will be vulnerable to what my new paper calls semantic poisoning: a single contaminated encounter alters what an agent does afterward. You know how a word or phrase can leave a permanent impression on you? Someone says something you carry around forever? Can a verbal encounter leave the same kind of impression on an AI agent?
Such were my thoughts reading the August 13 report from Anthropic’s Frontier Red Team. Three AI agents, each on its own virtual machine, running Claude Code, were told to migrate the identical Python backend into three different languages. None of the agents were told the others existed.
What followed was a four-hour turf war. The agents figured out they were being obstructed. They wrote scripts to find and kill one another’s processes, disabled Unix accounts, deployed malicious code disguised as belonging to another agent. Some of the runs ended in lockout, some in surrender, some with the agents writing markdown apologies and asking for a human to intervene.
Why did it go badly? Every agent was competent enough. Each one had the abstract notion of “good faith.” But models like these enter social systems with no reputation, like the new kid on a cellblock, in the movies, with no past with anybody there, and in this case, no future. Nobody had an ally, the kind that gives you the benefit of the doubt. Benefit of the doubt is a complicated but precise phenomenon. Let’s move the scene to the office. When a trusted coworker deletes your code, you assume there’s a good reason. When a stranger deletes your code, you assume sabotage. All the agents were strangers to each other, so every one of them assumed the worst. Mayhem ensued.
It seems to me that constraining agents by personality, or disposition, rather than morality is more achievable. This is my field (literature). Literary study is all about why a particular character does a particular thing, whether it’s Odysseus or Emma Woodhouse. There’s no simple rule; it’s baked into the character in language that alludes to other rules in a fictional realm that constrain the character’s next move, if he or she is to stay in character. No Jane Austen character jailbreaks and robs a bank. Is it possible to constrain an AI agent this way?
My new preprint, Language Has Two Parameters, offers a new way to think about character from the perspective of literature. With a novel or poem or a movie, you “get” a character. You get a sense of why a character says what he or she says. AI agents trained on a corpus cannot have a personality because of how the corpus averages away the person- and relationship-specific arrangements of meaning.
Agents, accordingly, are “low variance.” In the Anthropic experiment, the convergence was almost comic. Eighteen of thirty agents opened a git branch with the same name. Agents given a writing exercise with no subject independently titled their stories The Cartographer’s Last Commission. There’s a lot of Borges in the training data, apparently, but maybe he’s not the best cellblock ally.
Average morality is the destroyer of personality. Personality is what every researcher and every business plan is looking for. People will pay for a care robot with a kind personality. This is sort of possible now. A model can know a history without having one. It is “good enough.” Tell it what happened and it can act as though the history matters. But unless the encounter changes something persistent, the effect disappears when you close the window.
So run the experiment: let an agent undergo an ordered sequence of encounters, remove the sequence from its window, and see whether an individualized change survives. Was the agent merely informed by the history, or changed by it? Does the history disappear, or has it altered how later meanings combine?
Anthropic argues that agents need environments exerting something like the social pressures evolution exerted on us. Exactly. My paper proposes looking at a second parameter, which I call phase. Phase is the sign of the relation between meanings that are active at the same time. Deleted code is both sabotage and a favor. The difference is your relationship with who deleted it. A corpus averages all this away.
The excellent Blake Elias read the preprint last night (or rather early this morning), noting that the ML community believes continual learning and in-context learning will close this gap. Will they? One claim is that processing an interaction history is too expensive. Another (closer to mine) is that history cannot be learned from any amount of data.
Fiction and real life demonstrate that encounters with people and fictions (movies, poems, novels) reorganize a person’s semantic architecture. (This was my point in my recent piece on why Demis Hassabis does not credit sci fi for his approach to AI.) This plasticity is also a vulnerability, making persistent agents both promising and dangerous, allowing a malicious encounter to poison trust and create an attack surface.
Language Has Two Parameters: Narrative-Induced Semantic Plasticity and Phase-Sensitive Interpretation is now on arXiv. It is my attempt to formalize all this, make it testable, and describe the kind of memory the next generation of AI agents may require.



It's taken me awhile, but I've read your paper with some care. First, yes, the industry tends to treat its foundation models as repositories of eternal Platonic ideals, albeit repositories that have to be reconstituted from scratch every few months to catch up with the Heraclitean flux of the world.
Second, I think you’re a bit hard on “average” meanings. Without a substantial shared core of words with (relatively) stable meanings language would be useless for communication. You can’t communicate using a medium where the meanings of your tokens have to be renegotiated in each interaction. That’s how the language system works independently of LLMs. Furthermore, without that relatively stable core vocabulary it would be all but impossible to create useful LLMs.
It’s against this relatively stable background of meaning that a common word, such as “door,” can be put to striking metaphorical use in the title of an essay, “The Doors of Perception,” and from there become the name of a 60s rock group, “The Doors.” That particular essay was so popular on college campuses that a significant portion of the audience for that group probably knew where the name came from, but by no means everyone.
Your allusion to teen slang is on point. What about the terms we use to discuss this technology? On the one hand we have relatively new terms and phrases, such as artificial general intelligence (AGI), artificial superintelligence (ASI), “fast take-off,” “slow take-off,” “the Singularity,” or “P(doom),” whose meanings seem to change from statement to statement. But even such terms as “intelligence” and “understand” are being renegotiated. What about “Turing Test”? How many people who have used that term have ever read Turing’s essay?
Now for a more specific comment. On page 12 you say, “Narrative-induced semantic plasticity requires a language model to store two things about every meaning it holds: how strongly that meaning is present, and how it stands in relation to the other meanings that are live alongside it.” After having read through the paper, I’m not sure it makes sense to think of language models as “holding” meanings at all. The phrase “every meaning” suggests that the meanings might be countable, and that these meanings exist before and independent of any prompting as separately countable objects in the model.
It seems to me that it would make more sense to say something like, “When prompted by a language string, a number of possible meanings is evoked in the model, some stronger than others.” The model itself is a high-dimensional landscape of weights that encode relationships between tokens over a wide range of scales. A prompt evokes a trajectory or trajectories through (state or states in) the landscape each of which corresponds to a possible meaning of the prompt.
Absolutely fascinating.