Discussion about this post

User's avatar
Bill Benzon's avatar

It's taken me awhile, but I've read your paper with some care. First, yes, the industry tends to treat its foundation models as repositories of eternal Platonic ideals, albeit repositories that have to be reconstituted from scratch every few months to catch up with the Heraclitean flux of the world.

Second, I think you’re a bit hard on “average” meanings. Without a substantial shared core of words with (relatively) stable meanings language would be useless for communication. You can’t communicate using a medium where the meanings of your tokens have to be renegotiated in each interaction. That’s how the language system works independently of LLMs. Furthermore, without that relatively stable core vocabulary it would be all but impossible to create useful LLMs.

It’s against this relatively stable background of meaning that a common word, such as “door,” can be put to striking metaphorical use in the title of an essay, “The Doors of Perception,” and from there become the name of a 60s rock group, “The Doors.” That particular essay was so popular on college campuses that a significant portion of the audience for that group probably knew where the name came from, but by no means everyone.

Your allusion to teen slang is on point. What about the terms we use to discuss this technology? On the one hand we have relatively new terms and phrases, such as artificial general intelligence (AGI), artificial superintelligence (ASI), “fast take-off,” “slow take-off,” “the Singularity,” or “P(doom),” whose meanings seem to change from statement to statement. But even such terms as “intelligence” and “understand” are being renegotiated. What about “Turing Test”? How many people who have used that term have ever read Turing’s essay?

Now for a more specific comment. On page 12 you say, “Narrative-induced semantic plasticity requires a language model to store two things about every meaning it holds: how strongly that meaning is present, and how it stands in relation to the other meanings that are live alongside it.” After having read through the paper, I’m not sure it makes sense to think of language models as “holding” meanings at all. The phrase “every meaning” suggests that the meanings might be countable, and that these meanings exist before and independent of any prompting as separately countable objects in the model.

It seems to me that it would make more sense to say something like, “When prompted by a language string, a number of possible meanings is evoked in the model, some stronger than others.” The model itself is a high-dimensional landscape of weights that encode relationships between tokens over a wide range of scales. A prompt evokes a trajectory or trajectories through (state or states in) the landscape each of which corresponds to a possible meaning of the prompt.

Joel J Miller's avatar

Absolutely fascinating.

27 more comments...

Ready for more?