🧠 Does generative AI have ‘emergent’ properties?
AI chatbots are just knowledge sausages — read to understand what on earth I mean by that.
Hi there, and welcome back to Untangled, a newsletter and podcast about technology, people, and power. Let’s recap June, shall we?
I posed the important question: is AI ‘hallucinating,’ or are we? That essay racked up 12K views and generated 70 new subscribers.
I took on the question on everyone’s mind: how does the Netflix show ‘Ultimatum’ explain complex algorithmic systems?? 🙃
I wrote about Substack, Bluesky, and the trend toward ‘decentralized content moderation.’
I celebrated the mini-milestone of 4K subscribers by offering a steep discount and opening up the archives. The discount has ended, and the archives are now closed, but if you want to read them at your leisure, sign up for the paid edition.
One last thing - I launched a referral program in June. You can now invite your friends to join Untangled and get the paid version for free. Since launching, I altered the reward structure to make it a li’l easier to win free access to Untangled. Check it out!
Now, on to the show.
A few issues back, I explored what ‘information’ actually means, and argued that it’s a big ol’ problem when we treat metaphors not as a representation of something but reality itself. Then I interrogated the notion that AI is ‘hallucinating.’ Well, in this issue, I’m turning my attention to another consequential metaphor that obscures more than it reveals: emergence. I’m on a metaphor hunt y’all! Let’s dig in.
The phrase ‘emergent properties’ is everywhere. Take this 60 minutes clip as an example.
The narrator in the clip explains that Google’s large language model learned how to translate all of Bengali without being trained on any Bengali text. This is the idea of emergent behavior — that at a certain scale, models can do things that aren’t expected of them; that given enough data, we can no longer predict what they’re capable of. Emergence has become the core animating idea of sentience and artificial general intelligence — if a model can teach itself to learn a new language, the question becomes, what else might it learn to do?
But in this example, the large language model didn’t unexpectedly teach itself anything. As a number of AI researchers made clear, it was trained on Bengali text. Even Twitter subsequently fact-checked the clip and added context, writing “It is not correct to state that it ‘spoke a foreign language it was never trained to know.’” That’s pretty definitive, but this still leaves open the possibility that AI systems could exhibit ‘emergent properties.’ So, uh, could they?
The concept of an ‘emergent property’ comes from the literature on complex systems. In 2011, I became fascinated by the study of complex systems, attending workshops at the Santa Fe Institute, and trainings with Dave Snowden of Cognitive Edge. Invariably, the conversation would start with ants. Yes, ants. The way ants work together is a perfect example of a complex system: each li’l ant is performing actions that together form complex structures that help the colony survive; patterns organically emerge from the interactions of simple components, and this all happens without a central control mechanism. Or as physicist Phil Anderson succinctly put it, the study of complex systems is the study of “why is more, different”?
In addition to insect colonies, researchers in this field also study brains, food webs, social networks, cities, and even entire economies. What they’re finding is actually quite interesting — these systems behave in similar ways, and they involve a common set of properties. Take flocks of birds that fly in a coherent, predictable shape. As Shane Parrish writes, “They don’t manage this by following the instructions of a leader; instead, each bird instinctively follows certain rules, like keeping an even distance between themselves and their neighbors.” Because these patterns follow certain rules, they can be modeled and simulated.
This is all well and good, but those who study emergence can’t often explain how simple actions become more than the sum of their parts. Back to the ants. As
writes in her book Complexity: A Guided Tour, scientists can observe and document the patterned structures, but they can’t explain:“…neither their individual nor group behavior: exactly how the individual actions of the ants produce large, complex structures, how the ants signal one another, and how the colony as a whole adapts to changing circumstances (e.g. changing weather or attacks on the colony.)”
This has led to a distinction in the field between ‘weak emergence’ (i.e. what emerges within a system is based on identifiable rules) and ‘strong emergence’ (i.e. what emerges within a system can’t be attributed to identifiable rules). For example, birds flying in a flock is weak and ant colonies are strong. This distinction is a useful one, but critics are right to point out that if we can’t identify how patterns emerge, then the field is rather limited in its contributions. As ecologist Deborah Gordon argues:
“Recently, ideas about complexity, self-organization, and emergence — when the whole is greater than the sum of its parts — have come into fashion as alternatives for metaphors of control. But such explanations offer only smoke and mirrors, functioning merely to provide names for what we can’t explain.”
There is a big gap between finding fun patterns across systems on the one hand and science on the other — that gap can look a lot like magic.
This brings us back to AI. Those invested in AI, like Google CEO Sundar Pichai, want you to think of AI like magic. Digital humanities scholar Lauren Klein recently explained how this narrative — wherein code is magic and the creators are a kind of magician — goes all the way back to the 1950s. If it’s magic at a certain scale, then we must pursue the development of ever larger models, trained on bigger datasets. If it’s magic, then naturally we must rely on its creators to help us see the shape of the near future. If it’s magic, then artificial general intelligence (AGI) is just around the corner. But what if all this magic is actually a mirage?
A recent paper by Stanford researchers explained that so-called ‘emergent abilities’ can be understood as a “mirage” produced by the researchers themselves. Basically, the researchers found that whether a model shows what some might refer to as ‘emergent abilities’ hinges on whether the model used metrics that were nonlinear, linear, or discontinuous.
Take the example of the Exact String Match metric. As the name suggests, this metric is used to test whether the outcome generated by the model produces the exact sequence of words that they were aiming for. This is a discontinuous metric because it’s all or nothing — if the target string matches exactly, ta-da, it gets a ‘1.’ But if it’s off just a smidge? Nada. It doesn’t get a .99, it gets a 0. This is how Google researchers explain the metric:
“The EXACT_STRING_MATCH metric can lead to apparent sudden breakthroughs because of its inherent all-or-nothing discontinuity. It only gives credit for a model output that exactly matches the target string. Examining other metrics, such as BLEU, BLEURT, or ROUGE, can reveal more gradual progress.”
In other words, larger models might exhibit emergent behavior or “apparent sudden breakthroughs” if they get the target output exactly right, whereas smaller models will get zero credit even if they get the answer almost exactly right. Now, the Stanford researchers are clear that they aren’t disproving emergence, only that there is a totally plausible explanation for the model’s behavior: the ghost in the machine, so to speak, is just a developer turning a nob.
What can be done about all this?
When it comes to AI, what might we replace “emergent behavior” with? Yale computer science professor Theodore Kim has a colorful suggestion: “knowledge sausages.” Here’s Kim:
“Claiming that complex outputs arising from even more complex inputs is ‘emergent behavior’ is like finding a severed finger in a hot dog and claiming the hot dog factory has learned to create fingers. Modern AI chatbots are not magical artifacts without precedent in human history. They are not producing something out of nothing. They do not reveal insights into the laws that govern human consciousness and our physical universe. They are industrial-scale knowledge sausages.”
Yes, if you’re wondering, I’m jealous that I didn’t come up with the phrase “knowledge sausages” first. The point is: weird, unpredictable patterns aren’t self-justifying. They can be induced by the metrics a developer chooses. If AI researchers are going to back up claims of emergence, they would need a robust, generalizable theory of how the system is functioning, and the ability to explain how the patterns emerge. Until then, it’s knowledge sausages all the way down.
As always, thank you to Georgia Iacovou, my editor.