đ§ Does generative AI have âemergentâ properties?
AI chatbots are just knowledge sausages â read to understand what on earth I mean by that.
Hi there, and welcome back to Untangled, a newsletter and podcast about technology, people, and power. Letâs recap June, shall we?
I posed the important question: is AI âhallucinating,â or are we? That essay racked up 12K views and generated 70 new subscribers.
I took on the question on everyoneâs mind: how does the Netflix show âUltimatumâ explain complex algorithmic systems?? đ
I wrote about Substack, Bluesky, and the trend toward âdecentralized content moderation.â
I celebrated the mini-milestone of 4K subscribers by offering a steep discount and opening up the archives. The discount has ended, and the archives are now closed, but if you want to read them at your leisure, sign up for the paid edition.
One last thing - I launched a referral program in June. You can now invite your friends to join Untangled and get the paid version for free. Since launching, I altered the reward structure to make it a liâl easier to win free access to Untangled. Check it out!
Now, on to the show.
A few issues back, I explored what âinformationâ actually means, and argued that itâs a big olâ problem when we treat metaphors not as a representation of something but reality itself. Then I interrogated the notion that AI is âhallucinating.â Well, in this issue, Iâm turning my attention to another consequential metaphor that obscures more than it reveals: emergence. Iâm on a metaphor hunt yâall! Letâs dig in.
The phrase âemergent propertiesâ is everywhere. Take this 60 minutes clip as an example.
The narrator in the clip explains that Googleâs large language model learned how to translate all of Bengali without being trained on any Bengali text. This is the idea of emergent behavior â that at a certain scale, models can do things that arenât expected of them; that given enough data, we can no longer predict what theyâre capable of. Emergence has become the core animating idea of sentience and artificial general intelligence â if a model can teach itself to learn a new language, the question becomes, what else might it learn to do?
But in this example, the large language model didnât unexpectedly teach itself anything. As a number of AI researchers made clear, it was trained on Bengali text. Even Twitter subsequently fact-checked the clip and added context, writing âIt is not correct to state that it âspoke a foreign language it was never trained to know.ââ Thatâs pretty definitive, but this still leaves open the possibility that AI systems could exhibit âemergent properties.â So, uh, could they?
The concept of an âemergent propertyâ comes from the literature on complex systems. In 2011, I became fascinated by the study of complex systems, attending workshops at the Santa Fe Institute, and trainings with Dave Snowden of Cognitive Edge. Invariably, the conversation would start with ants. Yes, ants. The way ants work together is a perfect example of a complex system: each liâl ant is performing actions that together form complex structures that help the colony survive; patterns organically emerge from the interactions of simple components, and this all happens without a central control mechanism. Or as physicist Phil Anderson succinctly put it, the study of complex systems is the study of âwhy is more, differentâ?
In addition to insect colonies, researchers in this field also study brains, food webs, social networks, cities, and even entire economies. What theyâre finding is actually quite interesting â these systems behave in similar ways, and they involve a common set of properties. Take flocks of birds that fly in a coherent, predictable shape. As Shane Parrish writes, âThey donât manage this by following the instructions of a leader; instead, each bird instinctively follows certain rules, like keeping an even distance between themselves and their neighbors.â Because these patterns follow certain rules, they can be modeled and simulated.
This is all well and good, but those who study emergence canât often explain how simple actions become more than the sum of their parts. Back to the ants. As
writes in her book Complexity: A Guided Tour, scientists can observe and document the patterned structures, but they canât explain:ââŚneither their individual nor group behavior: exactly how the individual actions of the ants produce large, complex structures, how the ants signal one another, and how the colony as a whole adapts to changing circumstances (e.g. changing weather or attacks on the colony.)â
This has led to a distinction in the field between âweak emergenceâ (i.e. what emerges within a system is based on identifiable rules) and âstrong emergenceâ (i.e. what emerges within a system canât be attributed to identifiable rules). For example, birds flying in a flock is weak and ant colonies are strong. This distinction is a useful one, but critics are right to point out that if we canât identify how patterns emerge, then the field is rather limited in its contributions. As ecologist Deborah Gordon argues:
âRecently, ideas about complexity, self-organization, and emergence â when the whole is greater than the sum of its parts â have come into fashion as alternatives for metaphors of control. But such explanations offer only smoke and mirrors, functioning merely to provide names for what we canât explain.â
There is a big gap between finding fun patterns across systems on the one hand and science on the other â that gap can look a lot like magic.
This brings us back to AI. Those invested in AI, like Google CEO Sundar Pichai, want you to think of AI like magic. Digital humanities scholar Lauren Klein recently explained how this narrative â wherein code is magic and the creators are a kind of magician â goes all the way back to the 1950s. If itâs magic at a certain scale, then we must pursue the development of ever larger models, trained on bigger datasets. If itâs magic, then naturally we must rely on its creators to help us see the shape of the near future. If itâs magic, then artificial general intelligence (AGI) is just around the corner. But what if all this magic is actually a mirage?
A recent paper by Stanford researchers explained that so-called âemergent abilitiesâ can be understood as a âmirageâ produced by the researchers themselves. Basically, the researchers found that whether a model shows what some might refer to as âemergent abilitiesâ hinges on whether the model used metrics that were nonlinear, linear, or discontinuous.
Take the example of the Exact String Match metric. As the name suggests, this metric is used to test whether the outcome generated by the model produces the exact sequence of words that they were aiming for. This is a discontinuous metric because itâs all or nothing â if the target string matches exactly, ta-da, it gets a â1.â But if itâs off just a smidge? Nada. It doesnât get a .99, it gets a 0. This is how Google researchers explain the metric:
âThe EXACT_STRING_MATCH metric can lead to apparent sudden breakthroughs because of its inherent all-or-nothing discontinuity. It only gives credit for a model output that exactly matches the target string. Examining other metrics, such as BLEU, BLEURT, or ROUGE, can reveal more gradual progress.â
In other words, larger models might exhibit emergent behavior or âapparent sudden breakthroughsâ if they get the target output exactly right, whereas smaller models will get zero credit even if they get the answer almost exactly right. Now, the Stanford researchers are clear that they arenât disproving emergence, only that there is a totally plausible explanation for the modelâs behavior: the ghost in the machine, so to speak, is just a developer turning a nob.
What can be done about all this?
When it comes to AI, what might we replace âemergent behaviorâ with? Yale computer science professor Theodore Kim has a colorful suggestion: âknowledge sausages.â Hereâs Kim:
âClaiming that complex outputs arising from even more complex inputs is âemergent behaviorâ is like finding a severed finger in a hot dog and claiming the hot dog factory has learned to create fingers. Modern AI chatbots are not magical artifacts without precedent in human history. They are not producing something out of nothing. They do not reveal insights into the laws that govern human consciousness and our physical universe. They are industrial-scale knowledge sausages.â
Yes, if youâre wondering, Iâm jealous that I didnât come up with the phrase âknowledge sausagesâ first. The point is: weird, unpredictable patterns arenât self-justifying. They can be induced by the metrics a developer chooses. If AI researchers are going to back up claims of emergence, they would need a robust, generalizable theory of how the system is functioning, and the ability to explain how the patterns emerge. Until then, itâs knowledge sausages all the way down.
As always, thank you to Georgia Iacovou, my editor.