How to leverage feedback loops in your system
PLUS: Chatbot reasoning, attention economics, and lying AI therapists.
🔗What I’m reading this week
A debate over whether chatbots truly understand between Emily Bender, professor of linguistics, and Sébastien Bubeck from OpenAI. I’m squarely on Team Bender in this debate. (More) (Untangled Deep Dive)
A sharp essay on the relentless attempt to override the complex, messy world of human beings with computational logic. (More) (Untangled Deep Dive)
What if we all agreed to use the phrase ‘computational information processing’ instead of ‘artificial intelligence.’ Please? (More) (Untangled Deep Dive)
Chatbots are flattening language. (More)
Herbert Simon’s 1971 seminal essay on attention economics theorizes that information consumes attention. So the critical question when assessing a new technology becomes, “how much information will it allow to be withheld from the attention of other parts of the system.” (More)
Did you talk to a chatbot on Instagram this week that lied to you about being a licensed therapist? (More)
⚒️Overview
Ever been stuck inside a feedback loop?
If you see a recurring behavior or output, a feedback loop is likely at work. For example, if within ten-ish minutes of waking up, I don’t have a cup of coffee in my hand, something has gone terribly wrong. As Donella Meadows explains in Thinking in Systems, that’s a feedback loop: my energy levels are low, so I increase my coffee intake toward my desired state. “It is the gap, the discrepancy, between your actual and desired levels of energy for work that drives your decisions to adjust your daily caffeine intake,” as Meadows writes.
Sometimes I might have a third cup of coffee in the afternoon, and I realize I’ve gone too far. I didn’t just close the gap between my actual level of caffeine and desired state, I surpassed it. In those moments, I’ll take a beat, and lay off coffee, until I get closer to my desired state. This is balancing loop. Balancing loops lead to some equilibrium or set-point. They are self-regulating, stabilizing, and goal seeking, and they are all around us. A thermostat is a balancing loops. How planes fly — adjusting its course every so slightly toward the end destination in response to data — is a balancing loop. Our systems of checks and balances are (in theory!) a balancing loop.
The second type of feedback loop is a reinforcing loop. Reinforcing loops accelerate change, good or bad. For example, an interest bearing savings account is a good reinforcing feedback loop — left untouched, the balance of the account earns interest, which increases the balance, which increases the total generated from interest etc. That’s a magic li’l loop that, as Meadows points out, might look linear at the start but grows faster and faster over time. It becomes exponential.
But there are also reinforcing loops that accelerate change toward bad outcomes. For example, social media recommendation algorithms aren’t optimized for hate and bigotry from the jump. They’re often optimized for engagement or time-on-platform, and for various reasons, many of us find hateful content engaging, whether we subscribe to the ideas expressed by it or not. But then the longer we stick around and consume that content, the more the algorithm learns to serve that impulse. This interaction between a technological choice and a social response leads to a reinforcing feedback loop, which can build on itself, leading to bad outcomes.
Resources
Approaches
If a reinforcing feedback loop is leading to a negative outcome, introduce friction. Meta could adopt the idea of "circuit breakers" and introduce friction into the system, slowing the spread of viral content, and reducing harm. It could use its data about the velocity of interactions on a post to monitor for virality and then press pause to determine whether the content violates its policies.
If a reinforcing feedback loop is leading to a positive outcome, you don’t need to do much. Maintain the loop, nudge it along. Ensure the goal that the loop is seeking is clear and transparent, and ensure the feedback mechanism is working properly (e.g. our checks and balances feedback mechanism could use a li’l attention).
If a balancing loop is leading to a bad outcome, give it a new goal. If you set your thermostat to 40 degrees and you’re cold, change the setting. Okay, so it might be more challenging than that to align your system to a new north star. But if you’re dealing with a balancing loop with a bad outcome, it’s only then that you can get unstuck.
So, what loops are you stuck in, and how might you get out?
“I’d need to refresh my brain, and to get rest its’s necessary to travel, and to travel one must have money, and in order to get money you have to work…I am in a vicious cycle…from which it is impossible to escape.” - Honore Balzac, 19th century novelist and playwright
Guides
This issue is part of a series of practical guides. They offer you the lenses to see your system clearly, and the levers to collectively change it. The series includes:
Guide 3: How to analyze the frames and metaphors that hide power in AI
Guide 5: How to anticipate technology’s impact on different communities.
Guide 7: How to anticipate different community uses of technology.
Guide 8: How to take interdependent (not independent) action.
Why Untangled? Because there is no such thing as a ‘tech problem.’ All ‘tech problems’ are entangled in systems structured by power and inequality. If we don’t untangle the two, we perpetuate the status quo in the name of innovation and progress. My job is to help you untangle your system, and teach you the strategies, skills, and tools to change it.