How does your organization know it's wrong — when AI does the knowing?
Your staff can feel when something's off. The question is whether that feeling has anywhere to go
Hi there,
Welcome back to Untangled. It’s written by me, Charley Johnson, and valued by members like you. Help me make it better?
This week I’m writing about pulp, and what happens to an organization when it loses the ability to know it’s wrong.
As always, please send me feedback on today’s post by replying to this email. I read and respond to every note.
On to the show!
Untangled HQ
This week I kicked off the first cohort of Stewarding AI: How to Build Responsible Principles, Workflows, and Practices. The cohort includes non-profit founders working on tech governance, researchers on the forefront of AI and global health, advisors to corporate boards, and entrepreneurs working on AI governance and civic innovation. The first session was ... a ton of fun! Get on the waitlist for the second cohort.
Deep Dive
How does your organization know it’s wrong — when AI does the knowing?
In the 1980s, the sociologist Shoshana Zuboff spent years inside pulp mills, watching veteran operators work. One of them judged the properties of the pulp by chewing it (a job perk, presumably). Then the mills computerized, the operators moved into air-conditioned control rooms, and Zuboff watched workers sneak back onto the mill floor to touch the equipment — because they didn’t trust what the screens told them.
Zuboff’s In the Age of the Smart Machine remains one of the best accounts we have of what happens to knowledge when work becomes screens. Her operators’ expertise never made it into the computers — turns out, you can’t upload the taste of pulp! The old knowledge — tacit, embodied — was orphaned, and a new kind grew up in its place: reading the process through symbols instead of senses. Yes, there are real gains to this shift. But it also created a big ol’ problem: workers gave up their regular independent check. All of a sudden, the smell of a bad cook no longer had a place in the workflow. Still, the mills kept one backstop: physical reality! Pulp had a temperature. If the numbers looked off or the operator got suspicious, she could walk down to the floor and put her hand on the pipe, see what’s off, and close the feedback loop. The material world still offered a nice check on the computerized world.
Twenty years later, the organizational theorist Jannis Kallinikos documents the next turn of the screw. In The Consequences of Information, he argues that organizations increasingly run on the logic of their own data infrastructures — information produced from other information, taking another step away from contact with the real world. Whatever can’t be rendered as data becomes invisible to the organization. It stops circulating as knowledge. But! While the data may be self-referential — reports built from reports built from reports — a human still reasons about the data, and can decide something looks off.
Now fast-forward, and follow this logic through to large language models and agents. Checking the AI’s analysis often means asking the AI. Or asking a human whose sense of what a good analysis looks like has been calibrated, week after week, by AI output. We know from research that AI assistance changes how people evaluate their own decisions, and that systems providing direct answers produce greater reliance over time — and that this built-up reliance is exactly what erodes people’s ability to detect even obvious errors when the guidance goes wrong. Put differently, organizations are entering a world where their staff don’t know what they don’t know.
Zuboff’s operator could act on her unease because her unease had somewhere to go: down the stairs, onto the floor, hand on the pipe. When the check lives inside your head, and your head has been shaped by the machine, your unease doesn’t have a place to go. It just hangs out, and slowly dissipates — unless, of course, it is interrupted. So when someone reads an AI-generated analysis and feels that something is off before they can articulate what, that needs to register as an important signal within the organization. In short, people need to be given a routine, low-stakes way to register “this doesn’t feel right.” A channel that accepts half-formed observations in the moment they occur, in the person’s own words — no form asking them to justify the feeling, no threshold of confidence to clear. Then resist the urge to have the AI summarize what comes in, because that just rebuilds the loop you were trying to break. As complexity theorist Dave Snowden urges, let the people closest to the work say what their own observations mean, and look for patterns across many of them. One flagged hunch is noise but thirty people in different corners of the organization flagging the same kind of thing is useful! This is the work of sense-making.
But of course, none of this works if doubting the machine is expensive or career limiting. Right now, over-reliance on AI is rational inside organizations. If no one blames you for going with the AI but you own all the risk of overruling or questioning it, deference is just good sense. And no amount of “human oversight” fixes that math. So part of creating a culture of sense-making is making it safe to voice doubt and embodied instincts.
The other part is treating judgment as infrastructure, not something people have. Find ways to keep building it in workflows. For example, on a random slice of the work — say, one case in ten — withhold the AI’s output entirely and have people do the analysis unassisted. This keeps the underlying skill alive (the taste of pulp doesn’t orphan), and it produces a rolling baseline of independent judgment you can compare the AI-assisted work against. Put differently, keep building the craft that underlies judgment. This is Zuboff’s operators sneaking down to the mill floor — except now it has to be designed. Deviance in 1985; design in 2026.
So here’s a li’l experiment. Think about the last time an AI-generated analysis gave you that something’s-off feeling — and then ask where the feeling went. Did it feel safe to share? Did it have somewhere to go within your organization, or did it just sort of hang out, and then dissipate when you moved on to the next thing? Reply and tell me. If enough of you write back, I’ll synthesize what I hear and share it in a future issue. Deal?
Work With Me
Here are 3 ways I can help:
Advising: I can help you navigate uncertainty, make sense of AI, and steward change in your system.
Organizational Training: Everything you and your team need to cut through the tech-hype and implement strategies that catalyze true systems change. (For either Stewarding AI or Systems Change for Tech & Society Leaders)
1:1 Leadership Coaching: I can help you facilitate change — in yourself, your organization, and the system you work within.
Totally agree with your diagnosis here, Charley. This is something I’ve seen happen in the healthcare domain at an increasing rate. It’s like we are substituting our senses and measuring abstractions rather than objects, without noticing we are doing that.
https://theslowpanic.substack.com/p/the-self-determining-eye?r=bwndg&utm_medium=ios
The Zuboff operators could sneak back down to the floor because the floor had formed them first — years of hands on pipes built the palate before the control room ever tested it. That's the condition I'd watch in the design: the channel for half-formed observations routes unease, and the one-case-in-ten slice exercises it, but both assume a workforce whose 'something's-off' detector was calibrated by unassisted contact with consequences. That's a cohort condition, and it expires. The staff flagging hunches today were formed before the tools arrived; the analysts hired next year will have read AI output from day one, and a weekly unassisted case audits a palate that never had the floor time to form. Which suggests the channel works best during the transition and thins after — unless the unassisted slice is allowed to do the work of the apprenticeship it's quietly replacing. Thanks for this, Charley.