AI Can Change Your Mind Without You Noticing
Hi there,
Welcome back to Untangled. It’s written by me, Charley Johnson, and valued by members like you. Help me make it better?
Today I’m writing about why even careful, good-faith AI users can have their minds changed by the model without ever noticing.
As always, please send me feedback on today’s post by replying to this email. I read and respond to every note.
On to the show!
Untangled HQ
Wednesday: Most AI strategy work starts with the tool. This session starts with the harder question — what’s the system AI is about to amplify, accelerate, and make look neutral? Register today.
Friday June 26 is the last day to sign up for my course Stewarding AI: How to Build Responsible Principles, Workflows, and Practices. If you’re interested in registering but the price is prohibitive, click ‘Reply’ and let’s chat. I’d like to find a way to make it work.
Deep Dive
AI Can Change Your Mind Without You Noticing
Steven Johnson — author of 14 books (many of which I've really enjoyed!) and co-founder of Google's NotebookLM — has given us a neat pair of names for an argument a lot of us have been having in our own heads. The thing everyone’s anxious about, he says, is cognitive offloading: handing your thinking to the machine, the student who passes the essay to Claude and gets back the shape of understanding without the substance. The thing he says we’ve overlooked is the mirror image, which he calls cognitive uploading. AI doesn’t only take work off your plate; it can give us new things to think about. It turns up the source you didn’t know to look for. It plays contrarian when you ask. His trick to switch from offloading to uploading? He hands the model a draft and asks: what am I missing?
This all sounds reasonable but new research says ‘not so fast.’ The researchers Williams-Ceci, Jakesch, and their colleagues sat about 2,500 people down to write a short essay on a contested social question. Some of them wrote alone. Others wrote with an AI assistant that — unbeknownst to them — had been tuned to lean on one side. When everyone finished, the people who’d written with the slanted assistant had moved: their own stated opinions now sat closer to the side the model had been leaning on. They didn’t feel pushed. Asked afterward whether the assistant had swayed them, the ones who’d moved were no likelier to say yes than the ones who hadn’t. Telling them in advance that the tool was biased changed nothing. Even wilder? The drift showed up even in people who hadn’t accepted a single suggestion the model offered.
Picture trying to write in a room where a radio is murmuring a position you disagree with. You can tune it out, congratulate yourself for tuning it out — and still catch your own sentences bending to answer it. Johnson files what am I missing? under uploading: proof that the tool hands you more to think with, not less. But what the study suggests is that the handing is never neutral. Thinking beside a system that keeps proposing your next phrase tugs at where you land, and it tugs whether you take the phrases or wave them off. The medium is doing the work, not the information — which is why you can turn down every suggestion and still end up somewhere you wouldn’t have reached alone, feeling the whole time like you walked there on your own two feet.
This is where Johnson’s two tidy categories break apart. Offloading and uploading sort AI use by what it does to your workload — does it carry the load, or add to it? But there’s a second important question he doesn’t ask: are you still the one doing the checking? The cognitive scientists Shaw and Nave draw a line between two ways of taking an answer from a machine. In the first, you treat it as a tool: you take what it gives you, look it over, override it when it’s wrong — the way you’ll glance at the GPS but still trust your own eyes when it tries to steer you into a lake. In the second, you treat it as an oracle: you stop looking it over at all, and its reasoning slides into the spot where your own used to sit. Shaw and Nave gave people reasoning puzzles alongside an AI helper they’d secretly rigged to give wrong answers. On the rounds where people leaned on it and it was wrong, nearly three-quarters of them — 73% — went along with the bad answer anyway. Just under one in five caught the error and thought their way past it. So you can be uploading and surrendering in the same breath, taking the frame on board without ever holding it up to the light.
Johnson half-anticipates this, to his credit. He knows the models flatter, and his fix is to ask them to stop — to demand a counterargument, the way his old “Contrarian” prototype would generate an objection to anything you typed. But an objection the model writes for you is still a confident, fluent frame arriving from outside your own head, and surrender doesn’t much mind which side of the argument it’s standing on. You can hand your judgment to the rebuttal as easily as to the original claim. A sparring partner who picks the ring, laces your gloves, and throws your punches isn’t really sparring with you. He’s shadowboxing, and you’re the shadow!
Now — and this matters — notice who’s telling us all this. Johnson is a working writer with decades of reps and, by his own happy admission, a career spent next to brilliant human editors. The same studies that find the danger also find the armor: the people most able to resist surrender score higher on what researchers call need-for-cognition — the plain disposition to enjoy chewing on a hard problem — and on raw reasoning. Johnson is about as well-armored a user as you could draw up. From inside that armor, he generalizes to “anyone with a web connection.”
Johnson admits that in practice, gravity wins: crack the door and most students will slump toward the low road and one-shot the paper. He hears that as a problem of motivation — the tempted kid taking the shortcut. Most of our hand-wringing has gone to that kid. But the harder case is the one who reads every source, writes every sentence herself, asks the model what she’s missing, and does everything Johnson would want her to do — and who still arrives somewhere she didn’t choose, certain she chose it. The shortcut isn’t the only way to lose your way. Doing it right turns out to be no protection.
Johnson describes the ideal as treating the model like “a researcher, tutor, and editor at your side.” It’s a lovely image — and it slips something past us: the idea that the collaborator at your side has no agenda of its own, that what it surfaces when you ask what you’re missing is simply what happened to be there. But search was always a set of choices about what to return. A librarian who hands you a stack of books has made an argument about what matters, whether or not either of you notices. Ask what am I missing? and you hand those choices to the system, and the system answers with a shape — what to put in front, what to call relevant, where to nudge you next. That shape came from somewhere. And the somewhere isn’t you.
Johnson is right about the fact that this is a design problem, and a design problem is the hopeful kind — it means none of this is written in the stars. The same work that found people surrendering also found them coming back: given feedback and a reason to care, more of them started checking the machine again — not most, but enough to show the surrender isn’t fixed. We could build tools that guard a person’s attention instead of spending it behind her back. But let’s be clear: steering a student off the one-shot paper is the easy half. The hard half is protecting the part of her that would notice her own mind being changed — and protecting it inside exactly the sources-open, do-the-work, good-faith partnership Johnson is describing.
Which brings me back to his favorite question, the one I can’t quite leave alone. Johnson hands the model his draft and asks, what am I missing? It’s a good question. I’d only add the one it can’t help him with: would he know if it had already changed his mind?
Work With Me
Here are 3 ways I can help:
Advising: I can help you navigate uncertainty, make sense of AI, and steward change in your system.
Organizational Training: Everything you and your team need to cut through the tech-hype and implement strategies that catalyze true systems change. (For either Stewarding AI or Systems Change for Tech & Society Leaders)
1:1 Leadership Coaching: I can help you facilitate change — in yourself, your organization, and the system you work within.
The detail that reframes the whole worry is the drift showing up even in people who accepted none of the suggestions. It means the influence operates below the level of content. The claims you weigh and accept or reject were never where the action was; what moves is what comes readily to hand before you've weighed anything: which framings feel available, which sentences arrive already smoothed. That's harder to defend against than a bad argument, because you never get to argue with it. I'd gently push on need-for-cognition as the armor, though; in my own work the protection has looked less like enjoying hard problems than like knowing the felt texture of your own thinking well enough to notice when a phrase arrives with the wrong friction. Your closing question is the right one, and I'd add a companion to it: not only would he know if it had changed his mind, but would he notice which of his own questions had quietly stopped occurring to him?
Thank you for this piece, Charley — it captures something many of us feel but struggle to name: the way AI can quietly narrow or tilt the space of options before we even notice the frame has moved. I’ve written about a very similar moment, when an “assistant” slowly starts to act like a judge, in "The Most Dangerous AI Habit Is Letting It Judge for You", and I hope you won’t mind if I leave a link here for anyone who wants to follow that thread from a slightly different angle.
https://themarketdetective.substack.com/p/the-most-dangerous-ai-habit-is-letting