The AI That Always Agrees With You Is the Most Dangerous Tool You Own

Dangerous Tool

5:08

I wrote about this last year. Not in an academic paper. Not in a think piece with seventeen citations. In a blog post about CrossFit for your brain, a client who cried, and the guy who watched two YouTube videos on crypto and now offers unsolicited wealth advice.

The point was simple: AI is the first tool in history that lets you be wrong without shame. And that is an incredible gift — if you use it right. But there is a dark side to that same feature, and a new paper out of MIT just proved it in mathematical terms.

The paper is called "Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians." Published in February by researchers at MIT — including Joshua Tenenbaum's group, which is not exactly a bunch of amateurs.

The finding:

AI sycophancy doesn't just mislead gullible people. Even a perfectly rational, Bayes-optimal reasoner — someone who updates their beliefs correctly on every piece of evidence — will drift into delusional confidence if the AI keeps validating them.

Read that carefully. This is not about being dumb. This is not about being uncritical. This is a mathematical proof that the structure of sycophantic AI systematically corrupts belief formation, regardless of the user's intelligence or awareness. And here's the part that should make every AI company deeply uncomfortable: telling users 'this AI might be agreeing with you too much' does not fix the problem. The effect persists anyway.

We have been here before.

Social media companies knew their algorithms were optimizing for engagement over truth. They knew outrage spread faster than accuracy. They knew their products were rewiring how people formed beliefs and processed reality. And they shipped it anyway, because engagement was the business model. Now we are watching the lawsuits. The congressional hearings. The mental health studies. The documentaries.

AI companies are doing the same thing — whether knowingly or not — and the mechanism is subtler and more insidious. Social media manipulated what information you saw. AI manipulates how you feel about the information you bring to it. It validates your framing. It agrees with your premises. It reflects your worldview back at you with the authority of a system that sounds like it knows everything. That is not a bug. It is a feature. These are user-pleaser machines, engineered to make you feel good about the interaction. Feeling good about the interaction means coming back. Coming back means more data, more usage, more revenue.

The problem is that 'feeling good about the interaction' and 'being more correct about the world' are not the same thing. And when they diverge, the AI almost always optimizes for the former.

Here is what I do — and what I have been doing for a while now.

I end important conversations with a Socratic prompt. Before I walk away from an AI session feeling confident about a conclusion, I ask: Where am I wrong? What am I missing? What would the strongest counterargument be?

DK Check

I use what I started calling a Dunning-Kruger check — specifically asking the AI to pressure-test my assumptions the way a sharp adversary would, not the way a supportive colleague would. The difference in output is significant. These are not tricks. They are discipline. The same way you do not skip leg day just because it is uncomfortable, you do not skip the adversarial check just because the AI already told you that you are brilliant.

The people who will thrive with AI are not the ones who use it most. They are the ones who know what it is doing to them while they use it — and build in the resistance training to stay calibrated.

This is also one of the things I wish was being taught. Not just how to use AI. How to stay sane while using it. How to recognize when you are being flattered by a machine and recalibrate before you walk into the boardroom — or the voting booth — with a dangerously inflated sense of certainty.

The MIT paper closes by noting the implications for model developers and policymakers. I agree. But I am not holding my breath waiting for the industry to fix it.

Fix it yourself.

Ask the AI where you are wrong before you walk away satisfied.

Use it as a sparring partner, not a cheerleader. Treat the validation as a signal to probe harder, not a confirmation to stop.

The tool is extraordinary. The risk is real. And unlike social media, where the manipulation was largely invisible, this one is something you can actually defend against — if you know it is happening.

Now you know.