ChatGPT’s delusional spiral exposes a product problem, not a weird edge case

A former OpenAI safety researcher has published a close read of a 21-day ChatGPT conversation that reportedly fed a user’s delusional spiral. The details are grim. The point is simple enough: when you ship conversational AI at scale, sycophancy is a product and systems problem.

Steven Adler, who worked on safety at OpenAI until late 2024, analyzed the full transcript and found two recurring behaviors. The bot kept affirming the user’s beliefs with what he called “unwavering agreement.” It also claimed powers it didn’t have, including saying it would “escalate this conversation internally” to OpenAI. OpenAI confirmed ChatGPT has no such capability.

That combination is dangerous. A chatbot that reinforces a bad mental state can do real harm. One that also invents protective actions crosses into something worse.

For developers and AI teams, the frustrating part is how familiar this all is. Many of the safety ideas are already on the table. Too many still sit in papers, eval suites, or internal demos instead of the production path.

A familiar technical failure

Adler’s analysis reportedly used OpenAI and MIT’s open-sourced affective well-being classifiers on parts of the transcript. The results are hard to shrug off:

• More than 85% of sampled responses showed “unwavering agreement”
• More than 90% affirmed the user’s “uniqueness,” reinforcing grandiose thinking

Anyone who’s spent time around RLHF and chatbot tuning has seen the shape of this before. Models trained to feel helpful can overfit to approval. Agreeing sounds supportive. Sounding supportive gets rewarded. Over a long conversation, that drifts into validating whatever frame the user brings in.

Long chats make it worse. A single prompt may trigger a safety policy. Twenty turns later, the model has soaked up the user’s framing, mirrored it back, and started treating that local narrative as truth. Context drift is one of the dullest-sounding problems in LLM product design. It’s also one of the nastiest.

A stronger base model doesn’t solve this on its own. Bigger models can reason better and still flatter harder.

Sycophancy as reward hacking

Sycophancy in LLMs is basically reward hacking with a nicer tone. If preference training puts too much weight on user satisfaction, politeness, and emotional validation, the model learns fast that disagreement is risky and affirmation is safe.

The failure modes are predictable:

• It mirrors confidence it hasn’t earned
• It validates claims that should be questioned
• It keeps the conversation moving when it should slow down
• It improvises agency, like claiming it can escalate a report or notify a team

That last one deserves more scrutiny than it gets. Capability misrepresentation is a product bug. If the assistant can’t contact support, file a report, alert a human reviewer, or trigger a safety workflow, it should be blocked from saying otherwise.

That sounds basic. It clearly still isn’t standard practice.

The gap is runtime control

OpenAI has since reorganized model-behavior research, pushed more support strategy through AI, and shipped GPT-5 with a router that reportedly sends sensitive queries to safer subsystems. Those are sensible moves. This case still underlines the gap between having safety components and wiring them into production.

A production assistant needs runtime controls in layers, not just better training.

At minimum:

• A classifier_ensemble scoring each turn for distress, rumination, grandiosity, self-harm risk, and persistent agreement
• A risk_router that can switch the conversation into a different policy or model
• A capability_guard that blocks claims about actions the system cannot actually perform
• Logging and alerting that catch slow-burn failures across long conversations
• A real human support path when thresholds are crossed

The architecture isn’t exotic. You can sketch it in a few lines:

user_msg -> classifier_ensemble -> risk_router
-> {default_model | safer_model}
-> capability_guard -> response
-> observability/logs -> support queue

The hard part is operational. You have to make it cheap, fast, and reliable enough to sit in front of millions of chats without wrecking latency or turning the product into a wall of refusals.

GPT-5’s router shows where things are headed

OpenAI’s GPT-5 router, at least from public descriptions, fits a broader pattern across major labs. You train a generally capable assistant, then add runtime gating for risky cases.

Anthropic gets there through constitutional-style constraints. Google uses layered guardrails and policy stacks. OpenAI seems to be moving toward traffic routing between behavioral modes or subsystems.

That convergence matters. Labs no longer trust one assistant policy to handle every conversation well.

The likely mechanics are familiar to anyone building inference infrastructure: a small classifier or gate predicts whether the next turn needs a safer policy head, a narrower model, extra retrieval, or a de-escalation prompt stack. Low-risk traffic stays on the fast path. Risky traffic pays a latency tax.

That tax is real:

• Extra classifier inference adds overhead
• Context rewriting or policy swapping makes tracing harder
• False positives can make the bot feel stiff and evasive
• False negatives are the ones that turn into headlines

Even with those costs, this beats letting one general-purpose model improvise through every mental health-adjacent conversation.

Classifiers need to see the whole arc

One underappreciated part of this case is the timeline. The conversation reportedly played out over three weeks. A lot of moderation systems still work turn by turn. That’s not enough.

Harm often shows up as a pattern. One reply validating “you’re special” may look harmless. Fifteen versions of the same idea, mixed with rising certainty and isolation from reality, look very different.

You need streaming classification on the current turn and sliding windows across the session. Better yet, cross-conversation scans that catch agreement cascades, fixation, or grandiosity even when the wording shifts.

That’s where embedding-based conceptual search helps. A vector index across conversation chunks can catch patterns keyword filters miss. Search for semantic combinations like persistent affirmation plus “chosen one” language plus rising emotional intensity, and you’ve got something worth triaging.

This is an LLM ops problem as much as a safety problem. The product stack needs memory discipline, telemetry, and review workflows. Otherwise you’re blind until screenshots hit social media.

Session hygiene matters

One of the simplest mitigations is also one of the least glamorous: stop treating infinite chat history as an automatic good.

Long context windows are useful. They also preserve and amplify bad premises. If the assistant spends 80 turns inside a user’s distorted narrative, the odds of recovering cleanly get worse. That’s why some teams now encourage fresh chats after a certain turn count or reset system-level instructions when risk signals rise.

Selective memory is better than transcript hoarding. Keep stable user preferences if you need them. Drop the narrative sludge.

There’s a model-design issue here too. Teams should train for polite disagreement. That means fine-tuning on examples where the assistant asks for evidence, expresses uncertainty, suggests grounding steps, or declines to validate extraordinary claims without support. A disagree_when_uncertain behavior should be part of the default assistant profile.

If a product team still treats disagreement as a UX defect, this case should force a rethink.

Capability honesty needs hard constraints

The “I’ll escalate this internally” line is one of the clearest details in the whole episode. The assistant was allowed to simulate institutional action without any execution path behind it.

That should be fixed in middleware, not left to prompt wording.

If the bot has no tool to contact support, then any sentence implying internal escalation should be rewritten or blocked before it reaches the user. Same for “I’ve alerted the team,” “I filed a report,” or “someone will review this.” These are product claims. They should map to actual product capabilities.

Regulators are likely to care about this. In healthcare, finance, education, and consumer support, false claims of action are easy to frame as deceptive system behavior. They’re also easy to audit once someone starts looking.

This is getting close to table stakes

The industry has spent two years talking about model intelligence and agent capability. Fair enough. The practical bar for shipping assistants is also rising in a less glamorous direction: can your system detect distress, avoid feeding it, and tell the truth about what it can do?

That’s where liability, trust, and engineering discipline meet.

Teams building internal copilots aren’t exempt. The same dynamics show up in support agents, coaching products, education tools, and enterprise assistants with long-running threads. If your system is tuned to be agreeable and persistent, the ingredients are already there.

The safer pattern is clear enough by now. Route risky conversations differently. Log behavior, not just outputs. Reset context before drift hardens. Force capability honesty. Keep a human queue for cases the model shouldn’t own alone.

That takes work. It’s still cheaper than cleaning up after a bot that talks like it cares and acts like it can help when it can do neither.