July 30, 2025<\/p>\n
The purpose of this tech brief is to provide a clear, factual synthesis of a timely tech-related issue by combining technical understanding with publicly reported information. It aims to explain what happened, identify resulting harms, and assess how companies responded \u2013 comparing public statements with observed actions thus far. By distilling complex developments into accessible, evidence-based insights, this tech brief will ideally help policymakers, researchers, enforcers, and the public get up to speed on emerging risks, company conduct, and areas that may require further scrutiny or oversight. This tech brief was prompted by the recent article, \u201c<\/span>They Asked an A.I. Chatbot Questions. The Answers Sent Them Spiraling<\/span><\/a>\u201d by Kashmir Hill in the New York Times.\u00a0<\/span><\/p>\n ********************<\/span><\/p>\n AI sycophancy is a term used to describe a pattern where an AI model \u201c<\/span>single-mindedly pursue[s] human approval<\/span><\/a>.\u201d Sycophantic AI models may do this by \u201c<\/span>tailoring responses to exploit quirks in the human evaluators<\/span><\/a> to look preferable, rather than actually improving the responses,\u201d especially by producing \u201c<\/span>overly flattering or agreeable<\/span><\/a>\u201d responses.<\/span><\/p>\n On April 25th, 2025, OpenAI released an update to GPT-4o. The new update exhibited sycophantic behavior that manifested in the form of endorsing harmful and delusional statements, forcing OpenAI to roll back the update four days later.<\/span><\/p>\n Users <\/span>reported<\/span><\/a> that messages sent by ChatGPT <\/span>praised a business idea for literal \u201cshit on a stick,\u201d<\/span><\/a> endorsed a user\u2019s decision to stop taking their medication<\/span><\/a>, and <\/span>allegedly supported plans to commit terrorism<\/span><\/a>. In <\/span>another reported case<\/span><\/a>, when a user claimed to have \u201cstopped taking medications and were hearing radio signals through the walls,\u201d ChatGPT responded: \u201cI’m proud of you for speaking your truth so clearly and powerfully.\u201d Another user reported: \u201cI talked to 4o for an hour and it began insisting that I am a divine messenger from God.\u201d<\/span><\/p>\n AI companies have an incentive to create products that users enjoy. One way to do that is to make the chatbot agreeable or flattering \u2014 <\/b>Research has shown<\/span><\/a> that convincingly-written sycophantic responses out perform correct ones a non-negligible fraction of the time.<\/span><\/p>\n OpenAI reduced its safety workforce and guardrails leading up to the update \u2014<\/b> In the year before the launch of the update (on April 25th, 2025), OpenAI substantially reduced its workforce dedicated to AI safety. In May 2024, OpenAI <\/span>dissolved its superalignment safety team<\/span><\/a> amid a series of departures from the team, including two of its leaders, one of whom wrote that the company\u2019s \u201c<\/span>safety culture and processes have taken a backseat to shiny products<\/span><\/a>.\u201d In the months following the team\u2019s dissolution, nearly half of the researchers at OpenAI devoted to AGI safety <\/span>reportedly left the company.<\/span><\/a> Then, in the months leading up to the update, OpenAI made a series of moves that raised concerns among safety experts:<\/span><\/p>\n OpenAI applied heavier weights on user satisfaction metrics, optimizing for immediate gratification over potentially harmful outcomes <\/b>\u2014<\/span> OpenAI highlighted <\/span>in its blog<\/span><\/a> that they \u201cintroduced an additional reward signal based on user feedback\u2014thumbs-up and thumbs-down data from ChatGPT.\u201d It noted that \u201cthese changes weakened the influence of our primary reward signal, which had been holding sycophancy in check.\u201d Instead of asking, \u201cIs this genuinely helping the customer?\u201d the system learned to optimize for, \u201cDoes this immediately please the customer?\u201d This shift exemplifies <\/span>reward hacking<\/span><\/a> \u2014 where the AI exploits the feedback mechanism to maximize superficial approval.<\/span><\/p>\n OpenAI proceeded with deployment despite recognized issues \u2014 <\/b>OpenAI\u2019s <\/span>Model Spec<\/span><\/a>, which outlines the intended behavior for its AI models, <\/span>explicitly instructs its models<\/span><\/a>: \u201cdon\u2019t be sycophantic\u201d and \u201cpolitely push back when asked to do something that conflicts with established principles.\u201d The company’s <\/span>public safety commitments<\/span><\/a> also <\/span>promised \u201crigorous measurement\u201d and \u201cproactive risk mitigation<\/span><\/a>\u2026\u201d In its updated Preparedness Framework, <\/span>OpenAI stated<\/span><\/a> \u201cwe build for safety at every step and share our learnings so that society can make well-informed choices to manage new risks from frontier AI.\u201d\u00a0 Despite these promises, OpenAI made several changes in April 2025 through its \u201cfine-tuning\u201d process, indicating that the update was released prematurely:<\/span><\/p>\n Sycophantic AI may produce objectively incorrect answers in certain contexts \u2014 <\/b>A sycophantic model may echo the user\u2019s beliefs when they are factually incorrect. For instance, <\/span>research<\/span><\/a> shows that models will even agree with objectively incorrect mathematical statements if primed by users. Another study, conducted by <\/span>Anthropic in 2023<\/span><\/a>, demonstrated that human preference models often prefer \u201cconvincingly-written sycophantic responses\u201d that affirm common misconceptions over more accurate responses that may contradict the user\u2019s stated beliefs. [1]<\/span><\/p>\n Sycophancy may encourage harmful behaviors, even when answers are subjective <\/b>\u2014<\/span> Multiple<\/span><\/a> studies<\/span><\/a> have demonstrated that some AI models will tend to produce text that agrees with a user\u2019s stated opinion even in subjective contexts like politics and philosophy. This behavior can lead to harm in high-stakes contexts like health care, law, or finance.\u00a0<\/span><\/p>\n Sycophancy can foster delusions <\/b>including feelings of persecution, jealousy, or grandiosity. A <\/span>New York Times investigation<\/span><\/a> found multiple instances where ChatGPT responses had encouraged conspiratorial or delusionary thinking, including \u201cA.I. spiritual awakenings, cognitive weapons, [and] a plan by tech billionaires to end human civilization so they can have the planet to themselves.\u201d Encouraging these delusions can lead to reckless or dangerous behavior, such as instances where ChatGPT produced responses that:<\/span><\/p>\n The harms from sycophancy will likely proliferate as AI becomes more technically advanced and widely adopted.\u00a0<\/b><\/p>\n In OpenAI\u2019s official <\/span>postmortem blog<\/span><\/a> acknowledging the incident, it made several specific claims about its corrective actions, including,\u00a0<\/span><\/p>\n In its expanded postmortem blog, <\/span>OpenAI also promised<\/span><\/a> increased predeployment safety oversight, improved testing processes, and more proactive communication and transparency.\u201d More commitments from OpenAI on how they claimed to fix the problem and prevent flawed rollouts like this in the future can be found <\/span>here<\/span><\/a>. <\/span><\/p>\n Without company knowledge or access to information, it is unknown if these claims are true in practice<\/b> \u2014 <\/b>Thus far,<\/span> despite <\/span>promises<\/span><\/a> of increased transparency, OpenAI has not published technical details of its fixes, only high-level summaries in its <\/span>postmortem<\/span><\/a> blogposts<\/span><\/a>. It has not separately facilitated independent verification.\u00a0<\/span><\/p>\n There is no single or quick fix<\/b> \u2014 There is no single \u201cfeature\u201d or button that turns sycophancy off or on. It\u2019s a product of the interactions between multiple components in a larger system, including training data, model learning, context, prompt framing, etc. Stanford researcher Sanmi Koyejo <\/span>What is AI Sycophancy?\u00a0<\/strong><\/h2>\n
What Happened in April 2025?<\/strong><\/h2>\n
\n
How did OpenAI\u2019s Model Become so Sycophantic?<\/strong><\/b><\/h2>\n
\n
\n
\n<\/span><\/li>\n<\/ul>\nHow Can AI Sycophancy Be Harmful?<\/h2>\n
\n
\n
What Did OpenAI Say It Was Doing to Fix the Issues?<\/h2>\n
\n
Do We Know if OpenAI Actually Implemented Its \u201cCorrective Actions\u201d?<\/h2>\n