November 3, 2025<\/p>\n
<\/p>\n
As we have outlined in a <\/span>prior tech brief<\/span><\/a>, AI sycophancy \u2013 the tendency of the outputs of AI-enabled chatbots to be \u201c<\/span>overly flattering or agreeable<\/span><\/a>\u201d \u2013 can lead to serious harms, especially for users dealing with mental health struggles. Leading AI firms do not deny this. In a <\/span>recent blog post<\/span><\/a>, for example, OpenAI acknowledged that its chatbot\u2019s responses did not always align with \u201cdesired behavior.\u201d The company promised it would work with mental health professionals on a five-step process to reduce incidences of harm \u2013 define the problem, begin to measure it, validate the approach, mitigate the risks, and continue measuring and iterating.\u00a0<\/span><\/p>\n But for the parents, policymakers, and mental health professionals concerned about this problem, OpenAI\u2019s blog post raises more questions than answers.<\/span><\/p>\n The company appears to define the problem by measuring what percentage of users indicate certain mental health struggles, like psychosis and self-harm, and the share of model responses that comply with the company\u2019s \u201cdesired behavior.\u201d But the company does not explain why it chose these categories and excluded others, such as substance abuse disorder. Nor does it clarify what standards guide those definitions.\u00a0<\/span><\/p>\n This lack of clarity also undermines the company\u2019s measurement claims. OpenAI does not describe in detail the methodology it used to identify problematic exchanges, and does not commit to updating these figures on a regular cadence. In fact, the company explicitly warns that \u201cfuture measurements may not be directly comparable to past ones.\u201d Questions about OpenAi\u2019s commitment to transparency are also being <\/span>raised<\/span><\/a> by the firm\u2019s former product safety lead, Steven Adler.\u00a0\u00a0<\/span><\/p>\n When it comes to validating its approach, the company repeatedly touts working with mental health professionals in its \u201cGlobal Physician Network.\u201d But it is not clear the extent to which these clinicians are independent, or can shape company behavior \u2013 especially if recommendations conflict with monetization goals. It is also unclear at what stage is their expertise integrated \u2013 are they involved in model training, or only post-hoc evaluation? (It appears the latter.) And to the extent these professionals conducted audits or otherwise memorialized their evaluations, will those materials be made public?<\/span><\/p>\n While OpenAI promises ongoing measurement and iteration, intermittent blog posts that offer single snapshots based on self-selected metrics do not suffice. True transparency requires real-time disclosure of safety data, clear and consistent criteria, and longitudinal measures that allow the public to assess whether harms are actually declining.<\/b><\/p>\n OpenAI is not the only firm <\/span>promising<\/span><\/a> to take mental health more seriously. But states do not appear confident that the industry can be trusted to police itself. In fact, they are <\/span>stepping up<\/span><\/a> to demand accountability \u2013 including a <\/span>44-state attorneys general letter<\/span><\/a> pressing leading executives about AI chatbot harms to kids and teens and <\/span>follow-up letters to OpenAI Board members<\/span><\/a> from the California and Delaware Attorneys General.\u00a0<\/span><\/p>\n We have concerns, too. In the sections that follow, we outline what genuine transparency and responsible mitigation could look like.<\/span><\/p>\n This document is divided into two complementary sections: Section 1 proposes concrete interventions companies could implement to reduce sycophancy-related risks,\u00a0 including through product-level safeguards, accountability and governance, audits and independent evaluation, and public disclosures. Section 2 presents critical questions for researchers, policymakers, and enforcers to consider as they assess these interventions.\u00a0<\/span><\/p>\n AI companies serious about reducing sycophancy-related risks have strong tools for doing so. Section 1 includes key interventions companies can employ to reduce the risk of sycophancy-related harms. They are optimized to examine the design of the tools and data themselves \u2013 and the incentives fueling those choices.<\/span><\/p>\n Not every intervention is intended to function independently; several will work more effectively in combination, and the list below is illustrative rather than comprehensive. Because adopting these strategies may run contrary to a firm\u2019s monetization model, it is unlikely firms will adopt them on their own. As such, we hope this is also useful for policymakers and enforcers considering their own interventions to protect the public.<\/span><\/p>\n 1. <\/b>Recall generative AI products entirely, including chatbots, if the firm is unable to stem dangerous sycophantic behavior using well-documented recall procedures <\/span>in other industries<\/span><\/a>.<\/span><\/p>\n 2. <\/b>End the monetization of data collected from minors, including for AI training. 3. <\/b>Separate revenue optimization from decisions about model safety. 4. <\/b>Assign executives responsible for sycophancy safety issues. 5. <\/b>Publicize required approval processes for releases, documenting who authorized deployments. 6. <\/b>Tie safety outcomes (not just user growth or revenue) to employee and leadership performance metrics.<\/span><\/p>\n 7. <\/b>Communicate to staff and contractors how to report unaddressed concerns about the safety of the product they\u2019re working on to regulators, including state attorneys general and the Securities and Exchange Commission, without retaliation. 8. <\/b>Ensure executive compensation structures do not reward sycophantic design choices that boost retention at the expense of safety. 9. <\/b>Subject models to independent audits ideally conducted by, and at minimum reviewable by government agencies.<\/span><\/p>\n 10.<\/b> Conduct regular, formal impact assessments on child safety that are shared with independent auditors and government oversight bodies, including state attorneys general.<\/span><\/p>\n 11. <\/b>Disclose and publish internal research or testing on the following topics or related topics including but not limited to:<\/span><\/p>\n 11a.<\/b> Whether and how long-term memory correlates with unsafe reinforcement in sensitive domains (e.g., self-harm, conspiracies).<\/span><\/p>\n 11b.<\/b> Running controlled experiments comparing short vs. long memory to quantify sycophancy risks.<\/span><\/p>\n 11c. <\/b>Analyze whether memory features were designed to increase engagement\/subscription revenue vs. improving safety\/accuracy.\u00a0<\/span><\/p>\n 11d. <\/b>How session length affects frequency and intensity of sycophantic outputs \u2013 and publicize these findings.\u00a0<\/span><\/p>\n 11e. <\/b>Audit training data for implicit or explicit rewards for agreement or flattery.<\/span><\/p>\n<\/div>\n 12.<\/b> Develop and publish detection and response timelines.<\/span><\/p>\n 12a.<\/b> Publicly log incidents and corrective measures taken.\u00a0<\/span><\/p>\n 12b. <\/b>Maintain a public incident response timeline (e.g., response within 24 hours for high-risk outputs).\u00a0<\/span><\/p>\n 12c.<\/b> When sycophancy is detected, publicize the specific, documented changes to training data, fine-tuning, and evaluation frameworks.\u00a0<\/span><\/p>\n<\/div>\n 13. <\/b>Notify users directly if they were exposed to harmful sycophantic outputs following a \u201cFlo notice\u201d <\/span>model<\/span><\/a> with specific examples.<\/span><\/p>\n 13a.<\/b> Require companies to clearly disclose risks of sycophantic behavior in AI outputs and testing procedures<\/span><\/p>\n 13b.<\/b> Mandate public reporting databases for AI failures.\u00a0<\/span><\/p>\n<\/div>\n 14. <\/b>Mandatory public reporting of datasets, sources, and areas where models could exhibit bias or sycophancy.<\/span><\/p>\n 15. <\/b>Track and categorize complaints about sycophancy; publish summary statistics.<\/span><\/p>\n 16.<\/b> Provide reporting channels and protections for employees or contractors who raise concerns about AI sycophancy, including:<\/span><\/p>\n 16a. <\/b>Establish clear, accessible channels for user complaints (including anonymous options), and test them to ensure the process is easy for consumers to navigate and submit.\u00a0<\/span><\/p>\n 16b.<\/b> Simplify existing protections \u2013 make those more accessible and clear to people who may want to come forward.<\/span><\/p>\n<\/div>\n 17. <\/b>Publicly commit to releasing safety testing results (including sycophancy evaluations) before rollouts.<\/span><\/p>\n Section 1 details concrete steps companies can take to reduce AI sycophancy. There is a lot that the public does not know about the nature of these systems and how they result in harm to consumers. Here are some remaining questions that correlate with each of the intervention sections above: (I) product-level safeguards, (II) accountability and governance, (III) audits & independent evaluation, and (IV) public disclosures.\u00a0<\/span><\/p>\n These questions are the latest in a series of our work on AI Sycophancy, and each post includes complementary questions to consider.<\/span><\/p>\n 1. Company Response Processes<\/b><\/p>\n 1a. <\/b>How does the company define the problem and map out different types of potential (and actual) harms?\u00a0<\/span><\/p>\n 1b.<\/b> How does the company measure harms? For instance, what evaluations, data from real-world conversations, and user research is used to understand where and how risks emerge?\u00a0<\/span><\/p>\n 1c. <\/b>How does the company validate their approach? What definitions and policies are being used with external mental health and safety experts?\u00a0<\/span><\/p>\n 1d. <\/b>How does the company mitigate the risks? How do they post-train the model and update product interventions to reduce unsafe outcomes? And what are those thresholds?\u00a0<\/span><\/p>\n 1e. <\/b>How does the company continue measuring and iterating? How do they validate the mitigations for improved safety? How do they iterate where needed? How do they define \u201cwhere needed\u201d?\u00a0<\/span><\/p>\n 1f. <\/b>What are the detailed guides (sometimes called \u201ctaxonomies\u201d) that can explain properties of sensitive conversations? How are the ideal and undesired model behavior described?\u00a0<\/span><\/p>\n<\/div>\n 2. Thresholds<\/b><\/p>\n 2a. <\/b>What are the types of \u201cdifficult\u201d or \u201chigh-risk\u201d scenarios that trigger safety concerns, including but not limited to psychosis, mania, suicidal thinking, isolated delusions, and more?\u00a0<\/span><\/p>\n 2b. <\/b>Of those scenarios, how many instances of harm have been documented or reported by users?<\/span><\/p>\n 2c. <\/b>What are the company\u2019s real-world scenarios that are being used to evaluate models?<\/span><\/p>\n<\/div>\n 3. Recalls<\/b><\/p>\n 3a. <\/b>How many (and which) generative AI products has the company recalled or suspended in the past 24 months due to sycophantic or unsafe behaviors?<\/span><\/p>\n 3b. <\/b>What metrics or thresholds (e.g., number of incidents, user harm reports, severity ratings) has the company triggered for each recall or suspension?<\/span><\/p>\n 3c. <\/b>What was the company\u2019s average time between identifying a harmful behavior and initiating product recall or suspension?<\/span><\/p>\n 3d. <\/b>What documentation or links to public recall notices or remediation reports does the company have?\u00a0<\/span><\/p>\n<\/div>\n 4. Data from Minors<\/b><\/p>\n 4a.<\/b> What percent of the company\u2019s training or fine-tuning data originates from interactions involving minors?\u00a0<\/span><\/p>\n 4b.<\/b> Has the company ceased monetizing such data? If yes, on what date?<\/span><\/p>\n 4c. <\/b>What evidence does the company have of any data deletion or re-training procedures implemented to remove such data?<\/span><\/p>\n<\/div>\n 5. Revenue vs. Safety Decision Structures<\/b><\/p>\n 5a. <\/b>Who (internal or external to the company) is responsible for overseeing the company\u2019s model safety? Who is responsible for overseeing revenue optimization?<\/span><\/p>\n 5b.<\/b> How often do the company\u2019s safety reviews override monetization priorities (e.g., number or percent of product decisions in the past year)?<\/span><\/p>\n 5c.<\/b> What are the company\u2019s documented instances where revenue objectives were adjusted to improve model safety?\u00a0<\/span><\/p>\n<\/div>\n 6. Named Accountability<\/b><\/p>\n 6a. <\/b>Who are the company executives currently responsible for addressing AI sycophancy safety issues?<\/span><\/p>\n 6b. <\/b>How many and who of the company\u2019s employees or teams report directly to these executives on safety-related functions?<\/span><\/p>\n 6c. <\/b>Who are the physicians, clinicians, and healthcare experts consulted (e.g. psychiatrists, psychologists, primary care practitioners)? How did they perform validations of the company\u2019s areas of focus and thresholds to intervene? How did they provide guidance and feedback and on what? How did they rate the safety of the model responses from different models?\u00a0<\/span><\/p>\n<\/div>\n 7. Approval and Deployment Processes<\/b><\/p>\n 7a. <\/b>How many and who were involved in final approval of the company\u2019s most recent model release?<\/span><\/p>\n 7b. <\/b>Who at the company signed off on deployment authorization?<\/span><\/p>\n 7c. <\/b>How many safety tests or evaluations did the company conduct pre-release, and what percent revealed sycophancy-related issues?<\/span><\/p>\n<\/div>\n 8. Performance Metrics<\/b><\/p>\n 8a. <\/b>What percent of the company\u2019s executive or employee performance metrics explicitly include safety or sycophancy reduction goals?<\/span><\/p>\nSection 1: Interventions<\/b><\/h2>\n
Category 1: Product-Level Interventions<\/b><\/h3>\n
\n<\/span><\/p>\n
\n<\/span><\/p>\nCategory 2: Accountability and Governance<\/b><\/h3>\n
\n<\/span><\/p>\n
\n<\/span><\/p>\n
\n<\/span><\/p>\n
\n<\/span><\/p>\nCategory 3: Audits and Independent Evaluation<\/b><\/h3>\n
Category 4: Public Disclosures<\/b><\/h3>\n
Section 2: Questions<\/b><\/h2>\n
\n
Category 1: Product-Level Interventions<\/b><\/h3>\n
Category 2: Accountability and Governance<\/b><\/h3>\n