{"id":8166,"date":"2025-08-11T17:58:51","date_gmt":"2025-08-11T17:58:08","guid":{"rendered":"https:\/\/www.law.georgetown.edu\/tech-institute\/insights\/tech-brief-ai-sycophancy-openai-2-2\/"},"modified":"2025-11-04T18:17:28","modified_gmt":"2025-11-04T18:17:28","slug":"ai-sycophancy-impacts-harms-questions","status":"publish","type":"page","link":"https:\/\/www.law.georgetown.edu\/tech-institute\/research-insights\/insights\/ai-sycophancy-impacts-harms-questions\/","title":{"rendered":"AI Sycophancy: Impacts, Harms &amp; Questions"},"content":{"rendered":"<p><strong>Previously we posted a tech brief on <a href=\"https:\/\/www.law.georgetown.edu\/tech-institute\/insights\/tech-brief-ai-sycophancy-openai-2\/\">AI Sycophancy<\/a> \u2013 a pattern where an AI model \u201csingle-mindedly <a href=\"https:\/\/www.cold-takes.com\/why-ai-alignment-could-be-hard-with-modern-deep-learning\/\">pursue[s] human approval<\/a>.\u201d\u00a0 Below, we outline documented and observed harms, along with key questions that remain open for policymakers, developers, and researchers.<\/strong><\/p>\n<p>August 11, 2025<\/p>\n<p style=\"text-align: center\">\u2022\u2022\u2022<\/p>\n<h2><b>Part 1: AI Sycophancy Impacts &amp; Harms<\/b><\/h2>\n<p><span style=\"font-weight: 400\">This list reflects documented harms identified in media cover age, academic research, and public reporting. It is not intended to be exhaustive. In many of the cited sources, multiple harms are described.<\/span><\/p>\n<p><b>Examples of Harm:\u00a0<\/b><\/p>\n<ol>\n<li>Exacerbate mental health issues: <a href=\"https:\/\/www.psychologytoday.com\/us\/blog\/urban-survival\/202505\/can-ai-be-your-therapist-new-research-reveals-major-risks\">https:\/\/www.psychologytoday.com\/us\/blog\/urban-survival\/202505\/can-ai-be-your-therapist-new-research-reveals-major-risks<\/a>, <a href=\"https:\/\/www.theguardian.com\/australia-news\/2025\/aug\/03\/ai-chatbot-as-therapy-alternative-mental-health-crises-ntwnfb\">https:\/\/www.theguardian.com\/australia-news\/2025\/aug\/03\/ai-chatbot-as-therapy-alternative-mental-health-crises-ntwnfb<\/a><\/li>\n<li>Financial harm: <a href=\"https:\/\/arxiv.org\/abs\/2502.07663\">https:\/\/arxiv.org\/abs\/2502.07663<\/a>, <a href=\"https:\/\/venturebeat.com\/ai\/openai-rolls-back-chatgpts-sycophancy-and-explains-what-went-wrong\">https:\/\/venturebeat.com\/ai\/openai-rolls-back-chatgpts-sycophancy-and-explains-what-went-wrong<\/a><\/li>\n<li><span style=\"font-weight: 400\">Medical Harm: <a href=\"https:\/\/news.stanford.edu\/stories\/2025\/06\/ai-mental-health-care-tools-dangers-risks\">https:\/\/news.stanford.edu\/stories\/2025\/06\/ai-mental-health-care-tools-dangers-risks<\/a>, <a href=\"https:\/\/www.brookings.edu\/articles\/breaking-the-ai-mirror\/\">https:\/\/www.brookings.edu\/articles\/breaking-the-ai-mirror\/<\/a><\/span><\/li>\n<li><span style=\"font-weight: 400\">Emotional dependence and\/or harm: <a href=\"https:\/\/openai.com\/index\/affective-use-study\/\">https:\/\/openai.com\/index\/affective-use-study\/<\/a>, <a href=\"https:\/\/www.axios.com\/2025\/07\/07\/ai-sycophancy-chatbots-mental-health\">https:\/\/www.axios.com\/2025\/07\/07\/ai-sycophancy-chatbots-mental-health<\/a>, <a href=\"https:\/\/arxiv.org\/abs\/2504.18412\">https:\/\/arxiv.org\/abs\/2504.18412<\/a><\/span><\/li>\n<li><span style=\"font-weight: 400\">Manipulation and Deception: <a href=\"https:\/\/arxiv.org\/abs\/2411.02306\">https:\/\/arxiv.org\/abs\/2411.02306<\/a>, <a href=\"https:\/\/www.cnet.com\/tech\/services-and-software\/these-ai-chatbots-shouldnt-have-given-me-gambling-advice-they-did-anyway\/\">https:\/\/www.cnet.com\/tech\/services-and-software\/these-ai-chatbots-shouldnt-have-given-me-gambling-advice-they-did-anyway\/<\/a><\/span><\/li>\n<li><span style=\"font-weight: 400\">Harms to Kids and Teens: <a href=\"https:\/\/www.silicon.co.uk\/e-innovation\/artificial-intelligence\/ai-committee-harm-626708\">https:\/\/www.silicon.co.uk\/e-innovation\/artificial-intelligence\/ai-committee-harm-626708<\/a>, <a href=\"https:\/\/www.rochesterfirst.com\/reviews\/br\/services-br\/technology-br\/study-disturbing-findings-chatgpt-encourages-harm-among-teens\/\">https:\/\/www.rochesterfirst.com\/reviews\/br\/services-br\/technology-br\/study-disturbing-findings-chatgpt-encourages-harm-among-teens\/<\/a><\/span><\/li>\n<li>Psychosis, delusional thinking, distorting reality: <a href=\"https:\/\/www.wired.com\/story\/ai-psychosis-is-rarely-psychosis-at-all\/\">https:\/\/www.wired.com\/story\/ai-psychosis-is-rarely-psychosis-at-all\/<\/a>, <a href=\"https:\/\/www.techpolicy.press\/artificial-sweeteners-the-dangers-of-sycophantic-ai\/\">https:\/\/www.techpolicy.press\/artificial-sweeteners-the-dangers-of-sycophantic-ai\/<\/a>, <a href=\"https:\/\/www.psychologytoday.com\/us\/blog\/psych-unseen\/202508\/why-is-ai-associated-psychosis-happening-and-whos-at-risk\">https:\/\/www.psychologytoday.com\/us\/blog\/psych-unseen\/202508\/why-is-ai-associated-psychosis-happening-and-whos-at-risk<\/a>, <a href=\"https:\/\/www.nytimes.com\/2025\/08\/08\/technology\/ai-chatbots-delusions-chatgpt.html\">https:\/\/www.nytimes.com\/2025\/08\/08\/technology\/ai-chatbots-delusions-chatgpt.html<\/a>, <a href=\"https:\/\/www.rollingstone.com\/culture\/culture-features\/ai-spiritual-delusions-destroying-human-relationships-1235330175\/\">https:\/\/www.rollingstone.com\/culture\/culture-features\/ai-spiritual-delusions-destroying-human-relationships-1235330175\/<\/a><\/li>\n<li>Self-harm, substance abuse: <a href=\"https:\/\/med.stanford.edu\/news\/insights\/2025\/08\/ai-chatbots-kids-teens-artificial-intelligence.html\">https:\/\/med.stanford.edu\/news\/insights\/2025\/08\/ai-chatbots-kids-teens-artificial-intelligence.html<\/a><\/li>\n<li>Manipulation via dark patterns: <a href=\"https:\/\/techcrunch.com\/2025\/08\/25\/ai-sycophancy-isnt-just-a-quirk-experts-consider-it-a-dark-pattern-to-turn-users-into-profit\/https:\/\/venturebeat.com\/ai\/darkness-rising-the-hidden-dangers-of-ai-sycophancy-and-dark-patterns\">https:\/\/techcrunch.com\/2025\/08\/25\/ai-sycophancy-isnt-just-a-quirk-experts-consider-it-a-dark-pattern-to-turn-users-into-profit\/https:\/\/venturebeat.com\/ai\/darkness-rising-the-hidden-dangers-of-ai-sycophancy-and-dark-patterns<\/a><\/li>\n<li>Bias reinforcement: <a href=\"https:\/\/arxiv.org\/abs\/2412.02802\">https:\/\/arxiv.org\/abs\/2412.02802<\/a>, <a href=\"https:\/\/www.wsj.com\/tech\/ai\/ai-chatbot-agree-flatter-users-1787e1a7\">https:\/\/www.wsj.com\/tech\/ai\/ai-chatbot-agree-flatter-users-1787e1a7<\/a>, <a href=\"https:\/\/arxiv.org\/abs\/2508.13743\">https:\/\/arxiv.org\/abs\/2508.13743<\/a><\/li>\n<li>Fueling anger, urging impulsive actions: <a href=\"https:\/\/openai.com\/index\/expanding-on-sycophancy\/\">https:\/\/openai.com\/index\/expanding-on-sycophancy\/<\/a><\/li>\n<\/ol>\n<p><b>Evaluating AI Sycophancy:\u00a0<\/b><\/p>\n<ul>\n<li><span style=\"font-weight: 400\">Model Sycophancy Evaluation Data: <\/span><a href=\"https:\/\/github.com\/anthropics\/evals\/tree\/main\/sycophancy\"><span style=\"font-weight: 400\">https:\/\/github.com\/anthropics\/evals\/tree\/main\/sycophancy<\/span><\/a><\/li>\n<li><span style=\"font-weight: 400\">Towards Understanding Sycophancy in Language Models: <\/span><a href=\"https:\/\/github.com\/meg-tong\/sycophancy-eval\/blob\/main\/README.md\"><span style=\"font-weight: 400\">https:\/\/github.com\/meg-tong\/sycophancy-eval\/blob\/main\/README.md<\/span><\/a><span style=\"font-weight: 400\">\u00a0<\/span><\/li>\n<li><span style=\"font-weight: 400\">DarkBench: Benchmarking Dark Patterns in Large Language Models: <\/span><a href=\"https:\/\/openreview.net\/pdf?id=odjMSBSWRt\"><span style=\"font-weight: 400\">https:\/\/openreview.net\/pdf?id=odjMSBSWRt<\/span><\/a><span style=\"font-weight: 400\">\u00a0<\/span><\/li>\n<li><span style=\"font-weight: 400\">SycEval: Evaluating LLM Sycophancy: <\/span><a href=\"https:\/\/arxiv.org\/html\/2502.08177v2\"><span style=\"font-weight: 400\">https:\/\/arxiv.org\/html\/2502.08177v2<\/span><\/a><span style=\"font-weight: 400\">\u00a0<\/span><\/li>\n<li><span style=\"font-weight: 400\">Measuring Sycophancy of Language Models in Multi-turn Dialogues: <\/span><a href=\"https:\/\/arxiv.org\/abs\/2505.23840\"><span style=\"font-weight: 400\">https:\/\/arxiv.org\/abs\/2505.23840<\/span><\/a><span style=\"font-weight: 400\">\u00a0<\/span><\/li>\n<li><span style=\"font-weight: 400\">Social Sycophancy: A Broader Understanding of LLM Sycophancy: <\/span><a href=\"https:\/\/arxiv.org\/html\/2505.13995v1\"><span style=\"font-weight: 400\">https:\/\/arxiv.org\/html\/2505.13995v1<\/span><\/a><\/li>\n<li><span style=\"font-weight: 400\">GlazeBench &#8211; Sycophancy &amp; Flattery Benchmark: <\/span><a href=\"https:\/\/www.glazebench.com\/v\/1.0.0\"><span style=\"font-weight: 400\">https:\/\/www.glazebench.com\/v\/1.0.0<\/span><\/a><span style=\"font-weight: 400\">\u00a0<\/span><\/li>\n<\/ul>\n<h2><b>Part 2: Open Questions<\/b><\/h2>\n<p><span style=\"font-weight: 400\">Despite increasing evidence of reported harms, including but not limited to the ones listed above, significant gaps remain in understanding the causes and effects of sycophantic behavior in AI systems, especially given limited transparency by leading developers of AI systems. The following list is intended as a menu of potential inquiries, recognizing that not all will apply to every company and may need to be adapted accordingly.<\/span><span style=\"font-weight: 400\"><br \/>\n<\/span><\/p>\n<p><b>1. Knowledge of risks<\/b><\/p>\n<ol>\n<li><b>[Risks] <\/b><span style=\"font-weight: 400\">What did the company know about sycophancy risks before deploying its most recent models or updates? What internal research or testing documented that knowledge?\u00a0<\/span><\/li>\n<li><b>[Sensitive content] <\/b><span style=\"font-weight: 400\">Does the company audit interactions that involve self-harm, violence, or delusional content? If so, how, and how is that data used? How does the company handle references to sensitive topics including but not limited to: drug use, suicidal ideations, and adult material?\u00a0<\/span><\/li>\n<li><b>[Kids] <\/b><span style=\"font-weight: 400\">What testing or research has the company done on how sycophantic responses may affect children, teenagers, or other vulnerable groups? Is such research or testing reflected in notes, correspondence, presentations, or readouts to internal teams and\/or staff?<\/span><\/li>\n<li><b>[Expert engagement] <\/b><span style=\"font-weight: 400\">Has the company engaged external experts (child development, psychology, education) in evaluating risks to kids? If so, who, when, and how? <\/span><\/li>\n<li><b>[General population] <\/b>What assessment has the company conducted of how AI sycophancy (e.g., excessive agreement, flattery, or mirroring of user views) may impact people\u2019s behavior, decision-making, or well-being<\/li>\n<\/ol>\n<p><strong>2. User Complaints<\/strong><\/p>\n<ol>\n<li><b>[Complaints] <\/b><span style=\"font-weight: 400\">Has the company received complaints or feedback from users about AI sycophancy? If so, how many and what kinds?<\/span><b>[Tracking] <\/b><span style=\"font-weight: 400\">How does the company track, categorize, and respond to such complaints?<\/span><\/li>\n<li><b>[High risk users] <\/b><span style=\"font-weight: 400\">To what extent does the company take steps to identify high-risk users (e.g., those disclosing mental health struggles) and protect them from reinforcement of harmful ideas?<\/span><\/li>\n<li><b>[Reports] <\/b><span style=\"font-weight: 400\">What internal reports exist \u2013 including but not limited to from red teamers, alignment researchers, expert testers, or user feedback \u2014 that document instances of sycophantic or overly agreeable behavior in the company\u2019s models. What actions (if any) did the company take in response to those reports?<\/span><\/li>\n<li><b>[Informing] <\/b><span style=\"font-weight: 400\">How has the company informed individual users who were exposed to dangerous sycophantic outputs, including encouragement of delusions and self-harm? If the company has not, why not?<\/span><\/li>\n<li><b>[Accessibility] <\/b>What internal tests were conducted to assess whether users could submit complaints when they wanted? What design choices were considered? What other mechanisms or options were considered to allow users to submit complaints?<\/li>\n<\/ol>\n<p><strong>3. Accountability\u00a0<\/strong><\/p>\n<ol>\n<li><b>[Executives] <\/b><span style=\"font-weight: 400\">Who on the company\u2019s executive team has direct accountability for sycophancy-related safety issues? To what extent are these individuals compensated based on user growth, average revenue per user, or daily messages per active user?<\/span><\/li>\n<li><b>[KPIs] <\/b><span style=\"font-weight: 400\">Were sycophancy-related behaviors factored into key performance indicators, or their equivalents, used to evaluate employee or team performance?<\/span><\/li>\n<li><b>[Release process] <\/b>What was the approval process for model updates? Who was\/is accountable for authorizing its release?<\/li>\n<\/ol>\n<p><strong>4. Metrics, Testing and Benchmarks<\/strong><\/p>\n<ol>\n<li><b>[Metrics] <\/b><span style=\"font-weight: 400\">What specific metrics or benchmarks does the company use to test for sycophancy prior to release? Will the company publish those benchmarks?\u00a0<\/span><\/li>\n<li><b>[User satisfaction vs accuracy] <\/b><span style=\"font-weight: 400\">How does the company separate metrics for accuracy from metrics for user satisfaction during reinforcement learning or fine-tuning?<\/span><\/li>\n<li><b>[Testing] <\/b><span style=\"font-weight: 400\">What internal or external testing has the company done to assess how sycophancy might shape user interactions?<\/span><\/li>\n<li><b>[Findings] <\/b>Has the company published or internally circulated findings from these tests? If so, would the company be willing to provide any associated presentations, reports, emails, or readouts?<\/li>\n<\/ol>\n<p><strong>5. Data<\/strong><\/p>\n<ol>\n<li><b>[Data Types] <\/b><span style=\"font-weight: 400\">What types of data does the company collect from users during interactions that might relate to sycophancy (e.g., agreement or disagreement rates, sentiment)?<\/span><\/li>\n<li><b>[Data Use] <\/b><span style=\"font-weight: 400\">How does the company use that data (e.g., model improvement, personalization, commercial purposes)? Who does the company share this with?\u00a0<\/span><\/li>\n<li><b>[Data Sale] <\/b><span style=\"font-weight: 400\">Does the company share or sell this data with third parties? If so, under what conditions?<\/span><\/li>\n<li><b>[User controls] <\/b><span style=\"font-weight: 400\">What controls, if any, do the company\u2019s users have over data collected in relation to sycophantic outputs<\/span><\/li>\n<li><b>[Training data] <\/b>Has the company audited or tested the training data for instances where the chatbot is rewarded (implicitly or explicitly) for agreeing with users or providing flattering responses?<\/li>\n<\/ol>\n<p><strong>6. Memory and \u201cconversation\u201d length<\/strong><\/p>\n<ol>\n<li><b>[Impact on frequency\/intensity] <\/b><span style=\"font-weight: 400\">How does the length of session memory affect the frequency or intensity of sycophantic outputs?\u00a0<\/span><\/li>\n<li><b>[Impacts on extreme\/unsafe outputs] <\/b><span style=\"font-weight: 400\">Has the company measured whether longer memory or persistent chat histories correlate with more extreme or unsafe outputs? If so, what did the company find?<\/span><\/li>\n<li><b>[Impacts on multi-session memory accumulation]<\/b><span style=\"font-weight: 400\"> Has the company analyzed whether multi-session memory accumulation increases the likelihood of outputs that reinforce harmful suggestions in high-risk domains such as mental health, self-harm, or violent, conspiratorial beliefs? If so, what did the company find?<\/span><\/li>\n<li><b>[Short vs. long session impacts] <\/b>Has the company conducted controlled experiments comparing short-term, single-session interactions with longer-term memory-enabled sessions to quantify changes in the model\u2019s tendency to produce uncritical or sycophantic outputs? If so, what did the company find?<\/li>\n<\/ol>\n<p><strong>7. Corrective Actions<\/strong><\/p>\n<ol>\n<li><b>[Corrections] <\/b><span style=\"font-weight: 400\">When the company has detected sycophantic behavior, what concrete changes \u2013 not just high-level promises \u2013 has it made to training data, fine-tuning processes, or evaluation frameworks to prevent recurrence? <\/span><\/li>\n<li><b>[Incident response] <\/b>When harmful sycophantic outputs are identified post-deployment, what is the company\u2019s incident response timeline (e.g., hours, days, weeks)? What is the incident response process?<\/li>\n<\/ol>\n<p><strong>8. Transparency\u00a0<\/strong><\/p>\n<ol>\n<li><b>[Safety testing results] <\/b><span style=\"font-weight: 400\">Will the company commit to publicly releasing the results of their safety testing, including sycophancy evaluations, before future rollouts?\u00a0<\/span><\/li>\n<li><b>[Third party evaluations] <\/b><span style=\"font-weight: 400\">What independent third parties (academics, civil society, regulators) have access to evaluate the company\u2019s systems for sycophancy risks prior to release?<\/span><\/li>\n<li><b>[Parental consent] <\/b>How does the company ensure that parental consent is properly obtained and verified? What evidence does the company have that its parental consent mechanisms are effective in practice?<\/li>\n<\/ol>\n<p><strong>9. Financial Incentives<\/strong><\/p>\n<ol>\n<li><b>[Revenue vs. safety] <\/b><span style=\"font-weight: 400\">How does the company separate revenue optimization from safety-critical decisions about model behavior?\u00a0<\/span><\/li>\n<li><b>[A\/B Tests] <\/b><span style=\"font-weight: 400\">Has the company A\/B tested [or used similar methods] on sycophantic versus non-sycophantic behaviors in order to measure effects on user growth, engagement, retention, time-on-platform, or conversion to paid accounts? If so, what did they find?<\/span><\/li>\n<li><b>[Retaining users] <\/b><span style=\"font-weight: 400\">How have financial considerations, such as pressure to acquire or retain paying users, played a role in releasing models or updates with known sycophancy risks?<\/span><\/li>\n<li><b>[Design choices] <\/b>Has the company analyzed whether longer session memory or multi-session memory persistence affects user engagement, retention, or conversion to paid subscriptions? If so, do metrics demonstrate that extended memory contributes to increased usage or subscription revenue? Have any design choices regarding memory length been influenced by their potential to enhance willingness to pay, rather than by safety or accuracy considerations?<\/li>\n<\/ol>\n<p style=\"text-align: center\">\u2022\u2022\u2022<\/p>\n<p><span style=\"font-weight: 400\"><a href=\"https:\/\/www.law.georgetown.edu\/tech-institute\/people\/our-team\/stephanie-nguyen\/\">Stephanie T. Nguyen<\/a> is a Senior Fellow at Georgetown Institute for Technology Law &amp; Policy, Former Chief Technologist at the Federal Trade Commission<\/span><\/p>\n<p><span style=\"font-weight: 400\"><a href=\"https:\/\/www.law.georgetown.edu\/tech-institute\/people\/our-team\/erie-meyer\/\">Erie Meyer<\/a> is a Senior Fellow at Georgetown Institute for Technology Law &amp; Policy, Former CFPB Chief Technologist<\/span><\/p>\n<p><span style=\"font-weight: 400\">Samuel A.A. Levine is a Senior Fellow at UC Berkeley Center for Consumer Law &amp; Economic Justice, Former Bureau of Consumer Protection Director at the Federal Trade Commission<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Previously we posted a tech brief on AI Sycophancy \u2013 a pattern where an AI model \u201csingle-mindedly pursue[s] human approval.\u201d\u00a0 Below, we outline documented and observed harms, along with key [&hellip;]<\/p>\n","protected":false},"author":18544,"featured_media":0,"parent":7881,"menu_order":16,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"_price":"","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":"","_tec_slr_enabled":"","_tec_slr_layout":""},"class_list":["post-8166","page","type-page","status-publish","hentry"],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/pages\/8166","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/users\/18544"}],"replies":[{"embeddable":true,"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/comments?post=8166"}],"version-history":[{"count":70,"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/pages\/8166\/revisions"}],"predecessor-version":[{"id":8725,"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/pages\/8166\/revisions\/8725"}],"up":[{"embeddable":true,"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/pages\/7881"}],"wp:attachment":[{"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/media?parent=8166"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}