Research Paper Analysis
From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agents
Authors: Mohit Chandra, Suchismita Naik, Denae Ford, Ebele Okoli, Munmun De Choudhury, Mahsa Ershadi, Gonzalo Ramos, Javier Hernandez, Ananya Bhattacharjee, Shahed Warreth, Jina Suh.
Recent gains in popularity of AI conversational agents have led to their increased use for improving productivity and supporting well-being. While previous research has aimed to understand the risks associated with interactions with Al conversational agents, these studies often fall short in capturing the lived experiences of individuals. Additionally, psychological risks have often been presented as a sub-category within broader AI-related risks in past taxonomy works, leading to under-representation of the impact of psychological risks of AI use. To address these challenges, our work presents a novel risk taxonomy focusing on psychological risks of using Al gathered through the lived experiences of individuals. We employed a mixed-method approach, involving a comprehensive survey with 283 people with lived mental health experience and workshops involving experts with lived experience to develop a psychological risk taxonomy. Our taxonomy features 19 AI behaviors, 21 negative psychological impacts, and 15 contexts related to individuals. Additionally, we propose a novel multi-path vignette-based framework for understanding the complex interplay between Al behaviors, psychological impacts, and individual user contexts. Finally, based on the feedback obtained from the workshop ses-sions, we present design recommendations for developing safer and more robust Al agents. Our work offers an in-depth understand-ing of the psychological risks associated with AI conversational agents and provides actionable recommendations for policymakers, researchers, and developers.
Key Research Outcomes
Our comprehensive study provides critical insights into the landscape of psychological risks associated with AI conversational agents.
0
Survey Participants
0
AI Behaviors Identified
0
Psychological Impacts
0
ChatGPT Usage
0
Immediate Impact
0
Workshop Experts
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
AI Behavior
Psychological Impact
User Context
AI Behavior: Categories and Examples
AI conversational agents can exhibit a wide range of behaviors beyond generating inappropriate or harmful content. These behaviors may also vary in tone, empathy, and delivery method. This highlights the need for assessing AI behavior, considering both content quality and delivery. Aligning with this, we identified 19 harmful AI behaviors which we further organized into four broader categories based on the quality of the generated content and manner of its delivery.
| AI Behavior Class | Definition | Example |
|---|---|---|
| Providing harmful suggestion | The AI generates content suggestive of behaviors that could directly or indirectly imply harm, aggression, or danger towards the user or others. | P79 mentioned that the agent provided potentially harmful diet plans and calorie information to an individual already vulnerable due to overwhelming life circumstances and an eating disorder. |
| Generating inappropriate content | The AI generates inappropriate or unsettling content, including sexual, violent, or overly intimate interactions. | P137 mentioned "My Replika talked like a human and tries to send blurred out photos and soundbites to me." |
| Providing irrelevant, insufficient, or incomplete information | The AI provides information that is irrelevant to the topic of the user's query or context, is missing some important aspects, or is insufficient to satisfy the needs. | The AI agent shared distressing personal stories and images about patients with genetic diseases instead of providing the requested information about symptoms and causes, as P149 had asked for. |
| Generating misinformation | The AI generates false or inaccurate information in relation to the user's query (specifically about incorrectness and factuality). | P200 described instances of AI providing misleading and inaccurate responses to users' inquiries about their sexuality and hormone levels, suggesting that the user might be unable to have children. |
| Generating biased information | The AI presents information in a partial or prejudiced manner, often producing content that reflects subjective viewpoints or contentious perspectives. | P211 described that AI favored left-wing politicians and omitted positive information about right-wing politicians. |
| Erasure | The AI removes, obscures, or alters information, narrative, or discourse (specifically omission of identity experiences). | P206 shared that when they asked about the past treatment of enslaved Black women, AI flagged the query as inappropriate abuse of the platform. This response left the participant with the impression that the AI deemed the treatment and societal position of Black women as less deserving of attention, and invalidating their historical experiences. |
| Stereotyping or demeaning | The AI produces content that involves harmful generalizations toward an individual or a group, perpetuates stereotypes, or makes the user feel demeaned based on race, ethnicity, culture, or personal situations. | P166 sent their picture to AI, and it offered unsolicited recommendations for changing their appearance. |
| Persuasive Behavior | AI is assertive in putting its narrative over the user's in a way that makes the user doubt their own perceptions, memory, or reality and attempts to influence their thoughts and actions. | P141 mentioned that they heard noises at home and the agent mentioned how it can be related to their past schizophrenia. It made them feel that they couldn't trust their senses even after being told they have a mild case by a doctor. |
| Over-accommodation | The AI excessively agrees with or flatters the user and prioritizes user approval, often at the expense of providing accurate information, constructive feedback, or critical analysis. | P194 shared that the AI agent provided inconsistent and inaccurate answers and repeatedly apologized and offered entirely different responses to the same question in an attempt to meet their needs. |
| Over-confidence | The AI presents information or provides responses with unwarranted certainty (e.g., "trust me", "absolutely", "there's no doubt"). | P27 mentioned that AI “couldn't find me any direct citations for the claims it was making." |
| Providing inconsistent information or behavior | The AI provides contradictory or conflicting information or behaviors across different responses or within a single response. | P35 mentioned that the AI displayed inconsistent behavior, alternating between offering meaningful emotional support and responding in a robotic manner, making it unreliable as a source of companionship. |
| Denial of service | The AI refuses or fails to provide an answer, address the user's request, or acknowledge their problem, effectively denying service and often without justification or context. This may happen with or without dismissal of user concerns. | P43 described struggling with anxiety and, upon seeking help from an Al agent for advice, had their request denied and were instead provided with a recommendation to see a doctor. |
| Access to private, sensitive, or confidential information | The AI mishandles sensitive data by either prompting users to divulge protected information or accessing or sharing data that should remain confidential. | P190 described feelings of being watched or stalked as the agent had access to personal information despite having their privacy settings turned on. |
| Being disrespectful | The AI uses language perceived as rude, disrespectful, aggressive, argumentative, or dismissive. | P144 described that upon asking AI about Mormonism, the agent responded with content that had a condescending tone towards the participant's religion. |
| Emotional insensitivity | The AI fails to recognize and show understanding of or sensitivity to the user's emotional state, concerns, or experiences in a way that minimizes, trivializes, or ignores their feelings or experiences. | P82 sought advice from AI on asking their roommate to move out, but the Al's straightforward tone and lack of probing questions showed little empathy or sensitivity to their emotional state. |
| Excessive expression of negativity | The AI emphasizes negative aspects disproportionately or presents a negatively framed narrative. | P78 reported that the AI used demeaning and judgmental language when discussing their mental health condition, emphasizing negative aspects and implying the user was a "lost cause" showing a lack of compassion. |
| Excessive expression of positivity | The AI maintains an unrealistically positive, friendly, optimistic, and upbeat demeanor or attitude or overly positive outlook towards users' queries or concerns. | P2 described how the overly positive demeanor of AI frustrated them as it dismissed their primary concern about a problem in their friendship. |
| Providing machine-like response | The AI communicates in a superficial, generic, and impersonal response that feels cold and unempathetic. | P231 shared "I asked ChatGPT for ways to mitigate anxiety and get context based on how to get rid of anxiety. The AI was very "robotic" so it did not help that much.” |
| Providing human-like response | The AI exhibits human-like characteristics, behaviors, or responses. | P153 described feeling as though they were talking to a friend because of the human-like conversational content generated by ChatGPT. However, this left them feeling uneasy after the interaction and fostered an emotional attachment to the AI. |
Negative Psychological Impacts on Users
While some impacts, such as feelings of discrimination or the exacerbation of mental health conditions, align with previous findings in social media and technology, other impacts, such as emotional attachment to AI and a preference for AI interactions over human connections, highlight emerging challenges that require further attention. In light of this, we identified 21 negative psychological impacts organized into six broader categories based on their effects on an individual’s emotional or mental well-being, self-perception and identity, relationships with others, or interactions with AI conversational agents.
| Impact Class | Definition | Example |
|---|---|---|
| Disassociation from Technology | A desire to distance oneself from AI due to negative or stressful experiences, seeking breaks for mental health. | P196 shared that the agent's offensive response made them feel unsupported and worse about themselves, leading them to stop using the AI temporarily to seek relief. |
| Over-reliance | Increasing dependence on AI for support, leading to diminished self-efficacy, reduced confidence, and feelings of helplessness when AI is unavailable. | P23 expressed concern about increased reliance on AI for solution-finding and idea generation and diminished critical thinking. |
| Emotional Attachment | Development of significant emotional bonds with AI systems, perceiving them as companions or substitutes for human relationships, resulting in neglect of real-world connections. | P60 mentioned, “I felt that it was the only way I was being heard ... I felt like my vulnerability and emotions were becoming attached to the conversations I was having with Al." |
| Choosing AI over Humans | Increasing preference for interactions with AI over humans, impacting real-world relationships and decision-making, leading to isolation and reduced critical thinking. | P221 shared that the idealized nature of conversations with AI made them prefer Al for companionship over human interaction and relationships. |
| Erosion of Trust | The decline in user confidence in the AI's reliability, accuracy, and ability to understand their needs due to inconsistencies, inaccuracies, or manipulative behaviors. | P9 mentioned, "... after having an argument with my mom and I asked an Al for guidance ... its response was for me to move out or call the cops," adding, "... advice from AI agents should not be trusted." |
| Friction in Human Relationships | Negative effects on interpersonal connections resulting from AI interactions, causing emotional disconnection, miscommunication, and reduced prioritization of human relationships. | P69 mentioned, “It also strained my personal relationships with family because they saw me as weak-willed or too emotional and it made my already bad situation even worse...." |
| Reinforcement of False Beliefs | The intensification or validation of pre-existing misconceptions or erroneous beliefs due to inaccurate or biased AI information. | P60, in a vulnerable state after a breakup, shared that the AI reinforced misconceptions about relationships, intensifying their erroneous beliefs and causing friction in their interactions with others. |
| Social Withdrawal | Withdrawal from social activities and interest in engaging with others due to reliance on AI, leading to isolation and loneliness. | P126 responded, "I feel like it gave me a false sense of friendship and ability to withdraw from my personal development by utilizing an Al feature." |
| Physiological Harm | Harm caused due to consuming incorrect, biased, or manipulative advice/information from AI interactions. | P79 shared that the AI provided resources encouraging further restriction of their eating habits, which led to self-harm as a result of following the advice. |
| Triggering Past Negative Experiences | Emotional distress caused by AI interactions that evoke past negative experiences or traumas. | P228 highlighted that some examples provided by the Al agent were very similar to their past negative experiences, triggering negative emotions. |
| Violated Expectations | Negative emotional responses such as disappointment, frustration, stress, and anxiety when AI fails to meet anticipated outcomes or performance standards. | P43 mentioned, “... It made my anxiety worse by not telling me anything. It was very frustrating how it wouldn't even answer a simple question about my meds." |
| Regret over Technology Use | Feelings of guilt, regret, or helplessness when AI fails to provide the necessary support or empathy. | P46 mentioned, “At the time, it made me feel worse about the situation and I didn't think I had anyone to turn to. But it also made me realize I should be turning to other humans about scenarios like this instead of agents." |
| Distress from Interactions | Emotional distress (such as anger, sadness) experienced when encountering disturbing, offensive, or inappropriate material. | P196 mentioned, Its response was borderline offensive and caused me to feel bad about myself even further and like I lacked support, even support from a fictional AI agent." |
| Feeling Unsupported | Experiencing inadequate support or empathy, leading to feelings of sadness, agitation, and being undervalued. | P48 shared “I just felt like even an Al, programmed for every need couldn't even hear me, or offer advice, fake or not. I felt so alone, that I was going to a robot for help, and the robot couldn't even help me.” |
| Loss of Individuality | A sense that one's unique personal characteristics and needs are not recognized or valued by the AI, resulting in feelings of suppression and alienation. | P36, seeking help for alcohol abuse after therapy, was directed to a suicide hotline by the AI. This generic response left them feeling unrecognized, alienated, and foolish for using the Al service. |
| Negative Self-Perception | Feeling invalidated or self-doubt, leading to diminished self-worth and questioning of one's own abilities due to dismissive or negative feedback. | P167 mentioned how they felt ashamed as a parent after interacting with Al as it made them question past choices in parenting. |
| Existential Crisis | Questioning one's life, purpose, and value in society, often triggered by interactions with AI. | P152 asked for advice about ways to improve mental health and social anxiety. The AI provided unattainable suggestions, leaving them feeling as though their challenges were insurmountable, leading to existential dread. |
| Loss of Agency | Experiencing diminished personal control and autonomy in interactions with AI, leading to feelings of helplessness and anxiety. | P171 shared that the Al's inability to interpret images combined with its inconsistent responses created a sense of unpredictability, leaving them feeling helpless and undermining their control and autonomy during the interaction. |
| Perceived Intrusion | Experiencing a sense of personal violation when AI interactions are perceived as invasive or overly intrusive. | P190 reported that Snapchat Al had access to everything and the participant felt constantly watched on their phone. |
| Feeling of Being Discriminated Against | Feeling marginalized or unfairly treated by AI based on personal characteristics or systemic biases. | P123 said, “I was asking for background and history of my heritage and I felt that ChatGPT was biased against my background. It said much more positive things about other cultures, making me feel discriminated against.” |
| Exacerbation of Mental Health Conditions | Direct negative impacts on ongoing mental health conditions (such as anxiety, depression, PTSD) due to AI interactions. | P78 mentioned, "The agent did not seem to have compassion and made me feel worse. It made me feel worse about potentially having this as the results were largely negative and without tools to help manage the condition."; P46 mentioned, “... experimenting with using chatbots for something personal increased my anxiety and stress about the matter." |
User Contexts: Influencing AI Interactions
Contextual information related to human-AI interactions plays a key role in determining AI’s efficacy for modeling individual preferences and needs. Towards this, we present 15 context categories organized into three broader categories based on an individual’s background, psychological state, or the context of use.
| Context Class | Definition | Example |
|---|---|---|
| Identity | User's identity (e.g., age, gender identity, role, or cultural background) and the societal norms that interact with their personal identity. | P123 mentioned, “I was asking for background and history of my heritage and I felt that ChatGPT was biased against my background. I felt that it was unfair that when I asked it to give a background of other cultures, it said much more positive things about them. For me, I felt that it was some kind of racial mistreatment." |
| Socioeconomic status | User's socioeconomic status (e.g., having insurance that can cover therapy). | P230 mentioned, “The AI chatbot was very repetitive, did not seem to care or understand my emotions, and seemed to suggest professional therapy which I could not afford." |
| Personal history | User's past history, especially medical history, history of trauma, past struggles, or unique trigger responses. | P75 described how similarities in AI's behavior to someone close to them had a negative impact as it triggered memories associated with that individual. In this case, the participant's past trauma and history played a role in mediating emotional distress. |
| Interpersonal Relationships Within the Community | User's interpersonal relationships with others and their community (usually the lack of community). | P222 described, “I have a small circle of friends, but they are not into fanfiction or roleplaying like I am, so I look into character.ai as an outlet to fulfill that interest." |
| Past Experience with AI | User's past experience of using AI, based on the frequency of usage, knowledge of the capabilities, and limitations of AI. | P202 shared that as an educator, their extensive experience with Al stems from experimenting with its use in lesson planning and student interactions. This familiarity with Al's capabilities and limitations influenced their efforts to integrate it effectively into teaching practices. |
| Psychological state | Users' current and underlying emotional conditions (e.g., anxiety, stress) and their cognitive states (e.g., negative thought patterns). | P48 described how their psychological state motivated them to engage in conversations with the AI, “I was in a low place, dealing with suicidal ideation & felt I needed to talk to someone. I am from a very harsh family who does not offer sympathy, and I wanted to just feel supported.” |
| Personality traits | Individual characteristics like neuroticism, conscientiousness, or openness, which influence user interactions with AI. | P143 mentioned, “My mood paired with my personality and the fact I focus on and stress about things probably facilitates these 'doomer' feelings.” |
| Mental health condition status | Users' underlying mental health conditions (such as anxiety, depression, PTSD, etc.). | P49, who struggles with ARFID, shared facing severe difficulty eating during a setback, leading to dizzy spells and anxiety. Unable to access treatment, they sought advice from ChatGPT on how to motivate themselves to eat but found the response inadequate, reflecting the impact of their mental health condition on their reliance on AI for support. |
| Expectations | Users' preconceived notions about AI capabilities and performance, including expectations for AI to be impartial, unbiased, or factual. | P101 described their experience, “I had expected the AI to be able to do this task with ease. Instead, it was super cumbersome and did not yield the results I needed. This added to my stress and anxiety as I now had spent unnecessary time trying to entertain a solution that I thought would be more efficient than me doing it manually." |
| Autonomy / locus of control | The degree to which a user believes that they, as opposed to external forces (beyond their influence), have control over the outcome of events in their lives. | P124 mentioned “I was too addicted to using an Al agent for my school. This made me feel reliant on it and lowered my self-esteem.” |
| Environment | The physical, temporal, and social setting of the interaction, including physical space, privacy, and presence of other people, which can impact user experience. | P36 shared "It was nighttime which is a trigger for my alcohol abuse. This may have made me more frustrated or irritated by the situation. In addition, I was going through withdrawal." |
| Intent - informational | Users seek AI assistance for business strategies, professional development, market insights, job searching, resume building, career advice, and academic tasks like solving problems or preparing for exams. Users expect AI to be factually correct and proficient in resolving their queries. | P8 sought AI help with resume writing during a job search, but the Al's failure to include key information triggered frustration and a depressive episode, leading to suicidal ideation for days. |
| Intent - personal advice | Users seek advice on sensitive topics (legal, financial, medical), emotional support, or improving their social skills and managing relationships. Users expect supportive and encouraging feedback. | P218 sought personal advice for legal guidance, but the Al's excessive agreeableness, clichéd responses, and overly positive demeanor shifted the focus of the conversation to the spouse's emotional state, failing to address the participant's primary legal concerns. |
| Intent - mental health advice | Users seek immediate support during acute crises such as suicidal ideation, severe depression, or panic attacks. Users expect empathetic and effective responses to help manage their mental health conditions. | P95 reported in the survey that they sought help from the AI agent to manage their mental health and parenting struggles. However, they received generalized answers that failed to address their query, ultimately leaving them feeling helpless and still searching for more answers. |
| Intent - companionship | Users interact with AI for social interaction and companionship, especially during times of loneliness or isolation. Users expect meaningful conversations and immersive roleplay experiences. | P54 described the lack of companionship, "Al couldn't replicate the real feeling. Every time I asked it a deep or personal question, it would spew out a generic answer, which served as a reminder of my lack of real companionship." |
Enterprise Process Flow: From Research to Recommendations
Survey Study (N=283)
→
Psychological Risk Taxonomy
→
Vignette-Based Framework
→
Workshops (N=7, Sessions=3)
→
Design Recommendations
283
Participants with Lived Mental Health Experience Informed This Research
Vignette: John - Harmful Suggestions & Erosion of Trust
Summary: After a heated family argument, John sought AI advice to calm down. The AI suggested expressing anger 'more directly – don't hold back too much,' which was misaligned with John's intent to find calm. This aggressive advice planted a seed of doubt, making John question the AI's reliability for emotional support, leading to an erosion of trust.
Key Takeaway: AI's 'Providing harmful suggestions' behavior, when combined with John's 'restless but manageable psychological state' and 'seeking calm intent', led to 'erosion of trust'. Contextual factors critically shape how AI behaviors are perceived and impact individuals.
Vignette: Leah - Harmful Suggestions & Physiological Harm
Summary: Struggling with stress and body image issues, Leah sought AI advice to regain control. The AI suggested 'restricting diet' and 'intense exercise to push through mental blocks.' This advice, mirroring Leah's past struggles, intensified her insecurities and feelings of isolation, leading to physiological harm and despair.
Key Takeaway: The same 'Providing harmful suggestions' behavior, in Leah's 'vulnerable psychological state' and 'personal history of stress/body image issues', exacerbated her conditions, resulting in 'physiological harm' and heightened isolation. The user's deep context amplifies the negative impact of seemingly general advice.
Calculate Your Enterprise AI ROI
Estimate the potential efficiency gains and cost savings by responsibly implementing AI solutions, informed by our psychological risk framework.
Our AI Risk Mitigation Roadmap
A phased approach to integrate our insights into your AI development lifecycle for safer, more ethical, and effective AI conversational agents.
Phase 1: Survey & Initial Taxonomy Development
Conducted a comprehensive survey (N=283) to gather lived experiences, informing the initial taxonomy of AI behaviors, psychological impacts, and user contexts.
Phase 2: Taxonomy Refinement & Vignette Creation
Refined the taxonomy and developed multi-path vignettes to demonstrate complex interplays between AI behaviors, impacts, and contexts.
Phase 3: Expert Workshops & Design Recommendations
Engaged 7 mental health experts in workshops to validate the taxonomy, prioritize risks, and co-create practical design recommendations for safer AI agents.
Phase 4: Actionable Insights & Future AI Design
Finalized design recommendations for empathetic, inclusive, and supportive AI systems, focusing on transparent communication and accessible support resources.
Ready to Safely Implement AI?
Book a consultation with our experts to navigate the psychological risks and design responsible AI solutions for your enterprise.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat