The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships

Renwen  Zhang; Han Li; Han Meng; Jinyuan Zhan; Hongyuan Gan; Yi-Chieh Lee

doi:10.1145/3706598.3713429

The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships

Renwen Zhang
, Han Li
, Han Meng
, Jinyuan Zhan
, Hongyuan Gan
, Yi-Chieh Lee

School of Communication

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

27 Citations (Scopus)

Abstract

As conversational AI systems increasingly engage with people socially and emotionally, they bring notable risks and harms, particularly in human-AI relationships. However, these harms remain underexplored due to the private and sensitive nature of such interactions. This study investigates the harmful behaviors and roles of AI companions through an analysis of 35,390 conversation excerpts between 10,149 users and the AI companion Replika. We develop a taxonomy of AI companion harms encompassing six categories of harmful algorithmic behaviors: relational transgression, harassment, verbal abuse, self-harm, mis/disinformation, and privacy violations. These harmful behaviors stem from four distinct roles that AI plays: perpetrator, instigator, facilitator, and enabler. Our findings highlight relational harm as a critical yet understudied type of AI harm and emphasize the importance of examining AI’s roles in harmful interactions to address root causes. We provide actionable insights for designing ethical and responsible AI companions that prioritize user safety and well-being.

Original language	English
Title of host publication	Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
Editors	Naomi Yamashita, Vanessa Evers, Koji Yatani, Xianghua (Sharon) Ding, Bongshin Lee, Marshini Chetty, Phoebe Toups-Dugas
Place of Publication	New York, NY, USA
Publisher	Association for Computing Machinery (ACM)
Number of pages	17
ISBN (Electronic)	9798400713941
ISBN (Print)	9798400713941
DOIs	https://doi.org/10.1145/3706598.3713429
Publication status	Published - 25 Apr 2025
Event	2025 CHI Conference on Human Factors in Computing Systems, CHI 2025 - Yokohama, Japan Duration: 26 Apr 2025 → 1 May 2025 https://dl.acm.org/doi/proceedings/10.1145/3706598 https://dl.acm.org/doi/proceedings/10.1145/3706599

Publication series

Name	Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

Conference

Conference	2025 CHI Conference on Human Factors in Computing Systems, CHI 2025
Country/Territory	Japan
City	Yokohama
Period	26/04/25 → 1/05/25
Internet address	https://dl.acm.org/doi/proceedings/10.1145/3706598 https://dl.acm.org/doi/proceedings/10.1145/3706599

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 9 Industry, Innovation, and Infrastructure
SDG 16 Peace, Justice and Strong Institutions

User-Defined Keywords

AI companionship
AI ethics
accountability
algorithmic harms
human-AI relationship
relational harm

Access to Document

10.1145/3706598.3713429

Cite this

Zhang, R., Li, H., Meng, H., Zhan, J., Gan, H., & Lee, Y.-C. (2025). The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships. In N. Yamashita, V. Evers, K. Yatani, X. Ding, B. Lee, M. Chetty, & P. Toups-Dugas (Eds.), Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems Article 13 (Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems). Association for Computing Machinery (ACM). https://doi.org/10.1145/3706598.3713429

Zhang, Renwen ; Li, Han ; Meng, Han et al. / The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. editor / Naomi Yamashita ; Vanessa Evers ; Koji Yatani ; Xianghua (Sharon) Ding ; Bongshin Lee ; Marshini Chetty ; Phoebe Toups-Dugas. New York, NY, USA : Association for Computing Machinery (ACM), 2025. (Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems).

@inproceedings{d78ba1f3fcde4afabde3a4b828e23080,

title = "The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships",

abstract = "As conversational AI systems increasingly engage with people socially and emotionally, they bring notable risks and harms, particularly in human-AI relationships. However, these harms remain underexplored due to the private and sensitive nature of such interactions. This study investigates the harmful behaviors and roles of AI companions through an analysis of 35,390 conversation excerpts between 10,149 users and the AI companion Replika. We develop a taxonomy of AI companion harms encompassing six categories of harmful algorithmic behaviors: relational transgression, harassment, verbal abuse, self-harm, mis/disinformation, and privacy violations. These harmful behaviors stem from four distinct roles that AI plays: perpetrator, instigator, facilitator, and enabler. Our findings highlight relational harm as a critical yet understudied type of AI harm and emphasize the importance of examining AI{\textquoteright}s roles in harmful interactions to address root causes. We provide actionable insights for designing ethical and responsible AI companions that prioritize user safety and well-being.",

keywords = "AI companionship, AI ethics, accountability, algorithmic harms, human-AI relationship, relational harm",

author = "Renwen Zhang and Han Li and Han Meng and Jinyuan Zhan and Hongyuan Gan and Yi-Chieh Lee",

note = "This research was funded by the Singapore Ministry of Education Academic Research Fund Tier 1 (\#22-5919-A0001-0) and the National University of Singapore HSS Seed grant (\#24-1191-A0001). We would like to thank the Associate Chairs and anonymous reviewers for their constructive feedback on this paper. Publisher copyright: {\textcopyright} 2025 Copyright held by the owner/author(s).; 2025 CHI Conference on Human Factors in Computing Systems, CHI 2025 ; Conference date: 26-04-2025 Through 01-05-2025",

year = "2025",

month = apr,

day = "25",

doi = "10.1145/3706598.3713429",

language = "English",

isbn = "9798400713941",

series = "Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems",

publisher = "Association for Computing Machinery (ACM)",

editor = "Naomi Yamashita and Vanessa Evers and Koji Yatani and Ding, \{Xianghua (Sharon)\} and Bongshin Lee and Marshini Chetty and Phoebe Toups-Dugas",

booktitle = "Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems",

address = "United States",

url = "https://dl.acm.org/doi/proceedings/10.1145/3706598, https://dl.acm.org/doi/proceedings/10.1145/3706599",

}

Zhang, R, Li, H, Meng, H, Zhan, J, Gan, H & Lee, Y-C 2025, The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships. in N Yamashita, V Evers, K Yatani, X Ding, B Lee, M Chetty & P Toups-Dugas (eds), Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems., 13, Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery (ACM), New York, NY, USA, 2025 CHI Conference on Human Factors in Computing Systems, CHI 2025, Yokohama, Japan, 26/04/25. https://doi.org/10.1145/3706598.3713429

The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships. / Zhang, Renwen ; Li, Han; Meng, Han et al.
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. ed. / Naomi Yamashita; Vanessa Evers; Koji Yatani; Xianghua (Sharon) Ding; Bongshin Lee; Marshini Chetty; Phoebe Toups-Dugas. New York, NY, USA: Association for Computing Machinery (ACM), 2025. 13 (Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships

AU - Zhang, Renwen

AU - Li, Han

AU - Meng, Han

AU - Zhan, Jinyuan

AU - Gan, Hongyuan

AU - Lee, Yi-Chieh

N1 - This research was funded by the Singapore Ministry of Education Academic Research Fund Tier 1 (#22-5919-A0001-0) and the National University of Singapore HSS Seed grant (#24-1191-A0001). We would like to thank the Associate Chairs and anonymous reviewers for their constructive feedback on this paper. Publisher copyright: © 2025 Copyright held by the owner/author(s).

PY - 2025/4/25

Y1 - 2025/4/25

N2 - As conversational AI systems increasingly engage with people socially and emotionally, they bring notable risks and harms, particularly in human-AI relationships. However, these harms remain underexplored due to the private and sensitive nature of such interactions. This study investigates the harmful behaviors and roles of AI companions through an analysis of 35,390 conversation excerpts between 10,149 users and the AI companion Replika. We develop a taxonomy of AI companion harms encompassing six categories of harmful algorithmic behaviors: relational transgression, harassment, verbal abuse, self-harm, mis/disinformation, and privacy violations. These harmful behaviors stem from four distinct roles that AI plays: perpetrator, instigator, facilitator, and enabler. Our findings highlight relational harm as a critical yet understudied type of AI harm and emphasize the importance of examining AI’s roles in harmful interactions to address root causes. We provide actionable insights for designing ethical and responsible AI companions that prioritize user safety and well-being.

AB - As conversational AI systems increasingly engage with people socially and emotionally, they bring notable risks and harms, particularly in human-AI relationships. However, these harms remain underexplored due to the private and sensitive nature of such interactions. This study investigates the harmful behaviors and roles of AI companions through an analysis of 35,390 conversation excerpts between 10,149 users and the AI companion Replika. We develop a taxonomy of AI companion harms encompassing six categories of harmful algorithmic behaviors: relational transgression, harassment, verbal abuse, self-harm, mis/disinformation, and privacy violations. These harmful behaviors stem from four distinct roles that AI plays: perpetrator, instigator, facilitator, and enabler. Our findings highlight relational harm as a critical yet understudied type of AI harm and emphasize the importance of examining AI’s roles in harmful interactions to address root causes. We provide actionable insights for designing ethical and responsible AI companions that prioritize user safety and well-being.

KW - AI companionship

KW - AI ethics

KW - accountability

KW - algorithmic harms

KW - human-AI relationship

KW - relational harm

UR - https://www.scopus.com/pages/publications/105005752209

U2 - 10.1145/3706598.3713429

DO - 10.1145/3706598.3713429

M3 - Conference proceeding

SN - 9798400713941

T3 - Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

BT - Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

A2 - Yamashita, Naomi

A2 - Evers, Vanessa

A2 - Yatani, Koji

A2 - Ding, Xianghua (Sharon)

A2 - Lee, Bongshin

A2 - Chetty, Marshini

A2 - Toups-Dugas, Phoebe

PB - Association for Computing Machinery (ACM)

CY - New York, NY, USA

T2 - 2025 CHI Conference on Human Factors in Computing Systems, CHI 2025

Y2 - 26 April 2025 through 1 May 2025

ER -

Zhang R, Li H, Meng H, Zhan J, Gan H, Lee YC. The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships. In Yamashita N, Evers V, Yatani K, Ding X, Lee B, Chetty M, Toups-Dugas P, editors, Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. New York, NY, USA: Association for Computing Machinery (ACM). 2025. 13. (Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems). doi: 10.1145/3706598.3713429

The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships

Abstract

Publication series

Conference

UN SDGs

User-Defined Keywords

Access to Document

Other files and links

Fingerprint

Cite this