A cross-domain study on the user experience of ChatGPT-based recommendations

Conversational recommender systems (CRS) allow users to express preferences and provide feedback through natural language. With the emergence of ChatGPT, there is growing interest in leveraging its capabilities to enhance user engagement and improve recommendation quality. Research suggests that well-crafted prompts are essential for ChatGPT to accurately interpret tasks and generate high-quality responses. Therefore, we define Prompt Guidance (PG) as offering an example of crafting queries (prompts) that clearly specify the context, constraints, and procedures for the recommendations users may seek. Offering Prompt Guidance (PG) to novices can improve the overall quality of conversations. Meanwhile, in the field of recommender systems, Recommendation Domain (RD) can influence user behavior and perception. This study explores how PG and RD interact to affect user experience (UX) in ChatGPT-based recommendations. To investigate this, we conducted an empirical study with 100 participants, analyzing dialogue data to uncover user intents and system actions. Using a within-subject (e.g., book vs. job recommendations) and between-subject (providing prompts vs. none) design, we evaluated UX under different experimental conditions. Results show that PG enhances explainability, adaptability, ease of use, and transparency. In terms of the RD factor, the system promotes higher user engagement, greater novelty, and a stronger intention to try suggested items in book recommendations compared to job recommendations. Overall, ChatGPT-based recommendations not only induce novel dialogue behaviors but also improve UX in multiple aspects, such as usefulness and alignment with user preferences. These findings provide valuable insights into designing CRS that leverage ChatGPT’s potential to deliver effective and engaging user experiences.