CRS-Que: A User-centric Evaluation Framework for Conversational Recommender Systems

Yucheng Jin*, Li Chen, Wanling Cai, Xianglin Zhao

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review


An increasing number of recommendation systems try to enhance the overall user experience by incorporating conversational interaction. However, evaluating conversational recommender systems (CRSs) from the user’s perspective remains elusive. The GUI-based system evaluation criteria may be inadequate for their conversational counterparts. This article presents our proposed unifying framework, CRS-Que, to evaluate the user experience of CRSs. This new evaluation framework is developed based on ResQue, a popular user-centric evaluation framework for recommender systems. Additionally, it includes user experience metrics of conversation (e.g., understanding, response quality, humanness) under two dimensions of ResQue (i.e., Perceived Qualities and User Beliefs). Following the psychometric modeling method, we validate our framework by evaluating two conversational recommender systems in different scenarios: music exploration and mobile phone purchase. The results of the two studies support the validity and reliability of the constructs in our framework and reveal how conversation constructs and recommendation constructs interact and influence the overall user experience of the CRS. We believe this framework could help researchers conduct standardized user-centric research for conversational recommender systems and provide practitioners with insights into designing and evaluating a CRS from users’ perspectives.
Original languageEnglish
Article number2
Pages (from-to)1-34
Number of pages34
JournalACM Transactions on Recommender Systems
Early online date2 Nov 2023
Publication statusPublished - 7 Mar 2024

User-Defined Keywords

  • Recommender systems
  • conversational systems
  • user experience
  • music recommenders
  • questionnaire
  • user-centric evaluation


Dive into the research topics of 'CRS-Que: A User-centric Evaluation Framework for Conversational Recommender Systems'. Together they form a unique fingerprint.

Cite this