Task-Oriented User Evaluation on Critiquing-Based Recommendation Chatbots

Wanling Cai*, Yucheng Jin, Li Chen

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review

5 Citations (Scopus)


Dialogue-based conversational recommender systems (DCRSs) have become a new trend in recommender systems (RSs), allowing users to communicate with the system in natural language to facilitate feedback provision and product exploration. However, little work has been done to empirically study user perception of and interaction with such systems and, more importantly, how to best support users in providing feedback on the recommendation they receive. In this article, we aim to develop effective critiquing mechanisms for DCRS to improve its feedback elicitation process (i.e., allowing users to critique the current recommendation during the dialogue). Specifically, we have implemented three prototype systems featuring three different critiquing techniques, respectively, i.e., user-initiated critiquing, progressive system-suggested critiquing, and cascading system-suggested critiquing. We have then conducted two task-oriented user studies involving 292 subjects to evaluate the three prototypes. In particular, we consider two typical types of user tasks in RSs: basic recommendation task (BRT, i.e., looking for items according to the user's preferences), and exploration-oriented task (EOT, i.e., exploring different types of items). Results show that EOT stimulates more user interaction, while BRT results in higher user satisfaction. Moreover, when users perform EOT, the type of critiquing techniques is more likely to influence user perception and moderate the relationships between certain interaction metrics and users' perceived serendipity. The findings suggest effective critiquing techniques to enhance the interaction between users and the recommendation chatbot when the system makes recommendations for different purposes.

Original languageEnglish
Pages (from-to)354-366
Number of pages13
JournalIEEE Transactions on Human-Machine Systems
Issue number3
Early online date7 Jan 2022
Publication statusPublished - Jun 2022

Scopus Subject Areas

  • Human Factors and Ergonomics
  • Control and Systems Engineering
  • Signal Processing
  • Human-Computer Interaction
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

User-Defined Keywords

  • Chatbot
  • conversational recommender systems
  • critiquing
  • feedback elicitation
  • user evaluation


Dive into the research topics of 'Task-Oriented User Evaluation on Critiquing-Based Recommendation Chatbots'. Together they form a unique fingerprint.

Cite this