A user-centric evaluation framework for recommender systems

Pearl Pu*, Li Chen, Rong Hu

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

640 Citations (Scopus)

Abstract

This research was motivated by our interest in understanding the criteria for measuring the success of a recommender system from users' point view. Even though existing work has suggested a wide range of criteria, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a unifying evaluation framework, called ResQue (Recommender systems' Quality of user experience), which aimed at measuring the qualities of the recommended items, the system's usability, usefulness, interface and interaction qualities, users' satisfaction with the systems, and the influence of these qualities on users' behavioral intentions, including their intention to purchase the products recommended to them and return to the system. We also show the results of applying psychometric methods to validate the combined criteria using data collected from a large user survey. The outcomes of the validation are able to 1) support the consistency, validity and reliability of the selected criteria; and 2) explain the quality of user experience and the key determinants motivating users to adopt the recommender technology. The final model consists of thirty two questions and fifteen constructs, defining the essential qualities of an effective and satisfying recommender system, as well as providing practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources.

Original languageEnglish
Title of host publicationRecSys'11 - Proceedings of the 5th ACM Conference on Recommender Systems
Pages157-164
Number of pages8
DOIs
Publication statusPublished - 2011
Event5th ACM Conference on Recommender Systems, RecSys 2011 - Chicago, IL, United States
Duration: 23 Oct 201127 Oct 2011

Publication series

NameRecSys'11 - Proceedings of the 5th ACM Conference on Recommender Systems

Conference

Conference5th ACM Conference on Recommender Systems, RecSys 2011
Country/TerritoryUnited States
CityChicago, IL
Period23/10/1127/10/11

User-Defined Keywords

  • e-commerce recommender
  • post-study questionnaire
  • quality of user experience
  • recommender systems

Fingerprint

Dive into the research topics of 'A user-centric evaluation framework for recommender systems'. Together they form a unique fingerprint.

Cite this