User Bias in Beyond-Accuracy Measurement of Recommendation Algorithms

Ningxia Wang, Li Chen

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

12 Citations (Scopus)


There are various biases in recommender systems. Recognizing biases, as well as unfairness caused by problematic biases, is the first step of system optimization. Related studies on algorithmic biases are mainly from the perspective of either items or users. For the latter (we call it "algorithmic user bias"), existing works have considered algorithms' accuracy performances measured by accuracy metrics like RMSE. However, algorithmic user biases in beyond-accuracy measurements have rarely been studied, even though beyond-accuracy oriented recommendation algorithms have been increasingly investigated, with the purpose of breaking through the personalization limits of traditional accuracy-oriented algorithms (such as the typical "filter bubble"phenomenon). To fill in the research gap, in this work, we employ a large-scale survey dataset collected from a commercial platform, in which more than 11,000 users' ratings on the recommendation's 5 performance objectives (i.e., relevance, diversity, novelty, unexpectedness, and serendipity) and 8 kinds of user characteristics (i.e., gender, age, big-5 personality traits, and curiosity) are available. We study user biases of four algorithms (i.e., HOT, Rel-CF, Nov-CF, and Ser-CF) in terms of those five measurements between user groups of the eight user characteristics. We further look into users' behavior patterns like the preference of using more positive ratings, in order to interpret the observed biases. Finally, based on the observed algorithmic user bias and users' behavior patterns, we analyze the possible factors leading to the biases and recognize problematic biases that may lead to unfairness.

Original languageEnglish
Title of host publicationRecSys 2021 - 15th ACM Conference on Recommender Systems
PublisherAssociation for Computing Machinery (ACM)
Number of pages10
ISBN (Electronic)9781450384582
Publication statusPublished - 27 Sept 2021
Event15th ACM Conference on Recommender Systems, RecSys 2021 - Virtual, Online, Netherlands
Duration: 27 Sept 20211 Oct 2021

Publication series

NameProceeings of ACM Conference on Recommender Systems


Conference15th ACM Conference on Recommender Systems, RecSys 2021
CityVirtual, Online

Scopus Subject Areas

  • Control and Systems Engineering
  • Computer Networks and Communications
  • Hardware and Architecture

User-Defined Keywords

  • Algorithmic bias
  • Beyond-accuracy objectives
  • Curiosity
  • Fairness
  • Personality
  • Recommender systems
  • Serendipity
  • User bias


Dive into the research topics of 'User Bias in Beyond-Accuracy Measurement of Recommendation Algorithms'. Together they form a unique fingerprint.

Cite this