Deep Learning From Multiple Noisy Annotators as A Union

Hongxin Wei, Renchunzi Xie, Lei Feng*, Bo Han, Bo An

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


Crowdsourcing is a popular solution for large-scale data annotations. So far, various end-to-end deep learning methods have been proposed to improve the practical performance of learning from crowds. Despite their practical effectiveness, most of them have two major limitations--they do not hold learning consistency and suffer from computational inefficiency. In this article, we propose a novel method named UnionNet, which is not only theoretically consistent but also experimentally effective and efficient. Specifically, unlike existing methods that either fit a given label from each annotator independently or fuse all the labels into a reliable one, we concatenate the one-hot encoded vectors of crowdsourced labels provided by all the annotators, which takes all the labeling information as a union and coordinates multiple annotators. In this way, we can directly train an end-to-end deep neural network by maximizing the likelihood of this union with only a parametric transition matrix. We theoretically prove the learning consistency and experimentally show the effectiveness and efficiency of our proposed method.

Original languageEnglish
Number of pages11
JournalIEEE Transactions on Neural Networks and Learning Systems
Publication statusE-pub ahead of print - 29 Apr 2022

Scopus Subject Areas

  • Software
  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications

User-Defined Keywords

  • Annotators
  • crowdsourcing
  • Deep learning
  • Labeling
  • Learning systems
  • Neural networks
  • Noise measurement
  • noisy labels
  • Standards
  • Training
  • transition matrix


Dive into the research topics of 'Deep Learning From Multiple Noisy Annotators as A Union'. Together they form a unique fingerprint.

Cite this