Deep Learning From Multiple Noisy Annotators as A Union

Hongxin Wei, Renchunzi Xie, Lei Feng*, Bo Han, Bo An

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review

13 Citations (Scopus)


Crowdsourcing is a popular solution for large-scale data annotations. So far, various end-to-end deep learning methods have been proposed to improve the practical performance of learning from crowds. Despite their practical effectiveness, most of them have two major limitations--they do not hold learning consistency and suffer from computational inefficiency. In this article, we propose a novel method named UnionNet, which is not only theoretically consistent but also experimentally effective and efficient. Specifically, unlike existing methods that either fit a given label from each annotator independently or fuse all the labels into a reliable one, we concatenate the one-hot encoded vectors of crowdsourced labels provided by all the annotators, which takes all the labeling information as a union and coordinates multiple annotators. In this way, we can directly train an end-to-end deep neural network by maximizing the likelihood of this union with only a parametric transition matrix. We theoretically prove the learning consistency and experimentally show the effectiveness and efficiency of our proposed method.

Original languageEnglish
Pages (from-to)10552-10562
Number of pages11
JournalIEEE Transactions on Neural Networks and Learning Systems
Issue number12
Early online date29 Apr 2022
Publication statusPublished - Dec 2023

Scopus Subject Areas

  • Software
  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications

User-Defined Keywords

  • Annotators
  • crowdsourcing
  • Deep learning
  • Labeling
  • Learning systems
  • Neural networks
  • Noise measurement
  • noisy labels
  • Standards
  • Training
  • transition matrix


Dive into the research topics of 'Deep Learning From Multiple Noisy Annotators as A Union'. Together they form a unique fingerprint.

Cite this