A Holistic View of Label Noise Transition Matrix in Deep Learning and Beyond

Yong Lin, Renjie Pi*, Weizhong Zhang, Xiaobo Xia, Jiahui Gao, Xiao Zhou, Tongliang Liu, Bo Han

*Corresponding author for this work

Research output: Contribution to conferenceConference paperpeer-review

7 Citations (Scopus)

Abstract

In this paper, we explore learning statistically consistent classifiers under label noise by estimating the noise transition matrix (T).We first provide a holistic view of existing T-estimation methods including those with or without anchor point assumptions.We unified them into the Minimum Geometric Envelope Operator (MGEO) framework, which tries to find the smallest T (in terms of a certain metric) that elicits a convex hull to enclose the posteriors of all the training data.Although MGEO methods show appealing theoretical properties and empirical results, we find them prone to failing when the noisy posterior estimation is imperfect, which is inevitable in practice.Specifically, we show that MGEO methods are in-consistent even with infinite samples if the noisy posterior is not estimated accurately.In view of this, we make the first effort to address this issue by proposing a novel T-estimation framework via the lens of bilevel optimization, and term it RObust Bilevel OpTimzation (ROBOT).ROBOT paves a new road beyond MGEO framework, which enjoys strong theoretical properties: identifibility, consistency and finite-sample generalization guarantees.Notably, ROBOT neither requires the perfect posterior estimation nor assumes the existence of anchor points.We further theoretically demonstrate that ROBOT is more robust in the case where MGEO methods fail.Experimentally, our framework also shows superior performance across multiple benchmarks.Our code is released at https://github.com/pipilurj/ROBOT.

Original languageEnglish
Pages1-24
Number of pages24
Publication statusPublished - 1 May 2023
Event11th International Conference on Learning Representations, ICLR 2023 - Kigali, Rwanda
Duration: 1 May 20235 May 2023
https://iclr.cc/Conferences/2023
https://openreview.net/group?id=ICLR.cc/2023/Conference

Conference

Conference11th International Conference on Learning Representations, ICLR 2023
Country/TerritoryRwanda
CityKigali
Period1/05/235/05/23
Internet address

Scopus Subject Areas

  • Language and Linguistics
  • Computer Science Applications
  • Education
  • Linguistics and Language

Cite this