Co-teaching: Robust training of deep neural networks with extremely noisy labels

Bo Han; Quanming Yao; Xingrui Yu; Gang Niu; Miao Xu; Weihua Hu; Ivor W. Tsang; Masashi Sugiyama

doi:10.5555/3327757.3327944

Co-teaching: Robust training of deep neural networks with extremely noisy labels

Bo Han^*
, Quanming Yao^*
, Xingrui Yu
, Gang Niu
, Miao Xu
, Weihua Hu
, Ivor W. Tsang
, Masashi Sugiyama

^*Corresponding author for this work

Department of Computer Science

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

2040 Citations (Scopus)

Abstract

Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called “Co-teaching” for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.

Original language	English
Title of host publication	32nd Conference on Neural Information Processing Systems, NeurIPS 2018
Editors	Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi
Place of Publication	Red Hook
Publisher	ML Research Press
Pages	8536-8546
Number of pages	11
ISBN (Print)	9781510884472
DOIs	https://doi.org/10.5555/3327757.3327944
Publication status	Published - 3 Dec 2018
Event	32nd Conference on Neural Information Processing Systems, NeurIPS 2018 - Palais des Congrès de Montréal, Montreal, Canada Duration: 2 Dec 2018 → 8 Dec 2018 https://neurips.cc/Conferences/2018 (Conference website) https://proceedings.neurips.cc/paper/2018 (Conference proceeding)

Publication series

Name	Advances in Neural Information Processing Systems
Publisher	Neural Information Processing Systems Foundation
Volume	31
ISSN (Print)	1049-5258

Conference

Conference	32nd Conference on Neural Information Processing Systems, NeurIPS 2018
Abbreviated title	NeurIPS 2018
Country/Territory	Canada
City	Montreal
Period	2/12/18 → 8/12/18
Internet address	https://neurips.cc/Conferences/2018 (Conference website) https://proceedings.neurips.cc/paper/2018 (Conference proceeding)

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 9 Industry, Innovation, and Infrastructure

Access to Document

10.5555/3327757.3327944

Cite this

Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I. W., & Sugiyama, M. (2018). Co-teaching: Robust training of deep neural networks with extremely noisy labels. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, & N. Cesa-Bianchi (Eds.), 32nd Conference on Neural Information Processing Systems, NeurIPS 2018 (pp. 8536-8546). (Advances in Neural Information Processing Systems; Vol. 31). ML Research Press. https://doi.org/10.5555/3327757.3327944

Han, Bo ; Yao, Quanming ; Yu, Xingrui et al. / Co-teaching : Robust training of deep neural networks with extremely noisy labels. 32nd Conference on Neural Information Processing Systems, NeurIPS 2018. editor / Samy Bengio ; Hanna M. Wallach ; Hugo Larochelle ; Kristen Grauman ; Nicolò Cesa-Bianchi. Red Hook : ML Research Press, 2018. pp. 8536-8546 (Advances in Neural Information Processing Systems).

@inproceedings{e26775dbf93444cbb1cd364c1ac312b1,

title = "Co-teaching: Robust training of deep neural networks with extremely noisy labels",

abstract = "Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called “Co-teaching” for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.",

author = "Bo Han and Quanming Yao and Xingrui Yu and Gang Niu and Miao Xu and Weihua Hu and Tsang, \{Ivor W.\} and Masashi Sugiyama",

note = "MS was supported by JST CREST JPMJCR1403. IWT was supported by ARC FT130100746, DP180100106 and LP150100671. BH would like to thank the financial support from RIKEN-AIP. XRY was supported by NSFC Project No. 61671481 Publisher Copyright: {\textcopyright} 2018 Curran Associates Inc.All rights reserved.; 32nd Conference on Neural Information Processing Systems, NeurIPS 2018, NeurIPS 2018 ; Conference date: 02-12-2018 Through 08-12-2018",

year = "2018",

month = dec,

day = "3",

doi = "10.5555/3327757.3327944",

language = "English",

isbn = "9781510884472",

series = "Advances in Neural Information Processing Systems",

publisher = "ML Research Press",

pages = "8536--8546",

editor = "Samy Bengio and Wallach, \{Hanna M.\} and Hugo Larochelle and Kristen Grauman and Nicol{\`o} Cesa-Bianchi",

booktitle = "32nd Conference on Neural Information Processing Systems, NeurIPS 2018",

url = "https://neurips.cc/Conferences/2018, https://proceedings.neurips.cc/paper/2018",

}

Han, B, Yao, Q, Yu, X, Niu, G, Xu, M, Hu, W, Tsang, IW & Sugiyama, M 2018, Co-teaching: Robust training of deep neural networks with extremely noisy labels. in S Bengio, HM Wallach, H Larochelle, K Grauman & N Cesa-Bianchi (eds), 32nd Conference on Neural Information Processing Systems, NeurIPS 2018. Advances in Neural Information Processing Systems, vol. 31, ML Research Press, Red Hook, pp. 8536-8546, 32nd Conference on Neural Information Processing Systems, NeurIPS 2018, Montreal, Quebec, Canada, 2/12/18. https://doi.org/10.5555/3327757.3327944

Co-teaching: Robust training of deep neural networks with extremely noisy labels. / Han, Bo; Yao, Quanming; Yu, Xingrui et al.
32nd Conference on Neural Information Processing Systems, NeurIPS 2018. ed. / Samy Bengio; Hanna M. Wallach; Hugo Larochelle; Kristen Grauman; Nicolò Cesa-Bianchi. Red Hook: ML Research Press, 2018. p. 8536-8546 (Advances in Neural Information Processing Systems; Vol. 31).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - Co-teaching

T2 - 32nd Conference on Neural Information Processing Systems, NeurIPS 2018

AU - Han, Bo

AU - Yao, Quanming

AU - Yu, Xingrui

AU - Niu, Gang

AU - Xu, Miao

AU - Hu, Weihua

AU - Tsang, Ivor W.

AU - Sugiyama, Masashi

N1 - MS was supported by JST CREST JPMJCR1403. IWT was supported by ARC FT130100746, DP180100106 and LP150100671. BH would like to thank the financial support from RIKEN-AIP. XRY was supported by NSFC Project No. 61671481 Publisher Copyright: © 2018 Curran Associates Inc.All rights reserved.

PY - 2018/12/3

Y1 - 2018/12/3

N2 - Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called “Co-teaching” for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.

AB - Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called “Co-teaching” for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.

UR - https://proceedings.neurips.cc/paper_files/paper/2018/hash/a19744e268754fb0148b017647355b7b-Abstract.html

UR - https://www.scopus.com/pages/publications/85064825780

UR - https://dl.acm.org/doi/10.5555/3327757.3327944

U2 - 10.5555/3327757.3327944

DO - 10.5555/3327757.3327944

M3 - Conference proceeding

AN - SCOPUS:85064825780

SN - 9781510884472

T3 - Advances in Neural Information Processing Systems

SP - 8536

EP - 8546

BT - 32nd Conference on Neural Information Processing Systems, NeurIPS 2018

A2 - Bengio, Samy

A2 - Wallach, Hanna M.

A2 - Larochelle, Hugo

A2 - Grauman, Kristen

A2 - Cesa-Bianchi, Nicolò

PB - ML Research Press

CY - Red Hook

Y2 - 2 December 2018 through 8 December 2018

ER -

Han B, Yao Q, Yu X, Niu G, Xu M, Hu W et al. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Bengio S, Wallach HM, Larochelle H, Grauman K, Cesa-Bianchi N, editors, 32nd Conference on Neural Information Processing Systems, NeurIPS 2018. Red Hook: ML Research Press. 2018. p. 8536-8546. (Advances in Neural Information Processing Systems). doi: 10.5555/3327757.3327944

Co-teaching: Robust training of deep neural networks with extremely noisy labels

Abstract

Publication series

Conference

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this