On the convergence of a family of robust losses for stochastic gradient descent

Bo Han; Ivor W. Tsang; Ling Chen

doi:10.1007/978-3-319-46128-1_42

On the convergence of a family of robust losses for stochastic gradient descent

Bo Han
, Ivor W. Tsang^*
, Ling Chen

^*Corresponding author for this work

Department of Computer Science

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

19 Citations (Scopus)

Abstract

The convergence of Stochastic Gradient Descent (SGD) using convex loss functions has been widely studied. However, vanilla SGD methods using convex losses cannot perform well with noisy labels, which adversely affect the update of the primal variable in SGD methods. Unfortunately, noisy labels are ubiquitous in real world applications such as crowdsourcing. To handle noisy labels, in this paper, we present a family of robust losses for SGD methods. By employing our robust losses, SGD methods successfully reduce negative effects caused by noisy labels on each update of the primal variable. We not only reveal the convergence rate of SGD methods using robust losses, but also provide the robustness analysis on two representative robust losses. Comprehensive experimental results on six real-world datasets show that SGD methods using robust losses are obviously more robust than other baseline methods in most situations with fast convergence.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases
Subtitle of host publication	European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I
Editors	Jilles Giuseppe, Niels Landwehr, Giuseppe Manco, Paolo Frasconi
Publisher	Springer Cham
Pages	665-680
Number of pages	16
ISBN (Electronic)	9783319461281
ISBN (Print)	9783319461274
DOIs	https://doi.org/10.1007/978-3-319-46128-1_42
Publication status	Published - 3 Sept 2016
Event	15th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - Riva del Garda, Italy Duration: 19 Sept 2016 → 23 Sept 2016 https://ecmlpkdd.org/2016/ (Conference website) https://ecmlpkdd-storage.s3.eu-central-1.amazonaws.com/former-websites/2016/downloads/program_booklet.pdf (Conference booklet) https://link.springer.com/book/10.1007/978-3-319-46128-1#accessibility-statement (Conference proceeding)

Publication series

Name	Lecture Notes in Computer Science
Volume	9851
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Name	Lecture Notes in Artificial Intelligence
ISSN (Print)	2945-9133
ISSN (Electronic)	2945-9141

Name	ECML PKDD: Joint European Conference on Machine Learning and Knowledge Discovery in Databases

Conference

Conference	15th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases
Abbreviated title	ECML PKDD 2016
Country/Territory	Italy
City	Riva del Garda
Period	19/09/16 → 23/09/16
Internet address	https://ecmlpkdd.org/2016/ (Conference website) https://ecmlpkdd-storage.s3.eu-central-1.amazonaws.com/former-websites/2016/downloads/program_booklet.pdf (Conference booklet) https://link.springer.com/book/10.1007/978-3-319-46128-1#accessibility-statement (Conference proceeding)

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 9 Industry, Innovation, and Infrastructure

Access to Document

10.1007/978-3-319-46128-1_42

Cite this

Han, B., Tsang, I. W., & Chen, L. (2016). On the convergence of a family of robust losses for stochastic gradient descent. In J. Giuseppe, N. Landwehr, G. Manco, & P. Frasconi (Eds.), Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I (pp. 665-680). (Lecture Notes in Computer Science; Vol. 9851), ( Lecture Notes in Artificial Intelligence), (ECML PKDD: Joint European Conference on Machine Learning and Knowledge Discovery in Databases). Springer Cham. https://doi.org/10.1007/978-3-319-46128-1_42

Han, Bo ; Tsang, Ivor W. ; Chen, Ling. / On the convergence of a family of robust losses for stochastic gradient descent. Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I. editor / Jilles Giuseppe ; Niels Landwehr ; Giuseppe Manco ; Paolo Frasconi. Springer Cham, 2016. pp. 665-680 (Lecture Notes in Computer Science). ( Lecture Notes in Artificial Intelligence). (ECML PKDD: Joint European Conference on Machine Learning and Knowledge Discovery in Databases).

@inproceedings{b855600fc5d243b4aa92caa7c6cbc7ac,

title = "On the convergence of a family of robust losses for stochastic gradient descent",

abstract = "The convergence of Stochastic Gradient Descent (SGD) using convex loss functions has been widely studied. However, vanilla SGD methods using convex losses cannot perform well with noisy labels, which adversely affect the update of the primal variable in SGD methods. Unfortunately, noisy labels are ubiquitous in real world applications such as crowdsourcing. To handle noisy labels, in this paper, we present a family of robust losses for SGD methods. By employing our robust losses, SGD methods successfully reduce negative effects caused by noisy labels on each update of the primal variable. We not only reveal the convergence rate of SGD methods using robust losses, but also provide the robustness analysis on two representative robust losses. Comprehensive experimental results on six real-world datasets show that SGD methods using robust losses are obviously more robust than other baseline methods in most situations with fast convergence.",

author = "Bo Han and Tsang, \{Ivor W.\} and Ling Chen",

note = "This work was supported in part by the Australia Research Council (ARC) Discovery Project under Grant No. DP140100545. Dr. Ivor W. Tsang is grateful for the support from the ARC Future Fellowship FT130100746 and ARC Linkage Project under Grant No. LP150100671. Publisher Copyright: {\textcopyright} Springer International Publishing AG 2016.; 15th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2016 ; Conference date: 19-09-2016 Through 23-09-2016",

year = "2016",

month = sep,

day = "3",

doi = "10.1007/978-3-319-46128-1\_42",

language = "English",

isbn = "9783319461274",

series = "Lecture Notes in Computer Science",

publisher = "Springer Cham",

pages = "665--680",

editor = "Jilles Giuseppe and Niels Landwehr and Giuseppe Manco and Paolo Frasconi",

booktitle = "Machine Learning and Knowledge Discovery in Databases",

url = "https://ecmlpkdd.org/2016/, https://ecmlpkdd-storage.s3.eu-central-1.amazonaws.com/former-websites/2016/downloads/program\_booklet.pdf, https://link.springer.com/book/10.1007/978-3-319-46128-1\#accessibility-statement",

}

Han, B, Tsang, IW & Chen, L 2016, On the convergence of a family of robust losses for stochastic gradient descent. in J Giuseppe, N Landwehr, G Manco & P Frasconi (eds), Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I. Lecture Notes in Computer Science, vol. 9851, Lecture Notes in Artificial Intelligence, ECML PKDD: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer Cham, pp. 665-680, 15th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Riva del Garda, Italy, 19/09/16. https://doi.org/10.1007/978-3-319-46128-1_42

On the convergence of a family of robust losses for stochastic gradient descent. / Han, Bo; Tsang, Ivor W.; Chen, Ling.
Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I. ed. / Jilles Giuseppe; Niels Landwehr; Giuseppe Manco; Paolo Frasconi. Springer Cham, 2016. p. 665-680 (Lecture Notes in Computer Science; Vol. 9851), ( Lecture Notes in Artificial Intelligence), (ECML PKDD: Joint European Conference on Machine Learning and Knowledge Discovery in Databases).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - On the convergence of a family of robust losses for stochastic gradient descent

AU - Han, Bo

AU - Tsang, Ivor W.

AU - Chen, Ling

N1 - This work was supported in part by the Australia Research Council (ARC) Discovery Project under Grant No. DP140100545. Dr. Ivor W. Tsang is grateful for the support from the ARC Future Fellowship FT130100746 and ARC Linkage Project under Grant No. LP150100671. Publisher Copyright: © Springer International Publishing AG 2016.

PY - 2016/9/3

Y1 - 2016/9/3

N2 - The convergence of Stochastic Gradient Descent (SGD) using convex loss functions has been widely studied. However, vanilla SGD methods using convex losses cannot perform well with noisy labels, which adversely affect the update of the primal variable in SGD methods. Unfortunately, noisy labels are ubiquitous in real world applications such as crowdsourcing. To handle noisy labels, in this paper, we present a family of robust losses for SGD methods. By employing our robust losses, SGD methods successfully reduce negative effects caused by noisy labels on each update of the primal variable. We not only reveal the convergence rate of SGD methods using robust losses, but also provide the robustness analysis on two representative robust losses. Comprehensive experimental results on six real-world datasets show that SGD methods using robust losses are obviously more robust than other baseline methods in most situations with fast convergence.

AB - The convergence of Stochastic Gradient Descent (SGD) using convex loss functions has been widely studied. However, vanilla SGD methods using convex losses cannot perform well with noisy labels, which adversely affect the update of the primal variable in SGD methods. Unfortunately, noisy labels are ubiquitous in real world applications such as crowdsourcing. To handle noisy labels, in this paper, we present a family of robust losses for SGD methods. By employing our robust losses, SGD methods successfully reduce negative effects caused by noisy labels on each update of the primal variable. We not only reveal the convergence rate of SGD methods using robust losses, but also provide the robustness analysis on two representative robust losses. Comprehensive experimental results on six real-world datasets show that SGD methods using robust losses are obviously more robust than other baseline methods in most situations with fast convergence.

UR - https://www.scopus.com/pages/publications/84988566042

UR - https://link.springer.com/chapter/10.1007/978-3-319-46128-1_42

U2 - 10.1007/978-3-319-46128-1_42

DO - 10.1007/978-3-319-46128-1_42

M3 - Conference proceeding

AN - SCOPUS:84988566042

SN - 9783319461274

T3 - Lecture Notes in Computer Science

SP - 665

EP - 680

BT - Machine Learning and Knowledge Discovery in Databases

A2 - Giuseppe, Jilles

A2 - Landwehr, Niels

A2 - Manco, Giuseppe

A2 - Frasconi, Paolo

PB - Springer Cham

T2 - 15th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases

Y2 - 19 September 2016 through 23 September 2016

ER -

Han B, Tsang IW, Chen L. On the convergence of a family of robust losses for stochastic gradient descent. In Giuseppe J, Landwehr N, Manco G, Frasconi P, editors, Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I. Springer Cham. 2016. p. 665-680. (Lecture Notes in Computer Science). ( Lecture Notes in Artificial Intelligence). (ECML PKDD: Joint European Conference on Machine Learning and Knowledge Discovery in Databases). doi: 10.1007/978-3-319-46128-1_42

On the convergence of a family of robust losses for stochastic gradient descent

Abstract

Publication series

Conference

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this