Variational imitation learning with diverse-quality demonstrations

Voot Tangkaratt; Bo Han; Mohammad Emtiyaz Khan; Masashi Sugiyama

Variational imitation learning with diverse-quality demonstrations

Voot Tangkaratt^*, Bo Han, Mohammad Emtiyaz Khan, Masashi Sugiyama

^*Corresponding author for this work

Department of Computer Science

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

18 Citations (Scopus)

Abstract

Learning from demonstrations can be challenging when the quality of demonstrations is diverse, and even more so when the quality is unknown and there is no additional information to estimate the quality. We propose a new method for imitation learning in such scenarios. We show that simple quality-estimation approaches might fail due to compounding error, and fix this issue by jointly estimating both the quality and reward using a variational approach. Our method is easy to implement within reinforcement-learning frameworks and also achieves state-of-The-Art performance on continuous-control benchmarks. Our work enables scalable and data-efficient imitation learning under more realistic settings than before.

Original language	English
Title of host publication	Proceedings of the 37th International Conference on Machine Learning, ICML 2020
Editors	Hal Daumé III, Aarti Singh
Publisher	ML Research Press
Pages	9349-9359
Number of pages	11
ISBN (Electronic)	9781713821120
Publication status	Published - Jul 2020
Event	37th International Conference on Machine Learning, ICML 2020 - Virtual, Online Duration: 13 Jul 2020 → 18 Jul 2020 https://proceedings.mlr.press/v119/

Publication series

Name	Proceedings of Machine Learning Research
Volume	119
ISSN (Print)	2640-3498

Conference

Conference	37th International Conference on Machine Learning, ICML 2020
Period	13/07/20 → 18/07/20
Internet address	https://proceedings.mlr.press/v119/

Scopus Subject Areas

Computational Theory and Mathematics
Human-Computer Interaction
Software

Access to Document

http://proceedings.mlr.press/v119/tangkaratt20a/tangkaratt20a.pdf

Cite this

Tangkaratt, V., Han, B., Khan, M. E., & Sugiyama, M. (2020). Variational imitation learning with diverse-quality demonstrations. In H. Daumé III, & A. Singh (Eds.), Proceedings of the 37th International Conference on Machine Learning, ICML 2020 (pp. 9349-9359). (Proceedings of Machine Learning Research; Vol. 119). ML Research Press. http://proceedings.mlr.press/v119/tangkaratt20a/tangkaratt20a.pdf

@inproceedings{03bf5748589e494bb199fa7db28e61bd,

title = "Variational imitation learning with diverse-quality demonstrations",

abstract = "Learning from demonstrations can be challenging when the quality of demonstrations is diverse, and even more so when the quality is unknown and there is no additional information to estimate the quality. We propose a new method for imitation learning in such scenarios. We show that simple quality-estimation approaches might fail due to compounding error, and fix this issue by jointly estimating both the quality and reward using a variational approach. Our method is easy to implement within reinforcement-learning frameworks and also achieves state-of-The-Art performance on continuous-control benchmarks. Our work enables scalable and data-efficient imitation learning under more realistic settings than before.",

author = "Voot Tangkaratt and Bo Han and Khan, {Mohammad Emtiyaz} and Masashi Sugiyama",

note = "Funding Information: We thank the anonymous reviewers for their constructive feedback. BH was supported by the Early Career Scheme (ECS) through the Research Grants Council of Hong Kong under Grant No.22200720, HKBU Tier-1 Start-up Grant and HKBU CSD Start-up Grant. MS was supported by KAKENHI 17H00757.; 37th International Conference on Machine Learning, ICML 2020 ; Conference date: 13-07-2020 Through 18-07-2020",

year = "2020",

month = jul,

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "ML Research Press",

pages = "9349--9359",

editor = "{Daum{\'e} III}, Hal and Aarti Singh",

booktitle = "Proceedings of the 37th International Conference on Machine Learning, ICML 2020",

url = "https://proceedings.mlr.press/v119/",

}

Tangkaratt, V, Han, B, Khan, ME & Sugiyama, M 2020, Variational imitation learning with diverse-quality demonstrations. in H Daumé III & A Singh (eds), Proceedings of the 37th International Conference on Machine Learning, ICML 2020. Proceedings of Machine Learning Research, vol. 119, ML Research Press, pp. 9349-9359, 37th International Conference on Machine Learning, ICML 2020, 13/07/20. <http://proceedings.mlr.press/v119/tangkaratt20a/tangkaratt20a.pdf>

Variational imitation learning with diverse-quality demonstrations. / Tangkaratt, Voot; Han, Bo; Khan, Mohammad Emtiyaz et al.
Proceedings of the 37th International Conference on Machine Learning, ICML 2020. ed. / Hal Daumé III; Aarti Singh. ML Research Press, 2020. p. 9349-9359 (Proceedings of Machine Learning Research; Vol. 119).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - Variational imitation learning with diverse-quality demonstrations

AU - Tangkaratt, Voot

AU - Han, Bo

AU - Khan, Mohammad Emtiyaz

AU - Sugiyama, Masashi

N1 - Funding Information: We thank the anonymous reviewers for their constructive feedback. BH was supported by the Early Career Scheme (ECS) through the Research Grants Council of Hong Kong under Grant No.22200720, HKBU Tier-1 Start-up Grant and HKBU CSD Start-up Grant. MS was supported by KAKENHI 17H00757.

PY - 2020/7

Y1 - 2020/7

N2 - Learning from demonstrations can be challenging when the quality of demonstrations is diverse, and even more so when the quality is unknown and there is no additional information to estimate the quality. We propose a new method for imitation learning in such scenarios. We show that simple quality-estimation approaches might fail due to compounding error, and fix this issue by jointly estimating both the quality and reward using a variational approach. Our method is easy to implement within reinforcement-learning frameworks and also achieves state-of-The-Art performance on continuous-control benchmarks. Our work enables scalable and data-efficient imitation learning under more realistic settings than before.

AB - Learning from demonstrations can be challenging when the quality of demonstrations is diverse, and even more so when the quality is unknown and there is no additional information to estimate the quality. We propose a new method for imitation learning in such scenarios. We show that simple quality-estimation approaches might fail due to compounding error, and fix this issue by jointly estimating both the quality and reward using a variational approach. Our method is easy to implement within reinforcement-learning frameworks and also achieves state-of-The-Art performance on continuous-control benchmarks. Our work enables scalable and data-efficient imitation learning under more realistic settings than before.

UR - https://proceedings.mlr.press/v119/tangkaratt20a.html

UR - http://www.scopus.com/inward/record.url?scp=85105288970&partnerID=8YFLogxK

M3 - Conference proceeding

AN - SCOPUS:85105288970

T3 - Proceedings of Machine Learning Research

SP - 9349

EP - 9359

BT - Proceedings of the 37th International Conference on Machine Learning, ICML 2020

A2 - Daumé III, Hal

A2 - Singh, Aarti

PB - ML Research Press

T2 - 37th International Conference on Machine Learning, ICML 2020

Y2 - 13 July 2020 through 18 July 2020

ER -

Variational imitation learning with diverse-quality demonstrations

Abstract

Publication series

Conference

Scopus Subject Areas

Access to Document

Other files and links

Fingerprint

Cite this