Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint

Hao Liu, Minshuo Chen, Siawpeng Er, Wenjing Liao, Tong Zhang, Tuo Zhao*

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

3 Citations (Scopus)

Abstract

Overparameterized neural networks enjoy great representation power on complex data, and more importantly yield sufficiently smooth output, which is crucial to their generalization and robustness. Most existing function approximation theories suggest that with sufficiently many parameters, neural networks can well approximate certain classes of functions in terms of the function value. The neural network themselves, however, can be highly nonsmooth. To bridge this gap, we take convolutional residual networks (ConvResNets) as an example, and prove that large ConvResNets can not only approximate a target function in terms of function value, but also exhibit sufficient first-order smoothness. Moreover, we extend our theory to approximating functions supported on a low-dimensional manifold. Our theory partially justifies the benefits of using deep and wide networks in practice. Numerical experiments on adversarial robust image classification are provided to support our theory.
Original languageEnglish
Title of host publicationProceedings of 39th International Conference on Machine Learning (ICML’22)
EditorsKamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, Sivan Sabato
PublisherML Research Press
Pages13669-13703
Number of pages35
Publication statusPublished - 17 Jul 2022
Event39th International Conference on Machine Learning, ICML 2022 - Baltimore Convention Center , Baltimore, Maryland, United States
Duration: 17 Jul 202223 Jul 2022
https://icml.cc/Conferences/2022

Publication series

NameProceedings of Machine Learning Research
Volume162
ISSN (Print)2640-3498

Conference

Conference39th International Conference on Machine Learning, ICML 2022
Country/TerritoryUnited States
CityBaltimore, Maryland
Period17/07/2223/07/22
Internet address

Fingerprint

Dive into the research topics of 'Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint'. Together they form a unique fingerprint.

Cite this