TY - JOUR

T1 - Simultaneous neural network approximation for smooth functions

AU - Hon, Sean

AU - Yang, Haizhao

N1 - Funding Information:
The authors would like to thank the editor and the two anonymous reviewers for their valuable comments and constructive suggestions, which have greatly improved this paper. S. Hon was partially supported by the Hong Kong RGC under Grant 22300921 , a start-up grant from the Croucher Foundation, and a Tier 2 Start-up Grant from the Hong Kong Baptist University . H. Yang was partially supported by the US National Science Foundation under award DMS-1945029 .
Publisher Copyright:
© 2022 Elsevier Ltd

PY - 2022/10

Y1 - 2022/10

N2 - We establish in this work approximation results of deep neural networks for smooth functions measured in Sobolev norms, motivated by recent development of numerical solvers for partial differential equations using deep neural networks. Our approximation results are nonasymptotic in the sense that the error bounds are explicitly characterized in terms of both the width and depth of the networks simultaneously with all involved constants explicitly determined. Namely, for f∈Cs([0,1]d), we show that deep ReLU networks of width O(NlogN) and of depth O(LlogL) can achieve a nonasymptotic approximation rate of O(N−2(s−1)/dL−2(s−1)/d) with respect to the W1,p([0,1]d) norm for p∈[1,∞). If either the ReLU function or its square is applied as activation functions to construct deep neural networks of width O(NlogN) and of depth O(LlogL) to approximate f∈Cs([0,1]d), the approximation rate is O(N−2(s−n)/dL−2(s−n)/d) with respect to the Wn,p([0,1]d) norm for p∈[1,∞).

AB - We establish in this work approximation results of deep neural networks for smooth functions measured in Sobolev norms, motivated by recent development of numerical solvers for partial differential equations using deep neural networks. Our approximation results are nonasymptotic in the sense that the error bounds are explicitly characterized in terms of both the width and depth of the networks simultaneously with all involved constants explicitly determined. Namely, for f∈Cs([0,1]d), we show that deep ReLU networks of width O(NlogN) and of depth O(LlogL) can achieve a nonasymptotic approximation rate of O(N−2(s−1)/dL−2(s−1)/d) with respect to the W1,p([0,1]d) norm for p∈[1,∞). If either the ReLU function or its square is applied as activation functions to construct deep neural networks of width O(NlogN) and of depth O(LlogL) to approximate f∈Cs([0,1]d), the approximation rate is O(N−2(s−n)/dL−2(s−n)/d) with respect to the Wn,p([0,1]d) norm for p∈[1,∞).

KW - Approximation theory

KW - Deep neural networks

KW - ReLU activation functions

KW - Sobolev norm

UR - http://www.scopus.com/inward/record.url?scp=85134874925&partnerID=8YFLogxK

U2 - 10.1016/j.neunet.2022.06.040

DO - 10.1016/j.neunet.2022.06.040

M3 - Article

AN - SCOPUS:85134874925

VL - 154

SP - 152

EP - 164

JO - Neural Networks

JF - Neural Networks

SN - 0893-6080

ER -