TY - JOUR
T1 - Simultaneous neural network approximation for smooth functions
AU - Hon, Sean
AU - Yang, Haizhao
N1 - Funding Information:
The authors would like to thank the editor and the two anonymous reviewers for their valuable comments and constructive suggestions, which have greatly improved this paper. S. Hon was partially supported by the Hong Kong RGC under Grant 22300921 , a start-up grant from the Croucher Foundation, and a Tier 2 Start-up Grant from the Hong Kong Baptist University . H. Yang was partially supported by the US National Science Foundation under award DMS-1945029 .
Publisher Copyright:
© 2022 Elsevier Ltd
PY - 2022/10
Y1 - 2022/10
N2 - We establish in this work approximation results of deep neural networks for smooth functions measured in Sobolev norms, motivated by recent development of numerical solvers for partial differential equations using deep neural networks. Our approximation results are nonasymptotic in the sense that the error bounds are explicitly characterized in terms of both the width and depth of the networks simultaneously with all involved constants explicitly determined. Namely, for f∈Cs([0,1]d), we show that deep ReLU networks of width O(NlogN) and of depth O(LlogL) can achieve a nonasymptotic approximation rate of O(N−2(s−1)/dL−2(s−1)/d) with respect to the W1,p([0,1]d) norm for p∈[1,∞). If either the ReLU function or its square is applied as activation functions to construct deep neural networks of width O(NlogN) and of depth O(LlogL) to approximate f∈Cs([0,1]d), the approximation rate is O(N−2(s−n)/dL−2(s−n)/d) with respect to the Wn,p([0,1]d) norm for p∈[1,∞).
AB - We establish in this work approximation results of deep neural networks for smooth functions measured in Sobolev norms, motivated by recent development of numerical solvers for partial differential equations using deep neural networks. Our approximation results are nonasymptotic in the sense that the error bounds are explicitly characterized in terms of both the width and depth of the networks simultaneously with all involved constants explicitly determined. Namely, for f∈Cs([0,1]d), we show that deep ReLU networks of width O(NlogN) and of depth O(LlogL) can achieve a nonasymptotic approximation rate of O(N−2(s−1)/dL−2(s−1)/d) with respect to the W1,p([0,1]d) norm for p∈[1,∞). If either the ReLU function or its square is applied as activation functions to construct deep neural networks of width O(NlogN) and of depth O(LlogL) to approximate f∈Cs([0,1]d), the approximation rate is O(N−2(s−n)/dL−2(s−n)/d) with respect to the Wn,p([0,1]d) norm for p∈[1,∞).
KW - Approximation theory
KW - Deep neural networks
KW - ReLU activation functions
KW - Sobolev norm
UR - http://www.scopus.com/inward/record.url?scp=85134874925&partnerID=8YFLogxK
U2 - 10.1016/j.neunet.2022.06.040
DO - 10.1016/j.neunet.2022.06.040
M3 - Journal article
AN - SCOPUS:85134874925
SN - 0893-6080
VL - 154
SP - 152
EP - 164
JO - Neural Networks
JF - Neural Networks
ER -