Mean field theory for deep dropout networks: Digging up gradient backpropagation deeply

Wei Huang, Richard Yi Da Xu, Weitao Du, Yutian Zeng, Yunce Zhao

Research output: Chapter in book/report/conference proceedingConference contributionpeer-review

Abstract

In recent years, the mean field theory has been applied to the study of neural networks and has achieved a great deal of success. The theory has been applied to various neural network structures, including CNNs, RNNs, Residual networks, and Batch normalization. Inevitably, recent work has also covered the use of dropout. The mean field theory shows that the existence of depth scales that limit the maximum depth of signal propagation and gradient backpropagation. However, the gradient backpropagation is derived under the gradient independence assumption that weights used during feed forward are drawn independently from the ones used in backpropagation. This is not how neural networks are trained in a real setting. Instead, the same weights used in a feed-forward step needs to be carried over to its corresponding backpropagation. Using this realistic condition, we perform theoretical computation on linear dropout networks and a series of experiments on dropout networks with different activation functions. Our empirical results show an interesting phenomenon that the length gradients can backpropagate for a single input and a pair of inputs are governed by the same depth scale. Besides, we study the relationship between variance and mean of statistical metrics of the gradient and shown an emergence of universality. Finally, we investigate the maximum trainable length for deep dropout networks through a series of experiments using MNIST and CIFAR10 and provide a more precise empirical formula that describes the trainable length than original work.

Original languageEnglish
Title of host publicationECAI 2020 - 24th European Conference on Artificial Intelligence, including 10th Conference on Prestigious Applications of Artificial Intelligence, PAIS 2020 - Proceedings
EditorsGiuseppe De Giacomo, Alejandro Catala, Bistra Dilkina, Michela Milano, Senen Barro, Alberto Bugarin, Jerome Lang
PublisherIOS Press BV
Pages1215-1222
Number of pages8
ISBN (Electronic)9781643681016
ISBN (Print)9781643681009
DOIs
Publication statusPublished - 24 Aug 2020
Event24th European Conference on Artificial Intelligence, ECAI 2020, including 10th Conference on Prestigious Applications of Artificial Intelligence, PAIS 2020 - Online, Santiago de Compostela, Spain
Duration: 29 Aug 20208 Sep 2020
https://ebooks.iospress.nl/volume/ecai-2020-24th-european-conference-on-artificial-intelligence (Conference proceedings)

Publication series

NameFrontiers in Artificial Intelligence and Applications
Volume325
ISSN (Print)0922-6389

Conference

Conference24th European Conference on Artificial Intelligence, ECAI 2020, including 10th Conference on Prestigious Applications of Artificial Intelligence, PAIS 2020
Country/TerritorySpain
CitySantiago de Compostela
Period29/08/208/09/20
Internet address

Scopus Subject Areas

  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Mean field theory for deep dropout networks: Digging up gradient backpropagation deeply'. Together they form a unique fingerprint.

Cite this