Norm-Based Generalisation Bounds for Deep Multi-Class Convolutional Neural Networks

Antoine Ledent, Waleed Mustafa, Yunwen Lei, Marius Kloft

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

12 Citations (Scopus)

Abstract

We show generalisation error bounds for deep learning with two main improvements over the state of the art. (1) Our bounds have no explicit dependence on the number of classes except for logarithmic factors. This holds even when formulating the bounds in terms of the Frobenius-norm of the weight matrices, where previous bounds exhibit at least a square-root dependence on the number of classes. (2) We adapt the classic Rademacher analysis of DNNs to incorporate weight sharing—a task of fundamental theoretical importance which was previously attempted only under very restrictive assumptions. In our results, each convolutional filter contributes only once to the bound, regardless of how many times it is applied. Further improvements exploiting pooling and sparse connections are provided. The presented bounds scale as the norms of the parameter matrices, rather than the number of parameters. In particular, contrary to bounds based on parameter counting, they are asymptotically tight (up to log factors) when the weights approach initialisation, making them suitable as a basic ingredient in bounds sensitive to the optimisation procedure. We also show how to adapt the recent technique of loss function augmentation to replace spectral norms by empirical analogues whilst maintaining the advantages of our approach.

Original languageEnglish
Title of host publication35th AAAI Conference on Artificial Intelligence, AAAI 2021
PublisherAssociation for the Advancement of Artificial Intelligence
Pages8279-8287
Number of pages9
ISBN (Electronic)9781713835974
ISBN (Print)9781577358664
DOIs
Publication statusPublished - 18 May 2021
Event35th AAAI Conference on Artificial Intelligence, AAAI 2021 - Virtual, Online
Duration: 2 Feb 20219 Feb 2021
https://aaai.org/Conferences/AAAI-21/
https://ojs.aaai.org/index.php/AAAI/issue/archive

Publication series

NameProceedings of the AAAI Conference on Artificial Intelligence
Number9
Volume35
ISSN (Print)2159-5399
ISSN (Electronic)2374-3468

Conference

Conference35th AAAI Conference on Artificial Intelligence, AAAI 2021
Period2/02/219/02/21
Internet address

Scopus Subject Areas

  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Norm-Based Generalisation Bounds for Deep Multi-Class Convolutional Neural Networks'. Together they form a unique fingerprint.

Cite this