Transfer Learning With Singular Value Decomposition of Multichannel Convolution Matrices

Tak Shing Au Yeung, Ka Chun Cheung, Michael K. Ng, Simon See, Andy Yip

Research output: Contribution to journalLetterpeer-review


The task of transfer learning using pretrained convolutional neural networks is considered. We propose a convolution-SVD layer to analyze the convolution operators with a singular value decomposition computed in the Fourier domain. Singular vectors extracted from the source domain are transferred to the target domain, whereas the singular values are finetuned with a target data set. In this way, dimension reduction is achieved to avoid overfitting, while some flexibility to fine-tune the convolution kernels is maintained. We extend an existing convolution kernel reconstruction algorithm to allow for a reconstruction from an arbitrary set of learned singular values. A generalization bound for a single convolution-SVD layer is devised to show the consistency between training and testing errors. We further introduce a notion of transfer learning gap. We prove that the testing error for a single convolution-SVD layer is bounded in terms of the gap, which motivates us to develop a regularization model with the gap as the regularizer. Numerical experiments are conducted to demonstrate the superiority of the proposed model in solving classification problems and the influence of various parameters. In particular, the regularization is shown to yield a significantly higher prediction accuracy.

Original languageEnglish
Pages (from-to)1678-1712
Number of pages35
JournalNeural Computation
Issue number10
Early online date8 Sept 2023
Publication statusPublished - Oct 2023

Scopus Subject Areas

  • Arts and Humanities (miscellaneous)
  • Cognitive Neuroscience


Dive into the research topics of 'Transfer Learning With Singular Value Decomposition of Multichannel Convolution Matrices'. Together they form a unique fingerprint.

Cite this