Nonlinear Eigen-Approach for Optimization on Stiefel Manifolds Arising from Machine Learning

Modern machine learning at its core can be viewed as constrained optimization to ex- tract knowledge from real-world data. Optimization on Stiefel manifold ubiquitously arises from supervised/unsupervised multi-view subspace learning (MvSL), where or- thonormality of projection matrices is highly desired for distance preservation, robustness, and better noise reduction, among others. Optimization problems on Stiefel manifolds are numerically challenging. Generic optimization methods can be applied but they are generally too expensive and not feasible for large scale data science applications. For that reason, it is quite common for practitioners and even researchers to solve a relaxed problem by changing orthogonality constraints so that the relaxed problem is equivalent to a linear eigenvalue problem for which a number of mature numerical techniques can be readily applied. Unfortunately, the practice solves a wrong problem and often yields suboptimal performance. In this project, we propose to establish a practical solution framework that solves optimization problems on Stiefel manifolds as they are. We do so via an effective eigenvector dependent nonlinear eigenvalue problem (NEPv) approach which can fully utilize their structures as those from MvSL typically have. Numerically, self-consistent-field (SCF) iterations will be deployed to solve the involved NEPv efficiently by leveraging vast resources out there for eigenvalue problems. Theoretically, methods based our solution framework provably are either globally convergent or at least locally convergent with the help of level-shifting technique.

We plan to create a software suite that implements our methods for optimization problems on Stiefel manifolds commonly from MvSL and for the ones that will appear in our continuing efforts in the MvSL research. It is expected that at the completion of this project, the state of the art in MvSL is significantly raised. Applications of artificial intelligence have been infiltrating every corner of our society, and Hong Kong as a part of the greater bay area has a unique position to lead the development of new cutting edge technologies in the big data era. This project will and can provide solid mathematical and numerical foundations of orthogonal multi-view machine learning for years to come.