Sparse PCA via matrix (2,1)-norm regularization with an application to feature selection

This paper is concerned with sparse PCA via the matrix (2,1)-norm regularization (PCA_2,1). It can produce a row-sparse projection, a useful tool in machine learning when it comes to, for example, feature selection, that aims to choose most relevant features. Mathematically, PCA_2,1 is a non-smooth optimization problem on the Stiefel manifold. For a suitably chosen regularization parameter, the optimal projection matrix has many negligible rows. A practical NEPv approach (nonlinear eigenvalue problem with eigenvector dependency) is proposed to iteratively compute the optimal projection matrix. It is shown that the approach is globally convergent in the sense that the objective is monotonically increasing during the iterative process and any accumulation point of the iterates is a stationary point to the optimization problem. Extensive numerical experiments, with an application to feature selection, have been conducted to demonstrate the performance of the practical NEPv approach, with comparison against existing feature selection methods in terms of classification accuracy. The numerical results demonstrate that PCA_2,1 is highly effective and often produces superior classification results to existing feature selection methods that are in use today.