Double Penalized H-Likelihood for Selection of Fixed and Random Effects in Mixed Effects Models

The goal of this paper is to develop a double penalized hierarchical likelihood (DPHL) with a modified Cholesky decomposition for simultaneously selecting fixed and random effects in mixed effects models. DPHL avoids the use of data likelihood, which usually involves a high-dimensional integral, to define an objective function for variable selection. The resulting DPHL-based estimator enjoys the oracle properties with no requirement on the convexity of loss function. Moreover, a two-stage algorithm is proposed to effectively implement this approach. An H-likelihood-based Bayesian information criterion (BIC) is developed for tuning parameter selection. Simulated data and a real data set are examined to illustrate the efficiency of the proposed method.