Kernel-based generalized cross-validation in non-parametric mixed-effect models

Although generalized cross-validation (GCV) has been frequently applied to select bandwidth when kernel methods are used to estimate non-parametric mixed-effect models in which non-parametric mean functions are used to model covariate effects, and additive random effects are applied to account for overdispersion and correlation, the optimality of the GCV has not yet been explored. In this article, we construct a kernel estimator of the non-parametric mean function. An equivalence between the kernel estimator and a weighted least square type estimator is provided, and the optimality of the GCV-based bandwidth is investigated. The theoretical derivations also show that kernel-based and spline-based GCV give very similar asymptotic results. This provides us with a solid base to use kernel estimation for mixed-effect models. Simulation studies are undertaken to investigate the empirical performance of the GCV. A real data example is analysed for illustration.