基于核密度估计的有监督学习算法及其理论 Kernel Density Estimation Driven Supervised Learning: Algorithms and Theory

学习理论旨在提供数学框架以分析机器学习算法的泛化误差，在对各种有监督学习算法的深刻理解发挥了重要作用，如最小二乘回归，分位数回归和分类算法等，其随着深度学习的快速发展而受到了越来越多的关注。根据不同的学习任务我们使用不同的损失函数，其中常用于分位数回归和二分类算法的损失函数由于其不光滑性往往会给相应的优化问题带来很大的困难。为了解决这个问题，我们在此项目中提出一类基于核密度估计的有监督学习算法并在学习理论的框架下分析其理论性质。选择假设空间为再生核希尔伯特空间，我们将首先考虑基于核密度估计的正则化算法并导出其泛化误差的最优学习率。然后我们将研究基于核密度估计的在线学习算法以处理流数据并进行收敛性分析，这有利于解决大规模数据问题。最后，我们将探讨基于核密度估计的深度学习算法以理解深度神经网络生成的假设空间的有效性。
Learning theory aims to provide a mathematical framework for analyzing the generalization error of machine learning algorithms, playing a crucial role in the profound understanding of various supervised learning algorithms, such as least squares regression, quantile regression, and classification algorithms. With the rapid development of deep learning, learning theory has received increasing attention. Different loss functions are used for different learning tasks. However, the commonly used loss functions for quantile regression and binary classification algorithms often pose significant challenges due to their non-smoothness. To address this problem, in this project, we propose a class of supervised learning algorithms based on kernel density estimation and analyze their theoretical properties within the framework of learning theory. By considering the hypothesis space as the reproducing kernel Hilbert space, we first examine the regularized algorithm based on kernel density estimation and derive the optimal learning rate for its generalization error. Then, we investigate online learning algorithms based on kernel density estimation to handle streaming data and analyze their convergence, which is beneficial for addressing large-scale data problems. Finally, we explore deep learning algorithms based on kernel density estimation to understand the effectiveness of the hypothesis space generated by deep neural networks.