Sharper generalisation bounds for pairwise learning

• Main Authors: LEI, Yunwen, LEDENT, Antoine, KLOFT, Marius
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2020
• Subjects: Pairwise learning; Stochastic Gradient Descent; Stability analysis; Learning theory; Databases and Information Systems; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/7208; https://ink.library.smu.edu.sg/context/sis_research/article/8211/viewcontent/NeurIPS_2020_sharper_generalization_bounds_for_pairwise_learning_Paper.pdf
• Institution: Singapore Management University

Pairwise learning refers to learning tasks with loss functions depending on a pair of training examples, which includes ranking and metric learning as specific examples. Recently, there has been an increasing amount of attention on the generalization analysis of pairwise learning to understand its practical behavior. However, the existing stability analysis provides suboptimal high-probability generalization bounds. In this paper, we provide a refined stability analysis by developing generalization bounds which can be √nn-times faster than the existing results, where nn is the sample size. This implies excess risk bounds of the order O(n−1/2) (up to a logarithmic factor) for both regularized risk minimization and stochastic gradient descent. We also introduce a new on-average stability measure to develop optimistic bounds in a low noise setting. We apply our results to ranking and metric learning, and clearly show the advantage of our generalization bounds over the existing analysis.