Fine-grained analysis of structured output prediction

• Main Authors: MUSTAFA, Waleed, LEI, Yunwen, LEDENT, Antoine, and KLOFT, Marius
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: Structured output prediction; multi-label; Neural Networks; Multi-class; sequence-to-sequence; Stochastic Gradient Descent; Artificial Intelligence and Robotics; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/7207; https://ink.library.smu.edu.sg/context/sis_research/article/8210/viewcontent/Structured_output.pdf
• Institution: Singapore Management University

In machine learning we often encounter structured output prediction problems (SOPPs), i.e. problems where the output space admits a rich internal structure. Application domains where SOPPs naturally occur include natural language processing, speech recognition, and computer vision. Typical SOPPs have an extremely large label set, which grows exponentially as a function of the size of the output. Existing generalization analysis implies generalization bounds with at least a square-root dependency on the cardinality d of the label set, which can be vacuous in practice. In this paper, we significantly improve the state of the art by developing novel high-probability bounds with a logarithmic dependency on d. Furthermore, we leverage the lens of algorithmic stability to develop generalization bounds in expectation without any dependency on d. Our results therefore build a solid theoretical foundation for learning in large-scale SOPPs. Furthermore, we extend our results to learning with weakly dependent data.