Transfer learning for language model adaptation - Bari M. Saiful (Nanyang Technological University, 2023)
Language is the pathway to democratize the boundary of land and culture. Bridging the gap between languages is one of the biggest challenges of Artificial Intelligent (AI) systems. The current success of AI systems is dominated by the supervised learning paradigm where gradient-based learning algorithms (i.e., SGD, Adam) are designed to optimize complex high-dimensional planes. These algorithms learn from statistical observations that are typically collected with the intention of a specific task (i.e., product review, sentiment analysis). The use of task-dependent samples makes the learning procedure tedious as it requires manually annotated data. On the contrary, without having a sufficient amount of samples to represent the distribution, deep learning models tend to suffer from the lack of robustness. Due to the natural puzzle of randomness, not all sets of observations are observed in the data collection procedure, thus creating Out-of-Distribution (OOD) problem in learning algorithms.