Morphologically-aware vocabulary reduction of word embeddings

• Main Authors: CHIA, Chong Cher, TKACHENKO, Maksim, LAUW, Hady Wirawan
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Word embeddings; compression; vocabulary reduction; Databases and Information Systems; Numerical Analysis and Scientific Computing; Theory and Algorithms
• Online Access: https://ink.library.smu.edu.sg/sis_research/7608; https://ink.library.smu.edu.sg/context/sis_research/article/8611/viewcontent/wiiat22.pdf
• Institution: Singapore Management University

We propose SubText, a compression mechanism via vocabulary reduction. The crux is to judiciously select a subset of word embeddings which support the reconstruction of the remaining word embeddings based on their form alone. The proposed algorithm considers the preservation of the original embeddings, as well as a word’s relationship to other words that are morphologically or semantically similar. Comprehensive evaluation of the compressed vocabulary reveals SubText’s efficacy on diverse tasks over traditional vocabulary reduction techniques, as validated on English, as well as a collection of inflected languages.