General Words Representation Method for Modern Language Model

• Main Authors: Abbas Saliimi, Lokman, Mohamed Ariff, Ameedeen, Ngahzaifa, Ab. Ghani
• Format: Article
• Language: English
• Published: UTeM 2023
• Subjects: QA75 Electronic computers. Computer science; QA76 Computer software
• Online Access: http://umpir.ump.edu.my/id/eprint/38294/1/6241-Article%20Text-18137-1-10-20230324.pdf; http://umpir.ump.edu.my/id/eprint/38294/; https://jtec.utem.edu.my/jtec/article/view/6241
• Institution: Universiti Malaysia Pahang

This paper proposes a new word representation method emphasizes general words over specific words. The main motivation for developing this method is to address the weighting bias in modern Language Models (LMs). Based on the Transformer architecture, contemporary LMs tend to naturally emphasize spec...