Korean jamo-level byte-pair encoding for neural machine translation

• Main Author: Lee, Junyoung
• Other Authors: Wang Lipo
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Document and text processing
• Online Access: https://hdl.handle.net/10356/172737
• Institution: Nanyang Technological University

Tokenization is the very first step in most Natural Language Processing tasks, and is essential in addressing the fundamental out-of-vocabulary problem, as well as in changing the linguistic understanding. To exploit the characteristics of the Korean language for a more parameter-efficient tokenizat...