Punctuation prediction for Vietnamese texts using conditional random fields

• Main Authors: PHAM, Hong Quang, NGUYEN, Binh T., CUONG, Nguyen Viet
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2019
• Subjects: Conditional random field; Punctuation prediction; Sequence labeling; Vietnamese language; Numerical Analysis and Scientific Computing; South and Southeast Asian Languages and Societies
• Online Access: https://ink.library.smu.edu.sg/sis_research/7816; https://ink.library.smu.edu.sg/context/sis_research/article/8819/viewcontent/3368926.3369716_pv.pdf
• Institution: Singapore Management University

We investigate the punctuation prediction for the Vietnamese language. This problem is crucial as it can be used to add suitable punctuation marks to machine-transcribed speeches, which usually do not have such information. Similar to previous works for English and Chinese languages, we formulate this task as a sequence labeling problem. After that, we apply the conditional random field model for solving the problem and propose a set of appropriate features that are useful for prediction. Moreover, we build two corpora from Vietnamese online news and movie subtitles and perform extensive experiments on these data. Finally, we ask four volunteers to insert punctuations into a small sample of our dataset. The experimental results show that this problem is challenging, even for a human, and our model can achieve near performance in comparison to a human.