Measuring language similarity using trigrams: Limitations of language identification

• Main Authors: Oco, Nathaniel, Ilao, Joel P., Roxas, Rachel Edita, Syliongka, Leif Romeritch
• Format: text
• Published: Animo Repository 2013
• Subjects: Computational linguistics; Philippine languages—Data processing; Similarity (Language learning); Computer Sciences
• Online Access: https://animorepository.dlsu.edu.ph/faculty_research/2738
• Institution: De La Salle University

Computational approaches in language identification often result in high number of false positives and low recall rates, especially if the languages involved come from the same subfamily. In this paper, we aim to determine the cause of this problem by measuring language similarity through trigrams. Religious and literary texts were used as training data. Our experiments involving language identification show that the number of common trigrams for a given language pair is inversely proportional to precision and recall rates, whereas the average word length is directly proportional to the number of true positives. Future directions include improving language modeling and providing an approach to increase precision and recall. © 2013 IEEE.