Dice's coefficient on trigram profiles as metric for language similarity

• Main Authors: Oco, Nathaniel, Syliongka, Leif Romeritch, Roxas, Rachel Edita, Ilao, Joel P.
• Format: text
• Published: Animo Repository 2013
• Subjects: Computational linguistics; Similarity (Language learning); Philippine languages—Data processing; Computer Sciences
• Online Access: https://animorepository.dlsu.edu.ph/faculty_research/2737
• Institution: De La Salle University

In this study, we present Dice's coefficient on trigram profiles as metric for language similarity. As testbed, we focused on eight Philippine languages. No known language similarity value for these languages exists. Documents containing transcribed audio recordings, news articles, religious and literary texts were taken from an online corpus and used as training data. Character trigram profiles were then generated using an n-gram generator and language similarity was computed. The results were matched against those reported in the literature and against the language family tree. To evaluate the metric, it was applied to five languages with known similarity values. The results were then compared with an existing lexical similarity metric. The average difference is 27%. Analyses of the results reveal that phonetic spelling play an important role in language similarity. As future work, the metric can be used on phonetic transcriptions. © 2013 IEEE.