A Fast, Compact, Accurate Model for Language Identification of Codemixed Text

Citations Over TimeTop 10% of 2018 papers

Abstract

We address fine-grained multilingual language identification: providing a language code for every token in a sentence, including codemixed text containing multiple languages. Such text is prevalent online, in documents, social media, and message boards. We show that a feed-forward network with a simple globally constrained decoder can accurately and rapidly label both codemixed and monolingual text in 100 languages and 100 language pairs. This model outperforms previously published multilingual approaches in terms of both accuracy and speed, yielding an 800x speed-up and a 19.5% averaged absolute gain on three codemixed datasets. It furthermore outperforms several benchmark systems on monolingual language identification.

Related Papers

• → Development of an approach to automatic language identification based on phone recognition(1996)24 cited
• → Automatic language identification based on Gaussian mixture model and universal background model(2003)1 cited
• The model for identification of a natural language of the text(2011)
• Robust Techniques Of Language Modeling For Spoken Language Identification(2009)
• The Control and Optimization of Natural Language(2008)