Roman-Urdu-Parl: Roman-Urdu and Urdu Parallel Corpus for Urdu Language Understanding

Citations Over TimeTop 19% of 2022 papers

Abstract

Availability of corpora is a basic requirement for conducting research in a particular language. Unfortunately, for a morphologically rich language like Urdu, despite being used by over a 100 million people around the globe, the dearth of corpora is a major reason for the lack of attention and advancement in research. To this end, we present the first-ever large-scale publicly available Roman-Urdu parallel corpus, Roman-Urdu-Parl, with 6.37 million sentence-pairs. It is a huge corpus collected from diverse sources, annotated using crowd-sourcing techniques, and also assured for quality. It has a total of 92.76 million Roman-Urdu words, 92.85 million Urdu words, Roman-Urdu vocabulary of 42.9 K words, and Urdu vocabulary of 43.8 K words. Roman-Urdu-Parl has been built to ensure that it not only captures the morphological and linguistic features of the language but also the heterogeneity and variations arising due to demographic conditions. We validate the authenticity and quality of our corpus by using it to address two natural language processing research problems, that is, on learning word embeddings and building a machine transliteration system. Our contribution of the corpus leads to exceptional results in both settings, for example, our machine transliteration system sets a new state-of-the-art with a Bilingual Evaluation Understudy (BLEU) score of 84.67. We believe that Roman-Urdu-Parl can serve as fuel for igniting and advancing works in many research areas related to the Urdu language.

Related Papers

• From Hindi to Urdu: A Social and Political History(2012)
• → Curate a transliteration corpus from transliteration/translation pairs(2008)2 cited
• Masterpieces of Urdu rubaiyat(1996)
• → Urdu to Hindi and Reverse Transliteration System(2011)
• Transliteration of loanwords in Chinese(2009)