Automatic Language Identification in Code-Switched Hindi-English Social Media Text

Natural Language Processing (NLP) tools typically struggle to process code-switched data and so linguists are commonly forced annotate such manually. As this becomes more readily available, automatic increasingly needed help speed up the annotation improve consistency. Last year, a toolkit was developed semi-automatically transcribed bilingual Vietnamese-English speech with token-based language information POS tags (hereafter CanVEC toolkit, L. Nguyen & Bryant, 2020). In work, we extend methodology another pair, Hindi-English, explore extent which can standardise automation process. Specifically, applied principles behind from International Conference on (ICON) 2016 shared task, consists of social media posts (Facebook, Twitter WhatsApp) that have been annotated (Molina et al., 2016). We used ICON-2016 annotations as gold-standard labels in identification task. Ultimately, our tool achieved an F1 score 87.99% data. then evaluated first 500 tokens each subset manually, found almost 40% all errors were caused entirely by problems gold-standard, i.e., system correct. It is thus likely overall accuracy higher than reported. This shows great potential for effectively automating corpora, different combinations, genres. finally discuss some limitations approach release code human evaluation together paper.