Detecting Offensive Tweets via Topical Feature Discovery over a Large Scale Twitter Corpus

In this paper, we propose a novel approach for detecting cussing-related offensive content in Twitter. Our approach exploits the lexical collocation of swearing language via statistical topic modeling on a huge Twitter corpus and detects offensive tweets with automatically generated features under a machine learning framework. Our approach performed stably and competitively under a variety of machine learning algorithms. For instance, our approach achieved a true positive rate (TP) of 75.1% over 4029 testing tweets using Logistic Regression, significantly outperforming the popular and highly effective keyword matching baseline which has a TP of 69.7%, while keeping the false positive rate (FP) on the same level as the baseline at about 3.77%. In addition to the good performance, our approach also provides an alternative to large scale hand annotation efforts required by supervised learning approaches.