Measures of dispersion for corpus data: an overview, a suggestion, and a research program II

In order to adjust observed frequencies of occurrence, previous studies have suggested a variety of measures of dispersion and adjusted frequencies. In part I of this article, I first summarily reviewed many of these measures as well as a variety of their shortcomings and then suggested an alternative measure, DP, for deviation of proportions, which I argued to be conceptually simpler, but at the same time more versatile than many competing measures. I then exemplified this measure on the basis of word frequency data on co-occurrence data of words and construction/patterns. However, in spite of the advantages of DP and in spite of its relevance for virtually all corpus-linguistic work, dispersion is still a very much under-researched topic: to the best of my knowledge, there is not a single study investigating how different measures compare to each other when applied to large datasets. The present article therefore is largely programmatic and exploratory: it sketches out a research program for the investigation of measures of dispersion and adjusted frequencies and takes some initial steps itself. More specifically, this paper addresses the issues of the integration of frequencies and dispersion measures and the quantitative comparison of dispersion measures and adjusted frequencies. Finally, the paper makes available a few online resources that will hopefully stimulate more research in this central area of corpus-linguistic methodology and help other researchers go beyond the first programmatic steps taken here.