Text mining with information-theoretic clustering

well as in other applications such as bioinformatics, we are interested in an angle between document vectors, hence, it is convenient to consider sets of normalized vectors. A wide variety of clustering algorithms applicable for objects of general nature exist,1 of these, the k-means algorithm might be the most widely accepted general clustering technique. In this article we suggest a new two-step clustering procedure that handles unit norm vectors. One of these steps is based on the k-means clustering algorithm. The k-means-type clustering algorithm’s final partition quality depends on a good initial partition choice.2,3 In the first step of the procedure, we use a spherical principal directions divisive partitioning (sPDDP) algorithm.3 Based on singular value decomposition (SVD), this algorithm clusters l2 unit vectors and generates “good” initial partitions for the second step of the procedure. The second step is a modification of a new divisive information–theoretic clustering algorithm4 (DITC), which uses the Kullback–Leibler (KL) divergence5 as a similarity measure between two unit l1 vectors with nonnegative coordinates. However, the DITC algorithm resembles the classical batch k-means algorithm and suffers from similar deficiencies. We discuss our modifications to the DITC algorithm to remedy some of its deficiencies. We call the proposed modification the Kullback–Leibler means algorithm, or KL-means.