On Potts Model Clustering, Kernel K-means, and Density Estimation

Many clustering methods, such as K-means, kernel K-means, and MNcut clustering, follow the same recipe: (1) choose a measure of similarity between observations; (ii) define a figure of merit assigning a large value to partitions of the data that put similar observations in the same cluster; (iii) optimize this figure of merit over partitions. Potts model clustering, introduced by Blatt, Wiseman, and Domany (1996) represents an interesting variation on this recipe. Blatt et al. define a new figure of merit for partitions that is formally similar to the Hamiltonian of the Potts model for ferromagnetism extensively studied in statistical physics. For each temperature T , the Hamiltonian defines a distribution assigning a probability to each possible configuration of the physical system or, in the language of clustering, to each partition. Instead of searching for a single partition optimizing the Hamiltonian, they sample a large number of partitions from this distribution for a range of temperatures. They propose a heuristic for choosing an appropriate temperature and from the sample of partitions associated with this chosen temperature, they then derive what we call a consensus clustering: two observations are put in the same consensus cluster if they belong to the same cluster in the majority of the random partitions. In a sense, the consensus clustering is an “average” of plausible configurations, and we would expect it to be more stable (over different samples) than the configuration optimizing the Hamiltonian. The goal of this paper is to contribute to the understanding of Potts model clustering and to propose extensions and improvements: (1) We show that the Hamiltonian used in Potts model clustering is closely related to the kernel K-means and MNCut criteria. (2) We propose a modification of the Hamiltonian penalizing unequal cluster sizes and show that it can be interpreted as a weighted version of the kernel K-means criterion. (3) We introduce a new version of the Wolff algorithm to simulate configurations from the distribution defined by the penalized Hamiltonian, leading to penalized Potts model clustering. (4) We note a link between kernel based clustering methods and non-parametric density estimation and exploit it to automatically determine locally adaptive kernel bandwidths. (5) We propose a new simple rule for selecting a good temperature T .