Testing k-wise independent distributions

A probability distribution over {0, 1} is k-wise independent if its restriction to any k coordinates is uniform. More generally, a discrete distribution D over Σ1 × · · · × Σn is called (non-uniform) k-wise independent if for any subset of k indices {i1, . . . , ik} and for any z1 ∈ Σi1 , . . . , zk ∈ Σik , PrX∼D[Xi1 · · ·Xik = z1 · · · zk] = PrX∼D[Xi1 = z1] · · ·PrX∼D[Xik = zk]. k-wise independent distributions look random “locally” to an observer of only k coordinates, even though they may be far from random “globally”. Because of this key feature, k-wise independent distributions are important concepts in probability, complexity, and algorithm design. In this thesis, we study the problem of testing (non-uniform) k-wise independent distributions over product spaces. For the problem of distinguishing k-wise independent distributions supported on the Boolean cube from those that are δ-far in statistical distance from any k-wise independent distribution, we upper bound the number of required samples by Õ(n/δ) and lower bound it by Ω(n k−1 2 /δ) (these bounds hold for constant k, and essentially the same bounds hold for general k). To achieve these bounds, we use novel Fourier analysis techniques to relate a distribution’s statistical distance from k-wise independence to its biases, a measure of the parity imbalance it induces on a set of variables. The relationships we derive are tighter than previously known, and may be of independent interest. We then generalize our results to distributions over larger domains. For the uniform case we show an upper bound on the distance between a distribution D from k-wise independent distributions in terms of the sum of Fourier coefficients of D at vectors of weight at most k. For the non-uniform case, we give a new characterization of distributions being k-wise independent and further show that such a characterization is robust based on our results for the uniform case. Our results yield natural testing algorithms for k-wise independence with time and sample complexity sublinear in terms of the support size of the distribution when k is a constant. The main technical tools employed include discrete Fourier transform and the theory of linear systems of congruences. Thesis Supervisor: Ronitt Rubinfeld Title: Professor of Electrical Engineering and Computer Science