Limiting spectral distribution of renormalized separable sample covariance matrices when p/ni → 0

We are concerned with the behavior of the eigenvalues of renormalized sample covariance matrices of the form Cn = √ n p ( 1 n A p XnBnX ∗ nA 1/2 p − 1 n tr(Bn)Ap ) as p, n→∞ and p/n→ 0, where Xn is a p×n matrix with i.i.d. real or complex valued entries Xij satisfying E(Xij) = 0, E|Xij | = 1 and having finite fourth moment. A p is a squareroot of the nonnegative definite Hermitian matrix Ap, and Bn is an n× n nonnegative definite Hermitian matrix. We show that the empirical spectral distribution (ESD) of Cn converges a.s. to a nonrandom limiting distribution under the assumption that the ESD of Ap converges to a distribution F that is not degenerate at zero, and that the first and second spectral moments of Bn converge. The probability density function of the LSD of Cn is derived and it is shown that it depends on the LSD of Ap and the limiting value of n −1tr(B2 n). We propose a computational algorithm for evaluating this limiting density when the LSD of Ap is a mixture of point masses. In addition, when the entries of Xn are sub-Gaussian, we derive the limiting empirical distribution of { √ n/p(λj(Sn) − ntr(Bn)λj(Ap))}pj=1 where Sn := n−1A 1/2 p XnBnX ∗ nA 1/2 p is the sample covariance matrix and λj denotes the j-th largest eigenvalue, when F A is a finite mixture of point masses. These results are utilized to propose a test for the covariance structure of the data where the null hypothesis is that the joint covariance matrix is of the form Ap ⊗ Bn for ⊗ denoting the Kronecker product, as well as Ap and the first two spectral moments of Bn are specified. The performance of this test is illustrated through a simulation study.