Signal-noise search RMT estimator with adaptive decision criterion for estimating the number of signals based on random matrix theory

RMT estimator estimates the number of signals via sequentially testing the likelihood of an eigenvalue as arising from a signal or from noise. However, the RMT estimator does not consider the interaction term among eigenvalues, so it tends to down-estimate the number of signals as some signals will be buried in this interaction term. In order to overcome this problem, in this paper we focus on developing novel RMT-based estimators by incorporating the Lawley's theory into random matrix theory. Firstly, we derive a novel decision statistics for signal number estimation by incorporating the Lawley's theory into random matrix theory, and then propose a signal-search RMT estimator for signal number estimation. Secondly, we analyze the effect of the interaction term among eigenvalues on the estimation performance of the signal-search RMT estimator and the RMT estimator. It shows that the signal-search RMT estimator has better detection performance than the RMT estimator when some signals are buried in the interaction term among eigenvalues, but has larger over-estimation probability than the RMT estimator when there are no signals or when all signals are strong enough to be detected by the RMT estimator. Thirdly, in order to overcome the individual drawbacks of these two estimators, we derive analytical formulas for the over-estimation probability of the signal-search RMT estimator and the down-estimation probability of the RMT estimator, and propose a signal-noise search RMT estimator which can adaptively select its decision criterion between the RMT estimator and the signal-search RMT estimator to make benefits of these two estimators. Finally, simulation results show that the signal-noise search RMT estimator significantly outperforms the existing estimators including the RMT estimator, the classic AIC and MDL estimators, and the modified AIC estimator.