Detection of Signals in Colored Noise: Roy's Largest Root Test for Non-central $F$-matrices

Abstract: This paper investigates the signal detection problem in colored noise with an unknown covariance matrix. In particular, we focus on detecting a non-random signal by capitalizing on the leading eigenvalue (a.k.a. Roy's largest root) of the whitened sample covariance matrix as the test statistic. To this end, the whitened sample covariance matrix is constructed via (m)-dimensional (p ) plausible signal-bearing samples and (m)-dimensional (n ) noise-only samples. Since the signal is non-random, the whitened sample covariance matrix turns out to have a {\it non-central} (F)-distribution with a rank-one non-centrality parameter. Therefore, the performance of the test entails the statistical characterization of the leading eigenvalue of the non-central (F)-matrix, which we address by deriving its cumulative distribution function (c.d.f.) in closed-form by leveraging the powerful orthogonal polynomial approach in random matrix theory. This new c.d.f. has been instrumental in analyzing the receiver operating characteristic (ROC) of the detector. We also extend our analysis into the high dimensional regime in which (m,n), and (p) diverge such that (m/n) and (m/p) remain fixed. It turns out that, when (m=n) and fixed, the power of the test improves if the signal-to-noise ratio (SNR) is of at least (O(p)), whereas the corresponding SNR in the high dimensional regime is of at least (O(p^2)). Nevertheless, more intriguingly, for (m<n) with the SNR of order (O(p)), the leading eigenvalue does not have power to detect {\it weak} signals in the high dimensional regime.