Supervised dimensionality reduction for big data

Abstract To solve key biomedical problems, experimentalists now routinely measure millions or billions of features (dimensions) per sample, with the hope that data science techniques will be able to build accurate data-driven inferences. Because sample sizes are typically orders magnitude smaller than dimensionality these data, valid inferences require finding a low-dimensional representation preserves discriminating information (e.g., whether individual suffers from particular disease). There is lack interpretable supervised reduction methods scale dimensions strong statistical theoretical guarantees. We introduce an approach extending principal components analysis by incorporating class-conditional moment estimates into projection. The simplest version, Linear Optimal Low-rank projection, incorporates means. prove, and substantiate both synthetic real benchmarks, Low-Rank Projection its generalizations lead improved representations for subsequent classification, while maintaining computational efficiency scalability. Using multiple brain imaging datasets consisting more 150 million features, several genomics 500,000 outperforms other scalable linear in terms accuracy, only requiring few minutes on standard desktop computer.