Living on the edge: Phase transitions in convex programs with random data
چکیده
Recent empirical research indicates that many convex optimization problems with random constraints exhibit a phase transition as the number of constraints increases. For example, this phenomenon emerges in the `1 minimization method for identifying a sparse vector from random linear samples. Indeed, this approach succeeds with high probability when the number of samples exceeds a threshold that depends on the sparsity level; otherwise, it fails with high probability. This paper provides the first rigorous analysis that explains why phase transitions are ubiquitous in random convex optimization problems. It also describes tools for making reliable predictions about the quantitative aspects of the transition, including the location and the width of the transition region. These techniques apply to regularized linear inverse problems with random measurements, to demixing problems under a random incoherence model, and also to cone programs with random affine constraints. These applications depend on foundational research in conic geometry. This paper introduces a new summary parameter, called the statistical dimension, that canonically extends the dimension of a linear subspace to the class of convex cones. The main technical result demonstrates that the sequence of conic intrinsic volumes of a convex cone concentrates sharply near the statistical dimension. This fact leads to an approximate version of the conic kinematic formula that gives bounds on the probability that a randomly oriented cone shares a ray with a fixed cone. 1. MOTIVATION AND CONTRIBUTIONS A phase transition is a sharp change in the character of a computational problem as its parameters vary. Recent work indicates that phase transitions emerge in a variety of random convex optimization problems from mathematical signal processing and computational statistics; for example, see [DT09b, Sto09, RFP10, CSPW11, MT12, DGM13]. This paper develops geometric tools that allow us to identify the location of these phase transitions using geometric invariants associated with the mathematical program. Our analysis provides the first fully rigorous account of transition phenomena in random linear inverse problems, random demixing problems, and random cone programs. 1.1. Vignette: Compressed sensing. To illustrate our goals, we discuss the compressed sensing problem, a familiar example where a phase transition is plainly visible in numerical experiments [DT09b]. Let x0 ∈Rd be an unknown vector with s nonzero entries. Suppose we have access to a vector z0 ∈Rm consisting of random linear samples of x0, where the number m of samples is smaller than the ambient dimension d . More precisely, assume that z0 = Ax0 where A is an m ×d matrix with independent standard normal entries. The aim is to reconstruct x0 given the measurement vector z0, the measurement matrix A, and the prior knowledge that x0 is sparse. One may regard this formulation as a toy model for understanding when it is possible to solve an underdetermined linear inverse problem with a structured unknown. A well-established approach [CDS01, CT06, Don06a] to the compressed sensing problem is the method of `1 minimization: minimize ‖x‖1 subject to z0 = Ax . (1.1) We say that the convex program (1.1) succeeds at solving the compressed sensing problem when it has a unique optimal point x̂ that equals the true unknown x0; otherwise, it fails. Figure 1.1 depicts the results of a computer experiment designed to estimate the empirical probability that (1.1) succeeds (with respect to the randomness in A) as the sparsity level s and the number m of samples range from zero to the ambient dimension d . The plot evinces that, for a given sparsity level s, the `1 ar X iv :1 30 3. 66 72 v1 [ cs .I T ] 2 6 M ar 2 01 3 2 D. AMELUNXEN, M. LOTZ, M. B. MCCOY, AND J. A. TROPP 0 25 50 75 100 0 25 50 75 100 FIGURE 1.1: Empirical phase transition in compressed sensing. The colormap indicates the empirical probability that the `1 minimization problem (1.1) successfully recovers a sparse vector x0 ∈R100 from the vector z0 = Ax0 of random linear measurements, where A is a standard normal matrix. The probability of success increases with brightness from certain failure (black) to certain success (white). minimization technique (1.1) almost always succeeds when we have an adequate number m of samples, while it almost always fails when we have fewer samples. See Appendix A for the experimental details. Figure 1.1 raises several interesting questions about the performance of the `1 minimization method for solving the compressed sensing problem: • What is the probability of success? For a given pair (s,m) of parameters, can we estimate the probability that (1.1) succeeds or fails? • Does a phase transition exist? Is there a simple curve m =ψ(s) that separates the parameter space into regions where (1.1) succeeds or fails most of the time? • Where is the edge of the phase transition? Can we find a formula for the location of this threshold between success and failure? • How wide is the transition region? For a given sparsity level s and ambient dimension d , how big is the range of m where the probability of success and failure are comparable? • Why does the transition exist? Is there a geometric explanation for the phase transition in compressed sensing? Can we export this reasoning to understand other problems? In Section 10, we summarize a large corpus of research that has attempted to address these questions. Unfortunately, the current results are fragmentary, even for the vanilla compressed sensing problem. This work provides a detailed answer to each of the questions we have posed. 1.2. Contributions. We approach phase transition phenomena by studying the intrinsic geometric properties of convex cones. This attack is appropriate for the compressed sensing problem because we can express the optimality condition for (1.1) in terms of a descent cone of the `1 norm. Our techniques apply more broadly because convex cones play a central role in convex optimization. Let us summarize the main contributions of this work. • We introduce a new summary parameter for convex cones, which we call the statistical dimension. This quantity canonically extends the linear dimension of a subspace to the class of convex cones. (See Definition 2.1, Proposition 4.1, Proposition 5.11, and Proposition 5.12.) Date: 26 March 2013. 2010 Mathematics Subject Classification. Primary: 90C25, 52A22, 60D05. Secondary: 52A20, 62C20. THE GEOMETRY OF PHASE TRANSITIONS IN CONVEX OPTIMIZATION 3 • We prove that a regularized linear inverse problem with random measurements, such as the compressed sensing problem, must exhibit a phase transition as the number of random measurements increases. The location and width of the transition are controlled by the statistical dimension of a cone associated with the regularizer and the unknown. (Theorem II, Theorem 7.1, and Proposition 9.1.) • The literature [MT12] describes convex programming methods for decomposing a superposition of two structured, randomly oriented vectors into its constituents. These methods exhibit a phase transition whose properties depend on the total statistical dimension of two convex cones. (Theorem III and Theorem 7.1.) • A cone program with random affine constraints displays a phase transition as the number of constraints increases. We can predict the transition using the statistical dimension of the cone. (Theorem 8.1.) • In Section 4, we explain how to compute the statistical dimension for several important families of cones that arise in convex optimization and mathematical signal processing. These calculations are not substantially novel, but we provide the first rigorous proof that they are sharp in high dimensions. (Propositions 4.2, 4.3, 4.7, 4.8, and 4.9 and Theorem 4.5.) These applied results are supported by foundational research in conic geometry. Our theoretical analysis identifies a new geometric phenomenon that governs all the phase transitions mentioned above. • We prove that the sequence of conic intrinsic volumes of a general convex cone concentrates at the statistical dimension of the cone. This result provides a new family of nontrivial inequalities among the conic intrinsic volumes. (Theorem 6.1 and Section 6.1.) • The concentration result implies an approximate version of the kinematic formula for cones. This result uses the statistical dimension to bound the probability that a randomly oriented cone shares a ray with a fixed cone. (Theorem I and Theorem 7.1.) As an added bonus, we provide the final ingredient needed to resolve a series of conjectures [DMM09a, DJM11, DGM13] about the coincidence between the minimax risk of denoising and the location of phase transitions in linear inverse problems. Indeed, Oymak & Hassibi [OH12] have recently shown that the minimax risk is equivalent with the statistical dimension of an appropriate cone, and our results prove that the phase transition occurs at precisely this spot. See Section 10.4 for further details. 1.3. Roadmap. Section 2 explains the connection between conic geometry and phase transitions in two applications. We analyze both problems, and we showcase computer experiments that confirm the accuracy of our predictions. Section 3 introduces notation and background material. In Section 4, we explain how to compute the statistical dimension for several important families of cones, and we prove that the calculations are sharp. Section 5 summarizes the central concepts from conic geometric probability, including the definition of conic intrinsic volumes. Sections 6 and 7 form the technical kernel of the paper. Here, we establish that the sequence of conic intrinsic volumes concentrates at the statistical dimension, and we derive an approximate kinematic formula as a consequence. Afterward, we return to applications. Section 8 shows that cone programs with random affine constraints exhibit a phase transition. Section 9 uses our theory to prove that a certain linear inverse problem is bound to fail. Finally, we canvass the related work in Section 10. The paper contains five substantial appendices, which house many of the technical details. Appendix A provides information about the computer experiments. Appendix B justifies the methods for computing the statistical dimension of a descent cone. Appendix C completes several statistical dimension calculations. Appendix D contains most of the analysis required to show that the sequence of conic intrinsic volumes concentrates. Finally, Appendix E establishes a connection between the statistical dimension and another geometric quantity called the Gaussian width. 2. CONIC GEOMETRY AND PHASE TRANSITIONS The existence of phase transitions depends on a striking new fact about the geometry of cones. Indeed, our entire approach can be summarized in the following precept: Each convex cone C admits a “dimension” parameter δ(C ). For certain purposes, the cone behaves like a linear subspace with dimension [δ(C )]. This section introduces the quantity δ(C ), which we call the statistical dimension of the cone. Then we state an approximate kinematic formula, which uses the statistical dimension to express the probability that 4 D. AMELUNXEN, M. LOTZ, M. B. MCCOY, AND J. A. TROPP two randomly oriented cones intersect nontrivially. This theorem parallels familiar statements about the intersection of two randomly oriented subspaces. Afterward, we apply the approximate kinematic formula to explain why phase transitions arise in random instances of regularized linear inverse problems (such as compressed sensing) and in a related class of demixing problems. 2.1. The statistical dimension of a convex cone. For certain purposes, the dimension of a linear subspace carries all pertinent information about the subspace. In particular, we can resolve questions in stochastic geometry, such as the probability that two random subspaces have a nontrivial intersection, as soon as we know the dimensions of the subspaces. Consider the more general problem of determining the probability that a randomly oriented subspace shares a ray with a fixed convex cone. Prima facie, it seems impossible that we could answer this question without detailed information about the cone. Nonetheless, there is a single parameter that encapsulates all the relevant geometry of the cone. Definition 2.1 (Statistical dimension). The statistical dimension δ(C ) of a closed convex cone C ⊂Rd is defined as δ(C ) := E[‖ΠC (g )‖2 ], (2.1) where g ∈Rd is a standard normal vector, ‖·‖ is the Euclidean norm, and ΠC denotes the Euclidean projection onto the cone C : ΠC (x) := arg min {‖x − y‖ : y ∈C}. We define the statistical dimension of a general convex cone to be the statistical dimension of its closure. As we show in Section 5.3, the statistical dimension canonically extends the linear dimension of a subspace to the class of convex cones. Let us highlight a few properties that support this claim. First, the statistical dimension of a subspace L ⊂Rd satisfies δ(L) = E[‖ΠL(g )‖2]= dim(L). (2.2) Second, the statistical dimension of a cone C ⊂Rd is rotationally invariant: δ(UC ) = δ(C ) for each orthogonal matrix U ∈Rd×d . (2.3) Third, the statistical dimension increases with the size of the cone in the sense that C ⊂ K implies that δ(C ) ≤ δ(K ). See Proposition 4.1 for details about these and other properties of the statistical dimension. Section 4 explains how to calculate the statistical dimension for several important families of convex cones. The examples include self-dual cones, circular cones, descent cones of the `1 norm, and descent cones of the Schatten 1-norm.1 In all of these cases, we prove rigorously that our estimates for the statistical dimension are sharp in high dimensions. Remark 2.2 (Related concepts). The statistical dimension of a cone is closely related to its Gaussian width, another summary parameter for cones that has been proposed in the literature [RV08, Sto09, CRPW12]. See Section 10.3 for further discussion of Gaussian widths. Recent work of Oymak & Hassibi [OH12] also illuminates a connection between the statistical dimension and the minimax risk for denoising problems; see Section 10.4. 2.2. Kinematics and statistical dimension. Conic integral geometry [SW08, Ch. 6] studies properties of convex cones that are invariant under rotation and reflection. A crowning achievement in this field is the conic kinematic formula [SW08, Thm. 6.5.6], which yields the exact probability that two randomly oriented convex cones intersect nontrivially. As recognized in [Ame11, MT12], this result is tailor-made for studying random instances of convex optimization problems. Our main applied result is an approximate kinematic formula, expressed in terms of the statistical dimension. Theorem I (Approximate kinematic formula). Fix a tolerance η ∈ (0,1). Suppose that C ,K ⊂ Rd are closed convex cones, one of which is not a subspace. Draw an orthogonal matrix Q ∈Rd×d uniformly at random. Then δ(C )+δ(K ) ≤ d −aη p d =⇒ P{C ∩QK = {0}}≥ 1−η; δ(C )+δ(K ) ≥ d +aη p d =⇒ P{C ∩QK = {0}}≤ η. The quantity aη := 4 √ log(4/η). For example, a0.01 0 { y ∈Rd : f (x +τy) ≤ f (x)}. Chandrasekaran et al. [CRPW12, Prop. 2.1] characterize when the optimization problem (2.5) succeeds in terms of a descent cone. This result is simply a geometric statement of the primal optimality condition. Fact 2.6 (Optimality condition for linear inverse problems). The vector x0 is the unique optimal point of the convex program (2.5) if and only if D( f , x0)∩null(A) = {0}. 2The extended real numbers R :=R∪ {±∞}. A proper convex function has at least one finite value, and it does not take the value −∞. 6 D. AMELUNXEN, M. LOTZ, M. B. MCCOY, AND J. A. TROPP