Convexity in R N Main Notes 1

Transcription

1 John Nachbar Washington University December 16, 2016 Convexity in R N Main Notes 1 1 Introduction. These notes establish basic versions of the Supporting Hyperplane Theorem (Theorem 5) and the Separating Hyperplane Theorem (Theorem 6) for sets in R N. The supplemental notes on convexity in R N give more powerful versions, establishing conditions that are necessary as well as sufficient. All of the results here carry over immediately to general finite-dimensional vector spaces; see also Remark 2. For extensions to infinite-dimensional normed vector spaces, see the notes on the Hahn-Banach theorem. 2 Convex Sets. Definition 1. A set A R N is convex iff for any a, b A and any θ [0, 1] the point θa + (1 θ)b is also in A. Geometrically, A is convex iff A contains the line segment joining any two points in A. By an induction argument, A is convex iff for any (non-empty) finite subset {a 1,..., a m }, and weights θ i with θ i 0 and m i=1 θ i = 1, the weighted sum m i=1 θ ia i is also in A. m i=1 θ ia i is called a convex combination of a 1,..., a m. Definition 2. Let A, B R N. Let θ R. A + B = {x R N : a A, b B such that x = a + b}. θa = {x R N : a A such that x = θa}. Remark 1. A B means A + ( B) not A \ B = {x A : x / B}. For example, if A = {1} and B = {2}, then A B = { 1} but A \ B =. Theorem 1. Let A be a set of convex sets in R N, let A, B A, and let θ R. 1. A + B is convex. 2. θa is convex. 1 cbna. This work is licensed under the Creative Commons Attribution-NonCommercial- ShareAlike 4.0 License. 1

2 3. A A A is convex. 4. A is convex. Proof. I prove that A is convex. I leave the others as exercises. Consider any a, â A and any θ [0, 1]. Let x = θa + (1 θ)â. Take sequences (a t ) and (â t ) in A such that a t a and â t â. (If a A then one can take a t = a and similarly for â.) Let x t = θa t + (1 θ)â t. Since A is convex, x t A. By continuity, x t x. Since A is closed, x A. Theorem 2. For any x R N and any ε > 0, N ε (x) is convex. Proof. This is just a special case of a general fact mentioned in the notes on metric spaces: in any normed vector space, N ε (x) is convex. In R, it is easy to prove that A is convex iff it is an interval (possibly infinite, not necessarily containing one or both endpoints), which holds iff A is connected. In R N, one can show that convex sets are connected, but not necessarily conversely. 3 A Basic Separation Result. Definition 3. A, B R N can be strictly separated iff there exists a v R N and an r R such that v a > r > v b for every a A, b B. Note that since v a > r, v 0. Geometrically, strict separation means the following. Any N 1 dimensional plane in R N can be represented as a set {x R N : v x = r}, where v R N, v 0 and r R. The vector v gives the tilt of the plane and the number r determines the position of the plane. If r = 0, then the plane passes through the origin; in this case, I denote the plane as T v : T v = {x R N : v x = 0}. Thus, the plane T v is the set of points orthogonal to v. For r 0, the plane is a parallel copy of T v, shifted away from the origin. Strict separation means that there is a plane such that the set A lies strictly on one side of the plane and B lies strictly on the other side. Many of the results on convex sets are corollaries of the following theorem. Theorem 3 (The Basic Separation Theorem BST). Let A R N be non-empty, closed, and convex. If the origin is not in A, then A and the origin can be strictly separated. 2

3 Proof. Take v to be the element of A that has smallest norm. That is, v is a solution (in fact, the solution) to min x. x A To see that this problem has a solution, note that the norm is continuous. A solution would therefore exist if A were compact. A is not necessarily compact, since it may not be bounded, but this is not really a problem. Since A is not empty, there is some a A. Since 0 A, but a A, a 0. Take  = A N a (0). Then  is non-empty (it contains a) and compact (it is the intersection of a closed set and a compact set) and so the modified problem min x x  has a solution. Call this solution v. By construction, the norm of v is smaller than the norm of any x in A \ Â, hence v solves the original minimization problem as well. Since 0 / A, but v A, v 0. Since v 0, v > 0. I claim that v a v v for any a A, which proves the result for any r (0, v v), since then v a > r > 0 = v 0. I argue by contraposition. Suppose that there is an x A such that v x < v v, hence v (x v) < 0. The directional derivative of the Euclidean norm at the point v in the direction x v is, by definition, v + θ(x v) v D x v v = lim. θ 0 θ On the other hand, D x v v = D v (x v). (See the notes on Multivariate Differentiation.) One can compute that hence, D v = 1 v v, v + θ(x v) v lim = 1 v (x v), θ 0 θ v which is strictly negative. This implies that for θ close to 0, setting x θ = v + θ(x v) = θx + (1 θ)v, x θ v < 0, θ hence x θ < v. Since v minimizes norm on A, this implies that x θ / A. Since x, v A, this shows that A is not convex. Since A is, in fact, convex, the proof follows by contraposition. Geometrically, the proof works as follows. Figure 1 provides illustration. Let v 3

4 T v v S S Separation No Separation v T v + {v} x x θ 0 T v Figure 1: Proof of BST. be the point in A that minimizes norm and let S be the sphere defined by S = {x R N : x = v }. Since S is a level set of the norm, S and since the gradient of the norm (namely v/ v ) is positively collinear with v, the set of vectors tangent to S at v is the plane, v T T x* v + {x*} T v + {v} v = {x R N : v x = v v}, See the illustration. T v + {v} splits R N into two half-spaces, one containing the origin and the other away from the origin. The proof verifies that if v x < v v, so that x lies on the half-space containing the origin, then the line segment between x and v must cut through S, implying the existence of a point x θ = θx + (1 θ)v / A for which x θ < v. Since v minimizes norm on A, this implies that x θ / A. But if x A, this implies that A is not convex. By contraposition, v a v v for all a A: all of A must lie in the half-space away from the origin. A B The next example shows that convexity is not necessary for BST. For simplicity, I set all of the next examples in N = 1. Similar examples can be constructed for any N. Note that when N = 1, v a = va. Example 1. Let N = 1 and let A = {1, 2}. Then A is closed and the origin is not in A but A not convex. Nevertheless, taking v = 1 A, va vv (i.e., a 1) for all a A, so the conclusion of Theorem 3 holds. E co(e) E On the other hand, some condition is needed, beyond A is closed and 0 / A, as the next example illustrates. Example 2. Let N = 1 and let A = { 1, 1}. Then A is closed and the origin is not in A but A cannot be strictly separated from the origin: if v > 0 then v( 1) < 0 while v(1) > 0; of course, v(0) = 0. The mirror image problem arises if v < 0. Similarly, it is not necessary that A be closed, but again some condition is needed, as the next example illustrates. Example 3. Let N = 1 and let A = (0, 1]. Then A is convex and the origin is not in A but A cannot be strictly separated from the origin. In particular, if v > 0 then va > v0 = 0 for every a A. But there is no r such that va > r > 0 for every a A. 4

5 The supplemental notes on convexity address these examples by providing conditions for separation that are necessary as well as sufficient. Remark 2. Although the results in these notes are stated for R N, they hold in any finite-dimensional vector space V. In particular, if V is a vector subspace of R N, then note that in the proof of Theorem 3, which is the foundation of all of the other support and separation theorems in these notes, the vector v is an element of A and hence an element of the vector space V that contains A. 4 The Sum of Closed Sets. For later use, I need the following fact about the sum of closed (but not necessarily convex) sets. Theorem 4. Let A, B R N be closed. If at least one set is compact, then A + B is closed. Proof of Theorem 4. Let C = A + B and suppose that (c t ) is a sequence in C that converges to c R N. I must show that c C. Since c t C there is an a t A and a b t B such that c t = a t + b t. For concreteness, suppose that A is compact. Then (a t ) has a subsequence that converges to a point of A. Call this subsequential limit a. Along this subsequence, continuity implies that b t = c t a t converges to c a. Define b = c a. Since B is closed, b B. Therefore, c = a + b A + B = C, as was to be shown. Example 4. If A and B are closed but neither is compact, then C = A + B may not be closed. For example, let A = {x R 2 : x 1 > 0 and x 2 1/x 1 } and let B = {x R 2 : x 2 = 0}. Then A and B are closed but C = A + B is not closed. In fact, C is the half space {(x 1, x 2 ) R 2 : x 2 > 0}; every point on the x 1 axis is a limit point of C but no point on the x 1 axis is an element of C. 5 A Support Theorem. Definition 4. A R N is supported at x iff v R N, v 0, such that v a v x for all a A. That is, a set A is supported at x if the set lies to one side of (but possibly touching) a plane that passes through x. The following theorem gives a sufficient condition for a set A to be supported at x. The theorem belongs to a class of results called supporting hyperplane theorems. For us, hyperplane is just another word for plane. Theorem 5 (Supporting Hyperplane Theorem). Let A R N be non-empty, closed, and convex. If x is not interior to A, then A is supported at x. 5

6 Proof. If A = then the result holds trivially (take any v 0). Henceforth assume that A. Since x is not interior to A, for any ε > 0 there is a x ε N ε (x ) A c. This implies that there is a sequence (x t ) in A c converging to x. For each x t, consider the set C t = A {x t }. Since A is closed and convex and {x t } is (trivially) compact and convex, C t is closed (Theorem 4) and convex (Theorem 1). Moreover, since x t / A, 0 / C t. Apply Theorem 3 to get v t R N and r t > 0 such that v t c > r t > 0 for all c C t. This implies that for all a A, v (a x t ) > 0, hence v t a > v t x t. Although x t x, there is no guarantee that v t converges to anything. This is easily fixed, however. Since v t 0 (again, v t c > r > 0 implies v t 0), ˆv t = v t / v t is well defined. Since v t a > v t x t, ˆv t a > ˆv t x t. Since ˆv t belongs to a compact set, namely the unit sphere, (ˆv t ) has a convergent subsequence, converging to, say, v on the unit sphere. Since v is on the unit sphere, v 0. By continuity of inner product it follows that, for all a A, which is what I needed to show. v a v x, Remark 3. In the Supporting Hyperplane Theorem, the hard case is when x is in A but not interior to A (i.e., x is on the boundary of A). If x A then the Separating Hyperplane Theorem (Theorem 6) implies that x and A can be strictly separated. The Supporting Hyperplane Theorem can be generalized to handle A that is not closed. I address this more difficult case in the supplemental notes on convexity. The hurdle is that I need to prove that if x is not interior to A then it is not interior to A. This is true if A is convex but not in general. 6 A Separation Theorem. The following theorem gives a sufficient condition for two sets to be strictly separated. The theorem can be viewed as a generalization of BST. The theorem belongs to a class of results called separating hyperplane theorems. Again, for us, hyperplane is just another word for plane. Theorem 6 (Separating Hyperplane Theorem). Suppose that A, B R N are nonempty, closed, convex, and disjoint. If at least one is compact, then A and B can be strictly separated. 6

7 Proof. Consider C = A B. C is non-empty, convex (Theorem 1), and closed (Theorem 4). Since A B =, 0 C. Applying Theorem 3, there is a v R N and a ˆr > 0 such that v c > ˆr > 0 for all c C. This implies that v a > v b + ˆr for all a A and all b B. Define r = inf v a a A r = sup v b. b B r and r are well defined because, for example, the set of v a is bounded below by any v b and hence the inf exists. Then r r + ˆr > r. Choose any r (r, r). Then v a > r > v b for any a A, b B, as was to be shown. As established in the supplemental notes on convexity, one can relax many of the conditions on A and B and still separate them, but not necessarily strictly. Example 5. As in Example 4, suppose A = {x R 2 : x 1 > 0 and x 2 1/x 1 } and B = {x R 2 : x 2 = 0}. Then A and B are closed, convex, and disjoint but neither is compact. To separate A and B, v must be collinear with (0, 1). Without loss of generality, suppose v = (0, 1). Then v b = 0 for all b B. On the other hand, inf a A v a = 0. This implies that there cannot be strict separation. Example 6. Strict separation can also fail if one of the sets is not closed, even if that set is bounded, the other set is compact, and the sets are disjoint. For example, let A = (1, 2) and let B = [0, 1]. Clearly we cannot have strict separation. 7