LP Geometry We now briefly turn to a discussion of LP geometry extending the geometric ideas developed in Section for dimensional LPs to n dimensions. In this regard, the key geometric idea is the notion of a hyperplane. Definition. A hyperplane in R n is any set of the form where a R n, β R, and a 0. H(a, β) = {x : a T x = β} We have the following important fact whose proof we leave as an exercise for the reader. Fact. H R n is a hyperplane if and only if the set H x 0 = {x x 0 : x H} where x 0 H is a subspace of R n of dimension (n ). and Every hyperplane H(a, β) generates two closed half spaces: H + (a, β) = {x R n : a T x β} H (a, β) = {x R n : a T x β}. Note that the constraint region for a linear program is the intersection of finitely many closed half spaces: setting H j = {x : e T j x 0} for j =,...,n and we have H n+i = {x : a ij x j b i } {x : Ax b, 0 x} = for i =,..., m n+m Any set that can be represented in this way is called a convex polyhedron. Definition. Any subset of R n that can be represented as the intersection of finitely many closed half spaces is called a convex polyhedron. Therefore, a linear programming is simply the problem of either maximizing or minimizing a linear function over a convex polyhedron. We now develop some of the underlying geometry of convex polyhedra. i= H i.
Fact.4 Given any two points in R n, say x and y, the line segment connecting them is given by [x, y] = {( λ)x + λy : 0 λ }. Definition.5 A subset C R n is said to be convex if [x, y] C whenever x, y C. Fact.6 A convex polyhedron is a convex set. We now consider the notion of vertex, or corner point, for convex polyhedra in R. For this, consider the polyhedron C R defined by the constraints (.6) c : x x c : x 4x 0 c : x + x 6. x 5 4 v c v C c c v 4 5 x The vertices are v = ( 8, ) 6 7 7, v = (0, ), and v = ( 4, ) 8 5 5. One of our goals in this section is to discover an intrinsic geometric property of these vertices that generalizes to n dimensions and simultaneously captures our intuitive notion of what a vertex is. For this we examine our notion of convexity which is based on line segments. Is there a way to use line segments to make precise our notion of vertex?
Consider any of the vertices in the polyhedron C defined by (.7). Note that any line segment in C that contains one of these vertices must have the vertex as one of its end points. Vertices are the only points that have this property. In addition, this property easily generalizes to convex polyhedra in R n. This is the rigorous mathematical formulation for our notion of vertex that we seek. It is simple, has intuitive appeal, and yields the correct objects in dimensions and. Definition.7 Let C be a convex polyhedron. We say that x C is a vertex of C if whenever x [u, v] for some u, v C, it must be the case that either x = u or x = v. This definition says that a point is a vertex if and only if whenever that point is a member of a line segment contained in the polyhedron, then it must be one of the end points of the line segment. In particular, this implies that vertices must lie in the boundary of the set and the set must somehow make a corner at that point. Our next result gives an important and useful characterization of the vertices of convex polyhedra. Theorem.8 (Fundamental Representation Theorem for Vertices) A point x in the convex polyhedron C = {x R s Tx g}, where T = (t ij ) s n and g R s, is a vertex of this polyhedron if and only if there exist an index set I {,...,s} with such that x is the unique solution to the system of equations (.7) t ij x j = g i i I. Moreover, if x is a vertex, then one can take I = n in (.7), where I denotes the number of elements in I. Proof: We first prove that if there exist an index set I {,...,s} such that x = x is the unique solution to the system of equations (.7), then x is a vertex of the polyhedron C. We do this by proving the contraposition, that is, we assume that x C is not a vertex and show that it cannot be the unique solution to any system of the form (.7) with I {,,..., s}. If x is not a vertex of C, then there exist u, v C and 0 < λ < such that x = ( λ)u + λv. Let A(x) denote the set of active indices at x: { } A(x) = i t ij x j = g i. For every i A( x) (.8) t ij x j = g i, t ij u j g i, and t ij v j g i.
Therefore, 0 = g i t ij x j = ( λ)g i + λg i [ = ( λ) g i 0. t ij (( λ)u + λv) ] t ij u j [ ] + λ g i t ij v j Hence, which implies that 0 = ( λ) [ g i = g i ] t ij u j + λ t ij u j and g i = [ g i t ij v j ] t ij v j [ since both g i ] [ n t iju j and g i ] n t ijv j are non-negative. That is, A( x) A(u) A(v). Now if I {,,..., s} is such that (.7) holds at x = x, then I A( x). But then (.7) must also hold for x = u and x = v since A( x) A(u) A(v). Therefore, x is not a unique solution to (.7) for any choice of I {,,..., s}. Let x C. We now show that if x is a vertex of C, then there exist an index set I {,...,s} such that x = x is the unique solution to the system of equations (.7). Again we establish this by contraposition, that is, we assume that if x C is such that, for every index set I {,,..., s} for which x = x satisfies the system (.7) there exists w R n with w x such that (.7) holds with x = w, then x cannot be a vertex of C. To this end take I = A( x) and let w R n with w x be such that (.7) holds with x = w and I = A( x), and set u = w x. Since x C, we know that t ij x j < g i i {,,..., s} \ A( x). Hence, by continuity, there exists τ (0, ] such that (.9) t ij ( x j + tu j ) < g i Also note that t ij ( x j ± τu j ) = ( t ij x j ) ± τ( t ij w j i {,,..., s} \ A( x) and t τ. 4 t ij x j ) = g i ± τ(g i g i ) = g i i A( x).
Combining these equivalences with (.9) we find that x+τu and x τu are both in C. Since x = (x + τu) + (x τu) and τu 0, x cannot be a vertex of C. It remains to prove the final statement of the theorem. Let x be a vertex of C and let I {,,..., s} be such that x is the unique solution to the system (.7). First note that since the system (.7) is consistent and its solution unique, we must have I n; otherwise, there are infinitely many solutions since the system has a non-trivial null space when n > I. So we may as well assume that I > n. Let J I be such that the vectors t i = (t i, t i,...,t in ) T, i J is a maximally linearly independent subset of the set of vectors t i = (t i, t i,...,t in ) T, i I. That is, the vectors t i i J form a basis for the subspace spanned by the vectors t i, i I. Clearly, J n since these vectors reside in R n and are linearly independent. Moreover, each of the vectors t r for r I \ J can be written as a linear combination of the vectors t i for i J ; t r = i J λ ri t i, r I \ J. Therefore, g r = t T r x = i J λ ri t T i x = i J λ ri g i, r I \ J, which implies that any solution to the system (.0) t T i x = g i, i J is necessarily a solution to the larger system (.7). But then the smaller system (.0) must have x as its unique solution; otherwise, the system (.7) has more than one solution. Finally, since the set of solutions to (.0) is unique and J n, we must in fact have J = n which completes the proof. We now apply this result to obtain a characterization of the vertices for the constraint region of an LP in standard form. Corollary. A point x in the convex polyhedron described by the system of inequalities Ax b and 0 x, where A = (a ij ) m n, is a vertex of this polyhedron if and only if there exist index sets I {,..., m} and J {,...,n} with I + J = n such that x is the unique solution to the system of equations (.9) a ij x j = b i i I, and x j = 0 j J. 5
Proof: Take in the previous theorem. T = [ A I ] and [ b g 0 Recall that the symbols I and J denote the number of elements in the sets I and J, respectively. The constraint hyperplanes associated with these indices are necessarily a subset of the set of active hyperplanes at the solution to (.9). Theorem. is an elementary yet powerful result in the study of convex polyhedra. We make strong use of it in our study of the geometric properties of the simplex algorithm. As a first observation, recall from Math 08 that the coefficient matrix for the system (.9) is necessarily non-singular if this n n system has a unique solution. How do we interpret this system geometrically, and why does Theorem. make intuitive sense? To answer these questions, let us return to the convex polyhedron C defined by (.7). In this case, the dimension n is. Observe that each vertex is located at the intersection of precisely two of the bounding constraint lines. Thus, each vertex can be represented as the unique solution to a system of equations of the form a x + a x = b a x + a x = b, where the coefficient matrix [ a a a a is non-singular. For the set C above, we have the following: (a) The vertex v = ( 8, 6 ) is given as the solution to the system 7 7 ] x x = x 4x = 0, (b) The vertex v = (0, ) is given as the solution to the system and (c) The vertex v = ( 4 5, 8 5 x x = x + x = 6, ) is given as the solution to the system x 4x = 0 x + x = 6. ] 6
Theorem. indicates that any subsystem of the form (.9) for which the associated coefficient matrix is non-singular, has as its solution a vertex of the polyhedron (.0) Ax b, 0 x if this solution is in the polyhedron. We now connect these ideas to the operation of the simplex algorithm. The system (.0) describes the constraint region for an LP in standard form. It can be expressed componentwise by a ij x j b i i =,...,m 0 x j j =,...,n. The associated slack variables are defined by the equations (.) x n+i = b i a ij x j i =,...,m. Let x = ( x,..., x n+m ) be any solution to the system (.) and set x = ( x,..., x n ) ( x gives values for the decision variables associated with the underlying LP). Note that if for some j J {,...,n} we have x j = 0, then the hyperplane H j = {x R n : e T j x = 0} is active at x, i.e., x H j. Similarly, if for some i I {,,..., m} we have x n+i = 0, then the hyperplane H n+i = {x R n : a ij x j = b i } is active at x, i.e., x H n+i. Next suppose that x is a basic feasible solution for the LP (P) maxc T x subject to Ax b, 0 x. Then it must be the case that n of the components x k, k {,,..., n + m} are assigned to the value zero since every dictionary has m basic and n non-basic variables. That is, every basic feasible solution is in the polyhedron defined by (.0) and is the unique solution to a system of the form (.9). But then, by Theorem., basic feasible solutions correspond precisely to the vertices of the polyhedron defining the constraint region for the LP P!! This amazing geometric fact implies that the simplex algorithm proceeds by moving from vertex to adjacent vertex of the polyhedron given by (.0). This is the essential underlying geometry of the simplex algorithm for linear programming! 7
By way of illustration, let us observe this behavior for the LP (.) maximize x + 4x subject to x + x x x 4 0 x, 0 x 4. The constraint region for this LP is graphed on the next page. x V V 4 V V 5 V V 6 x The simplex algorithm yields the following pivots: 8
- 0 0 0 vertex - 0 0 0 4 V = (0, 0) 0 0 0 0 0 0 0 0 4 4 0 0 0 0 0-0 0 0 vertex 0 0 0 0 6 V = (0, ) 0 0 0 0 0-0 0 0-4 0 0 0-8 0 0 0 0 4 vertex 0 0 0 0 6 V = (, 4) 0 0 0 0 0 0 0 0 0 0-9 The Geometry of Degeneracy 0 0 0 0 4 vertex 0 0 0 - V 4 = (, 4) 0 0 0-4 0 0 0 0 0 0 0 0 - -4-5 We now give a geometric interpretation of degeneracy in linear programming. Recall that a basic feasible solution, or vertex, is said to be degenerate if one or more of the basic variables is assigned the value zero. In the notation of (.) this implies that more than n of the hyperplanes H k, k =,,..., n+m are active at this vertex. By way of illustration, suppose we add the constraints x + x and x + x 7 to the system of constraints in the LP (.). The picture of the constraint region now looks as follows: 9
x V V 4 V V 5 V V 6 x Notice that there are redundant constraints at both of the vertices V and V 4. Therefore, as we pivot we should observe that the tableaus associated with these vertices are degenerate. 0
- 0 0 0 0 0 vertex - 0 0 0 0 0 4 V = (0, 0) - 0 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 4 4 0 0 0 0 0 0 0-0 0 0 0 0 vertex 0 0 0 0 0 0 6 V = (0, ) 0-0 0 0 0 0-0 0 0 0 5 0 0 0 0 0 0 0-0 0 0 0 0-4 0 0 0 0 0-8 0-0 0 0 0 4 vertex 0 0 0 0 0 0 6 V = (, 4) 0-0 0 0 0 0 0 0-0 0 0 0 0-0 0 0 0 0-0 0 0 degenerate 0 0 7 0-0 0 0-9 0 0 0 0 0 0 4 vertex 0 0 0 0 0 6 V = (, 4) 0 0 0-0 0 0 0 0 0 0-0 0 0 0 0-0 0 0-0 0 0 degenerate 0 0 0 0 0 0-7 -9 0 0 0 0 0 0 4 vertex 0 0 0 0-0 5 V 4 = (, 4) 0 0 0 0 0-0 0 0 0 0 - optimal 0 0 0 0 0-0 degenerate 0 0 0 0 0-4 0 0 0 0 0-0 - -5 In this way we see that a degenerate pivot arises when we represent the same vertex as the intersection point of a different subset of n active hyperplanes. Cycling implies that we are cycling between different representations of the same vertex. In the example given above, the third pivot is a degenerate pivot. In the third tableau, we represent the vertex
V = (, 4) as the intersection point of the hyperplanes and x + x = (since x = 0) x + x =. (since x 5 = 0) The third pivot brings us to the 4th tableau where the vertex V = (, 4) is now represented as the intersection of the hyperplanes and x + x = (since x 5 = 0) x =4 (since x 8 = 0). Observe that the final tableau is both optimal and degenerate. Just for the fun of it let s try pivoting on the only negative entry in the 5th row of this tableau (we choose the 5th row since this is the row that exhibits the degeneracy). Pivoting we obtain the following tableau. 0 0 0 0 0 0 0 4 0 0 0 0 0-0 0 0 0 0 0 0 0 0 0 0-0 0 0 0 0 - - 0 0 0 0 0 0-4 0 0 0 0 0 0 - - -5 Observe that this tableau is also optimal, but it provides us with a different set of optimal dual variables. In general, a degenerate optimal tableau implies that the dual problem has infinitely many optimal solutions. Fact: If an LP has an optimal tableau that is degenerate, then the dual LP has infinitely many optimal solutions. We will arrive at an understanding of why this fact is true after we examine the geometry of duality. The Geometry of Duality Consider the linear program (.) maximize x + x subject to x + x 4 x x 0 x, x. This LP is solved graphically below.
n = (, ) c = (, ) n = (, ) The solution is x = (, ). In the picture, the vector n = (, ) is the normal to the hyperplane x + x = 4, the vector n = (, ) is the normal to the hyperplane x x =, and the vector c = (, ) is the objective normal. Geometrically, the vector c lies between the vectors n and n. That is to say, the vector c can be represented as a non-negative linear combination of n and n : there exist y 0 and y 0 such that c = y n + y n, or equivalently, ( ) ( ) ( = y + y [ ] [ ] y =. y ) Solving for (y, y ) we have
- - - - 0 5 7 - - 0 7 5 0 6 5 0 7 5 or y = 6, y 5 = 7. I claim that the vector y = 5 (6, 7 ) is the optimal solution to the dual! 5 5 Indeed, this result follows from the complementary slackness theorem and gives another way to recover the solution to the dual from the solution to the primal, or equivalently, to check whether a point that is feasible for the primal is optimal for the primal. Theorem.4 (Geometric Duality Theorem) Consider the LP (P) maximize c T x subject to Ax b, 0 x. where A is an m n matrix. Given a vector x that is feasible for P, define Z( x) = {j {,,..., n} : x j = 0} and E( x) = {i {,..., m} : a ij x j = b i }. The indices Z( x) and E( x) are the active indices at x and correspond to the active hyperplanes at x. Then x solves P if and only if there exist non-negative scalars r j, j Z( x) and y i, i E( x) such that (.4) c = r j e j + y i a i j Z( x) i E( x) where for each i =,..., m, a i = (a i, a i,...,a in ) T is the ith column of the matrix A T, and, for each j =,...,n, e j is the jth unit coordinate vector. Moreover, the vector ȳ R m given by { yi for i E( x) (.5) ȳ i = 0 otherwise, solves the dual problem (D) maximize b T x subject to A T y c, 0 y. 4
Proof: Let us first suppose that x solves P. Then there is a ȳ R n solving the dual D with c T x = ȳ T A x = b T ȳ by the Strong Duality Theorem. We need only show that there exist r j, j Z( x) such that (.4) and (.5) hold. The Complementary Slackness Theorem implies that (.6) ȳ i = 0 for i {,,..., m} \ E( x) and (.7) m ȳ i a ij = c j for j {,...,n} \ Z( x). i= Note that (.6) implies that ȳ satisfies (.5). Define r = A T ȳ c. Since ȳ is dual feasible we have both r 0 and ȳ 0. Moreover, by (.7), r j = 0 for j {,..., n} \ Z( x), while r j = ȳ i a ij c j 0 for j Z( x), i= or equivalently, (.8) c j = r j + m ȳ i a ij for j Z( x). i= Combining (.8) with (.7) and (.6) gives c = r j e j + A T ȳ = j Z( x) j Z( x) r j e j + i E( x) ȳ i a i, so that (.4) and (.5) are satisfied with ȳ solving D. Next suppose that x is feasible for P with r j, j Z( x) and ȳ i, i E( x) non-negative and satisfying (.4). We must show that x solves P and ȳ solves D. Let ȳ R m be such that its components are given by the ȳ i s for i E( x) and by (.5) otherwise. Then the non-negativity of the r j s in (.4) imply that A T ȳ = i E( x) j Z( x) ȳ i a i j Z( x) i E( x) r j e j + i E( x) ȳ i a i = c, so that ȳ is feasible for D. Moreover, c T x = r j e T j x + ȳ i a T i x = ȳ T A x = ȳ T b, where the final equality follows from the definition of the vector ȳ and the index set E( x). Hence, by the Weak Duality Theorem x solves P and ȳ solves D as required. 5
Remark: As is apparent from the proof, the Geometric Duality Theorem is nearly equivalent to the complementary Slackness Theorem even though it provides a superficially different test for optimality. (.5) We now illustrate how to apply this result with an example. Consider the LP maximize x +x x +x 4 subject to x +x x +4x 4 4x x +x 4 x +x x 4 x x +x x 5 4 0 x, x, x, x 4. Does the vector x = (, 0,, 0) T solve this LP? If it does, then according to Theorem.4 we must be able to construct the solution to the dual of (.5) by representing the objective vector c = (,,, ) T as a non-negative linear combination of the outer normals to the active hyperplanes at x. Since the active hyperplanes are x + x x + 4x 4 = x + x x 4 = x = 0 x 4 = 0. This means that y = y 4 = 0 and y and y are obtained by solving 0 0 0 y 0 y 0 0 r =. 4 0 r 4 Row reducing, we get 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 Therefore, y = and y =. We now check to see if the vector ȳ = (, 0,, 0) does indeed solve the dual to (.5);. (.6) minimize y + y + y + 4y 4 subject to y y4 y + 4y y y 4 y y + y + y 4 4y + y y y 4 0 y, y, y, y 4. 6
Clearly, ȳ is feasible for (.6). In addition, b T ȳ = = c T x. Therefore, ȳ solves (.6) and x solves (.5) by the Weak Duality Theorem. 7