By W.E. Diewert. July, Linear programming problems are important for a number of reasons:

Save this PDF as:

Size: px
Start display at page:

Download "By W.E. Diewert. July, Linear programming problems are important for a number of reasons:"

Transcription

1 APPLIED ECONOMICS By W.E. Diewert. July, 3. Chapter : Linear Programming. Introduction The theory of linear programming provides a good introduction to the study of constrained maximization (and minimization) problems where some or all of the constraints are in the form of inequalities rather than equalities. Many models in economics can be expressed as inequality constrained optimization problems. A linear program is a special case of this general class of problems where both the objective function and the constraint functions are linear in the decision variables. Linear programming problems are important for a number of reasons: Many general constrained optimization problems can be approximated by a linear program. The mathematical prerequisites for studying linear programming are minimal; only a knowledge of matrix algebra is required. Linear programming theory provides a good introduction to the theory of duality in nonlinear programming. Linear programs appear in many economic contexts but the exact form of the problems varies across applications. We shall present several equivalent formulations of the basic linear programming problem in this introductory section. In the following section, we provide a geometric interpretation of a linear program (LP) in activities space. In subsequent sections, we will present George Dantzig s (963) simplex algorithm for solving an LP. Our first formulation of the basic linear programming problem is: () min x {c T x : Ax = b ; x N } where c T [c,c,,c N ] and b T [b,b,,b M ] are N and M dimensional vectors of constants, A [a mn ] is an M by N matrix of constants and x T [x,x,,x N ] is a nonnegative N dimensional vector of decision or choice variables. Thus this first form for a linear programming problem is the problem of minimizing a linear function c T x in the vector of nonnegative variables x N subject to M linear equality constraints, which are written in the form Ax = b. Linear programming was developed by George B. Dantzig in 947 as a technique for planning the diversified activities of the U.S. Air Force. Robert Dorfman, Paul A. Samuelson and Robert M. Solow (958; 3). Dorfman, Samuelson and Solow go on to note that Dantzig s fundamental paper was circulated privately for several years and finally published as Dantzig (95). A complete listing of Dantzig s early contributions to developing the theory of linear programming can be found in Dantzig (963; ).

2 Our second formulation of an LP is: () max x, x {x : x + c T x = ; Ax = b ; x N } where x is a new scalar variable, which is defined by the equality constraint in (); i.e., x c T x and so minimizing c T x is equivalent to maximizing x. It can be seen that the first and second formulations of an LP are completely equivalent. Our third formulation of an LP is the following problem: (3) max x {c T x : Ax b ; x N }. The above problem can be transformed into a problem of the type defined by () as follows: (4) max x, x, s {x : x c T x = ; Ax + I M s = b ; x N ; s M } where I M is the M by M identity matrix. Note that we have converted the inequality constraints Ax b into equality constraints by adding an M dimensional vector of nonnegative slack variables s to Ax. Note that equality constraints such as Ax = b can be converted into inequality constraints by writing Ax = b as two sets of inequality constraints: Ax b and Ax b. Note also that it does not matter whether we are maximizing the objective function c T x or minimizing c T x since a problem involving the maximization of c T x is equivalent to a problem involving the minimization of c T x. In the above problems, the vectors of variables x and s were restricted to be nonnegative. This is not an essential restriction, since if a variable, x say, is allowed to be unrestricted, it may be written in terms of two nonnegative variables, say s and s, as x = s s. However, in most economic problems, restricting the decision variables x to be nonnegative will be a reasonable assumption and it will not be necessary to resort to the x = s s construction. Thus the essence of a linear programming problem is that the objective function (the function that we are maximizing or minimizing) is linear and the constraint functions are linear equalities or inequalities. It turns out that our second formulation of an LP is the most convenient one for actually solving an LP (the simplex algorithm due to Dantzig) but the third formulation is the most convenient one for proving the duality theorem for linear programs.. The Geometric Interpretation of a Linear Program in Activities Space

3 3 Consider the following LP: (5) min x {c T x : Ax b ; x } where c T [,], b T [,] and the two rows of A are A [,] and A [,]. Thus we wish to minimize x + x subject to the four inequality constraints: x ; x + x ; x ; and x. We define the constraint set or feasible region in x space or activities space to be the set of x s which satisfy the constraints in (5). It is the shaded set in Figure below. We also graph the level sets of the objective function x + x ; i.e., these are the family of straight lines indexed by k, L(k) {x : c T x = k}. These level sets form a system of parallel straight lines. Obviously, the optimal x solution to the LP problem will be that x which belongs to the feasible region and which also belongs to the lowest level set of the objective function. Figure : A Linear Program in Activities Space x Feasible Region x +x (x *,x * ) = (,3) is the optimal solution L(4) {(x,x ) : x + x = 4} L(3) {x,x ) : x + x = 3} x x Note that the optimal solution to the LP lies on the boundary of the feasible region and that the optimal level set of the objective function is tangent to the feasible region. Finally, note that the optimal solution to the LP is at a vertex of the feasible region. This is typically what happens. 3. The Simplex Algorithm for Solving Linear Programs In this section, we outline Dantzig s (963; chapters 5-7) simplex algorithm for solving linear programming problems. Dantzig s method is not only of interest from a computational point of view, but also from a theoretical point of view, since it enables us Actually, we present a version of Dantzig s (963; chapter 9) revised simplex algorithm.

4 4 to present an entirely algebraic proof of the duality theorem for linear programming problems as we shall see later. However, before the method can be explained, we need some additional definitions and results. Definition : Any solution x to the system of linear equations Ax = b and inequalities x N is called a feasible solution to the LP defined by (). We are considering the system of linear equations, Ax = b, which occurs in () above where A is an M by N matrix. In what follows, we assume that the number of equations M is less than the number of x n variables, which is N. Hence A is a singular matrix. Definition : Any nonnegative solution x N to the system of equations Ax = b with at least N M components equal to zero is called a basic feasible solution to the LP defined by (). Theorem ; Carathéodory (9; ), Fenchel (953; 37), Dantzig (963; 3-4): If Ax = b, x N has a feasible solution, then it has a basic feasible solution. Proof: Assume that M < N and that there exists an x * [x *,,x N * ] T N such that Ax * = b. If x * has M or less nonzero components, then x * is a basic feasible solution and we are done. Thus we assume that x * has K > M nonzero components. By reordering the variables if necessary, we assume that the first K components of x * are the nonzero components. Now look at the following system of equations: (6) k= K A k λ k = M where A k is the kth column of the A matrix. The matrix [A, A,, A K ] is M by K where K > M and hence this matrix is singular or alternatively, the K columns in this matrix are linearly dependent. Hence, there is a nonzero λ * [λ *,,λ K * ] T solution to (6). Without loss of generality, we can assume that at least one component of λ * is positive. 3 We also have x k * > for k =,,,K. Hence the number α defined by (7) below must be positive: (7) α max k {λ k * /x k * : k =,,,K} >. Note that: (8) α λ k * /x k * or αx k * λ k * for k =,,,K. Since x * N is a feasible solution for Ax = b, we have: (9) b = Ax * = K * k= A k x k = K k= A k x * k α K * k= A k λ k using (6) 3 If all components of λ * were or negative, then λ * would satisfy (6), with all components or positive.

5 5 = α k= K A k [αx k * λ k * ]. But for all k which attain the max in (7), we will have αx k * λ k * equal to. Thus y k * α [αx k * λ k * ] for k =,,,K is a feasible solution for Ax = b with at least one additional zero component compared to our initial feasible solution x *. We can continue this process of creating extra zero components as long as K, the number of nonzero variables, exceeds M, the number of equations. The procedure may stop when K = M, but at that stage, we have a basic feasible solution to Ax = b. Q.E.D. In order to initiate the simplex algorithm, we need to start the algorithm with a basic feasible solution. But the above results say that given an arbitrary feasible solution, we can construct a basic feasible solution in a finite number of steps and thus initiate the simplex algorithm. We can rewrite the basic linear program defined by () above as follows: () max x x, x,..., x {x : e x + N n= A * n x n = b * }, N where c T [c,,c N ], A [A, A,, A N ] is the original A matrix and () A * cn n A n for n =,,N, b * and e b. M In what follows, we also need to define the following M unit vectors of dimension M+: () e T [,,,,] ; e T [,,,,,] ; ; e M T [,,,,]. Thus e m is an M+ dimensional vector consisting of elements except entry m+ is. Assume that a feasible solution to () exists. Then by Theorem, a basic feasible solution exists. By relabelling variables if necessary, we can assume that the first M columns of A correspond to the nonzero variables in this basic feasible solution. Thus we have x, x, x,,x M such that: (3) e x + m= M A m * x m = b *. At this point, we make two additional assumptions that will be relaxed at a later stage. Nonsingular Basis Matrix Assumption: [e, A *, A *,, A M * ] B exists. Nondegenerate Basis Matrix Assumption: x >, x >,,x M > ; i.e., the x m which satisfy (3) for m =,,,M are all positive.

6 6 We define the initial M+ by M+ basis matrix B as B [e, A *, A *,, A M * ]. By the nonsingular basis matrix assumption, we have the existence of B and (4) B B = B [e, A *, A *,, A M * ] = I M+ = [e,e,,e M ]. Now premultiply both sides of the constraint equations in () by B. In view of (4), the resulting system of M+ equations is the following one: (5) e x + e x + e x + + e M x M + B A M+ * x M+ + B A M+ * x M+ + + B A N * x N = B b *. If we set x M+ =, x M+ =,, x N = in (5), then we obtain our initial basic feasible solution to the LP problem: (6) x m = B m b * > for m =,,,M where B m is row m+ of B and the strict inequalities in (6) follow from the nondegenerate basis matrix assumption. The question we now ask is: is our initial basic feasible solution to the LP () the optimal solution to the problem or not? To answer this question, look at equations (5) with x M+, x M+,, x N all equal to zero. If we try to increase any of these last N M x n variables from its initial level, then obviously, we will increase x only if B A n * < for some n > M. Thus we have the following: (7) Optimality Criterion: If B A n * for n = M+, M+,,N, then the current basis matrix and the corresponding solution (6) solve the LP (). If B A n * > for n = M+, M+,,N, then the current basis matrix and the corresponding solution provide the unique solution to the LP (). Since B A n * = for n =,,,M using (4), the first part of the optimality criterion can be changed to: (8) B A n * for n =,,,N. Suppose now that column s (not in the basis matrix) is such that (9) B A s * <. In view of the nondegeneracy assumption, it can be seen that we can increase the value of our objective function x and satisfy the inequality constraints in the LP () by increasing x s from its initial level. The question now is: which column should be dropped from the initial basis matrix B? To answer this question, look at the following system of equations:

7 7 () e x + e x + e x + + e M x M + B A s * x s = B b *. Suppose that B m A s * for m =,,,M. Then as we increase x s from its initial level, the last M components of B A s * are either or negative and these nonpositive numbers can be offset by increasing x, x,, x M from their initial values x, x,, x M. Thus we have a feasible solution to () which will allow the objective function to attain any large positive value. In this case, we obtain the following: () Unbounded Solution Criterion: If we have a column s such that B A s * < and B m A s * for m =,,,M, then we may increase x s to + and obtain an unbounded solution to the LP (). Suppose now that (9) holds but that B m A s * > for at least one index m. In this case, as we increase x s from its initial level, in order to satisfy equations (), we must decrease x m from its initial positive level x m = B m b * >. Thus an upper bound to the amount that we can increase x s before we violate the nonnegativity constraints x N is B m b * /B m A s *. Now take the minimum of all such upper bounds over all indexes m such that B m A s * > : () x s * min m {B m b * /B m A s * : m and m is such that B m A s * > }. The algebra in the above paragraph can be summarized as the following: (3) Dropping Criterion: If B A s * < for some column index s not in the initial basis matrix and B m A s * > for at least one m, then add column s to the basis matrix and drop any column r such that column r attains the minimum in (); i.e., r is such that r and (4) B r b * /B r A s * = min m {B m b * /B m A s * : m and m is such that B m A s * > }. Using the nondegeneracy assumption, it can be seen that x s * defined by () is positive and introducing column s into the basis matrix leads to a strict increase in the objective function for the LP (). If the minimum in () or (4) is attained by a unique column r, then it can be seen that the new basis matrix is uniquely determined. It is also possible to show that the new basis matrix will also satisfy the nondegeneracy assumption. The above criteria form the core of an effective algorithm for solving the LP (). 4 Obviously, a finite but tremendously inefficient algorithm for solving () would involve solving the following system of M equations in M unknowns: 4 There are a few gaps but we will fill them in later.

8 8 (5) j= M A x n j n j = b for all possible choices of M of the N columns of the A matrix, checking to see if the resulting x are nonnegative, evaluating the objective function at these basic feasible n j solutions and then the nonnegative solution that gave the lowest objective function M j = cn j xn j would be picked. What the simplex algorithm does is to search through basis matrices in such a way that we always increase our objective function. 5 Experience with the simplex algorithm has shown that an optimal basis matrix is generally reached in the order of M to 3M iterations of the algorithm. Thus the simplex algorithm is tremendously more efficient than simply searching through all of the basic feasible solutions for the linear program. 4. An Example of the Simplex Algorithm Let us use the simplex algorithm to solve the LP that was graphed in section above. Rewriting the problem in the form () leads to the following problem: (6) max x x, x, s, s {x : e x + A * x + A * x + A * 3 s + A * 4 s = b * }, where e, A *, A *, A * 3, A * 4 and b *. Define the initial basis matrix to be (7) B [e, A *, A * ] = c T A a where A is a two by two matrix. Using determinants, if A a the formula for A is given by (provided that a a a a ): a a, we know that (8) A = a a a a a a a a = for our example. Using (8), we can readily calculate B as follows: 5 In general, we either increase the objective function or leave it unchanged at each iteration of the simplex algorithm or we find an unbounded solution.

9 9 (9) B = c T c T A = = A A. Check for feasibility: (3) x x x * * * = B b * = = 4. 3 To check that we have a basic feasible solution that satisfies the nonnegativity constraints, we need to check that x * and x *. Equations (3) show that these constraints are indeed satisfied. Check for optimality: We need to check that the inner product of the first row of B with each column not in the basis matrix is positive or zero. (3) B A * 3 = [,, ] = > ; B A * 4 = [,, ] = >. The optimality conditions are satisfied and thus x * = and x * = 3 solve the LP. In the above example, it was easy to find a starting basic feasible solution so that we could start the simplex algorithm. However, finding a starting basic feasible solution may not be all that easy in a complicated problem. In fact, it may be the case that a given LP may not even have a feasible solution. Thus in the following section, we discuss Dantzig s Phase I procedure for finding a starting basic feasible solution. 5. Finding a Starting Basic Feasible Solution Consider again the first form () of a linear programming problem, min x {c T x : Ax = b ; x N }. Now multiply both sides of each constraint equation through by if necessary so that the right hand side vector b is nonnegative; i.e., so that b M. Now consider the following Phase I LP where we have introduced a vector of artificial variables s: (3) min x,s { M T s : Ax + I M s = b ; x N ; s M }

10 where M is an M dimensional column vector of ones and I M is an M by M identity matrix. A basic feasible starting solution to (3) is s = b M and x = N. Note that this starting basis matrix is nonsingular. Now use the simplex algorithm to solve (3). 6 Several cases can occur. Case : The minimum for (3) is greater than. In this case, there is no feasible solution for the original LP and we can stop at this point. Case : The minimum for (3) equals. In this case, the original LP has feasible solutions. However, we need to consider two subcases. Case A: The minimum for (3) is and no columns corresponding to the artificial variables are in the final basis matrix. In this case, we have an x N such that Ax = b and at most M components of x are nonzero. Thus we have a starting basic feasible solution for the original LP (). Case B: The minimum for (3) is and one or more columns corresponding to the artificial variables are in the final basis matrix. In this case, we have a starting basic feasible solution for the following linear programming problem, which turns out to be equivalent to the original problem (): (33) max x, x, s {x : x + c T x = ; Ax + I M s = b ; M T s = }. N M Note that (33) is the same as () except that the vector of artificial variables s has been inserted into the constraint equations Ax = b and the extra constraint M T s = has been added to the problem, along with the nonnegativity restrictions on s, s M. However, note that the constraints M T s = and s M together imply that s = M. Thus the artificial variables will be kept at levels for all iterations of the simplex algorithm applied to (33) and so solving (33) will also solve (). Thus solving the Phase I linear programming problem (3) will tell us whether a feasible solution to the original problem () exists or not and if a feasible solution to the original problem does exist, the solution to the Phase I problem will give us a starting basic feasible solution to solve the original linear programming problem (), which is called the Phase II problem by Dantzig (963). 6 Note that the unbounded solution case cannot occur for this Phase I problem because the objective function is bounded from below by.

11 We conclude this section by solving the Phase I problem for the example in the previous section. Our Phase I problem is the following one: (34),,,, max s s x x x {x : e x + A * x + A * x + A 3 * x 3 + A 4 * x 4 + A 5 * s + A 6 * s = b * } where e, A *, A *, A 3 *, A 4 *, A 5 *, A 6 * and b *. Our starting basis matrix is B defined as follows: (35) B [e, A 5 *, A 6 * ] = and so B =. Feasibility Check: (36) () () () s s x = B b * = = 3. Since s () and s (), the feasibility conditions are satisfied. Optimality Check: (37) B A * = ; OK B A * = ; Can introduce A * into the basis matrix B A 3 * = ; OK B A 4 * = ; OK B A 5 * = ; OK (must equal since A 5 * is in the initial basis matrix) B A 6 * = ; OK (must equal since A 6 * is in the initial basis matrix). There is only one column A s * such that B A s * <, namely A *. If there were two or more columns with B A s * <, then pick the s such that B A s * is the most negative. Thus column A * will enter the new basis matrix but which column will leave? We need to calculate:

12 Dropping Criterion: (38) x () min m {B m b * /B m A * : m and m is such that B m A * > } = B b * /B A * = / = since B A * =, B A * = and B b * =. In other words, the minimum in (38) is attained for m = and the corresponding column of the original B matrix leaves the basis matrix; i.e., the last column A 6 * leaves. Thus the new basis matrix becomes: (39) B () [e, A 5 *, A * ] = and so [B () ] =. Feasibility Check: (4) x s x () () () = B () b * = =. Since s () and x (), the feasibility conditions are satisfied. Optimality Check: (4) B () A * = ; Can introduce A * into the basis matrix B () A * = ; OK (must equal since A * is in the basis matrix) B () A 3 * = ; OK B () A 4 * = ; OK B () A 5 * = ; OK (must equal since A 5 * is in the basis matrix) B () A 6 * = ; OK. There is only one column A s * such that B () A s * <, namely A *. Thus column A * will enter the new basis matrix but which column will leave? We need to calculate: Dropping Criterion: (4) x () min m {B m () b * /B m () A * : m and m is such that B m A * > } = B () b * /B () A * = / =

13 3 since B () A * =, B () A * = and B () b * =. In other words, the minimum in (4) is attained for m = and the corresponding column of the B () matrix leaves the basis matrix; i.e., the second column A 5 * leaves and A * enters. Thus the new basis matrix becomes: (43) B () [e, A *, A * ] = and so [B () ] =. Feasibility Check: (44) x x x () () () = B () b * = =. 3 Since x () and x (), the feasibility conditions are satisfied. Optimality Check: (45) B () A * = ; OK (must equal since A * is in the basis matrix) B () A * = ; OK (must equal since A * is in the basis matrix) B () A 3 * = ; OK B () A 4 * = ; OK B () A 5 * = ; OK B () A 6 * = ; OK. Thus the optimality conditions for solving the Phase I problem are satisfied with A * and A * the final columns in the optimal basis matrix. Note that the objective function x increased at each iteration of the Phase I problem and we ended up with a basis matrix that we could use to start the Phase II problem. Note also that the basis matrix was invertible at each stage of the Phase I algorithm. In the following section, we will show that the basis matrix is always invertible, provided that we start with a basis matrix that is invertible, which we can always do. Thus our nonsingular basis matrix assumption made in section is not a restrictive assumption. 6. Nonsingularity of the Basis Matrix in the Simplex Algorithm Suppose that at some iteration of the simplex algorithm, the basis matrix is B [e, A *,, A M * ] is nonsingular; i.e., B exists. If B A n * for n =,,,N, then we have an optimal solution and the algorithm stops. If B A s * < for some s with B m A s * for m =,,,M, then we have an unbounded optimal solution and the algorithm stops. Now consider the remaining case; i.e., suppose that there exists a column index s such that

14 4 (46) B A s * < and B m A s * > for at least one m. Thus A s * enters the basis matrix and according to the dropping criterion, A r * leaves the basis matrix where r is such that B r b * /B r A s * = min m {B m b * /B m A s * : m and m is such that B m A s * > }. Thus we have: (47) B r A s * >. Since the initial basis matrix B [e, A *,, A M * ] is nonsingular, it must be possible to express the entering column A s * as a linear combination of the columns of B; i.e., there exists α [α, α,, α M ] T such that (48) A s * = Bα or α = B A s *. Using (47) and (48), we have (49) α r = B r A s * >. Now form the new basis matrix B () defined as follows: (5) B () [e, A *, A *,, A r *, A s *, A r+ *, A r+ *,, A M * ]. It is easy to see that (48) and α r > imply that A r * can be expressed as a linear combination of the columns of B () : (5) A r * = B () β ; β [β, β,,β M ] T where (5) β r = /α r = /B r A s * and β m = α m /α r = B m A s * /B r A s * for m r. Comparing B with B (), it can be seen that (53) B [e, A *,, A r *, A r *, A r+ *,, A M * ] = [e, A *,, A r *, A s *, A r+ *,, A M * ][e, e,, e r, β, e r+,, e M ] = B () E where E [e, e,, e r, β, e r+,, e M ] and the components of β are defined by (5). Note that since β r = /α r >, the determinant of E is equal to β r > and hence E exists. Thus B () = BE and (54) B () = EB and hence the inverse of the new basis matrix exists. Thus we have proven:

15 5 Theorem ; Dantzig (963): The basis matrix is nonsingular at each iteration of the simplex algorithm. Note that the inverse of the new basis matrix B () is easy to compute using (54): we need only premultiply the inverse of the old basis matrix B by the matrix E [e, e,, e r, β, e r+,, e M ], which differs from the identity matrix in only one column. 7 There is one remaining detail to be filled in to complete our discussion of Dantzig s simplex algorithm for solving a linear program. 7. The Degeneracy Problem Recall the nondegeneracy assumption made in section 3. Recall that degeneracy occurs when the dropping criterion does not yield a unique column to be dropped; i.e., when we compute the minimum in (4), the minimum is attained by more than one column index m. When this situation occurs, then it can happen at the next stage of the simplex algorithm that we do not get a drop in the objective function and two or more columns cycle in and out of the basis matrix without causing a drop in the objective function. Thus there is the theoretical possibility that the simplex algorithm could enter an infinite cycle and the algorithm might fail to converge. It seems from an a priori point of view that it would be extremely unlikely that this cycling phenomenon could occur. However, Dantzig reports that the degeneracy phenomenon is common: It is common experience, based on the solutions of thousands of practical linear programming problems by the simplex method, that nearly every problem at some stage of the process is degenerate. George Dantzig (963; 3). When degeneracy occurs, it is conceivable that we could have a sequence of basis matrices where the objective function does not change such that the initial basis matrix reappears at some stage, which could then lead to an endless cycle of the same sequence of basis matrices. There are at least two artificially constructed examples of linear programming problems where this cycling phenomenon occurred; see Hoffman (953) and Beale (955). Dantzig (963; 8-3) presents these two examples but he comments on the phenomenon as follows: To date, there has not been one single case of circling, except in the specially constructed examples of Hoffman and Beale. Apparently, circling is a very rare phenomenon in practice. For this reason, most instruction codes for electronic computers use no special device for perturbing the problem to avoid degeneracy and the possibility of circling. George Dantzig (963; 3). 7 See (5) for the definition of the components of the β vector.

16 6 It turns out that cycling or circling can be avoided if we try dropping a different sequence of columns after going through a cycle of column changes where the objective function did not change the first time around. Dantzig (963) develops various rules for choosing which column to drop which work and he has references to rules developed by others. For our purposes, the exact form of the dropping rule is not important; all we need to know is that a dropping rule exists such that the simplex algorithm will terminate in a finite number of iterations, even when nondegeneracy is not assumed. We turn now to a topic of great economic interest. 8. The Dual Linear Program In this section, we use the third formulation for a linear programming problem as our starting point. Thus we suppose that we are given as data an M by N matrix A, an N dimensional vector c and an M dimensional vector b. Then the primal linear programming problem is defined as the problem of maximizing the linear function c T x with respect to the vector of primal decision variables x subject to the inequality constraints Ax b and the nonnegativity constraints x N : (55) Primal Problem: max x {c T x : Ax b ; x N }. The dual to the above problem switches the roles of b and c, changes a maximization problem into a minimization problem, switches from a vector of primal variables x that operate on the columns of A to a vector of dual variables y which operate on the rows of A and writes the inequality constraints in the opposite direction: (56) Dual Problem: min y {y T b : y T A c T ; y M }. Note that the dual problem is also a linear programming problem. The primal and dual problems are related by the following theorem: Theorem 3; von Neumann (947), Dantzig (963; chapter 6): If the primal problem has a finite optimal solution x * say, then the dual problem also has a finite optimal solution y * say and moreover: (57) c T x * = y *T b ; i.e., the optimal values of the primal and dual objective functions are equal. Proof: We first put the primal problem into the standard simplex algorithm format. Define x c T x. After adding the vector of slack variables s to Ax, we find that the primal problem (55) is equivalent to the following problem: (58) max x, x, s T T c M x {x N M : x + = M A IM s }. b

17 7 Apply Phase I of the simplex algorithm to the LP defined by (58). Since we assume that the primal problem has a finite optimal solution, the optimal objective function for the Phase I problem will attain its lower bound of. Thus we will have: (59) = max x, x, s, z x T T T N M M {x N M M : x + M A I M I s M z = } b where we have added the vector of artificial variables z M to the constraints of (58). There are two cases that can occur at this point. Case : In this case, the Phase I problem yields a nonsingular initial basis matrix for the Phase II problem (58), which does not have any columns in it that correspond to the artificial variables z. Thus in this case, we may apply the simplex algorithm to (58) without including any artificial variables and since by assumption, there is a finite optimal solution to (58), the simplex algorithm will terminate and there will exist an M+ by M+ nonsingular basis matrix B (where the first column of B is the unit vector e ) such that the following optimality conditions hold: T T c M (6) B A I [ T N, T M ] and B e = M where B [, y *T ] say, is the first row of B. 8 Thus the optimality conditions (6) imply that y * satisfies the following inequalities: (6) y *T A c T and y *T I M M T. Thus y * is a feasible solution for the dual problem (56). At this point, we need the following auxiliary result: if x * and y * are feasible solutions for the primal and dual problems respectively, then we have the following bounds for the objective functions for the two problems: (6) c T x * y *T b ; i.e., the dual objective function evaluated at a feasible solution to the dual, y *T b, is an upper bound for the primal objective function evaluated at any feasible solution and the primal objective function evaluated at a feasible solution for the primal, c T x *, is a lower bound for the dual objective function evaluated at any feasible solution. To prove (6), note that x * feasible for the primal problem and y * feasible for the dual problem means that x * and y * satisfy the following inequalities: 8 We can deduce that the first component of B is using B e = since e is the first column of B.

18 8 (63) x * N ; Ax * b ; y * M ; y *T A c T. Thus (64) c T x * c T x * + y *T (b Ax * ) since y * M and Ax * b = y *T b + (c T y *T A)x * rearranging terms y *T b since x * N and y *T A c T. Now (6) shows that the y * defined by B [, y *T ] is a feasible solution for the dual. In view of (64), we need only show that y *T b = c T x * where x * solves the primal, and we will have y * as an optimal solution to the dual problem. Substitute the optimal Phase II solution to (58), x *, x * and s * into the constraints listed in (58) and then premultiply both sides of this matrix equation by B [, y *T ]. We obtain the following equation: (65) [, y *T ] * x + [, y *T T T * c M x ] * M A IM s = [, y *T ]. b Obviously, [, y *T ][, T M ] T = and so the first term on the left hand side of (65) is simply equal to x *. It turns out that the next set of terms on the left hand side of (65) is T T c M equal to since if a particular column in the matrix is in the optimal basis A IM matrix, the inner product of this column with the first row of B, [, y *T ], will be 9 and if a particular column in this matrix is not in the optimal basis matrix, then the corresponding x * n or s * m is zero and again the term will be. The term on the right hand side of (65) is equal to y *T b. Thus (65) simplifies to: (66) x * = y *T b. Since x * = c T x *, we have c T x * = y *T b and thus y * is indeed an optimal solution to the dual problem. Case : In this case, the solution to the Phase I problem contains one or more columns corresponding to the artificial variables z. In this case, the Phase I solution gives us a starting basic feasible solution to the following problem, which is equivalent to the primal problem (55): 9 This follows from B B = I M+.

19 9 (67) max x, x, s, z {x : N M M c x M + A T N T x + I T M M T M s + T M I T M M z = b }. We have set up the artificial variables in a slightly different way than was suggested in section 5 above. Since we already have added + I M s to Ax, we added I M z as the artificial variables. Between the s and z variables, we can obviously find a Phase I starting basic feasible solution to the constraints Ax + I M s I M z = b, no matter what the signs are for the components of the b vector. Now apply the Phase II simplex algorithm to (67). The constraints M T z = and z M will force the z variables to be kept at levels at all iterations of the Phase II algorithm. Since we assumed that a finite optimal solution to the original primal problem existed, the Phase II simplex algorithm will eventually terminate and there will exist an M+ by M+ basis matrix C such that the following optimality conditions will hold: (68) C c A T N T I T M M T M T M I M T M [ N T, M T, M T ] where C [, y *T, y * ] is the first row of C. The optimal solution to (67), x *, x *, s *, z * will satisfy the following relations: (69) x * = c T x * ; x * N ; s * M ; z * = M. The first two sets of inequalities in (68) imply that y * satisfies: (7) y *T A c T and y * M which implies that y * is feasible for the dual problem. Now evaluate the constraints in (67) at the optimal Phase II solution, x *, x *, s *, z * and premultiply the constraint equations through by C [, y *T, y * ]. We find that as in Case : (7) x * + = y *T b Where the second term is because columns not in the final Phase II basis matrix are multiplied by while the columns in the final basis matrix have a inner product with the first row of the inverse of the basis matrix, C [, y *T, y * ]. Since x * is feasible (and optimal) for the primal problem and y * is feasible for the dual problem, (69) and (7) and the auxiliary result imply that y * is a solution to the dual with y *T b = c T x *. Q.E.D.

20 Corollary: If the primal problem has a feasible solution and the dual problem has a feasible solution, then both the primal and dual problems have finite optimal solutions and the values of the optimal objective functions coincide. Proof: The auxiliary result (6) shows that if the primal and dual both have feasible solutions, then the primal objective function is bounded from above. Hence the unbounded solution case for the primal cannot occur. Now repeat the proof of Theorem 3. Q.E.D. Note that the above proof of the duality theorem for linear programs is entirely algebraic and rests on the mechanics of the simplex algorithm for solving linear programs. The proof of the above theorem is essentially due to Dantzig (963). A further implication of the above duality theorem is that if the primal solution has an unbounded optimal solution, then the dual cannot have a feasible solution. The duality theorem for linear programs has many economic applications as will be seen by studying the problems at the end of the chapter. We conclude this section by proving another result that will be helpful in providing an economic interpretation for the dual variables. Theorem 4: Basis Theorem for Linear Programs; Dantzig (963; ): Suppose that a nonsingular basis matrix B is optimal for the problem: (7) max x, x {x : e x + A * x = b * ; x N }. Suppose further that this basis matrix B generates a feasible solution for the following problem where the right hand side vector b * has been replaced by b : (73) max x, x {x : e x + A * x = b ; x N }. Then the basis matrix B is also optimal for the new problem (73). Proof: Since B is an optimal basis matrix for (7), then we have: (74) B m b * for m =,,,M (feasibility conditions satisfied); (75) B A n * for n =,,,N (optimality conditions satisfied). But by assumption, the initial optimal basis matrix B generates a feasible solution for the new b vector, b, so that we have: (76) B m b for m =,,,M.

21 But since the basis matrix has not changed, the optimality conditions (75) are still satisfied (these conditions do not depend directly on b) and so (75) and (76) imply that the old basis matrix B is still optimal for the new problem (73). Q.E.D. Theorem 4 is helpful in providing an economic interpretation for the dual variables. However, to see the connection, we have to consider a slightly different primal and dual LP problems compared to (55) and (56). Thus consider an equality constrained primal problem of the following form: (77) max x { c T x : Ax = b ; x N }. In order to find the dual problem to (77), we need to write the equality constraints Ax = b in an inequality format. Thus (77) is equivalent to the following problem: (78) max x { c T A b x : x A ; x N }. b Using our rules for forming the dual problem, it can be seen that the dual to (78) is: (79) min y, y {y T b y T b : [y T, y T A ] c T ; y M ; y M }. A Define the unrestricted vector of variables y y y and using this definition, we can rewrite the dual problem (79) as follows: (8) min y {y T b : y T A c T }. Thus (8) is the dual problem to the primal problem (77): equality constraints in the primal lead to a dual problem with unrestricted in sign dual variables (instead of nonnegative dual variables as before). Assume that there is a finite optimal solution to (77). Putting (77) into the standard simplex algorithm format leads to the following equivalent problem: c T (8) max x, x { x : x + x = ; x N }. M A b Suppose that the final optimal basis matrix for (8) turns out to include the first M columns of the A matrix. Thus we have c... cm (8) B = M A... A M

22 and we assume that B is nonsingular. Our final assumption is that the final basis matrix B for (8) is nondegenerate so that (83) x * m = B m > for m =,,,M. b We may repeat the proof of the Duality Theorem for linear programs in this set up. As before, we find that (84) B [, y *T ] where y * turns out to be a solution to the dual problem (8). The nondegeneracy assumptions (83) imply that y * must satisfy the following equations: (85) [, y *T cm ] A m = for m =,,,M. Equations (85) imply that the dual variables must be unique. Thus nondegeneracy of the final basis matrix implies that the optimal solution to the dual problem is unique. Recall that the feasibility restrictions (83) are satisfied by our basis matrix, which we rewrite as follows: (86) B m > for m =,,,M. b The following optimality conditions are also satisfied by our basis matrix B: c n (87) B for n =,,,N. A n Now we can apply the Basis Theorem to the above setup. If b is sufficient close to the initial b, the nonnegativity restrictions (86) will continue to hold using the old basis matrix B; i.e., we will have, b close to b: (88) B m b > for m =,,,M. The optimality conditions (87) will continue to hold for this new primal problem where b replaces b and hence the basis matrix B continues to be optimal for the new problem. Setting x M+ = x M+ = = x N = for the constraints in (8) and premultiplying both sides of the constraints by [, y *T ] where y * is defined by (84) leads to the following equation, where we have also used (85):

23 3 (89) x = y *T b. All of this shows that for b close to b, the optimal primal and dual objective functions, regarded as functions of b, are given by (9) V(b ) = y *T b. Thus for b = b, the vector of first order partial derivatives of the optimized primal and dual objective functions, with respect to the components of b is equal to: (9) b V(b) = y * where the unique vector of dual variables y * is defined by (84). Thus y m * is the marginal increase in the primal (and dual) objective functions due to a small increase in the mth component of b, b m, provided that we have a nondegenerate primal solution. Note that this interpretation for a dual variable is entirely analogous to the economic interpretation of a Lagrange multiplier in classical constrained optimization. 9. The Geometric Interpretation of a Linear Program in Requirements Space Recall the basic linear program defined by () above and recall that we rewrote in the form () above. For convenience, we repeat () below as problem (9): (9) max x x, x,..., x {x : e x + N n= A * n x n = b * }, N where c T [c,,c N ], A [A, A,, A N ] is the original A matrix and (93) A * cn n A n for n =,,N, b * and e b. M Define the super feasibility set S as the following set in M+ dimensional space: (94) S {z : z = n= N A n * x n ; x, x,, x N }. The set S is the set of all nonnegative linear combinations of the N column vectors A n *. Thus S is a set in M+ dimensional space. A set S is a cone if and only if it has the following property: See Samuelson (947; 65) and Diewert (984; 48) for this interpretation for a Lagrange multiplier in classical (equality constrained) optimization theory. Dorfman, Samuelson and Solow (958; 5-59) have a nice discussion on the meaning and interpretation of dual variables. Problem 6 at the end of the chapter provides an economic interpretation for the dual prices in the general case where conditions (86) hold only as weak inequalities rather than as strict inequalities. Contrast this definition with the feasible set of x s, which is the set of x such that x N and Ax = b.

24 4 (95) z S, λ implies λz S. It is easy to verify that the super feasibility set S defined by (94) is a cone. Note that this set S does not involve the right hand side vector b which appeared in the original constraints for the LP, Ax = b. The vector b is often called the requirements vector; i.e., a nonnegative vector x must be chosen so that production Ax meets the requirements b. The requirements line L is defined as follows: z z (96) L { z : z z M z M z = b b M where z is an unrestricted scalar variable}. Note that points z that belong to both L and S correspond to feasible solutions for the LP (9). An optimal solution for (9) will correspond to a point z such that: z S z L and The first component of z is as small as possible subject to z S and z L. Thus to solve (9), we look for the lowest point (with respect to the first component of z) on the requirements line that also belong to the super feasibility set S. It also turns out that any hyperplane that supports the set S (is tangent to S) at this lowest point of L provides a set of optimal dual variables. An example will help to illustrate these points. Suppose that we have 4 production units in a region or a country that uses varying amounts of two inputs and each production unit when run at unit scale produces output that is valued at dollar. The input requirements of the 4 production units when run at unit scale are the A n listed below: / (97) A ; A ; A 3 / ; A 4. Thus the first production unit uses units of the first input and units of the second input in order to produce outputs worth dollar, the second production unit uses / of a unit of the first input and unit of the second input in order to produce outputs worth dollar and so on. The 4 production units are controlled by a single firm which can allocate b > units of input and b > units of input across the 4 production units. Production in each plant is subject to constant returns to scale. The firm is interested in allocating the amounts of the two inputs across the 4 plants in order to maximize revenue; i.e., the firm would like to solve the following linear programming problem: It is also a convex set so it is a convex cone.

25 5 (98) max x {c T x : Ax = b ; x 4 } where c T [,,,], b T [b,b ], A [A, A, A 3, A 4 ] and x T [x,x,x 3,x 4 ] is the vector of nonnegative plant scales. To put the problem defined by (97) into standard simplex algorithm format, define the A n * by adding c n as an extra component to the A n : (99) A * ; A * / ; A 3 * / ; A 4 * ; b * b. b Using the above notation, the LP (98) can be rewritten as follows: () max x, x {x : e x + A * x + A * x + A * 3 x 3 + A * 4 x 4 = b * }. Now recall the definition of the super feasibility set S, (94). For our example LP (), this set will be a cone in three dimensional space. Now note that each of the 4 column vectors A *, A *, A 3 * and A 4 * will belong to S and the first component for each of these vectors is equal to. Hence if we intersect the cone S with the plane {(z,z,z ): z =, z and z are unrestricted, we obtain the shaded set in Figure below. 3 3 A three dimensional diagram would be more instructive but it is more difficult to graph. The reader should think of S as a 3 dimensional set in (b,b,b ) space. What we are doing in Figure is intersecting this three dimensional set with various (negative) heights or levels for b and then we plot these level sets in (b,b ) space.

26 Figure : The Simplex Algorithm in Requirements Space Cone A b Cone (b,b ) = (3/,3/) b = indifference curve 6 A b = indifferenc e curve (,) Cone 3 A 3 A 4 4 b The lower boundary of the shaded set is the counterpart to an indifference curve in consumer theory; this curve gives the set of (b,b ) points which will allow the firm to earn one unit of revenue without wasting resources. 4 The point A is the top point on this curve and it is joined by a straight line to the point A, which in turn is joined to the point A 3, which in turn is joined to the lowest point on the curve, A 4. Now suppose that the firm has (b,b ) = (3/, 3/) units of the two inputs available to use in the 4 production units. One can go through the algebra of the simplex algorithm in this case and one will * find that A and A 3 are the efficient plants to use in this case and the optimal x s are x =, x * =, x * 3 = and x * 4 =. The optimal objective function in this case is x * =. In terms of Figure, it can be seen that we get to the point (b,b ) = (3/, 3/) by moving up the dashed line through A until we hit A and by moving up the dashed line through A 3 until we hit A 3 and then we complete the parallelogram with the dotted lines until we hit (b,b ) = (3/, 3/). Note that the two dashed lines enclose a region of (b,b ) that we called cone. Using the Basis Theorem for linear programs, it can be seen that we will use only production units and 3 provided that (b,b ) falls into this region. Using a similar argument, it can be seen only production units and will be used if (b,b ) falls into the cone region and only production units 3 and 4 will be used if (b,b ) falls into the cone 3 region. Suppose that (b,b ) falls into the interior of the cone region. The corresponding (unique) dual (input) prices can be calculated by solving the following two equations in two unknowns: () [,y,y ]A * = ; [,y,y ]A 3 * =. 4 In terms of the set S, this curve is the set of (b,b ) points such that (,b,b ) belongs to the boundary of S.

27 7 Using our data listed in (99), we find that the cone dual solution is: () y () = /3 ; y () = /3. Thus the input prices are equal in this case where the resource availability vector falls into the cone region. This corresponds to the fact that the parallel indifference curves in this region all have slope equal to (which is equal to y () /y () ). Now suppose that (b,b ) falls into the interior of the cone region. The corresponding (unique) dual (input) prices can be calculated by solving the following two equations in two unknowns: (3) [,y,y ]A * = ; [,y,y ]A * =. Using our data listed in (99), we find that the cone dual solution is: (4) y () = ; y () = /. Thus the input price is twice the size of the input price in this region where input is relatively scarce compared to input. This corresponds to the fact that the parallel indifference curves in this region all have slope equal to (which is equal to y () /y () ). Finally suppose that (b,b ) falls into the interior of the cone 3 region. The corresponding (unique) dual (input) prices can be calculated by solving the following two equations in two unknowns: (5) [,y,y ]A 3 * = ; [,y,y ]A 4 * =. Using our data listed in (99), we find that the cone 3 dual solution is: (6) y (3) = / ; y (3) =. Thus the input price is twice the size of the input price in this region where input is relatively scarce compared to input. This corresponds to the fact that the parallel indifference curves in this region all have slope equal to / (which is equal to y (3) /y (3) ). What happens if the endowment vector (b,b ) happens to fall on the dashed line through the origin and A so that (b,b ) is on the boundary of both cone and cone? In this case, although the primal solution is unique (x * solves the primal where A x * = b), the dual solution is not. In this case the dual solution set is: (7) (y,y ) = λ (y (),y () ) + ( λ) (y (),y () ) ; λ ;

Linear Programming for Optimization. Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc.

1. Introduction Linear Programming for Optimization Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc. 1.1 Definition Linear programming is the name of a branch of applied mathematics that

Definition of a Linear Program

Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1

Practical Guide to the Simplex Method of Linear Programming

Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear

1 Solving LPs: The Simplex Algorithm of George Dantzig

Solving LPs: The Simplex Algorithm of George Dantzig. Simplex Pivoting: Dictionary Format We illustrate a general solution procedure, called the simplex algorithm, by implementing it on a very simple example.

4.6 Linear Programming duality

4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal

Duality in Linear Programming

Duality in Linear Programming 4 In the preceding chapter on sensitivity analysis, we saw that the shadow-price interpretation of the optimal simplex multipliers is a very useful concept. First, these shadow

3 Does the Simplex Algorithm Work?

Does the Simplex Algorithm Work? In this section we carefully examine the simplex algorithm introduced in the previous chapter. Our goal is to either prove that it works, or to determine those circumstances

Special Situations in the Simplex Algorithm

Special Situations in the Simplex Algorithm Degeneracy Consider the linear program: Maximize 2x 1 +x 2 Subject to: 4x 1 +3x 2 12 (1) 4x 1 +x 2 8 (2) 4x 1 +2x 2 8 (3) x 1, x 2 0. We will first apply the

Linear Programming. March 14, 2014

Linear Programming March 1, 01 Parts of this introduction to linear programming were adapted from Chapter 9 of Introduction to Algorithms, Second Edition, by Cormen, Leiserson, Rivest and Stein [1]. 1

Solving Linear Programs

Solving Linear Programs 2 In this chapter, we present a systematic procedure for solving linear programs. This procedure, called the simplex method, proceeds by moving from one feasible solution to another,

LECTURE 5: DUALITY AND SENSITIVITY ANALYSIS. 1. Dual linear program 2. Duality theory 3. Sensitivity analysis 4. Dual simplex method

LECTURE 5: DUALITY AND SENSITIVITY ANALYSIS 1. Dual linear program 2. Duality theory 3. Sensitivity analysis 4. Dual simplex method Introduction to dual linear program Given a constraint matrix A, right

1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where.

Introduction Linear Programming Neil Laws TT 00 A general optimization problem is of the form: choose x to maximise f(x) subject to x S where x = (x,..., x n ) T, f : R n R is the objective function, S

OPRE 6201 : 2. Simplex Method

OPRE 6201 : 2. Simplex Method 1 The Graphical Method: An Example Consider the following linear program: Max 4x 1 +3x 2 Subject to: 2x 1 +3x 2 6 (1) 3x 1 +2x 2 3 (2) 2x 2 5 (3) 2x 1 +x 2 4 (4) x 1, x 2

Linear Programming Notes V Problem Transformations

Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material

Mathematical finance and linear programming (optimization)

Mathematical finance and linear programming (optimization) Geir Dahl September 15, 2009 1 Introduction The purpose of this short note is to explain how linear programming (LP) (=linear optimization) may

7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

Linear Programming in Matrix Form

Linear Programming in Matrix Form Appendix B We first introduce matrix concepts in linear programming by developing a variation of the simplex method called the revised simplex method. This algorithm,

Linear Programming I

Linear Programming I November 30, 2003 1 Introduction In the VCR/guns/nuclear bombs/napkins/star wars/professors/butter/mice problem, the benevolent dictator, Bigus Piguinus, of south Antarctica penguins

Solving Systems of Linear Equations

LECTURE 5 Solving Systems of Linear Equations Recall that we introduced the notion of matrices as a way of standardizing the expression of systems of linear equations In today s lecture I shall show how

Duality of linear conic problems

Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least

Linear Dependence Tests

Linear Dependence Tests The book omits a few key tests for checking the linear dependence of vectors. These short notes discuss these tests, as well as the reasoning behind them. Our first test checks

Vector Spaces II: Finite Dimensional Linear Algebra 1

John Nachbar September 2, 2014 Vector Spaces II: Finite Dimensional Linear Algebra 1 1 Definitions and Basic Theorems. For basic properties and notation for R N, see the notes Vector Spaces I. Definition

MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.

MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column

The Graphical Method: An Example

The Graphical Method: An Example Consider the following linear program: Maximize 4x 1 +3x 2 Subject to: 2x 1 +3x 2 6 (1) 3x 1 +2x 2 3 (2) 2x 2 5 (3) 2x 1 +x 2 4 (4) x 1, x 2 0, where, for ease of reference,

Linear Programming. Solving LP Models Using MS Excel, 18

SUPPLEMENT TO CHAPTER SIX Linear Programming SUPPLEMENT OUTLINE Introduction, 2 Linear Programming Models, 2 Model Formulation, 4 Graphical Linear Programming, 5 Outline of Graphical Procedure, 5 Plotting

24. The Branch and Bound Method

24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NP-complete. Then one can conclude according to the present state of science that no

What is Linear Programming?

Chapter 1 What is Linear Programming? An optimization problem usually has three essential ingredients: a variable vector x consisting of a set of unknowns to be determined, an objective function of x to

Linear Programming. April 12, 2005

Linear Programming April 1, 005 Parts of this were adapted from Chapter 9 of i Introduction to Algorithms (Second Edition) /i by Cormen, Leiserson, Rivest and Stein. 1 What is linear programming? The first

EQUATIONS and INEQUALITIES

EQUATIONS and INEQUALITIES Linear Equations and Slope 1. Slope a. Calculate the slope of a line given two points b. Calculate the slope of a line parallel to a given line. c. Calculate the slope of a line

This exposition of linear programming

Linear Programming and the Simplex Method David Gale This exposition of linear programming and the simplex method is intended as a companion piece to the article in this issue on the life and work of George

Arrangements And Duality

Arrangements And Duality 3.1 Introduction 3 Point configurations are tbe most basic structure we study in computational geometry. But what about configurations of more complicated shapes? For example,

1. LINEAR EQUATIONS. A linear equation in n unknowns x 1, x 2,, x n is an equation of the form

1. LINEAR EQUATIONS A linear equation in n unknowns x 1, x 2,, x n is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b, where a 1, a 2,..., a n, b are given real numbers. For example, with x and

Chapter 5. Linear Inequalities and Linear Programming. Linear Programming in Two Dimensions: A Geometric Approach

Chapter 5 Linear Programming in Two Dimensions: A Geometric Approach Linear Inequalities and Linear Programming Section 3 Linear Programming gin Two Dimensions: A Geometric Approach In this section, we

DETERMINANTS. b 2. x 2

DETERMINANTS 1 Systems of two equations in two unknowns A system of two equations in two unknowns has the form a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 This can be written more concisely in

We call this set an n-dimensional parallelogram (with one vertex 0). We also refer to the vectors x 1,..., x n as the edges of P.

Volumes of parallelograms 1 Chapter 8 Volumes of parallelograms In the present short chapter we are going to discuss the elementary geometrical objects which we call parallelograms. These are going to

Operation Research. Module 1. Module 2. Unit 1. Unit 2. Unit 3. Unit 1

Operation Research Module 1 Unit 1 1.1 Origin of Operations Research 1.2 Concept and Definition of OR 1.3 Characteristics of OR 1.4 Applications of OR 1.5 Phases of OR Unit 2 2.1 Introduction to Linear

Linear Algebra Notes for Marsden and Tromba Vector Calculus

Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of

Lecture 3: Linear Programming Relaxations and Rounding

Lecture 3: Linear Programming Relaxations and Rounding 1 Approximation Algorithms and Linear Relaxations For the time being, suppose we have a minimization problem. Many times, the problem at hand can

Linear Programming. Widget Factory Example. Linear Programming: Standard Form. Widget Factory Example: Continued.

Linear Programming Widget Factory Example Learning Goals. Introduce Linear Programming Problems. Widget Example, Graphical Solution. Basic Theory:, Vertices, Existence of Solutions. Equivalent formulations.

SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA

SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA This handout presents the second derivative test for a local extrema of a Lagrange multiplier problem. The Section 1 presents a geometric motivation for the

Sensitivity Analysis 3.1 AN EXAMPLE FOR ANALYSIS

Sensitivity Analysis 3 We have already been introduced to sensitivity analysis in Chapter via the geometry of a simple example. We saw that the values of the decision variables and those of the slack and

3. Linear Programming and Polyhedral Combinatorics

Massachusetts Institute of Technology Handout 6 18.433: Combinatorial Optimization February 20th, 2009 Michel X. Goemans 3. Linear Programming and Polyhedral Combinatorics Summary of what was seen in the

MATHEMATICS (CLASSES XI XII)

MATHEMATICS (CLASSES XI XII) General Guidelines (i) All concepts/identities must be illustrated by situational examples. (ii) The language of word problems must be clear, simple and unambiguous. (iii)

Lecture 2: August 29. Linear Programming (part I)

10-725: Convex Optimization Fall 2013 Lecture 2: August 29 Lecturer: Barnabás Póczos Scribes: Samrachana Adhikari, Mattia Ciollaro, Fabrizio Lecci Note: LaTeX template courtesy of UC Berkeley EECS dept.

Linear Programming: Theory and Applications

Linear Programming: Theory and Applications Catherine Lewis May 11, 2008 1 Contents 1 Introduction to Linear Programming 3 1.1 What is a linear program?...................... 3 1.2 Assumptions.............................

1 Portfolio mean and variance

Copyright c 2005 by Karl Sigman Portfolio mean and variance Here we study the performance of a one-period investment X 0 > 0 (dollars) shared among several different assets. Our criterion for measuring

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

Linear Programming Problems

Linear Programming Problems Linear programming problems come up in many applications. In a linear programming problem, we have a function, called the objective function, which depends linearly on a number

Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

Using the Simplex Method to Solve Linear Programming Maximization Problems J. Reeb and S. Leavengood

PERFORMANCE EXCELLENCE IN THE WOOD PRODUCTS INDUSTRY EM 8720-E October 1998 \$3.00 Using the Simplex Method to Solve Linear Programming Maximization Problems J. Reeb and S. Leavengood A key problem faced

Solutions to Math 51 First Exam January 29, 2015

Solutions to Math 5 First Exam January 29, 25. ( points) (a) Complete the following sentence: A set of vectors {v,..., v k } is defined to be linearly dependent if (2 points) there exist c,... c k R, not

Standard Form of a Linear Programming Problem

494 CHAPTER 9 LINEAR PROGRAMMING 9. THE SIMPLEX METHOD: MAXIMIZATION For linear programming problems involving two variables, the graphical solution method introduced in Section 9. is convenient. However,

Math 313 Lecture #10 2.2: The Inverse of a Matrix

Math 1 Lecture #10 2.2: The Inverse of a Matrix Matrix algebra provides tools for creating many useful formulas just like real number algebra does. For example, a real number a is invertible if there is

Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T

Matrix Norms. Tom Lyche. September 28, Centre of Mathematics for Applications, Department of Informatics, University of Oslo

Matrix Norms Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 28, 2009 Matrix Norms We consider matrix norms on (C m,n, C). All results holds for

2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

Overview of Math Standards

Algebra 2 Welcome to math curriculum design maps for Manhattan- Ogden USD 383, striving to produce learners who are: Effective Communicators who clearly express ideas and effectively communicate with diverse

Lecture 2: Homogeneous Coordinates, Lines and Conics

Lecture 2: Homogeneous Coordinates, Lines and Conics 1 Homogeneous Coordinates In Lecture 1 we derived the camera equations λx = P X, (1) where x = (x 1, x 2, 1), X = (X 1, X 2, X 3, 1) and P is a 3 4

Linear Codes. In the V[n,q] setting, the terms word and vector are interchangeable.

Linear Codes Linear Codes In the V[n,q] setting, an important class of codes are the linear codes, these codes are the ones whose code words form a sub-vector space of V[n,q]. If the subspace of V[n,q]

Nonlinear Optimization: Algorithms 3: Interior-point methods

Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,

Linear Algebra Notes

Linear Algebra Notes Chapter 19 KERNEL AND IMAGE OF A MATRIX Take an n m matrix a 11 a 12 a 1m a 21 a 22 a 2m a n1 a n2 a nm and think of it as a function A : R m R n The kernel of A is defined as Note

Linear Programming Notes VII Sensitivity Analysis

Linear Programming Notes VII Sensitivity Analysis 1 Introduction When you use a mathematical model to describe reality you must make approximations. The world is more complicated than the kinds of optimization

PYTHAGOREAN TRIPLES KEITH CONRAD 1. Introduction A Pythagorean triple is a triple of positive integers (a, b, c) where a + b = c. Examples include (3, 4, 5), (5, 1, 13), and (8, 15, 17). Below is an ancient

3.3. Solving Polynomial Equations. Introduction. Prerequisites. Learning Outcomes

Solving Polynomial Equations 3.3 Introduction Linear and quadratic equations, dealt within Sections 3.1 and 3.2, are members of a class of equations, called polynomial equations. These have the general

Systems of Linear Equations

Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and

26 Linear Programming

The greatest flood has the soonest ebb; the sorest tempest the most sudden calm; the hottest love the coldest end; and from the deepest desire oftentimes ensues the deadliest hate. Th extremes of glory

9.4 THE SIMPLEX METHOD: MINIMIZATION

SECTION 9 THE SIMPLEX METHOD: MINIMIZATION 59 The accounting firm in Exercise raises its charge for an audit to \$5 What number of audits and tax returns will bring in a maximum revenue? In the simplex

by the matrix A results in a vector which is a reflection of the given

Eigenvalues & Eigenvectors Example Suppose Then So, geometrically, multiplying a vector in by the matrix A results in a vector which is a reflection of the given vector about the y-axis We observe that

SYSTEMS OF EQUATIONS

SYSTEMS OF EQUATIONS 1. Examples of systems of equations Here are some examples of systems of equations. Each system has a number of equations and a number (not necessarily the same) of variables for which

a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2.

Chapter 1 LINEAR EQUATIONS 1.1 Introduction to linear equations A linear equation in n unknowns x 1, x,, x n is an equation of the form a 1 x 1 + a x + + a n x n = b, where a 1, a,..., a n, b are given

4.5 Linear Dependence and Linear Independence

4.5 Linear Dependence and Linear Independence 267 32. {v 1, v 2 }, where v 1, v 2 are collinear vectors in R 3. 33. Prove that if S and S are subsets of a vector space V such that S is a subset of S, then

Induction. Margaret M. Fleck. 10 October These notes cover mathematical induction and recursive definition

Induction Margaret M. Fleck 10 October 011 These notes cover mathematical induction and recursive definition 1 Introduction to induction At the start of the term, we saw the following formula for computing

Section 1.1. Introduction to R n

The Calculus of Functions of Several Variables Section. Introduction to R n Calculus is the study of functional relationships and how related quantities change with each other. In your first exposure to

Mathematics Course 111: Algebra I Part IV: Vector Spaces

Mathematics Course 111: Algebra I Part IV: Vector Spaces D. R. Wilkins Academic Year 1996-7 9 Vector Spaces A vector space over some field K is an algebraic structure consisting of a set V on which are

Algebra Unpacked Content For the new Common Core standards that will be effective in all North Carolina schools in the 2012-13 school year.

This document is designed to help North Carolina educators teach the Common Core (Standard Course of Study). NCDPI staff are continually updating and improving these tools to better serve teachers. Algebra

Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

Solving Linear Systems, Continued and The Inverse of a Matrix

, Continued and The of a Matrix Calculus III Summer 2013, Session II Monday, July 15, 2013 Agenda 1. The rank of a matrix 2. The inverse of a square matrix Gaussian Gaussian solves a linear system by reducing

NOTES ON LINEAR TRANSFORMATIONS

NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all

Solution of Linear Systems

Chapter 3 Solution of Linear Systems In this chapter we study algorithms for possibly the most commonly occurring problem in scientific computing, the solution of linear systems of equations. We start

3. Evaluate the objective function at each vertex. Put the vertices into a table: Vertex P=3x+2y (0, 0) 0 min (0, 5) 10 (15, 0) 45 (12, 2) 40 Max

SOLUTION OF LINEAR PROGRAMMING PROBLEMS THEOREM 1 If a linear programming problem has a solution, then it must occur at a vertex, or corner point, of the feasible set, S, associated with the problem. Furthermore,

Lecture 6. Inverse of Matrix

Lecture 6 Inverse of Matrix Recall that any linear system can be written as a matrix equation In one dimension case, ie, A is 1 1, then can be easily solved as A x b Ax b x b A 1 A b A 1 b provided that

Linear Programming Supplement E

Linear Programming Supplement E Linear Programming Linear programming: A technique that is useful for allocating scarce resources among competing demands. Objective function: An expression in linear programming

4.6 Null Space, Column Space, Row Space

NULL SPACE, COLUMN SPACE, ROW SPACE Null Space, Column Space, Row Space In applications of linear algebra, subspaces of R n typically arise in one of two situations: ) as the set of solutions of a linear

Diagonalisation. Chapter 3. Introduction. Eigenvalues and eigenvectors. Reading. Definitions

Chapter 3 Diagonalisation Eigenvalues and eigenvectors, diagonalisation of a matrix, orthogonal diagonalisation fo symmetric matrices Reading As in the previous chapter, there is no specific essential

Vector and Matrix Norms

Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty

Convex Optimization SVM s and Kernel Machines

Convex Optimization SVM s and Kernel Machines S.V.N. Vishy Vishwanathan vishy@axiom.anu.edu.au National ICT of Australia and Australian National University Thanks to Alex Smola and Stéphane Canu S.V.N.

Date: April 12, 2001. Contents

2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........

Chapter 2 Solving Linear Programs

Chapter 2 Solving Linear Programs Companion slides of Applied Mathematical Programming by Bradley, Hax, and Magnanti (Addison-Wesley, 1977) prepared by José Fernando Oliveira Maria Antónia Carravilla A

Module1. x 1000. y 800.

Module1 1 Welcome to the first module of the course. It is indeed an exciting event to share with you the subject that has lot to offer both from theoretical side and practical aspects. To begin with,

Economics 2020a / HBS 4010 / HKS API-111 FALL 2010 Solutions to Practice Problems for Lectures 1 to 4

Economics 00a / HBS 4010 / HKS API-111 FALL 010 Solutions to Practice Problems for Lectures 1 to 4 1.1. Quantity Discounts and the Budget Constraint (a) The only distinction between the budget line with

Several Views of Support Vector Machines

Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007 Tikhonov Regularization We are considering algorithms of the form min

Continued Fractions and the Euclidean Algorithm

Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction

Solutions to Homework 10

Solutions to Homework 1 Section 7., exercise # 1 (b,d): (b) Compute the value of R f dv, where f(x, y) = y/x and R = [1, 3] [, 4]. Solution: Since f is continuous over R, f is integrable over R. Let x

Definition: A square matrix A is block diagonal if A has the form A 1 O O O A 2 O A =

The question we want to answer now is the following: If A is not similar to a diagonal matrix, then what is the simplest matrix that A is similar to? Before we can provide the answer, we will have to introduce

Chapter 6. Cuboids. and. vol(conv(p ))

Chapter 6 Cuboids We have already seen that we can efficiently find the bounding box Q(P ) and an arbitrarily good approximation to the smallest enclosing ball B(P ) of a set P R d. Unfortunately, both

International Doctoral School Algorithmic Decision Theory: MCDA and MOO

International Doctoral School Algorithmic Decision Theory: MCDA and MOO Lecture 2: Multiobjective Linear Programming Department of Engineering Science, The University of Auckland, New Zealand Laboratoire

Orthogonal Projections

Orthogonal Projections and Reflections (with exercises) by D. Klain Version.. Corrections and comments are welcome! Orthogonal Projections Let X,..., X k be a family of linearly independent (column) vectors

First Welfare Theorem

First Welfare Theorem Econ 2100 Fall 2015 Lecture 17, November 2 Outline 1 First Welfare Theorem 2 Preliminaries to Second Welfare Theorem Last Class Definitions A feasible allocation (x, y) is Pareto