B553 Lecture 7: Constrained Optimization, Lagrange Multipliers, and KKT Conditions
|
|
- Emery Pearson
- 7 years ago
- Views:
Transcription
1 B553 Lecture 7: Constrained Optimization, Lagrange Multipliers, and KKT Conditions Kris Hauser February 2, 2012 Constraints on parameter values are an essential part of many optimization problems, and arise due to a variety of mathematical, physical, and resource limitations. In optimization, they can require significant work to handle depending on their complexity. In general, constrained optimization algorithms are much more complex than their unconstrained counterparts. A constrained optimization is specified in a problem of the form min f(x) x R n such that x S (1) where S R n denotes the subset of valid parameters, known as the feasible set (Figure 1). S must be a closed set to guarantee the existence of a minimum. Recall how in the univariate case of optimizing a function within some interval [a, b], we had to test the endpoints of the interval as well as the critical points in the interior (a, b) for optimality. In the multivariate constrained setting, the optimizer must not only consider the possibility that the optimum is a local minimum, but also that the optimum lies on the boundary of the feasible set (Figure 2). The challenge is now that there are an infinite number of points on S. This lecture will introduce analytical techniques Lagrange multipliers for equality constraints and the Karush-Kuhn-Tucker (KKT) conditions for inequalities for identifying those critical points. Besides being analytically useful, these conditions are the starting point for most constrained 1
2 optimization algorithms. Note that like other critical point tests these are only first-order conditions for optimality, and are therefore necessary but not sufficient for finding minima. 1 Common types of constraints Several forms of constraints arise in practice. common ones (Figure 3). Here are some of the most Bound constraints. Axis-aligned bound constraints take the form l i x i u i for some lower and upper values l i and u i, i = 1,..., n. These are some of the easiest constraints to incorporate. Linear inequalities. Linear inequality constraints take the form Ax b for some m n matrix A and a length m vector b. Note that bound constraints are a special case of a linear inequality with [ ] I A = (2) I and b = [ u l where u and l are the vectors fo upper and lower bounds, respectively. Linear equalities. Linear equality constraints take the form Ax = b, where A and b have m rows. Note that this is usually an underdetermined system (otherwise S would consist of either a single point or the empty set). In theory these constraints can easily be removed by finding a representation that incorporates the nullspace of A, say x = x 0 + Ny, and converting the optimization over x into a smaller optimization over y. However, note that most optimization routines do not operate this way because of numerical errors in computing N. Nonlinear constraints: general form. In general, constraints may be nonlinear. In this setting we can (usually) write the constraints in the following form: ] (3) g i (x) = 0 for i = 1,..., m h j (x) 0 for j = 1,..., p (4) 2
3 where the g i and h j are continuous, differentiable scalar field functions. This is the form that we will assume for the rest of this class, because all prior constraint types are special cases. Convex constraints. A convex set S satisfies the following property. For any two points x and y in S, the point (1 u)x + uy (5) for u [0, 1] lies in S as well. In other words, the line segment between any two points in S must also lie in S. Later we will discuss efficient algorithms for solving problems in which the constraints g i and h j produce a convex feasible set, and also the objective function f is a convex function. In particular, we will show that descent methods converge to a global minimum. (Note that to achieve convexity, any equality constraints must be linear) Black-box constraints. Another type of constraint that is a black box that can be queried to test whether a point x lies inside it. No other mathematical property, like the magnitude of feasibility violation, derivatives, or even smoothness, is necessarily provided. These constraints typically arise as a result of rather complex procedures (e.g., simulations, geometric algorithms, etc) that do not have a convenient mathematical representation. These constraints are rarely considered in the numerical optimization literature, but often come up in large practical systems. 2 First-Order Conditions of Local Optimality We say that a feasible point x is a local minimum of the optimization problem (1) if f(x) is lower than any other feasible point in some neighborhood of S. That is, x is a local minimum if x S and there exists a neighborhood of radius ɛ so that f(x) < f(y) for any y in {y S 0 < d(x, y) < ɛ}. Unfortunately not all local minima are critical points of f, because we have to take into account how the constraints affect the neighborhood! We will show that there are alternative criteria that we can use to generate candidates for local minima. 3
4 2.1 Lagrange Multipliers Let us suppose for the moment that there are no inequality constraints, and instead that we are addressing the general equality-constrained problem min f(x) x R n such that g i (x) = 0 for i = 1,..., m. (6) We will assume that both f and g are differentiable. With One Constraint. First let us consider the m = 1 case. The principle of Lagrange multipliers states that any local minima or maxima x of (6) must simultaneously satisfy the following equations: f(x)+λ g 1 (x) = 0 g 1 (x) = 0 (7) for some value of λ. The variable λ is known as the Lagrange multiplier. These equations are saying that at x, the gradient direction of f is a multiple of the gradient direction of g, which is to say that they are parallel (Figure 4). You might visualize this as follows. Imagine yourself standing at x, which satisfies g 1 (x) = 0. Any direction v that you can move in to instantaneously change the value of f will have a non-zero dot-product with f due to the properties of the directional derivative. The constraint g 1, however, will stop you from moving in any direction unless it maintains g 1 (x) = 0. This is equivalent to saying that the dot-product of v with g 1 (x) must be zero. If g 1 (x) is not a multiple of f(x), then you can slide along level set g 1 (x) = 0 along a direction v that has a nonzero dot-product with f(x) (Figure 5). In other words, x is not a minimum. On the other hand, if g 1 (x) is a multiple of f(x), then there is no such direction to move in, because any valid sliding direction will not change the value of f. In other words, the constraint g 1 cancels out any kind of change that you could make in the value of f. It is important to note that there may be multiple points x that satisfy (7), each of which has different Lagrange multipliers λ. With Many Constraints. The following condition generalizes Lagrange 4
5 multipliers to multiple constraints: f(x) + λ 1 g 1 (x) + + λ m g m (x) = 0 g 1 (x) = 0. g m (x) = 0 (8) where λ 1,..., λ m are the Lagrange multipliers. This equation is saying that at x, f(x) Span({ g 1 (x),..., g m (x)}). The reason why this makes sense is that each of the constraints resists motion in the direction of its gradient. If f lies in this span, then a motion in any direction that locally changes f will be completely nullified by the constraints. All local minima must satisfy (8). Conversely, if the two equations of (8) are satisfied then x must be a local minimum, maximum, or a sort of saddle point restricted to S. So this is a necessary, but not sufficient, condition for optimality. Example. Suppose we wanted to find the closest points (x 1, y 1 ) and (x 2, y 2 ) on two unit circles, one centered at the origin and the other centered at (c x, c y ). The optimization variable is x = (x 1, y 1, x 2, y 2 ) and the constrained minimization problem is min f(x) = (x 1 x 2 ) 2 + (y 1 y 2 ) 2 such that g 1 (x) = x y1 2 1 = 0 g 2 (x) = (x 2 c x ) 2 + (y 2 c y ) 2 1 = 0 (9) The method of Lagrange multipliers states that we need to find a variable x that satisfies the constraints, and multipliers λ 1 and λ 2 to satisfy: We can compute the following gradients f(x) + λ 1 g 1 (x) + λ 2 g 2 (x) = 0. (10) f(x) = 2(x 1 x 2 ) 2(y 1 y 2 ) 2(x 1 x 2 ) 2(y 1 y 2 ) 5, (11)
6 g 1 (x) = g 2 (x) = 2x 1 2y (x 2 x c ) 2(y 2 y c ), (12). (13) Putting these together, we have the two simultaneous sets of equations x 1 x 2 + λ 1 x 1 = 0 y 1 y 2 + λ 1 y 1 = 0 (14) and x 1 x 2 λ 2 (x 2 x c ) = 0 y 1 y 2 λ 2 (y 2 y c ) = 0. (15) In other words, the vectors (x 1 x 2, y 1 y 2 ), (x 1, y 1 ), and (x 2 x c, y 2 y c ) must all be parallel. With some rearrangement, it also means that (x 1, y 1 ) and (x 2, y 2 ) must be parallel to (x c, y c ). Verify geometrically that all points on the circles that intersect the line through (x c, y c ) are either local minima, local maxima, or saddle points of the squared distance function. Interpreting Lagrange Multipliers. In some applications like physics and economics, Lagrange multipliers have a meaningful interpretation. Consider the m = 1 case. Let s interpret the constraint as stating g 1 (x) = c with c = 0. The Lagrange multiplier λ at a (global) minimum x states how fast the minimum value f would change if I were to relax the constraint by raising c at a constant rate. This amount would be λ (Figure 6). For example, in constrained physical simulation the Lagrange multipliers produce the forces required to maintain each constraint. Using Lagrange Multipliers in numerical optimization. If we define the following Lagrangian function on n + m variables L(x, λ 1,..., λ m ) = f(x) + m λ i g i (x). (16) i=1 6
7 then the constraint optimization problem can be cast as one of finding the critical points of L in R n+m. More compactly, if we let λ = (λ 1,..., λ m ), note that we would like to find a point (x, λ) such that L(x, λ) = [ x L(x, λ) λ L(x, λ) ] = f(x) + m i=1 λ i g i (x) g 1 (x). g m (x) (17) equals zero. The importance of this is that we have converted a constrained optimization into an unconstrained root-finding problem! There do exist Newton-like techniques for solving multivariate root-finding problems. If f and the g i s are twice differentiable, we can use the iterative method: (x t+1, λ t+1 ) = (x t, λ t ) 2 L(x t, λ t ) 1 L(x, λ). (18) The Hessian of the Lagrangian is given by the following matrix 2 f(x) + m i=1 λ i 2 g i (x) g 1 (x) g m (x) 2 g 1 (x) T L(x, λ) =. 0. (19) g m (x) T 2.2 Karush-Kuhn-Tucker Conditions The KKT conditions extend the ideas of Lagrange multipliers to handle inequality constraints in addition to equality constraints. These conditions provide a first-order optimality condition for the problem: min f(x) x R n such that g i (x) = 0 for i = 1,..., m h j (x) 0 for j = 1,..., p (20) where f and all the g i s and h j s are differentiable. With one inequality. Let us start by assuming m = 0 and p = 1. The peculiar thing about inequalities is that they operate in essentially two regimes depending on whether they affect a critical point or not (Figure 7). If x is a 7
8 local minimum of f(x) such that h 1 (x) < 0, then the constraint is satisfied for a neighborhood around x, and x is a local minimum of the constrained problem. On the other hand, there could be local minima at the boundary of the feasible set S, which consists of those points that satisfy h 1 (x) = 0. To find these critical points, we can treat h 1 like an equality constraint and use the method of Lagrange multipliers. So, we must be aware of the following two cases: 1. f(x) = 0 and h 1 (x) < h 1 (x) = 0 and there exists a Lagrange multiplier µ such that f(x) + µ h 1 (x) = 0. A compact way of writing these two conditions, which will be very useful in a moment, is through the following set of equalities and inequalities: f(x) + µ h 1 (x) = 0 h 1 (x) 0 µh 1 (x) = 0 (21) in which the term µh 1 (x) = 0 is known as the complementarity condition that enforces either µ to be zero or h 1 (x) to be zero, If we are only interested in finding local minima, we can also include the constraint µ 0. With many inequalities. To generalize this argument to p > 1, consider that each of the two cases outlined above can hold for each of the inequalities. So, we may potentially need to enumerate all partitions of inequalities into those that are strictly satisfied and those that are met with equality, and find critical points for each subset. But there are 2 n possible subsets (Figure 8)! To express this condition, we can use the compact form as follows. f(x) + µ 1 h 1 (x) + + µ p h p (x) = 0 h j (x) 0 for j = 1,..., p µ j h j (x) = 0 for j = 1,..., p (22) where µ 1,..., µ p are the KKT multipliers. For those critical points with h j (x) = 0, we say the inequality is active at x. If h j (x) < 0, then we say the inequality is inactive. Some of the first 8
9 numerical methods that we present in this class perform combinatorial search through the possible subsets of active constraints. General form. Equalities can be incorporated in a straightforward manner into the above equation, giving us the full set of KKT conditions. f(x) + m p λ i g i (x) + µ j h j (x) = 0 i=1 j=1 g i (x) = 0 for i = 1,..., m h j (x) 0 for j = 1,..., p µ j h j (x) = 0 for j = 1,..., p (23) where λ 1,..., λ m are the Lagrange multipliers and µ 1,..., µ p are the KKT multipliers. Note that the complementarity condition only needs to be satisfied on the inequalities. Use of KKT conditions in analytical optimization. The KKT conditions can be used to analytically prove that a point is an optimum of a constrained problem. One drawback is that there are a combinatorial number of subsets of active inequalities, and in the absence of further information all of these subsets must be considered as candidates for generating the optimal critical point! Use of KKT conditions in numerical optimization. Unfortunately, we are not able to use the KKT conditions to formulate an unconstrained root-finding problem like we did in the case of Lagrange multipliers. The reason is because the inequality constraints h j (x) 0 must be preserved, and there is no natural way to handle them in the root finding methods we have observed so far. Instead, in most optimization software the KKT conditions are usually used as a first stage of verifing that a candidate point found by some algorithm is truly a critical point. 3 Exercises 1. The entropy of a discrete probability distribution (p 1,..., p n ) over n values is given by E(p 1,..., p n ) = n i=1 p i ln p i. Of course, probabilities must sum to 1. Find the probability distribution that maximizes entropy using Lagrange multipliers. 9
10 2. Find a simple way to compute the solution to the n-dimensional constrained optimization min x c 2 such that l x u, where l and u are bound constraints. 3. Write the KKT conditions for finding the closest point in a 2D triangle with vertices a, b, c (boundary inclusive) to the origin. Assume a, b, c are given in counterclockwise order. What is the significance of the KKT multipliers? What does it mean if none of them are nonzero? One? Two? More? 10
Nonlinear Programming Methods.S2 Quadratic Programming
Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective
More informationIncreasing for all. Convex for all. ( ) Increasing for all (remember that the log function is only defined for ). ( ) Concave for all.
1. Differentiation The first derivative of a function measures by how much changes in reaction to an infinitesimal shift in its argument. The largest the derivative (in absolute value), the faster is evolving.
More informationMetric Spaces. Chapter 7. 7.1. Metrics
Chapter 7 Metric Spaces A metric space is a set X that has a notion of the distance d(x, y) between every pair of points x, y X. The purpose of this chapter is to introduce metric spaces and give some
More informationconstraint. Let us penalize ourselves for making the constraint too big. We end up with a
Chapter 4 Constrained Optimization 4.1 Equality Constraints (Lagrangians) Suppose we have a problem: Maximize 5, (x 1, 2) 2, 2(x 2, 1) 2 subject to x 1 +4x 2 =3 If we ignore the constraint, we get the
More information3. INNER PRODUCT SPACES
. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.
More informationBig Data - Lecture 1 Optimization reminders
Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Schedule Introduction Major issues Examples Mathematics
More informationNumerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen
(für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained
More informationSolutions Of Some Non-Linear Programming Problems BIJAN KUMAR PATEL. Master of Science in Mathematics. Prof. ANIL KUMAR
Solutions Of Some Non-Linear Programming Problems A PROJECT REPORT submitted by BIJAN KUMAR PATEL for the partial fulfilment for the award of the degree of Master of Science in Mathematics under the supervision
More informationLecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method
Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming
More informationSection 1.1. Introduction to R n
The Calculus of Functions of Several Variables Section. Introduction to R n Calculus is the study of functional relationships and how related quantities change with each other. In your first exposure to
More informationLinear Threshold Units
Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear
More informationDate: April 12, 2001. Contents
2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........
More information24. The Branch and Bound Method
24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NP-complete. Then one can conclude according to the present state of science that no
More informationNonlinear Optimization: Algorithms 3: Interior-point methods
Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,
More information2.3 Convex Constrained Optimization Problems
42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions
More informationDuality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725
Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T
More informationMathematical finance and linear programming (optimization)
Mathematical finance and linear programming (optimization) Geir Dahl September 15, 2009 1 Introduction The purpose of this short note is to explain how linear programming (LP) (=linear optimization) may
More informationSolutions to Math 51 First Exam January 29, 2015
Solutions to Math 5 First Exam January 29, 25. ( points) (a) Complete the following sentence: A set of vectors {v,..., v k } is defined to be linearly dependent if (2 points) there exist c,... c k R, not
More informationSpecial Situations in the Simplex Algorithm
Special Situations in the Simplex Algorithm Degeneracy Consider the linear program: Maximize 2x 1 +x 2 Subject to: 4x 1 +3x 2 12 (1) 4x 1 +x 2 8 (2) 4x 1 +2x 2 8 (3) x 1, x 2 0. We will first apply the
More informationLecture 2: August 29. Linear Programming (part I)
10-725: Convex Optimization Fall 2013 Lecture 2: August 29 Lecturer: Barnabás Póczos Scribes: Samrachana Adhikari, Mattia Ciollaro, Fabrizio Lecci Note: LaTeX template courtesy of UC Berkeley EECS dept.
More informationInterior Point Methods and Linear Programming
Interior Point Methods and Linear Programming Robert Robere University of Toronto December 13, 2012 Abstract The linear programming problem is usually solved through the use of one of two algorithms: either
More information1 Review of Least Squares Solutions to Overdetermined Systems
cs4: introduction to numerical analysis /9/0 Lecture 7: Rectangular Systems and Numerical Integration Instructor: Professor Amos Ron Scribes: Mark Cowlishaw, Nathanael Fillmore Review of Least Squares
More information1 if 1 x 0 1 if 0 x 1
Chapter 3 Continuity In this chapter we begin by defining the fundamental notion of continuity for real valued functions of a single real variable. When trying to decide whether a given function is or
More informationEconomics 121b: Intermediate Microeconomics Problem Set 2 1/20/10
Dirk Bergemann Department of Economics Yale University s by Olga Timoshenko Economics 121b: Intermediate Microeconomics Problem Set 2 1/20/10 This problem set is due on Wednesday, 1/27/10. Preliminary
More information2013 MBA Jump Start Program
2013 MBA Jump Start Program Module 2: Mathematics Thomas Gilbert Mathematics Module Algebra Review Calculus Permutations and Combinations [Online Appendix: Basic Mathematical Concepts] 2 1 Equation of
More informationSECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA
SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA This handout presents the second derivative test for a local extrema of a Lagrange multiplier problem. The Section 1 presents a geometric motivation for the
More informationSolutions to Homework 10
Solutions to Homework 1 Section 7., exercise # 1 (b,d): (b) Compute the value of R f dv, where f(x, y) = y/x and R = [1, 3] [, 4]. Solution: Since f is continuous over R, f is integrable over R. Let x
More informationt := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d).
1. Line Search Methods Let f : R n R be given and suppose that x c is our current best estimate of a solution to P min x R nf(x). A standard method for improving the estimate x c is to choose a direction
More informationNumerical Analysis Lecture Notes
Numerical Analysis Lecture Notes Peter J. Olver 5. Inner Products and Norms The norm of a vector is a measure of its size. Besides the familiar Euclidean norm based on the dot product, there are a number
More informationLinear Programming Notes V Problem Transformations
Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material
More informationAlternative proof for claim in [1]
Alternative proof for claim in [1] Ritesh Kolte and Ayfer Özgür Aydin The problem addressed in [1] is described in Section 1 and the solution is given in Section. In the proof in [1], it seems that obtaining
More informationSOLUTIONS. f x = 6x 2 6xy 24x, f y = 3x 2 6y. To find the critical points, we solve
SOLUTIONS Problem. Find the critical points of the function f(x, y = 2x 3 3x 2 y 2x 2 3y 2 and determine their type i.e. local min/local max/saddle point. Are there any global min/max? Partial derivatives
More informationTHE FUNDAMENTAL THEOREM OF ARBITRAGE PRICING
THE FUNDAMENTAL THEOREM OF ARBITRAGE PRICING 1. Introduction The Black-Scholes theory, which is the main subject of this course and its sequel, is based on the Efficient Market Hypothesis, that arbitrages
More informationLinear Algebra Notes for Marsden and Tromba Vector Calculus
Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of
More informationLinear Programming. March 14, 2014
Linear Programming March 1, 01 Parts of this introduction to linear programming were adapted from Chapter 9 of Introduction to Algorithms, Second Edition, by Cormen, Leiserson, Rivest and Stein [1]. 1
More informationMath 4310 Handout - Quotient Vector Spaces
Math 4310 Handout - Quotient Vector Spaces Dan Collins The textbook defines a subspace of a vector space in Chapter 4, but it avoids ever discussing the notion of a quotient space. This is understandable
More information(Quasi-)Newton methods
(Quasi-)Newton methods 1 Introduction 1.1 Newton method Newton method is a method to find the zeros of a differentiable non-linear function g, x such that g(x) = 0, where g : R n R n. Given a starting
More informationMechanics 1: Conservation of Energy and Momentum
Mechanics : Conservation of Energy and Momentum If a certain quantity associated with a system does not change in time. We say that it is conserved, and the system possesses a conservation law. Conservation
More informationComputational Geometry. Lecture 1: Introduction and Convex Hulls
Lecture 1: Introduction and convex hulls 1 Geometry: points, lines,... Plane (two-dimensional), R 2 Space (three-dimensional), R 3 Space (higher-dimensional), R d A point in the plane, 3-dimensional space,
More informationA NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION
1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of
More informationChapter 4. Moment - the tendency of a force to rotate an object
Chapter 4 Moment - the tendency of a force to rotate an object Finding the moment - 2D Scalar Formulation Magnitude of force Mo = F d Rotation is clockwise or counter clockwise Moment about 0 Perpendicular
More informationOPRE 6201 : 2. Simplex Method
OPRE 6201 : 2. Simplex Method 1 The Graphical Method: An Example Consider the following linear program: Max 4x 1 +3x 2 Subject to: 2x 1 +3x 2 6 (1) 3x 1 +2x 2 3 (2) 2x 2 5 (3) 2x 1 +x 2 4 (4) x 1, x 2
More informationGeneral Framework for an Iterative Solution of Ax b. Jacobi s Method
2.6 Iterative Solutions of Linear Systems 143 2.6 Iterative Solutions of Linear Systems Consistent linear systems in real life are solved in one of two ways: by direct calculation (using a matrix factorization,
More informationArrangements And Duality
Arrangements And Duality 3.1 Introduction 3 Point configurations are tbe most basic structure we study in computational geometry. But what about configurations of more complicated shapes? For example,
More informationLinear Programming I
Linear Programming I November 30, 2003 1 Introduction In the VCR/guns/nuclear bombs/napkins/star wars/professors/butter/mice problem, the benevolent dictator, Bigus Piguinus, of south Antarctica penguins
More informationWalrasian Demand. u(x) where B(p, w) = {x R n + : p x w}.
Walrasian Demand Econ 2100 Fall 2015 Lecture 5, September 16 Outline 1 Walrasian Demand 2 Properties of Walrasian Demand 3 An Optimization Recipe 4 First and Second Order Conditions Definition Walrasian
More information10. Proximal point method
L. Vandenberghe EE236C Spring 2013-14) 10. Proximal point method proximal point method augmented Lagrangian method Moreau-Yosida smoothing 10-1 Proximal point method a conceptual algorithm for minimizing
More informationSolving Systems of Linear Equations
LECTURE 5 Solving Systems of Linear Equations Recall that we introduced the notion of matrices as a way of standardizing the expression of systems of linear equations In today s lecture I shall show how
More information1.3. DOT PRODUCT 19. 6. If θ is the angle (between 0 and π) between two non-zero vectors u and v,
1.3. DOT PRODUCT 19 1.3 Dot Product 1.3.1 Definitions and Properties The dot product is the first way to multiply two vectors. The definition we will give below may appear arbitrary. But it is not. It
More informationBX in ( u, v) basis in two ways. On the one hand, AN = u+
1. Let f(x) = 1 x +1. Find f (6) () (the value of the sixth derivative of the function f(x) at zero). Answer: 7. We expand the given function into a Taylor series at the point x = : f(x) = 1 x + x 4 x
More informationLargest Fixed-Aspect, Axis-Aligned Rectangle
Largest Fixed-Aspect, Axis-Aligned Rectangle David Eberly Geometric Tools, LLC http://www.geometrictools.com/ Copyright c 1998-2016. All Rights Reserved. Created: February 21, 2004 Last Modified: February
More information17.3.1 Follow the Perturbed Leader
CS787: Advanced Algorithms Topic: Online Learning Presenters: David He, Chris Hopman 17.3.1 Follow the Perturbed Leader 17.3.1.1 Prediction Problem Recall the prediction problem that we discussed in class.
More informationElasticity Theory Basics
G22.3033-002: Topics in Computer Graphics: Lecture #7 Geometric Modeling New York University Elasticity Theory Basics Lecture #7: 20 October 2003 Lecturer: Denis Zorin Scribe: Adrian Secord, Yotam Gingold
More information(a) We have x = 3 + 2t, y = 2 t, z = 6 so solving for t we get the symmetric equations. x 3 2. = 2 y, z = 6. t 2 2t + 1 = 0,
Name: Solutions to Practice Final. Consider the line r(t) = 3 + t, t, 6. (a) Find symmetric equations for this line. (b) Find the point where the first line r(t) intersects the surface z = x + y. (a) We
More informationThe Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method
The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method Robert M. Freund February, 004 004 Massachusetts Institute of Technology. 1 1 The Algorithm The problem
More informationStochastic Inventory Control
Chapter 3 Stochastic Inventory Control 1 In this chapter, we consider in much greater details certain dynamic inventory control problems of the type already encountered in section 1.3. In addition to the
More informationFurther Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1
Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1 J. Zhang Institute of Applied Mathematics, Chongqing University of Posts and Telecommunications, Chongqing
More informationDuality of linear conic problems
Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least
More informationThnkwell s Homeschool Precalculus Course Lesson Plan: 36 weeks
Thnkwell s Homeschool Precalculus Course Lesson Plan: 36 weeks Welcome to Thinkwell s Homeschool Precalculus! We re thrilled that you ve decided to make us part of your homeschool curriculum. This lesson
More informationBiggar High School Mathematics Department. National 5 Learning Intentions & Success Criteria: Assessing My Progress
Biggar High School Mathematics Department National 5 Learning Intentions & Success Criteria: Assessing My Progress Expressions & Formulae Topic Learning Intention Success Criteria I understand this Approximation
More informationby the matrix A results in a vector which is a reflection of the given
Eigenvalues & Eigenvectors Example Suppose Then So, geometrically, multiplying a vector in by the matrix A results in a vector which is a reflection of the given vector about the y-axis We observe that
More informationα = u v. In other words, Orthogonal Projection
Orthogonal Projection Given any nonzero vector v, it is possible to decompose an arbitrary vector u into a component that points in the direction of v and one that points in a direction orthogonal to v
More informationVector Algebra CHAPTER 13. Ü13.1. Basic Concepts
CHAPTER 13 ector Algebra Ü13.1. Basic Concepts A vector in the plane or in space is an arrow: it is determined by its length, denoted and its direction. Two arrows represent the same vector if they have
More informationLinear Algebra Notes
Linear Algebra Notes Chapter 19 KERNEL AND IMAGE OF A MATRIX Take an n m matrix a 11 a 12 a 1m a 21 a 22 a 2m a n1 a n2 a nm and think of it as a function A : R m R n The kernel of A is defined as Note
More informationLinear Programming. Solving LP Models Using MS Excel, 18
SUPPLEMENT TO CHAPTER SIX Linear Programming SUPPLEMENT OUTLINE Introduction, 2 Linear Programming Models, 2 Model Formulation, 4 Graphical Linear Programming, 5 Outline of Graphical Procedure, 5 Plotting
More informationInner Product Spaces
Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and
More informationLecture 5 Principal Minors and the Hessian
Lecture 5 Principal Minors and the Hessian Eivind Eriksen BI Norwegian School of Management Department of Economics October 01, 2010 Eivind Eriksen (BI Dept of Economics) Lecture 5 Principal Minors and
More informationThe equivalence of logistic regression and maximum entropy models
The equivalence of logistic regression and maximum entropy models John Mount September 23, 20 Abstract As our colleague so aptly demonstrated ( http://www.win-vector.com/blog/20/09/the-simplerderivation-of-logistic-regression/
More informationInformation Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay
Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 17 Shannon-Fano-Elias Coding and Introduction to Arithmetic Coding
More informationTo give it a definition, an implicit function of x and y is simply any relationship that takes the form:
2 Implicit function theorems and applications 21 Implicit functions The implicit function theorem is one of the most useful single tools you ll meet this year After a while, it will be second nature to
More informationCritical points of once continuously differentiable functions are important because they are the only points that can be local maxima or minima.
Lecture 0: Convexity and Optimization We say that if f is a once continuously differentiable function on an interval I, and x is a point in the interior of I that x is a critical point of f if f (x) =
More informationName: ID: Discussion Section:
Math 28 Midterm 3 Spring 2009 Name: ID: Discussion Section: This exam consists of 6 questions: 4 multiple choice questions worth 5 points each 2 hand-graded questions worth a total of 30 points. INSTRUCTIONS:
More informationLinear Programming for Optimization. Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc.
1. Introduction Linear Programming for Optimization Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc. 1.1 Definition Linear programming is the name of a branch of applied mathematics that
More information6. Define log(z) so that π < I log(z) π. Discuss the identities e log(z) = z and log(e w ) = w.
hapter omplex integration. omplex number quiz. Simplify 3+4i. 2. Simplify 3+4i. 3. Find the cube roots of. 4. Here are some identities for complex conjugate. Which ones need correction? z + w = z + w,
More informationMath 215 HW #6 Solutions
Math 5 HW #6 Solutions Problem 34 Show that x y is orthogonal to x + y if and only if x = y Proof First, suppose x y is orthogonal to x + y Then since x, y = y, x In other words, = x y, x + y = (x y) T
More informationIntersection of Convex Objects: The Method of Separating Axes
Intersection of Convex Objects: The Method of Separating Axes David Eberly Geometric Tools, LLC http://www.geometrictools.com/ Copyright c 1998-2016. All Rights Reserved. Created: January 28, 2001 Last
More informationLecture 2: Homogeneous Coordinates, Lines and Conics
Lecture 2: Homogeneous Coordinates, Lines and Conics 1 Homogeneous Coordinates In Lecture 1 we derived the camera equations λx = P X, (1) where x = (x 1, x 2, 1), X = (X 1, X 2, X 3, 1) and P is a 3 4
More information15.062 Data Mining: Algorithms and Applications Matrix Math Review
.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop
More informationMATH2210 Notebook 1 Fall Semester 2016/2017. 1 MATH2210 Notebook 1 3. 1.1 Solving Systems of Linear Equations... 3
MATH0 Notebook Fall Semester 06/07 prepared by Professor Jenny Baglivo c Copyright 009 07 by Jenny A. Baglivo. All Rights Reserved. Contents MATH0 Notebook 3. Solving Systems of Linear Equations........................
More informationMethods for Finding Bases
Methods for Finding Bases Bases for the subspaces of a matrix Row-reduction methods can be used to find bases. Let us now look at an example illustrating how to obtain bases for the row space, null space,
More informationNew insights on the mean-variance portfolio selection from de Finetti s suggestions. Flavio Pressacco and Paolo Serafini, Università di Udine
New insights on the mean-variance portfolio selection from de Finetti s suggestions Flavio Pressacco and Paolo Serafini, Università di Udine Abstract: In this paper we offer an alternative approach to
More informationSeveral Views of Support Vector Machines
Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007 Tikhonov Regularization We are considering algorithms of the form min
More information15 Kuhn -Tucker conditions
5 Kuhn -Tucker conditions Consider a version of the consumer problem in which quasilinear utility x 2 + 4 x 2 is maximised subject to x +x 2 =. Mechanically applying the Lagrange multiplier/common slopes
More informationNorth Carolina Math 2
Standards for Mathematical Practice 1. Make sense of problems and persevere in solving them. 2. Reason abstractly and quantitatively 3. Construct viable arguments and critique the reasoning of others 4.
More informationThis unit will lay the groundwork for later units where the students will extend this knowledge to quadratic and exponential functions.
Algebra I Overview View unit yearlong overview here Many of the concepts presented in Algebra I are progressions of concepts that were introduced in grades 6 through 8. The content presented in this course
More informationHøgskolen i Narvik Sivilingeniørutdanningen STE6237 ELEMENTMETODER. Oppgaver
Høgskolen i Narvik Sivilingeniørutdanningen STE637 ELEMENTMETODER Oppgaver Klasse: 4.ID, 4.IT Ekstern Professor: Gregory A. Chechkin e-mail: chechkin@mech.math.msu.su Narvik 6 PART I Task. Consider two-point
More informationCS3220 Lecture Notes: QR factorization and orthogonal transformations
CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss
More informationVector and Matrix Norms
Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty
More informationLinear Programming. Widget Factory Example. Linear Programming: Standard Form. Widget Factory Example: Continued.
Linear Programming Widget Factory Example Learning Goals. Introduce Linear Programming Problems. Widget Example, Graphical Solution. Basic Theory:, Vertices, Existence of Solutions. Equivalent formulations.
More informationDRAFT. Algebra 1 EOC Item Specifications
DRAFT Algebra 1 EOC Item Specifications The draft Florida Standards Assessment (FSA) Test Item Specifications (Specifications) are based upon the Florida Standards and the Florida Course Descriptions as
More informationSection 1.7 22 Continued
Section 1.5 23 A homogeneous equation is always consistent. TRUE - The trivial solution is always a solution. The equation Ax = 0 gives an explicit descriptions of its solution set. FALSE - The equation
More informationSolving Linear Programs
Solving Linear Programs 2 In this chapter, we present a systematic procedure for solving linear programs. This procedure, called the simplex method, proceeds by moving from one feasible solution to another,
More informationdiscuss how to describe points, lines and planes in 3 space.
Chapter 2 3 Space: lines and planes In this chapter we discuss how to describe points, lines and planes in 3 space. introduce the language of vectors. discuss various matters concerning the relative position
More information1 Solving LPs: The Simplex Algorithm of George Dantzig
Solving LPs: The Simplex Algorithm of George Dantzig. Simplex Pivoting: Dictionary Format We illustrate a general solution procedure, called the simplex algorithm, by implementing it on a very simple example.
More informationSolutions for Review Problems
olutions for Review Problems 1. Let be the triangle with vertices A (,, ), B (4,, 1) and C (,, 1). (a) Find the cosine of the angle BAC at vertex A. (b) Find the area of the triangle ABC. (c) Find a vector
More informationOverview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model
Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written
More informationSupport Vector Machine (SVM)
Support Vector Machine (SVM) CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin
More informationAlgebra Unpacked Content For the new Common Core standards that will be effective in all North Carolina schools in the 2012-13 school year.
This document is designed to help North Carolina educators teach the Common Core (Standard Course of Study). NCDPI staff are continually updating and improving these tools to better serve teachers. Algebra
More informationLecture 2. Marginal Functions, Average Functions, Elasticity, the Marginal Principle, and Constrained Optimization
Lecture 2. Marginal Functions, Average Functions, Elasticity, the Marginal Principle, and Constrained Optimization 2.1. Introduction Suppose that an economic relationship can be described by a real-valued
More informationThe Graphical Method: An Example
The Graphical Method: An Example Consider the following linear program: Maximize 4x 1 +3x 2 Subject to: 2x 1 +3x 2 6 (1) 3x 1 +2x 2 3 (2) 2x 2 5 (3) 2x 1 +x 2 4 (4) x 1, x 2 0, where, for ease of reference,
More information