2.2 Stochastic Dynamic Programming
|
|
- Norah Watts
- 7 years ago
- Views:
Transcription
1 2.2 Stochastic Dynamic Programming In general, the result of a given action will be unknown, although we will often be able to at least estimate the distribution of the resulting state given the action taken and the present state. These probabilities will be called the transition probabilities. The reward or cost incurred at each stage can depend on the action taken and the state. 1 / 47
2 Calculation of the optimal value recursively Consider the recursive derivation of the optimal value. Suppose we have the optimal payoffs starting from each of the possible states at stage i + 1. We can calculate the expected cost (payoff) starting from any state at stage i by using the appropriate transition probabilities. Hence, we can calculate the optimal expected reward starting from any state at stage i. 2 / 47
3 Calculation of the optimal value inductively Now consider the inductive derivation of the optimal value. At each stage we need to calculate the optimal cost of arriving at a given state given the optimal payoffs at previous stages. In order to do this, we need to calculate the probabilities of coming from each of the previous possible states given the present state (these are not the transition probabilities). It follows that it will be easier to use the recursive procedure. 3 / 47
4 Determination of the optimal strategy The optimal policy (strategy) should define what action is to be taken in each possible combination of state and stage. This policy is built up by remembering the action that achieves the optimal cost/reward at each step of the calculation. We cannot in general infer a full description of the strategy used by an individual by observing their actions, since some possible states may not be reached. 4 / 47
5 2.2.2 The Bellman equation Assume that Set of possible states at stage i is S i. Set of actions available in state j is A j (the set of available actions is assumed to be independent of the moment). It is assumed that all of these sets are finite. It is assumed that the probability of a transition from state j to state k only depends on the action taken and not on the moment an action is taken. i.e. the transition probability of going from state j to state k after taking action a can be denoted p j,k (a). 5 / 47
6 Bellman equation for minimisation Suppose the optimal expected cost in the minimisation problem starting in state k at stage i is d i (k). It is simple to calculate the expected costs of taking an action in a given state at the final stage. The total costs starting at earlier stages in a given state can be calculated by recursion. 6 / 47
7 Bellman equation for minimisation We have d i 1 (j) = min a A j {c a + k S i p j,k (a)d i (k)}, where c a is the immediate expected cost of the action a. The sum is simply the expected costs starting at stage i given that the action a is taken (the sum is taken over the set of states possible at stage i). We minimise this sum over the set of possible actions in state j. 7 / 47
8 Bellman equation for maximisation Analogously, suppose the optimal expected reward in the maximisation problem starting in state k at stage i is R i (k). We have R i 1 (j) = max a A j {r a + k S i p j,k (a)r i (k)}, where r a is the expected immediate reward gained by taking action a. 8 / 47
9 Example At the beginning of a day a machine is in one of two states: G-Good and B-Bad. Two actions are available each morning: F -Fix and L-Leave. The cost of F is 4. No immediate costs are associated with the action L. There are no costs associated with being in state G, but the cost of being in state B at the end of the day is 3 (this cost is incurred on the same day and the state does not change overnight). 9 / 47
10 Example ctd. If the present state is G, then the state at the end of the day will be G with probability 0.95 if action F is taken and with probability 0.8 if the action L is taken. If the present state is B, then the state at the end of the day will be B with probability 0.1 if action F is taken and with probability 0.9 if the action L is taken. At the beginning of day 1 the machine is in state G. Derive the optimal policy to minimise the costs over a 3-day period. In order to solve such a problem, we first derive the transition matrix. 10 / 47
11 The transition matrix The rows in the transition matrix correspond to the possible combinations of present state and action (here with 2 states and 2 actions in each state, we have have four combinations {F, G}, {F, B}, {L, G} and {L, B}). The columns correspond to the state at the next stage (here G or B). The entries are the transition probabilities to the new state given the combination of the present state and action. The sum of the entries in a row must be equal to 1. G B F, G L, G F, B L, B / 47
12 Solution of Example We calculate the optimal policy by recursion. We need to derive the optimal action to be taken in each possible state at each moment. At the beginning of the last day the system can be either in state G or state B. We first calculate d 3 (G) and d 3 (B), the index gives the day on which the action was taken. 12 / 47
13 Solution of Example In state G, taking the action F we incur a cost of 4 and with probability 0.05 the machine finishes in state B and we then incur a cost of 3. Thus, the expected immediate cost is Taking the action L, with probability 0.2 the machine finishes in state B. Hence, the expected immediate cost is 0.6. Thus, d 3 (G) = 0.6 and the optimal action at stage 4 in state G is L. 13 / 47
14 Solution of Example Similarly, d 3 (B) = min{ , 0.9 3} = 2.7. Here, the first entry is the expected cost of using F and the second entry is the expected cost of using L (this convention is used in all the calculations below). Hence, d 3 (L) = 2.7 and the optimal action at stage 3 in state B is L. 14 / 47
15 Solution of Example Working backwards, the Bellman equation in state G at stage 2 is given by d 2 (G)=min{ d 3 (G) d 3 (B), d 3 (G) + 0.2d 3 (B)} =min{4.855, 1.62} = The optimal action at stage 2 in state G is L. Note that 4.15 is the expected immediate cost at stage 2 when action F is taken. This is made up of the cost of fixed plus the expected costs associated with the state of the machine at the end of the day (3 with probability 0.05). Given the present state is G and the action F, the next state will be G with probability 0.95, otherwise it will be F. Hence, the optimal expected future costs are 0.95d 3 (G) d 3 (B). 15 / 47
16 Solution of Example Similarly, d 2 (B)=min{ d 3 (G) + 0.1d 3 (B), d 3 (G) + 0.9d 3 (B)} =min{5.11, 5.19} = The optimal action at stage 2 in state B is F. 16 / 47
17 Solution of Example Working backwards, the Bellman equation in state G at stage 1 is given by d 1 (G)=min{ d 2 (G) d 2 (B), d 2 (G) + 0.2d 2 (B)} =min{5.9445, 2.918} = The optimal action at stage 1 in state G is L. 17 / 47
18 Solution of Example Similarly, d 1 (B)=min{ d 2 (G) + 0.1d 2 (B), d 3 (G) + 0.9d 2 (B)} =min{6.27, 8.46} = The optimal action at stage 1 in state B is F. 18 / 47
19 2.3 Infinite Horizon Problems The imposition of a finite time horizon is unnatural, since in general we are interested in long-term control procedures. In this case, we may define a problem as an infinite horizon problem. Of course, there are problems with such a formulation: 1. The total costs (rewards) obtained in such a problem would not be finite. 2. Such problems cannot be solved by recursion. 19 / 47
20 Formulation of costs (rewards) in an infinite horizon problem In order to eliminate the first problem, we can either a. Discount the costs (rewards) over time. The logic behind this is that people naturally prefer immediate rewards to delayed rewards (due to inflation, uncertainty etc.). A reward of x at stage i 0 + i is defined to be worth the same as a reward of xβ i at stage i 0, where 0 < β < 1 is the discount factor. b. Optimise the average cost (reward) per stage. The method of solution depends on the approach we use. 20 / 47
21 2.3.1 Dynamic Programming with Discounted Costs (Payoffs) It should be noted that discount factors may be used in finite horizon problems and the recursive method of solution used. The appropriate Bellman equations are: For minimisation problems d i 1 (j) = min a A j {c a + β k S i p j,k (a)d i (k)}. 21 / 47
22 Dynamic Programming with Discounted Costs (Payoffs) and Finite Horizons For maximisation problems R i 1 (j) = max a A j {r a + β k S i p j,k (a)r i (k)}, 22 / 47
23 Recursive calculations and discounted payoffs Using this procedure to calculate the optimal actions at stage n 1 the calculation clearly discounts the expected costs obtained in the final stage by a factor of β. This discount is incorporated in the optimal costs {d n 1 (j)}. At stage n 2, the costs obtained at stage n 1 are discounted by a factor of β and thus the costs incurred at stage n are discounted in total by a factor of β 2. Hence, these calculations implicitly take into account the discount factor used. 23 / 47
24 Infinite horizon problems with discounted payoffs We assume that: The set of possible states is the same at each stage. The set of permissible actions only depends on the present state. Let m be the (finite) number of possible states and S = {1, 2,..., m}. A j is the (finite) set of permissible actions in state j and A j = {a 1,j, a 2,j,..., a nj,j}, where n j is the number of available actions in state j. 24 / 47
25 Stationary strategies We only consider stationary strategies. These are strategies which choose actions that only depend on the present state and not on the stage. In order to specify a stationary strategy π, it suffices to define which action should be taken in each state, i.e. define a(j) for j = 1, 2,..., m. There is a finite number of stationary strategies. 25 / 47
26 Optimality and stationary strategies It is intuitively clear that there must a strategy in this class that is optimal in the class of all possible strategies. This is due to the fact that the problem faced by an individual at stage i 0 in state j is identical (self-similar) to the problem faced by an individual at stage i 0 + i in state j. Hence, if the action a(j) is optimal at stage i 0, then it must be optimal at stage i 0 + i. 26 / 47
27 Methods of solution We consider two methods of solving such problems 1. Policy iteration. 2. Value iteration. 27 / 47
28 Policy Iteration Consider the stationary policy π = {a(j)} m j=1 (i.e. a policy defines what action is taken in each state j). The idea of policy iteration is based on the self-similarity of infinite horizon problems. That is to say, if the expected sum of discounted costs starting at stage 0 in state j using the strategy π is C π (j), then the sum of the expected sum of discounted costs starting at stage 1 in state j will be βc π (j). 28 / 47
29 Derivation of expected costs under a policy Using this, the expected costs (payoffs) when starting in each of the possible states, C π (1), C π (2),..., C π (m), under the strategy π is defined by a set of m linear equations in m unknowns. The j-th equation is C π (j) = c a(j) + β m p j,k [a(j)]c π (k), k=1 where c a(j) are the immediate expected costs of using action a(j) and p j,k [a(j)] is the probability of transition from state j to state k given that action a(j) is taken. 29 / 47
30 Example Consider the infinite horizon version of Example in which β = Assume that the costs of the machine ending the day in a bad state are incurred on that day. Calculate the expected reward from adopting the policy π of never checking the machine, i.e. always choose L. 30 / 47
31 Calculation of the expected costs under a policy If the present state is G, then under the policy the state will be G at the next stage with probability 0.8 and B with probability 0.2. An immediate cost of 3 is only incurred in the second case. Hence, the expected immediate costs are 0.6. It follows that C π (G) = 0.6+β[0.8C π (G)+0.2C π (B)] = C π (G)+0.19C π (B). 31 / 47
32 Calculation of the expected costs under a policy-ctd. Arguing similarly, it follows that C π (B)=2.7 + β[0.1c π (G) + 0.9C π (B)] = C π (G) C π (B). Rearranging these two equations, we obtain 0.24C π (G)= C π (B) 0.095C π (G) C π (B)=2.7. Solving this system of equations, we obtain C π (G) 35.82, C π (B) / 47
33 Evalution of the optimal policy Since there is a finite number of stationary strategies, we could derive the optimal policy by deriving the optimal costs (rewards) starting in each state {C π (j)} m j=1 for each possible stationary strategy π. The optimal policy minimises the expected costs regardless of the starting state. However, when there are a large number of possible strategies, this is not an efficient procedure. 33 / 47
34 The policy iteration algorithm The algorithm defining the method of policy iteration is as follows: 0. Set i = 0 and some initial stationary policy π Calculate the expected costs using π i when starting in each of the possible states i.e. by solving the appropriate system of linear equations. 2. Calculate the expected cost starting in state j = 1 for each of the strategies of the form: Carry out action a in stage 1 and thereafter follow π i. [Note that if the action is the one defined by π i in state j, then this expected cost is simply C π i (j).] Such a strategy will be denoted by (a, π i ). Set a(j) to be the action that minimises these costs (maximises rewards). Repeat for j = 2, 3,..., m. 34 / 47
35 The policy iteration algorithm 3. The policy π i+1 is defined to be {a(j)} m j=1 (i.e. it is made up of the actions that minimised the costs in each state, calculated in Step 2). If π i+1 = π i, then stop. In this case, π i is the optimal policy. If π i+1 π i, then set i = i + 1 and return to / 47
36 Calculation of expected costs for a perturbed policy In Step 2, we need to calculate the expected costs starting in state j, using a in the first step and thereafter using π i. These are given by m C (a,π i ) (j) = c a + β p j,k (a)c π i (k). k=1 Note that C π i (k) was calculated in Step 1 (it gives the expected cost (reward) under the present policy). Hence, we only need to evaluate the appropriate expressions rather than solve a system of equations. 36 / 47
37 Example Derive the optimal policy of the problem given in Example / 47
38 Solution of Example Step 0 is to set some initial policy. Let π 0 be the policy always choose L. Step 1 involves the calculation of the expected costs starting in each of the possible states. From Example 2.3.1, we have C π 0 (G) 35.82, C π 0 (B) / 47
39 The policy improvement step Step 2 is to check to see whether this policy can be improved. Starting in state G instead of using L at the first stage we could use F. In this case, the state at the next stage will be G with probability 0.95, otherwise the state will be B. The action F is associated with an immediate cost of 4 and a transition to B (which occurs with probability 0.05) is associated with an immediate cost of 3. It follows that the expected immediate cost is / 47
40 The policy improvement step It follows that C (F,π 0) (G) = β[0.95c π 0 (G) C π 0 (B)] Since C π 0 (G) 35.82, we have a(g) = L. 40 / 47
41 The policy improvement step Similarly, starting in state B and using F at the first stage, we have C (F,π 0) (B) = β[0.9c π 0 (G) + 0.1C π 0 (B)] Since C π 0 (B) 42.09, it can be seen that a(b) = F. Hence, the policy π 1 is to fix the machine when it is in a bad state and leave it when it is in a good state. 41 / 47
42 Calculation of the expected costs under the new policy Returning to step 1, we calculate the expected costs under this new policy. As in Example 2.3.1, this requires the solution of m linear equations (where m is the number of states, here 2). As the action used in state G is L, we have C π 1 (G)=0.6 + β[0.8c π 1 (G) + 0.2C π 1 (B)] = C π 1 (G) C π 1 (B). 42 / 47
43 Calculation of the expected costs under the new policy In state B we use the action F (cost of 4). The state at the next stage will be G with probability 0.9 (no costs). Otherwise this state will be B, which is associated with an immediate cost of 3. Hence, the immediate expected cost is 4.3. We have C π 1 (B)=4.3 + β[0.9c π 1 (G) + 0.1C π 1 (B)] = C π 1 (G) C π 1 (B). 43 / 47
44 Calculation of the expected costs under the new policy This leads to the pair of linear equations 0.24C π 1 (G)= C π 1 (B) 0.855C π 1 (G) C π 1 (B)=4.3 Solving this system of equations gives C π 1 (B) = 28.22, C π 1 (G) = / 47
45 The policy improvement step We now see if it is possible to improve this policy. Starting in state G, we could change the action used in the first stage to F. The cost of this action is 4 and the probability of going to state B is 0.05, which is associated with an immediate cost of 3. The expected immediate cost is thus / 47
46 The policy improvement step We have C (F,π 1) (G) = β[0.95c π 1 (G) C π 1 (B)] Since C π 1 (G) = 24.84, it does not pay to change this action. 46 / 47
47 The policy improvement step Similarly, starting in state B, we could change the action taken to L. Arguing as above, C (L,π 1) (B) = β[0.1c π 1 (G) + 0.9C π 1 (B)] Since C π 1 (B) = 28.22, it does not pay to change this action. Hence, π 2 = π 1. The optimal policy is to fix the machine when it is in state B and leave it when it is in state G. 47 / 47
Reinforcement Learning
Reinforcement Learning LU 2 - Markov Decision Problems and Dynamic Programming Dr. Martin Lauer AG Maschinelles Lernen und Natürlichsprachliche Systeme Albert-Ludwigs-Universität Freiburg martin.lauer@kit.edu
More informationReinforcement Learning
Reinforcement Learning LU 2 - Markov Decision Problems and Dynamic Programming Dr. Joschka Bödecker AG Maschinelles Lernen und Natürlichsprachliche Systeme Albert-Ludwigs-Universität Freiburg jboedeck@informatik.uni-freiburg.de
More informationMATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.
MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column
More information2x + y = 3. Since the second equation is precisely the same as the first equation, it is enough to find x and y satisfying the system
1. Systems of linear equations We are interested in the solutions to systems of linear equations. A linear equation is of the form 3x 5y + 2z + w = 3. The key thing is that we don t multiply the variables
More informationDynamic Programming 11.1 AN ELEMENTARY EXAMPLE
Dynamic Programming Dynamic programming is an optimization approach that transforms a complex problem into a sequence of simpler problems; its essential characteristic is the multistage nature of the optimization
More informationSystems of Linear Equations
Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and
More information6.231 Dynamic Programming Midterm, Fall 2008. Instructions
6.231 Dynamic Programming Midterm, Fall 2008 Instructions The midterm comprises three problems. Problem 1 is worth 60 points, problem 2 is worth 40 points, and problem 3 is worth 40 points. Your grade
More information5 Homogeneous systems
5 Homogeneous systems Definition: A homogeneous (ho-mo-jeen -i-us) system of linear algebraic equations is one in which all the numbers on the right hand side are equal to : a x +... + a n x n =.. a m
More information5.5. Solving linear systems by the elimination method
55 Solving linear systems by the elimination method Equivalent systems The major technique of solving systems of equations is changing the original problem into another one which is of an easier to solve
More information1 The EOQ and Extensions
IEOR4000: Production Management Lecture 2 Professor Guillermo Gallego September 9, 2004 Lecture Plan 1. The EOQ and Extensions 2. Multi-Item EOQ Model 1 The EOQ and Extensions This section is devoted to
More informationa 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2.
Chapter 1 LINEAR EQUATIONS 1.1 Introduction to linear equations A linear equation in n unknowns x 1, x,, x n is an equation of the form a 1 x 1 + a x + + a n x n = b, where a 1, a,..., a n, b are given
More informationRow Echelon Form and Reduced Row Echelon Form
These notes closely follow the presentation of the material given in David C Lay s textbook Linear Algebra and its Applications (3rd edition) These notes are intended primarily for in-class presentation
More information6.207/14.15: Networks Lecture 15: Repeated Games and Cooperation
6.207/14.15: Networks Lecture 15: Repeated Games and Cooperation Daron Acemoglu and Asu Ozdaglar MIT November 2, 2009 1 Introduction Outline The problem of cooperation Finitely-repeated prisoner s dilemma
More informationBasic Proof Techniques
Basic Proof Techniques David Ferry dsf43@truman.edu September 13, 010 1 Four Fundamental Proof Techniques When one wishes to prove the statement P Q there are four fundamental approaches. This document
More informationA linear algebraic method for pricing temporary life annuities
A linear algebraic method for pricing temporary life annuities P. Date (joint work with R. Mamon, L. Jalen and I.C. Wang) Department of Mathematical Sciences, Brunel University, London Outline Introduction
More informationRegular Languages and Finite Automata
Regular Languages and Finite Automata 1 Introduction Hing Leung Department of Computer Science New Mexico State University Sep 16, 2010 In 1943, McCulloch and Pitts [4] published a pioneering work on a
More informationCOMP 250 Fall 2012 lecture 2 binary representations Sept. 11, 2012
Binary numbers The reason humans represent numbers using decimal (the ten digits from 0,1,... 9) is that we have ten fingers. There is no other reason than that. There is nothing special otherwise about
More informationProduct Mix as a Framing Exercise: The Role of Cost Allocation. Anil Arya The Ohio State University. Jonathan Glover Carnegie Mellon University
Product Mix as a Framing Exercise: The Role of Cost Allocation Anil Arya The Ohio State University Jonathan Glover Carnegie Mellon University Richard Young The Ohio State University December 1999 Product
More informationECON 459 Game Theory. Lecture Notes Auctions. Luca Anderlini Spring 2015
ECON 459 Game Theory Lecture Notes Auctions Luca Anderlini Spring 2015 These notes have been used before. If you can still spot any errors or have any suggestions for improvement, please let me know. 1
More informationWRITING PROOFS. Christopher Heil Georgia Institute of Technology
WRITING PROOFS Christopher Heil Georgia Institute of Technology A theorem is just a statement of fact A proof of the theorem is a logical explanation of why the theorem is true Many theorems have this
More informationAPPENDIX. Interest Concepts of Future and Present Value. Concept of Interest TIME VALUE OF MONEY BASIC INTEREST CONCEPTS
CHAPTER 8 Current Monetary Balances 395 APPENDIX Interest Concepts of Future and Present Value TIME VALUE OF MONEY In general business terms, interest is defined as the cost of using money over time. Economists
More information4/1/2017. PS. Sequences and Series FROM 9.2 AND 9.3 IN THE BOOK AS WELL AS FROM OTHER SOURCES. TODAY IS NATIONAL MANATEE APPRECIATION DAY
PS. Sequences and Series FROM 9.2 AND 9.3 IN THE BOOK AS WELL AS FROM OTHER SOURCES. TODAY IS NATIONAL MANATEE APPRECIATION DAY 1 Oh the things you should learn How to recognize and write arithmetic sequences
More informationReduced echelon form: Add the following conditions to conditions 1, 2, and 3 above:
Section 1.2: Row Reduction and Echelon Forms Echelon form (or row echelon form): 1. All nonzero rows are above any rows of all zeros. 2. Each leading entry (i.e. left most nonzero entry) of a row is in
More informationSensitivity Analysis 3.1 AN EXAMPLE FOR ANALYSIS
Sensitivity Analysis 3 We have already been introduced to sensitivity analysis in Chapter via the geometry of a simple example. We saw that the values of the decision variables and those of the slack and
More informationOptimal linear-quadratic control
Optimal linear-quadratic control Martin Ellison 1 Motivation The lectures so far have described a general method - value function iterations - for solving dynamic programming problems. However, one problem
More information1.2 Solving a System of Linear Equations
1.. SOLVING A SYSTEM OF LINEAR EQUATIONS 1. Solving a System of Linear Equations 1..1 Simple Systems - Basic De nitions As noticed above, the general form of a linear system of m equations in n variables
More information3. Time value of money. We will review some tools for discounting cash flows.
1 3. Time value of money We will review some tools for discounting cash flows. Simple interest 2 With simple interest, the amount earned each period is always the same: i = rp o where i = interest earned
More informationMATHEMATICS FOR ENGINEERS BASIC MATRIX THEORY TUTORIAL 2
MATHEMATICS FO ENGINEES BASIC MATIX THEOY TUTOIAL This is the second of two tutorials on matrix theory. On completion you should be able to do the following. Explain the general method for solving simultaneous
More informationOne Period Binomial Model
FIN-40008 FINANCIAL INSTRUMENTS SPRING 2008 One Period Binomial Model These notes consider the one period binomial model to exactly price an option. We will consider three different methods of pricing
More informationNOTES ON LINEAR TRANSFORMATIONS
NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all
More information160 CHAPTER 4. VECTOR SPACES
160 CHAPTER 4. VECTOR SPACES 4. Rank and Nullity In this section, we look at relationships between the row space, column space, null space of a matrix and its transpose. We will derive fundamental results
More informationA simple analysis of the TV game WHO WANTS TO BE A MILLIONAIRE? R
A simple analysis of the TV game WHO WANTS TO BE A MILLIONAIRE? R Federico Perea Justo Puerto MaMaEuSch Management Mathematics for European Schools 94342 - CP - 1-2001 - DE - COMENIUS - C21 University
More informationDiscrete Mathematics and Probability Theory Fall 2009 Satish Rao, David Tse Note 2
CS 70 Discrete Mathematics and Probability Theory Fall 2009 Satish Rao, David Tse Note 2 Proofs Intuitively, the concept of proof should already be familiar We all like to assert things, and few of us
More informationLinear Programming Notes VII Sensitivity Analysis
Linear Programming Notes VII Sensitivity Analysis 1 Introduction When you use a mathematical model to describe reality you must make approximations. The world is more complicated than the kinds of optimization
More informationContinued Fractions and the Euclidean Algorithm
Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction
More informationWhat is Linear Programming?
Chapter 1 What is Linear Programming? An optimization problem usually has three essential ingredients: a variable vector x consisting of a set of unknowns to be determined, an objective function of x to
More informationA linear combination is a sum of scalars times quantities. Such expressions arise quite frequently and have the form
Section 1.3 Matrix Products A linear combination is a sum of scalars times quantities. Such expressions arise quite frequently and have the form (scalar #1)(quantity #1) + (scalar #2)(quantity #2) +...
More informationSolving Linear Systems, Continued and The Inverse of a Matrix
, Continued and The of a Matrix Calculus III Summer 2013, Session II Monday, July 15, 2013 Agenda 1. The rank of a matrix 2. The inverse of a square matrix Gaussian Gaussian solves a linear system by reducing
More information3 Some Integer Functions
3 Some Integer Functions A Pair of Fundamental Integer Functions The integer function that is the heart of this section is the modulo function. However, before getting to it, let us look at some very simple
More information1 Review of Least Squares Solutions to Overdetermined Systems
cs4: introduction to numerical analysis /9/0 Lecture 7: Rectangular Systems and Numerical Integration Instructor: Professor Amos Ron Scribes: Mark Cowlishaw, Nathanael Fillmore Review of Least Squares
More informationMATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +
More information5.4 Solving Percent Problems Using the Percent Equation
5. Solving Percent Problems Using the Percent Equation In this section we will develop and use a more algebraic equation approach to solving percent equations. Recall the percent proportion from the last
More informationSolving Linear Programs
Solving Linear Programs 2 In this chapter, we present a systematic procedure for solving linear programs. This procedure, called the simplex method, proceeds by moving from one feasible solution to another,
More information5.3 The Cross Product in R 3
53 The Cross Product in R 3 Definition 531 Let u = [u 1, u 2, u 3 ] and v = [v 1, v 2, v 3 ] Then the vector given by [u 2 v 3 u 3 v 2, u 3 v 1 u 1 v 3, u 1 v 2 u 2 v 1 ] is called the cross product (or
More informationSpecial Situations in the Simplex Algorithm
Special Situations in the Simplex Algorithm Degeneracy Consider the linear program: Maximize 2x 1 +x 2 Subject to: 4x 1 +3x 2 12 (1) 4x 1 +x 2 8 (2) 4x 1 +2x 2 8 (3) x 1, x 2 0. We will first apply the
More informationSYSTEMS OF EQUATIONS AND MATRICES WITH THE TI-89. by Joseph Collison
SYSTEMS OF EQUATIONS AND MATRICES WITH THE TI-89 by Joseph Collison Copyright 2000 by Joseph Collison All rights reserved Reproduction or translation of any part of this work beyond that permitted by Sections
More informationVector and Matrix Norms
Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty
More informationSession 7 Fractions and Decimals
Key Terms in This Session Session 7 Fractions and Decimals Previously Introduced prime number rational numbers New in This Session period repeating decimal terminating decimal Introduction In this session,
More informationMATRIX ALGEBRA AND SYSTEMS OF EQUATIONS
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a
More informationInteger Programming Formulation
Integer Programming Formulation 1 Integer Programming Introduction When we introduced linear programs in Chapter 1, we mentioned divisibility as one of the LP assumptions. Divisibility allowed us to consider
More information9.2 Summation Notation
9. Summation Notation 66 9. Summation Notation In the previous section, we introduced sequences and now we shall present notation and theorems concerning the sum of terms of a sequence. We begin with a
More informationMathematical Induction
Mathematical Induction (Handout March 8, 01) The Principle of Mathematical Induction provides a means to prove infinitely many statements all at once The principle is logical rather than strictly mathematical,
More informationIntroduction to Matrix Algebra
Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary
More informationNPV with an Option to Delay and Uncertainty
NPV with an Option to Delay and Uncertainty In all the examples and the theory so far, we have assumed perfect foresight when making investment decisions and analyzing future variables. We have also ignored
More informationSTRATEGIC CAPACITY PLANNING USING STOCK CONTROL MODEL
Session 6. Applications of Mathematical Methods to Logistics and Business Proceedings of the 9th International Conference Reliability and Statistics in Transportation and Communication (RelStat 09), 21
More informationMath 115 Spring 2011 Written Homework 5 Solutions
. Evaluate each series. a) 4 7 0... 55 Math 5 Spring 0 Written Homework 5 Solutions Solution: We note that the associated sequence, 4, 7, 0,..., 55 appears to be an arithmetic sequence. If the sequence
More informationSolution to Homework 2
Solution to Homework 2 Olena Bormashenko September 23, 2011 Section 1.4: 1(a)(b)(i)(k), 4, 5, 14; Section 1.5: 1(a)(b)(c)(d)(e)(n), 2(a)(c), 13, 16, 17, 18, 27 Section 1.4 1. Compute the following, if
More informationSection 1.3 P 1 = 1 2. = 1 4 2 8. P n = 1 P 3 = Continuing in this fashion, it should seem reasonable that, for any n = 1, 2, 3,..., = 1 2 4.
Difference Equations to Differential Equations Section. The Sum of a Sequence This section considers the problem of adding together the terms of a sequence. Of course, this is a problem only if more than
More informationRecall that two vectors in are perpendicular or orthogonal provided that their dot
Orthogonal Complements and Projections Recall that two vectors in are perpendicular or orthogonal provided that their dot product vanishes That is, if and only if Example 1 The vectors in are orthogonal
More informationPsychology and Economics (Lecture 17)
Psychology and Economics (Lecture 17) Xavier Gabaix April 13, 2004 Vast body of experimental evidence, demonstrates that discount rates are higher in the short-run than in the long-run. Consider a final
More informationChapter 27: Taxation. 27.1: Introduction. 27.2: The Two Prices with a Tax. 27.2: The Pre-Tax Position
Chapter 27: Taxation 27.1: Introduction We consider the effect of taxation on some good on the market for that good. We ask the questions: who pays the tax? what effect does it have on the equilibrium
More informationMath 55: Discrete Mathematics
Math 55: Discrete Mathematics UC Berkeley, Fall 2011 Homework # 5, due Wednesday, February 22 5.1.4 Let P (n) be the statement that 1 3 + 2 3 + + n 3 = (n(n + 1)/2) 2 for the positive integer n. a) What
More informationCircuit Analysis using the Node and Mesh Methods
Circuit Analysis using the Node and Mesh Methods We have seen that using Kirchhoff s laws and Ohm s law we can analyze any circuit to determine the operating conditions (the currents and voltages). The
More informationArguments and Dialogues
ONE Arguments and Dialogues The three goals of critical argumentation are to identify, analyze, and evaluate arguments. The term argument is used in a special sense, referring to the giving of reasons
More informationLinear Algebra Notes
Linear Algebra Notes Chapter 19 KERNEL AND IMAGE OF A MATRIX Take an n m matrix a 11 a 12 a 1m a 21 a 22 a 2m a n1 a n2 a nm and think of it as a function A : R m R n The kernel of A is defined as Note
More informationNumerical Methods for Option Pricing
Chapter 9 Numerical Methods for Option Pricing Equation (8.26) provides a way to evaluate option prices. For some simple options, such as the European call and put options, one can integrate (8.26) directly
More informationMathematical Induction. Mary Barnes Sue Gordon
Mathematics Learning Centre Mathematical Induction Mary Barnes Sue Gordon c 1987 University of Sydney Contents 1 Mathematical Induction 1 1.1 Why do we need proof by induction?.... 1 1. What is proof by
More informationNetwork Planning and Analysis
46 Network Planning and Analysis 1. Objective: What can you tell me about the project? When will the project finish? How long will the project take (project total duration)? 2. Why is this topic Important
More informationChapter 2 An Introduction to Forwards and Options
Chapter 2 An Introduction to Forwards and Options Question 2.1. The payoff diagram of the stock is just a graph of the stock price as a function of the stock price: In order to obtain the profit diagram
More informationLinear Programming Notes V Problem Transformations
Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material
More information4.6 Linear Programming duality
4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal
More informationHow to Configure and Use MRP
SAP Business One How-To Guide PUBLIC How to Configure and Use MRP Applicable Release: SAP Business One 8.8 All Countries English October 2009 Table of Contents Purpose... 3 The MRP Process in SAP Business
More informationLinear Programming. Solving LP Models Using MS Excel, 18
SUPPLEMENT TO CHAPTER SIX Linear Programming SUPPLEMENT OUTLINE Introduction, 2 Linear Programming Models, 2 Model Formulation, 4 Graphical Linear Programming, 5 Outline of Graphical Procedure, 5 Plotting
More informationSection IV.1: Recursive Algorithms and Recursion Trees
Section IV.1: Recursive Algorithms and Recursion Trees Definition IV.1.1: A recursive algorithm is an algorithm that solves a problem by (1) reducing it to an instance of the same problem with smaller
More informationLinear Programming. March 14, 2014
Linear Programming March 1, 01 Parts of this introduction to linear programming were adapted from Chapter 9 of Introduction to Algorithms, Second Edition, by Cormen, Leiserson, Rivest and Stein [1]. 1
More informationIncreasing for all. Convex for all. ( ) Increasing for all (remember that the log function is only defined for ). ( ) Concave for all.
1. Differentiation The first derivative of a function measures by how much changes in reaction to an infinitesimal shift in its argument. The largest the derivative (in absolute value), the faster is evolving.
More informationTime Value of Money. 2014 Level I Quantitative Methods. IFT Notes for the CFA exam
Time Value of Money 2014 Level I Quantitative Methods IFT Notes for the CFA exam Contents 1. Introduction...2 2. Interest Rates: Interpretation...2 3. The Future Value of a Single Cash Flow...4 4. The
More informationBinomial lattice model for stock prices
Copyright c 2007 by Karl Sigman Binomial lattice model for stock prices Here we model the price of a stock in discrete time by a Markov chain of the recursive form S n+ S n Y n+, n 0, where the {Y i }
More informationMATH 551 - APPLIED MATRIX THEORY
MATH 55 - APPLIED MATRIX THEORY FINAL TEST: SAMPLE with SOLUTIONS (25 points NAME: PROBLEM (3 points A web of 5 pages is described by a directed graph whose matrix is given by A Do the following ( points
More informationChapter 11 Number Theory
Chapter 11 Number Theory Number theory is one of the oldest branches of mathematics. For many years people who studied number theory delighted in its pure nature because there were few practical applications
More informationDERIVATIVES AS MATRICES; CHAIN RULE
DERIVATIVES AS MATRICES; CHAIN RULE 1. Derivatives of Real-valued Functions Let s first consider functions f : R 2 R. Recall that if the partial derivatives of f exist at the point (x 0, y 0 ), then we
More information10.2 Series and Convergence
10.2 Series and Convergence Write sums using sigma notation Find the partial sums of series and determine convergence or divergence of infinite series Find the N th partial sums of geometric series and
More informationSolutions to Math 51 First Exam January 29, 2015
Solutions to Math 5 First Exam January 29, 25. ( points) (a) Complete the following sentence: A set of vectors {v,..., v k } is defined to be linearly dependent if (2 points) there exist c,... c k R, not
More informationThe Basics of FEA Procedure
CHAPTER 2 The Basics of FEA Procedure 2.1 Introduction This chapter discusses the spring element, especially for the purpose of introducing various concepts involved in use of the FEA technique. A spring
More informationMethods for Finding Bases
Methods for Finding Bases Bases for the subspaces of a matrix Row-reduction methods can be used to find bases. Let us now look at an example illustrating how to obtain bases for the row space, null space,
More informationBasic Components of an LP:
1 Linear Programming Optimization is an important and fascinating area of management science and operations research. It helps to do less work, but gain more. Linear programming (LP) is a central topic
More informationSection 6.1 - Inner Products and Norms
Section 6.1 - Inner Products and Norms Definition. Let V be a vector space over F {R, C}. An inner product on V is a function that assigns, to every ordered pair of vectors x and y in V, a scalar in F,
More informationPERPETUITIES NARRATIVE SCRIPT 2004 SOUTH-WESTERN, A THOMSON BUSINESS
NARRATIVE SCRIPT 2004 SOUTH-WESTERN, A THOMSON BUSINESS NARRATIVE SCRIPT: SLIDE 2 A good understanding of the time value of money is crucial for anybody who wants to deal in financial markets. It does
More informationSOLVING LINEAR SYSTEMS
SOLVING LINEAR SYSTEMS Linear systems Ax = b occur widely in applied mathematics They occur as direct formulations of real world problems; but more often, they occur as a part of the numerical analysis
More information8.2. Solution by Inverse Matrix Method. Introduction. Prerequisites. Learning Outcomes
Solution by Inverse Matrix Method 8.2 Introduction The power of matrix algebra is seen in the representation of a system of simultaneous linear equations as a matrix equation. Matrix algebra allows us
More information4.5 Linear Dependence and Linear Independence
4.5 Linear Dependence and Linear Independence 267 32. {v 1, v 2 }, where v 1, v 2 are collinear vectors in R 3. 33. Prove that if S and S are subsets of a vector space V such that S is a subset of S, then
More informationSolution to Exercise 7 on Multisource Pollution
Peter J. Wilcoxen Economics 437 The Maxwell School Syracuse University Solution to Exercise 7 on Multisource Pollution 1 Finding the Efficient Amounts of Abatement There are two ways to find the efficient
More informationVALUE 11.125%. $100,000 2003 (=MATURITY
NOTES H IX. How to Read Financial Bond Pages Understanding of the previously discussed interest rate measures will permit you to make sense out of the tables found in the financial sections of newspapers
More informationLecture 3: Finding integer solutions to systems of linear equations
Lecture 3: Finding integer solutions to systems of linear equations Algorithmic Number Theory (Fall 2014) Rutgers University Swastik Kopparty Scribe: Abhishek Bhrushundi 1 Overview The goal of this lecture
More informationOverview. Essential Questions. Precalculus, Quarter 4, Unit 4.5 Build Arithmetic and Geometric Sequences and Series
Sequences and Series Overview Number of instruction days: 4 6 (1 day = 53 minutes) Content to Be Learned Write arithmetic and geometric sequences both recursively and with an explicit formula, use them
More informationOn the Efficiency of Competitive Stock Markets Where Traders Have Diverse Information
Finance 400 A. Penati - G. Pennacchi Notes on On the Efficiency of Competitive Stock Markets Where Traders Have Diverse Information by Sanford Grossman This model shows how the heterogeneous information
More informationMathematical Induction
Mathematical Induction In logic, we often want to prove that every member of an infinite set has some feature. E.g., we would like to show: N 1 : is a number 1 : has the feature Φ ( x)(n 1 x! 1 x) How
More informationNPV Versus IRR. W.L. Silber -1000 0 0 +300 +600 +900. We know that if the cost of capital is 18 percent we reject the project because the NPV
NPV Versus IRR W.L. Silber I. Our favorite project A has the following cash flows: -1 + +6 +9 1 2 We know that if the cost of capital is 18 percent we reject the project because the net present value is
More informationThis asserts two sets are equal iff they have the same elements, that is, a set is determined by its elements.
3. Axioms of Set theory Before presenting the axioms of set theory, we first make a few basic comments about the relevant first order logic. We will give a somewhat more detailed discussion later, but
More informationIEOR 6711: Stochastic Models, I Fall 2012, Professor Whitt, Final Exam SOLUTIONS
IEOR 6711: Stochastic Models, I Fall 2012, Professor Whitt, Final Exam SOLUTIONS There are four questions, each with several parts. 1. Customers Coming to an Automatic Teller Machine (ATM) (30 points)
More information