Dynamic Optimization user s guide

Transcription

1 Dynamic Opimizaion user s guide Economics 212 Spring 1998 TA: Éva Nagypál These noes are an aemp o give an overview of dynamic opimizaion and he soluion mehods used in solving dynamic opimizaion problems. Also, hey are an aemp o highligh he connecion beween he differen soluion mehods (finie horizon vs. infinie horizon or discree vs. coninuous ime.) All hrough hese noes I will use he consumpion problem o illusrae soluion mehods and conceps, bu he descripion is mean o be much more general and o cover mos dynamic opimizaion problems ha you will have o solve in he firs year macro sequence. Inroducion In a dynamic programming problem an opimizing agen solves an ineremporal opimizaion problem. He chooses a sequence of decisions o maximize some objecive funcion. Normally we will denoe he period when he sequence of decisions is made ime 0. I is imporan o noe ha decisions regarding all periods are made a ime 0, no jus decisions regarding ime 0. Tha is why we say ha he opimizing agen chooses a sequence of decisions, one for each period. The main difference beween dynamic and saic opimizaion problems is ha in dynamic opimizaion problems his period s decision affecs fuure possibiliies and he opimizing agen wans o ake ino accoun his effec when making his decision oday. An example of dynamic opimizaion is ineremporal consumpion choice. In 202 you learned abou he consumer who opimizes in a saic environmen given some endowmen. This consumer had a budge consrain each period, and maximized period uiliy subjec o his budge consrain. An ineremporal consumpion problem is differen in ha he consumer can ransfer wealh beween periods and hus his period consumpion need no saisfy he saic budge consrain. Insead he has one budge consrain, an ineremporal or dynamic one. This period s consumpion hen - hrough asse accumulaion or decumulaion - affecs he choices available in subsequen periods. Our main objecive in seing up a dynamic programming problem is o express his ineremporal dependence by defining an appropriae sae space. An elemen of he sae space compleely summarizes he effecs of pas decisions on oday s decision, hus insead of keeping rack of all pas decisions when making oday s decision we can jus make a decision based on oday s sae. The sae space can conain a single sae variable, several sae variables or even such objecs as he disribuion of some variable. The problems you will be solving will always have one or more sae variables. The key concep ha allows us o solve dynamic opimizaion problems is he Principle of Opimaliy, which 1

2 saes ha an opimal policy has he propery ha whaever he iniial sae and decision are, he remaining decisions are an opimal policy wih regard o he sae resuling from he firs ransiion. This means ha if an opimizing agen solves a dynamic programming problem oday, i.e. he chooses a sequence of decisions, one for each period, he can be assured ha he opimal policy he chose oday will sill be opimal omorrow afer he acion based on he firs decision has been aken and he ransiion from oday s sae o omorrow s sae has occurred. Wihin dynamic programming here are hree dichoomies: 1. discree vs. coninuous ime 2. deerminisic vs. sochasic case 3. finie vs. infinie horizon We will firs look a discree ime problems (noe ha ofen imes he erm dynamic programming iself is reserved for discree ime problems.) We will discuss he deerminisic case boh for finie horizon and infinie horizon problems. Then we will discuss he addiional pifalls if uncerainy is inroduced in he seup. In he second par we will look a coninuous ime problems in a deerminisic seing. 1 Discree ime dynamic opimizaion 1.1 Deerminisic case Finie horizon problems We will discuss he deerminisic finie horizon case in deail, since much of he insigh regarding discree ime dynamic opimizaion can be bes undersood in his seing. Boh he infinie horizon case and he sochasic case are exensions of his relaively simple seup. As saed above, he very firs sep in all dynamic opimizaion problems is o define he sae space. This you can do by carefully considering he problem a hand and hinking hrough wha are he variables ha affec he agen s decision a any poin in ime. In many problems you will find ha deermining he righ se of sae variables is relaively sraigh forward. Le us denoe he sae space by S, and a paricular elemen of i a ime by s. Noe ha in he general case s will conain as one of is variables because indicaes how much ime has passed and hus how much ime is lef unil he las period N. Le us hen separae s ino and all oher sae variables, x, i.e. s = {x,}. 2

3 Once we have defined our sae space, we can reduce our dynamic opimizaion problem o a sequenial decision process. The sae a ime, {x,}, provides a complee descripion of he informaion needed o make he decision regarding ime. Associaed wih every sae here is a feasible se Γ (x ) of decisions d Γ (x ). (d is ofen called he conrol variable. The opimal decision is called opimal policy, oo. Noe ha he opimal d is a funcion of he sae variables a ime.) Given he sae x and decision d we hen mus have: 1. a reward or uiliy (also called feliciy) f(x,d,) 2. a ransiion x = T (x,d,) o a new sae in period +1. While his seup migh seem a bi oo formal, i is easy o see wha he above objecs are in a ypical consumpion problem. Consider a consumer who lives for N periods and in period 0 is faced wih he problem of deciding how much o consume in each period unil period N in order o maximize he sum of discouned uiliies. He has asses A 0 now, and has an income Y in each period, =0,1,...N. Hecan borrow and lend a a consan ineres rae, r. In his case i is easy o convince yourself ha he decision regarding ime is influenced by 2 variables: his available asses a he beginning of period, A, and his income in period, Y. Thus, his sae variable is x = {A,Y }. The decision variable is consumpion, hus d = C. The equaion of moion for he asses is A =(1+r)(Y C + A ) hus he ransiion funcion is T (x,d,)=t(a,y,c,)=[(1+r)(y C + A ),Y (herey is no affeced by previous decision or sae variables, i is jus a predeermined variable - noe here is no uncerainy here!) The reward in each period is he discouned uiliy, hus f(x,d,)=f(a,y,c,)=β u(a,y,c ) and if we assume ha uiliy does no depend on A or Y hen we can jus wrie his as β u(c ). The uiliy of a program, d =(d 0,d 1,...d N ), is hen given by he sum: N U(x 0,d)= f(x,d,)+f(x N+1 ) (1) =0 where x 0 is a given iniial sae, d Γ (x ), x evolves according o T and F is he erminal value. In he consumpion problem hink of F as he uiliy derived from leaving a beques for one s children. Given some echnical condiions here exiss a program, d, ha maximizes he above uiliy, in which case we can define he value funcion a ime as: [ N V (x,)= max d =(d,d,...d N) f(x τ,d τ,τ)+f(x N+1 ) τ= (2) 3

4 Thus he value funcion is he maximal oal uiliy obainable beween periods and N given ha he sae a ime is x. The crucial sep in any dynamic programming problem is o esablish he relaionship beween he above formulaion of he value funcion and he alernaive formulaion, he so-called Bellman equaion: V (x,)= max [f(x,d,)+v(t(x,d,),+ 1) (3) d Γ (x ) To esablish his equivalence we will proceed by inducion. Assume ha we know ha V (x,)= V(x,+ 1) and assume ha d =(d,d,...d N ) is he program ha maximizes (2) (recall ha we assumed he exisence of such a program.) By he Principle of Opimaliy, however, we know ha if he opimal decision oday is d =(d,d,...d N ), hen he sequence d =(d,d +2,...d N ) will be opimal saring omorrow. Thus we can wrie: V (x,) = N f(x τ,d τ,τ)+f(x N+1) =f(x,d,)+ τ= N τ= f(x τ,d τ,τ)+f(x N+1) = (4) = f(x,d,)+v (T(x,d,),)=f(x,d,)+V(T(x,d,),) where in he las sep we used our firs assumpion. Now if here was a ˆd Γ (x ) such ha f(x, ˆd,)+V(T(x, ˆd,),)>f(x,d,)+V(T(x,d,),) hen here would be a program ˆd =(ˆd,ˆd,... ˆd N ) ha would resul in higher value for V (x,)han d,where ˆd =(ˆd, ˆd +2,... ˆd N ) is he program ha maximizes V (T (x, ˆd,),) = V (T (x, ˆd,),+ 1). The exisence of such a program, however, would conradic he opimaliy of d. Therefore here can be no such ˆd and hus: V (x,) = max [f(x,d,)+v(t(x,d,),+ 1) (5) d Γ (x ) hus V (x,)=v(x,). To wrap up he proof, noe ha by definiion V (x N+1,N+1) = V (x N+1,N+1) (since V (x N+1+i,N +1+i)=0fori>0), and hus our inducion can be sared off a = N. Having esablished his equivalence, we can hen solve (3) which is a much easier problem han solving (2). How do we go abou solving (3)? We will review hree soluion mehods here, each wih a differen goal in mind. The firs one is he one ha you should use in finie horizon cases, while he second and hird one are included wih an eye on quesions ha will arise laer. 4

5 Mehod 1. We can use essenially he same mehod ha we used o prove he equivalence of (2) and (3): backward recursion. 1. Le = N. We know ha Using his: V (x N+1,N +1)=F(x N+1 ) V (x N,N)= max [f(x N,d N,N)+F(T(x N,d N,N)) d N Γ N(x N) This is a simple saic opimizaion problem ha we can solve. Le d N (x N )behe opimal soluion. Then V (x N,N)=f(x N,d N(x N ),N)+F(T(x N,d N(x N ),N)) is a known funcion. 2. Decrease by 1. Since we know V (x,+ 1) by he previous sep, solving (3) can be done using sandard opimizaion ools. Thus we can obain V (x,) 3. If >1reurnosep2. Mehod 2. Anoher mehod can be used if some echnical condiions are saisfied. This mehod is worhwhile o discuss here, since i will prove useful in he infinie horizon case when he backward recursion mehod canno be used any more. A heorem by Benvenise and Sheinkman says ha given cerain regulariy condiions if Γ (x) is convex for all and each x and if f(x, d, ) is a sricly concave and differeniable funcion of (x, d) foreachhen he value funcion is differeniable. Le us assume ha his is he case and le d be he opimal policy. Then for all x and all a he opimum by he firs order condiion for d we have: (x,d V,)+ (T(x,d,),) (x,d,)=0 (6) while aking he derivaive of he value funcion a he opimum wih respec o he sae variable yields: V (x,)= (x,d V,)+ (T(x,d,),) (x,d,) (7) Using (6) o subsiue ino (7) for V we ge: v (x,)= (x,d,) (x,d d,) (x,d,) (x,d,) (8) 5

6 Noe ha he Benvenise - Sheinkman heorem saes ha he value funcion is differeniable under cerain condiions, and hen applies he envelope heorem o ge he relaionship expressed in (8). Thus i is more han jus applying he envelope heorem. Of course, in mos applicaions ha you see all he condiions of he Benvenise - Sheinkman heorem hold, and hus applying i becomes equivalen o applying he envelope heorem. In many economic applicaions = 0 (i.e. his period s uiliy does no depend on his period s sae) x in which case (8) becomes: V (x,)= (x,d,) (x,d,) (x,d,) (9) (A noable excepion is he problem regarding durable consumpion on Problem Se 2.) Also, here are many economic applicaions where (x,d,)= (x,d,), for which cases (9) can be furher reduced o: V (x,)= (x,d,) (10) Repeaing he same seps for nex period s value funcion we ge: V (x,)= (x,d x,) (x,d d,) (x,d,) +1 (x,d,) (11) which can hen be used o eliminae V (T(x,d,),+ 1) from (6) o arrive a he Euler equaion: ( (x,d d,)+ (x,d x,) (12) (x,d d,) (x,d,) +1 (x,d,) ) (x,d,)=0 Given funcional forms, his Euler equaion can be solved for he opimal policy. This approach has he disadvanage, however, ha i requires more echnical assumpions and ofen is less inuiive han backward recursion. Mehod 3. We can also use he familiar Lagrangian mehod o solve he above problem. Besides familiariy his mehod has he addiional advanage ha i highlighs he relaionship beween he discree ime and he coninuous case. We are maximizing (1) subjec o + 1 consrains, x = T (x,d,)for=0,1,...n. Thus we can wrie up he Lagrangian as: N N L = f(x,d,)+f(x N+1 )+ λ [T (x,d,) x (13) =0 =0 6

7 Taking he derivaive wih respec o d for =0,1,...N we ge he firs order condiions: (x,d,)+λ (x,d,)=0 =0,1,...N (14) Taking he derivaive wih respec o x for =0,1,...N we ge he firs order condiions: (x,d,)+λ (x,d,) λ =0 =0,1,...N (15) From (14) we can express λ as: d λ = (x,d,) (x,d,) and subsiue i back ino (15) o ge: (x,d,) (x,d,) (x,d,)=λ (16) (x,d,) The above relaionship can be derived for λ, oo, which hen can be subsiued back ino (14) o give: ( (x,d d,)+ (x,d x,) (17) ) (x,d d,) (x,d,) +1 (x,d,) (x,d d,)=0 Noice ha his is he exac same equaion as (12), which implies ha he Lagrangian mehod is anoher way o ge o he Euler equaion. Also noe by comparing (16) and (8) ha λ = v which makes inuiive sense if we hink of he inerpreaion of he Lagrange muliplier as he shadow price of x, i.e. he marginal gain ha resuls from relaxing he consrain x = T (x 1,d 1, 1) Infinie horizon problems If in he above deerminisic problem we le N,f(x,d,)=β u(x,d )andt(x,d,)=t(x,d ) independen of ime, hen we have a problem of saionary discouned dynamic programming o which we urn now. Firs noe ha will no longer be an elemen of he sae variable. This simplifies he analysis somewha, since he value funcion will be ime-invarian. As for he res, he same line of hough applies as in he 7

8 finie case. Drawing on resuls from he finie horizon case, if we define V (x )= sup d =(d,d,...) [ β τ u(x τ,d τ ) τ= (18) such ha x = T (x,d )andd Γ(x )givenx 0 hen he analogy o he finie horizon case suggess ha his same funcion will saisfy he funcional equaion: V (x )= sup [u(x,d )+βv (T(s,d,)) (19) d Γ(x ) The erm β eners because if we wan o define he value funcion exacly analogous o (2), hen we would define i as: [ ˆV (x )= sup d =(d,d,...) τ= β τ u(x τ,d τ) =β V (x ) (20) Thus (3) hen implies ha ˆV (x,) = max d Γ (x ) [ f(x,d,)+ ˆV(T(x,d,),), which in erms of V is β V (x,) = max d Γ (x ) [ β u(x,d )+β V (T (x,d,),). Dividing by β we ge (19). In he infinie horizon case esablishing his relaionship rigorously is much more complicaed. I will only ouline he condiions which are required for he above equivalence o hold (for a deailed proof see Sokey and Lucas chaper 3.) Assumpion 1 Γ(x) is non-empy for any x X, wherexdenoes he se of possible saes. Assumpion 2 For all x X and all feasible sequence of decisions given x 0, lim n n =0 β u(x,d ) exiss (alhough i may be plus or minus infiniy.) These wo assumpions are echnical in naure. The firs one says ha i is possible o sar an allocaion process from any x X. The second one requires he uiliy funcion no o be oo erraic. Under hese wo assumpions he supremum in (18) exiss and is unique, and hus can be replaced by he maximum. Theorem 1 Le X, Γ, u and β saisfy Assumpions 1 and 2. Then he funcion V saisfies he funcional equaion (19). Theorem 2 Le X, Γ, u and β saisfy Assumpions 1 and 2. If V is a soluion o he funcional equaion and saisfies lim n β n V (x n)=0for all feasible sequence of decisions given x 0,henV =V. The very las condiion, lim n β n V (x n) = 0, is he ransversaliy condiion. To find he unique soluion o (18) we need o pick from he many funcions ha saisfy (19) he single one ha will saisfy he TVC. In he finie horizon case, we could solve backwards, which assured ha from he many soluions 8

9 ha saisfy (3) we picked he single one ha saisfied (2) also. In he infinie horizon case here is no erminal condiion and ha is why we need some oher condiion o ie down wha he value funcion can do in he limi. This condiion is he TVC. Ulimaely, however, i is no value funcions ha we are ineresed in, bu opimal policies. The equivalence of he opimal policy under (18) and (19) is based on he following wo heorems: Theorem 3 Le X, Γ, u and β saisfy Assumpions 1 and 2. Le d =(d,d,...) be a feasible plan given x ha aains he supremum in (18) for iniial sae x.then V (x )=u(x,d )+βv (x ) where x = T (x,d ) (21) Theorem 4 Le X, Γ, u and β saisfy Assumpions 1 and 2. Le d =(d,d,...) be a feasible plan given x ha saisfies (21) and lim sup β V (x ) 0 hen d aains he supremum in (18) for iniial sae x 0. Having esablished he equivalence of (18) and (19), we can use his relaionship o solve for he opimal policy funcion which we know exiss if he above wo assumpions hold. Le us look a (19), which is ofen referred o as he Bellman equaion. Since we know ha an opimal policy, d, exiss, we can wrie max insead of sup in (19): V (x )= max {u(x,d )+βv (T(x,d,))} (22) d Γ(x ) Under regulariy condiions ha are similar o he one menioned for he finie horizon case, he Benvenise - Sheinkman heorem is applicable and hus he value funcion is differeniable. Therefore we can use Mehod 2 oulined for he finie horizon case. To recap, his soluion mehod is he following: 9

10 1. Take he firs order condiion: u (x,d d )+β V (T(x,d,)) (x,d d ) = 0 (23) 2. Use he envelope heorem a he opimum: 3. Eliminae V V (x )= u (x,d x )+β V (T(x,d )) (x,d x ) (24) from (23) and (24) o ge: V (x )= u (x,d x ) u (x,d ) (x,d ) (x,d ) (25) 4. Repea he same seps for nex period s value funcion (forward (25)) o ge: V (x )= u (x,d ) u (x,d d ) (x,d ) +1 (x,d ) (26) 5. Use (26) o eliminae V (T(x,d )) from (23) o arrive o he Euler equaion: ( u u (x,d d )+β (x,d x ) u (x,d d ) (x,d ) +1 (x,d ) ) (x,d ) = 0 (27) Examples Ineremporal consumpion Once again consider a consumer who lives for N periods and in period 0 is faced wih he problem of deciding how much o consume in each period unil period N in order o maximize he sum of discouned uiliies. He has asses A 0 now, and has an income Y in each period, =0,1,...N. He can borrow and lend a a consan ineres rae, r. We have seen ha he sae variable is x = {A,Y }. In he case where income is predeermined, and is no affeced by previous consumpion decisions as is assumed o be he case here, we can disregard i as a par of he sae variable, and simply concenrae on asses. The equaion of moion for he asses is A =(1+r)(Y C + A ). The reward in each period is he discouned uiliy, hus f(x,d,)=f(a,y,c,)=β u(a,y,c ). Le u(a,y,c )= C1 1/σ, i.e. uiliy is of he consan relaive risk aversion form. Then f(x,d,)= 1 1/σ β u(c ). The linear/quadraic regulaor problem 10

11 You migh wonder how wha Sargen alked abou in 210 fis ino his framework. The problem he invesigaed is a special case of he above general framework. Here he uiliy funcion is quadraic and he ransiion funcions are linear. By making hese funcional assumpions we can ge a lo of mileage from his seup. Finie horizon The finie horizon problem is formulaed as follows: Le f(s [ [ [ R G s,d )= s d G and he ransiion funcion be s Q d = A s + B d,where as before s is he vecor of sae variables (lengh n) and d is he vecor of conrol variables (lengh k). R is[ an n n negaive semi-definie symmeric marix, Q is a k k negaive definie symmeric marix, R G and G is a (n + k) (n + k) negaive definie marix. Q Also le F (s N)=s P Ns,whereP N is a negaive semi-definie marix. The soluion o his problem akes he following form: V (x )=x Px where P = R+A P A (A P B +G )(Q +B P B ) 1 (B P A +G )(28) The opimal conrol is: d = K x where K =(Q +B P B ) 1 (B P A + G ) (29) wih P N given. While hese formulas seem raher complicaed, here derivaion does no require anyhing else bu he careful use of linear algebra and backward subsiuion. The equaion for P in (28) is called he Ricai equaion for he finie horizon case. A special case of he above is he ime-homogeneous case, where A, B, G, Q, R, S are all ime-invarian marices. In his case: and P = R + A P A (A P B + G )(Q + B P B) 1 (B P A + G) K =(Q+B P B) 1 (B P A + G) The infinie horizon case - incomplee We ge o he infinie horizon case from he finie horizon one by leing N go o infiniy. The basic quesion is wheher P and K will converge as N. 1.2 Sochasic case Finie horizon problems The inroducion of uncerainy alers our opimizaion problem in several ways. Firs, we need o redefine our objecive. This we will do by adoping expeced uiliy heory. To model uncerainy we inroduce 11

12 a vecor of random variables, z, which are observed a dae and we modify he feliciy funcion o be f(x,d,z,).theobjecivehenis: sup E d=(d 0,d 1,...d N) [ N f(x,d,z,)+f(x N+1,z N+1,N +1) I 0 =0 (30) such ha x = T (x,d,z,), d Γ (x,z ), where I 0 denoes all informaion available a ime 0. Second, we have o specify how uncerainy eners he decision process. In wha follows we will always assume ha uncerainy regarding period is resolved before period decisions are made. This assumpion is naural for he sandard consumpion problem. If we consider a model wih sochasic income, hen usually we assume ha uncerainy regarding he sochasic income sream is resolved before consumpion decisions need o be made. In oher cases, however, i is possible ha his period s uncerainy is resolved only afer his period s decision has been made. This is he case, for insance, when a firm mus commi o an invenory level before uncerainy abou demand and prices is resolved. I can be shown ha he second case can be reformulaed ino he firs one, and his is why we only discuss he firs case. I should be clear from above ha he very firs problems we will encouner in he sochasic case is ha s = {x,} will no necessarily be an adequae descripion of he sae in he sense ha i is I ha is used a ime o make decisions abou he fuure. There are special cases, however, when we can sill find an adequae descripor of he sae given a finie number of variables known a ime. 1. If {z } are independen random variables ha are no affeced by pas values of x or d hen s = {x,} remains a sufficien descripion of he sae. 2. If he process {z } has finie memory, i.e. is value a ime is affeced by a finie number of lags of z, x or d, hen we can sill describe he sae space using a finie number of sae variables. Le us consider a simple case in which z is a Markov process so ha he disribuion of z only depends on x 1 and z 1. Then we have a process wih a finie (one-period) memory. We can hen wrie he Bellman equaion as: V (x,z,)= sup (f(x,d,z,)+e[v(t(x,d,z,),z,) z,x ) (31) d Γ (x,z ) Noe ha we wroe sup raher han max. This is because in his case even if u is a very smooh funcion, aking expecaions on he righ hand side of (31) may resul in loss of coninuiy or smoohness due o he paricular disribuional properies of z. The exisence of he maximum and he exisence of a value 12

13 funcion may depend on he probabiliy disribuion of z. Wih hese qualifiacions in mind, we can wrie he maximizaion problem as: max E d [ N f(x,d,z,)+f(x N+1,z N+1,N +1) x 0,z 0 =0 (32) subjec o: x = T (x,d,z,) given x 0 and F and such ha d Γ (x,z ). The corresponding Bellman equaion is: V (x,z,)= max (f(x,d,z,)+e[v(t(x,d,z,),z,) z,x ) (33) d Γ (x,z ) Noe ha in his case he opimal policy a any ime d will be a funcion of he sae variables a ime. Since z is one of he sae variables, his means ha he opimizing agen does no choose deerminisically a ime 0 as when here was no uncerainy. Insead, his opimal policy is a coningen plan ha is condiional on he paricular realizaion of z and x (he laer also being a random variable a ime 0 since i is affeced by previous realizaions of z.) In he form of (33) he sochasic problem is in he same form as he deerminisic problem was in he previous par. Direc backward inducion can be used once again Infinie horizon case If in he above problem we le N,f(x,d,z,)=β u(x,z,d )andt(x,d,z,)=t(x,d,z ) independen of ime, hen we have a problem of sochasic saionary discouned dynamic programming. This is he mos common problem you will encouner. I should be clear by now ha his problem is a very difficul one concepually. Boh he qualificaions ha were encounered a he deerminisic infinie horizon case and he qualificaions of he sochasic finie horizon case apply. Noneheless, we normally do no concern ourselves oo much wih hese qualificaions, and assume everyhing we need along he way o be able o use our sandard resuls. We can hen define he value funcion as: V (x )= max E d =(d,d,...) [ β τ u(x τ,z τ,d τ ) x,z τ= (34) such ha x = T (x,z,d )andd Γ(x,z )givenx,z. Given he above qualificaions, his funcion 13

14 will saisfy he Bellman equaion: V (x,z )= max {[u(x,z,d )+βe [V (T(x,z,d ),z ) x,z } (35) d Γ(x,z ) To denoe he expecaion E[. z,x we can use he shorhand E. Solving his problem is very similar o solving deerminisic infinie horizon problems, excep for he elemen of uncerainy. 1. Take he firs order condiion: u (x,z,d )+βe [ v (T(x,z,d,),z ) (x,z,d ) = 0 (36) 2. Use he envelope heorem a he opimum: v (x,z )= u [ v (x,z,d x )+βe (T(x,z,d ),) (x,z,d x ) (37) [ 3. Eliminae E v from (36) and (37) o ge: v (x,z )= u (x,z,d u ) (x,z,d d ) (x,z,d ) (x,z,d ) (38) 4. Repea he same seps for nex period s value funcion (forward (38)) o ge: v (x,z )= = u (x,z,d ) u (x,z,d d ) (x,z,d ) +1 (x,z,d ) (39) 5. Use (39) o eliminae v (T(x,d )) from (36) o arrive o he Euler equaion: u (x,z,d )+βe u (x,z,d d ) [ u (x,z,d ) (40) (x,z,d ) +1 (x,z,d ) (x,z,d )=0 14

15 2 Coninuous ime dynamic opimizaion In coninuous ime you will be asked o solve deerminisic problems only. Solving sochasic coninuous ime problems goes beyond he scope of his handou and is no required in he firs year. The aached pages are from he Mahemaical Appendix o Barro and Sala-i-Marin: Economic Growh (McGraw-Hill,1995). I find i a good summary of coninuous ime problems. Noe he similariies beween he discree and coninuous ime problems. In boh cases we are in essence using he Lagrangian mehod. We saw in he firs par ha in discree ime using he Lagrangian mehod (mehod 3.) leads o he same Euler equaion as using he Bellman approach (mehod 2.) In fac he Bellman equaion is jus a more compac way o express he same idea - he incorporaion of he effec of his period s decision on fuure possibiliies (which in he Lagrangian mehod is done explicily by using he Lagrange mulipliers.) In coninuous ime using he Lagrange mehod leads o he concep of Hamilonians which is once again a compac way o express he ineremporal dependence of decisions. 15