Inaugural - Dissertation

Transcription

1 Inaugual - Dissetation zu Elangung de Doktowüde de Natuwissenschaftlich-Mathematischen Gesamtfakultät de Rupecht - Kals - Univesität Heidelbeg vogelegt von Diplom-Mathematike Makus Fische aus Belin Datum Tag de mündlichen Püfung: 2. Novembe 27

2

3 Discetisation of continuous-time stochastic optimal contol poblems with delay Gutachte: Pof. D. Makus Reiß Univesität Heidelbeg Pof. Salah-Eldin A. Mohammed Southen Illinois Univesity, Cabondale

4

5 i Abstact In the pesent wok, we study discetisation schemes fo continuous-time stochastic optimal contol poblems with time delay. The dynamics of the contol poblems to be appoximated ae descibed by contolled stochastic delay o functional diffeential equations. The value functions associated with such contol poblems ae defined on an infinite-dimensional function space. The discetisation schemes studied ae obtained by eplacing the oiginal contol poblem by a sequence of appoximating discete-time Makovian contol poblems with finite o finite-dimensional state space. Such a scheme is convegent if the value functions associated with the appoximating contol poblems convege to the value function of the oiginal poblem. Following a geneal method fo the discetisation of continuous-time contol poblems, sufficient conditions fo the convegence of discetisation schemes fo a class of stochastic optimal contol poblems with delay ae deived. The geneal method itself is cast in a fomal famewok. A semi-discetisation scheme fo a second class of stochastic optimal contol poblems with delay is poposed. Unde standad assumptions, convegence of the scheme as well as unifom uppe bounds on the discetisation eo ae obtained. The question of how to numeically solve the esulting discete-time finite-dimensional contol poblems is also addessed.

6 ii Zusammenfassung In de voliegenden Abeit untesuchen wi Schemata zu Disketisieung von zeitstetigen stochastischen Kontollpoblemen mit Zeitvezögeung. Die Dynamik solche Pobleme wid von gesteueten stochastischen Diffeentialgleichungen mit Gedächtnis beschieben. Die zugehöigen Wetfunktionen sind auf einem unendlich-dimensionenalen Funktionenaum definiet. Man ehält die Disketisieungsschemata, die wi betachten, indem man das Ausgangspoblem duch eine Folge appoximieende zeitdiskete Makovsche Kontollpobleme esetzt, deen Zustandsaum endlich-dimensional ode endlich ist. Ein solches Schema ist konvegent, wenn die Wetfunktionen de appoximieenden Steuungspobleme gegen die Wetfunktion des uspünglichen Poblems steben. Indem wi eine allgemeine Methode zu Disketisieung zeitstetige Kontollpobleme anwenden, ehalten wi hineichende Bedingungen fü die Konvegenz von Disketisieungsschemata fü eine Klasse von stochastischen Steueungspoblemen mit Zeitvezögeung. Die Methode zu Konvegenzanalyse selbst wid in einen fomalen Rahmen gefasst. Wi fühen dann ein Semidisketisieungsschema fü eine zweite Klasse von stochastischen Steueungspoblemen mit Zeitvezögeung ein. Unte üblichen Annahmen weden die Konvegenz des Schemas, abe auch gleichmäßige obee Schanken fü den Disketisieungsfehle hegeleitet. Schließlich widmen wi uns de Fage, wie die esultieenden endlich-dimensionalen Steueungspobleme numeisch gelöst weden können.

7 iii Danksagung Fü den Voschlag des Gebietes, in das die voliegende Abeit fällt, und die Beteuung und ununtebochene Untestützung in allen fachlichen und auch außefachlichen Fagen danke ich Pof. Makus Reiß. Mein Dank gebüht Pof.ssa Giovanna Nappo von de Univesität La Sapienza fü die fuchtbae Zusammenabeit und ihe Gastfeundschaft wähend meines Aufenthalts in Rom von Apil bis Septembe 26. Fü die Untestützung bei de Oganisation dieses Aufenthalts danke ich Pof. Pete Imkelle. Den Mitglieden de Foschungsguppe Stochastische Algoithmen und Nichtpaametische Statistik am Weiestaß Institut WIAS in Belin und den Mitglieden de Statistikguppe an de Univesität Heidelbeg danke ich fü die angenehme gemeinsam vebachte Zeit. Fü wetvolle Hinweise zu Theoie deteministische Kontollpobleme danke ich Pof. Mauizio Falcone. Fü fachliche Gespäche, Anegungen und Untestützung danke ich Stefan Ankichne, Chistian Bende, Chistine Gün, Jan Johannes, Alexande Linke, Jan Neddemeye, Eva Saskia Rohbach und Kasten Tabelow. Finanzielle Untestützung von Seiten de Deutschen Foschungsgemeinschaft DFG und des ESF-Pogamms Advanced Methods in Mathematical Finance ekenne ich dankend an. Schließlich und endlich danke ich Stella fü ihe Geduld und ih Veständnis.

8 iv

9 Contents Notation and abbeviations vii 1 Intoduction Stochastic optimal contol poblems with delay Stochastic delay diffeential equations Optimal contol poblems with delay Examples of optimal contol poblems with delay Linea quadatic contol poblems A simple model of esouce allocation Picing of weathe deivatives Delay poblems educible to finite dimension Appoximation of continuous-time contol poblems Aim and scope The Makov chain method Kushne s appoximation method An abstact famewok Optimisation and contol poblems Appoximation and convegence Application to stochastic contol poblems with delay The oiginal contol poblem Existence of optimal stategies Appoximating chains Convegence of the minimal costs An auxiliay esult Discussion Two-step time discetisation and eo bounds The oiginal contol poblem Fist discetisation step: Eule-Mauyama scheme Second discetisation step: piecewise constant stategies Bounds on the total eo Solving the contol poblems of degee N, M Conclusions and open questions v

10 vi CONTENTS A Appendix 79 A.1 On the Pinciple of Dynamic Pogamming A.2 On the modulus of continuity of Itô diffusions A.3 Poofs of constant coefficients eo bounds Bibliogaphy 95

11 vii Notation and abbeviations a b a b the smalle of the two numbes a, b the bigge of the two numbes a, b 1 A indicato function of the set A x x N N Z BX CX, Y CX DI Gauß backet of the eal numbe x, that is, the lagest intege not geate than x the least intege not smalle than the eal numbe x the set of natual numbes stating fom one the set of all non-negative integes the set of all integes the space of all bounded eal-valued functions on the set X the space of all continuous functions fom the topological space X to the topological space Y the space of all continuous eal-valued functions on the topological space X the Skoohod space of all eal-valued càdlàg functions on the inteval I C in Chapte 3: the space C[, ], R d C N in Chapte 3: the space C[ N, ], Rd ĈN A T càdlàg iff in Chapte 3: the space of all ϕ C which ae piecewise linea w.. t. the gid {k N k Z} [, ] tanspose of the matix A ight-continuous with left-hand limits Fench aconym if and only if w.. t. with espect to

12 viii

13 Chapte 1 Intoduction In this thesis, discetisation schemes fo the appoximation of continuous-time stochastic optimal contol poblems with time delay in the state dynamics ae studied. Optimal contol poblems of this kind ae infinite-dimensional contol poblems in a sense to be made pecise below; they aise in engineeing, economics and finance, among othes. We will deive esults about the convegence of discetisation schemes. Fo a moe specific semi-discetisation scheme, a pioi bounds on the discetisation eo will also be obtained. Such esults ae useful in the numeical solution of the oiginal contol poblems. Section 1.1 pesents the class of optimal contol poblems we will be concened with. In Section 1.2, some examples of optimal contol poblems with delay ae given. Section 1.3 povides an oveview ove appoaches and some esults fom the liteatue elated to the discetisation of continuous-time optimal contol poblems with o without delay. The oganisation of the main pat of the pesent wok, its aim and scope ae specified in Section Stochastic optimal contol poblems with delay Hee, we intoduce the type of optimal contol poblems we will be concened with in this thesis. An optimal contol poblem is composed of two pats: a contolled system and a pefomance citeion. Given an initial condition of the system and a stategy, the system poduces a unique output. A numeical value is assigned to each output accoding to the pefomance citeion. In this way, the pefomance of any stategy fo any given initial condition is measued. The objective is to find stategies which pefom as good as possible, and to calculate optimal pefomance values. A contolled system is usually modelled as a discete- o continuous-time paametised dynamical system. In continuous time, contolled systems ae often descibed by some kind of diffeential equation. The continuous-time contolled systems we ae inteested in, hee, ae modelled as stochastic o deteministic delay diffeential equations. We descibe this class of equations in Subsection 1.1.1; a standad efeence is Mohammed In Subsection 1.1.2, the class of stochastic optimal contol poblems with delay we study in this wok is intoduced. If the time delay is zeo, then those poblems educe to odinay stochastic optimal contol poblems. Fo this latte class of poblems a welldeveloped theoy exists; see, fo instance, Yong and Zhou 1999 o Fleming and Sone 1

14 2 CHAPTER 1. INTRODUCTION 26. Basic optimality citeia, in paticula the Pinciple of Dynamic Pogamming, ae also mentioned in Subsection Stochastic delay diffeential equations An odinay Itô stochastic diffeential equation SDE is an equation of the fom 1.1 dxt = b t, Xt dt + σt, Xt dw t, t, whee b is the dift coefficient, σ the diffusion coefficient and W. a Wiene pocess. When the diffusion coefficient σ is zeo, then Equation 1.1 takes on the fom of an odinay diffeential equation ODE. Let the state space be R d. The unknown function X. in Equation 1.1 is then an R d -valued stochastic pocess with continuous o càdlàg 1 tajectoies. The dift coefficient b is a function [, R d R d, the diffusion coefficient σ a matix-valued function [, R d R d d 1, and W is a d 1 -dimensional Wiene pocess defined on a filteed pobability space Ω, F, P adapted to the filtation F t t. In the notation, we often omit the dependence on ω Ω. Equation 1.1 is to be undestood as an integal equation. Standad assumptions on the coefficients b, σ guaantee that the initial value poblem 1.2 Xt = { X + b s, Xs ds + σs, Xs dw s, t >, x, t =, possesses, fo each x R d, a unique stong solution, that is, thee is a unique up to indistinguishability R d -valued stochastic pocess X = Xt t with continuous o càdlàg tajectoies which is defined on Ω, F, P and adapted to the filtation F t t such that Equation 1.2 is satisfied. The initial condition may also be stochastic, namely an F - measuable R d -valued andom vaiable. Standad assumptions guaanteeing stong existence and uniqueness of solutions to Equation 1.2 ae that b, σ ae jointly measuable, Lipschitz continuous in the second vaiable unifomly in the fist and that they satisfy a condition of sublinea gowth in the second vaiable unifomly in the fist; see paagaph in Kaatzas and Sheve 1991: p. 289, fo example. An impotant popety of solutions of SDEs is that they ae Makov pocesses w.. t. the given filtation. Anothe equally impotant popety is that they ae continuous semimatingales with semi-matingale decomposition given by the SDE itself. In addition to the notion of stong solution, thee is the notion of weak solution to an SDE. While stong solutions must live on the given pobability space and must be adapted to the given filtation, weak solutions ae only equied to exist on some suitable stochastic basis; fo example, the given filtation may be the one induced by the diving Wiene pocess, but solutions exist only when they ae adapted to some lage filtation. Thus, thee ae two notions of existence and also two notions of uniqueness fo an SDE, cf. Kaatzas and Sheve 1991: Sects. 5.2 & A function defined on an inteval is càdlàg iff it is ight-continuous with limits fom the left.

15 1.1. STOCHASTIC OPTIMAL CONTROL PROBLEMS WITH DELAY 3 The basic existence and uniqueness esults cay ove to the case of andom coefficients, that is, b, σ ae defined on [, R d Ω, povided b, σ ae F t -adapted. 2 A contolled SDE can be epesented in the fom 1.3 dxt = b t, Xt, ut dt + σt, Xt, ut dw t, t, whee u. is a contol pocess, that is, an F t -adapted function [, Ω Γ. Hee, Γ is a sepaable metic space, called the space of contol actions. The coefficients in Equation 1.3 ae deteministic functions [, R d Γ R d and [, R d Γ R d d 1, espectively. Fo any given contol pocess u., howeve, b.,., u., σ.,., u. ae adapted andom coefficients. A contol pocess u. such that the initial value poblem coesponding to the contolled equation, hee Equation 1.3, has a unique solution fo each initial condition of inteest will be called an admissible stategy o, simply, a stategy. Thoughout this thesis, we will epesent contol pocesses and stategies as Γ-valued functions defined on [, Ω, that is, defined on the poduct of time and scenaio space. In the deteministic case, contol pocesses educe to functions [, Γ, so-called open-loop contols. In the liteatue, contol pocesses ae often epesented as feedback contols, that is, as deteministic functions defined on the poduct of time and state space. This epesentation, though being natual fo the contol of Makov pocesses, leads to technical difficulties aleady fo discete-time contol poblems, see Betsekas and Sheve Feedback contols give ise to contol pocesses in the fom consideed hee. Systems with delay ae chaacteised by the popety that thei futue evolution, as seen fom any instant t, depends not only on t and the cuent state at t and possibly the contol, but also on states of the system a cetain amount of time into the past. We will assume thoughout that the system has bounded memoy; thus, thee is some finite > such that the futue evolution of the system as seen fom time t depends only on t and system states ove the peiod [t, t]. The paamete is the maximal length of the memoy o delay. Stochastic delay diffeential equations SDDEs model systems with delay. The dift and diffusion coefficient of an SDDE ae functions of time and tajectoy segments and, possibly, the contol action. Fo an R d -valued function ψ = ψ. living on the time inteval [,, the segment of length at time t [, is the function ψ t : [, ] R d, ψ t s := ψt+s, s [, ]. If ψ is a continuous function, then the segment ψ t at time t is a continuous function defined on [, ]. Likewise, if ψ is a càdlàg function, then the segment ψ t at time t is a càdlàg function defined on [, ]. Accodingly, if Xt t is an R d -valued stochastic pocess with continuous tajectoies, then the associated segment pocess X t t is a stochastic pocess taking its values in C := C[, ], R d, the space of all R d -valued continuous functions on the inteval [, ]. In this wok, the space C will always be equipped with the supemum nom induced by the standad nom on R d. 2 Stictly speaking, the statement about SDEs with andom coefficients is tue only if existence and uniqueness ae undestood in the stong sense. The notions of weak existence and weak uniqueness make sense also fo solutions to contolled SDEs with o without delay, cf. Section 3.1.

16 4 CHAPTER 1. INTRODUCTION The segment pocess associated with an R d -valued stochastic pocess with càdlàg tajectoies takes its values in the space D := D[, ], R d of all R d -valued càdlàg functions on [, ]. We will efe to the space of tajectoy segments as the segment space. As segment space we will choose eithe D o C. Notice that both spaces depend on the dimension d and the maximal length of the delay ; both d and may vay. In the notation just intoduced, an SDDE is of the fom 1.4 dxt = b t, X t dt + σt, Xt dw t, t. The coefficients b, σ ae now functions defined on [, D o, in the case of andom coefficients, on [, D Ω, whee D is the segment space. In ode to obtain unique solutions, as initial condition we have to pescibe not a point x R d, but an initial segment ϕ D. The initial segment might also be stochastic, namely a D-valued F -measuable andom vaiable. Let the segment space be the space C of continuous functions. Theoem II.2.1 in Mohammed 1984: p. 36 gives sufficient conditions such that, fo each initial segment ϕ C, the initial value poblem 1.5 Xt = { X + b s, X s ds + σs, X s dw s, t >, ϕt, t [, ], possesses a unique stong solution. Sufficient conditions ae that the coefficients b, σ ae measuable, ae Lipschitz continuous in thei segment vaiable unde the supemum nom on C unifomly in the time vaiable, satisfy a linea gowth condition and, in case they ae andom, ae F t -pogessively measuable. Existence and uniqueness esults fo SDDEs can also be deived fom the existence and uniqueness esults fo geneal functional SDEs as given, fo instance, in Potte 23: Ch. 5. Thee, the coefficients of the SDE ae allowed to be andom and to depend on the entie tajectoy of the solution fom time zeo up to the cuent time. Initial conditions, howeve, ae not tajectoy segments, but points in R d o R d -valued andom vaiables. Hence, to tansfe the esults, the dift and diffusion coefficient of the SDDE have to be edefined accoding to the given initial condition. A contolled SDDE can be epesented in the fom 1.6 dxt = b t, X t, ut dt + σt, X t, ut dw t, t, whee u. is a Γ-valued contol pocess as above and b, σ ae deteministic functions defined on [, D Γ. Existence and uniqueness ae again a consequence of the geneal esults applied to the andom coefficients b.,., u., σ.,., u.. Obseve that, in Equation 1.6, thee is no delay in the contol. At time t >, the coefficients b, σ depend on ut, and ut is F t -measuable. Systems with delay in the contol ae outside the scope of the pesent wok. Some kind of implementation delay, howeve, can be captued. Let w be some measuable function Γ R l. We can now add l additional dimensions to the state space R d and conside an SDDE of the fom dxt = b t, X t, Y t, ut dt + σt, X t, Y t, ut dw t, dy t = w ut dt,

17 1.1. STOCHASTIC OPTIMAL CONTROL PROBLEMS WITH DELAY 5 whee X. epesents the fist d components and Y. the emaining l components. The coefficients b, σ do not diectly depend on the tajectoy of u., but, though Y t, on segments of the pocess wusds t and the initial segment Y ; b, σ may, fo example, be functions of the diffeence Y t δ Y t, whee δ [,. In this way, distibuted implementation delay can be modelled. The solution of an SDDE like Equation 1.4 is, in geneal, not a Makov pocess. Suppose the coefficients of the SDDE ae deteministic and uncontolled o else a constant contol is applied, and let X. be a solution pocess. Then the segment pocess X t t associated with X. enjoys the Makov popety, cf. Theoem III.1.1 in Mohammed 1984: p. 51. The Makov semigoup of linea opeatos induced by the tansition pobabilities of the segment pocess is weakly, but not stongly continuous. In paticula, only the weak infinitesimal geneato exists. A epesentation of the weak infinitesimal geneato on a subset of its domain as a diffeential opeato can be deived, cf. Theoem III.4.3 in Mohammed 1984: pp The solution of an SDDE like Equation 1.4, although geneally not a Makov pocess, is an Itô diffusion and a continuous semi-matingale afte time zeo, and the Itô fomula is applicable as usual. Howeve, the usual Itô fomula does not apply to the segment pocess. It is possible to develop an Itô-like calculus also fo the segment pocesses associated with solutions of SDDEs, see Hu et al. 24 and Yan and Mohammed 25. In this thesis, the diving noise pocess of the continuous-time systems will always be a Wiene pocess. Extensions of some of the esults of this thesis, in paticula the convegence analysis of Section 2.3, to systems diven by moe geneal Lévy pocesses ae possible. In Chaptes 2 and 3, we will be concened with the discetisation of contolled systems with delay; hee, we give some efeences to woks concened with the discetisation of uncontolled systems with delay. An oveview of numeical methods fo uncontolled SDDEs is given in Buckwa 2. The simplest discetisation pocedue is the Eule-Mauyama scheme. The wok by Mao 23 gives the ate of convegence fo this scheme povided the SDDE has globally Lipschitz continuous coefficients and the dependence on the segments is in the fom of genealised distibuted delays; Poposition 3.3 in Section 3.2 of the pesent wok povides a patial genealisation of Mao s esults and uses aguments simila to those in Calzolai et al. 27. The most common fist ode scheme is due to Milstein; see Hu et al. 24 fo the ate of convegence of this scheme applied to SDDEs with point delay Optimal contol poblems with delay Recall that an optimal contol poblem is composed of a contolled system and a pefomance citeion. In what follows, the contolled system will always be descibed by a contolled SDDE like Equation 1.6 in Subsection As initial condition, an element of the segment space D has to be pescibed; the segment space D will be eithe C := C[, ], R d o D := D[, ], R d. When, in addition to the initial segment ϕ D, also the initial time t [, is allowed to vay, then the system output fo

18 6 CHAPTER 1. INTRODUCTION initial condition t, ϕ unde contol pocess u. is detemined by 1.7 { ϕ + Xt = b t +s, X s, us ds + σt +s, X s, us dw s, t >, ϕt, t [, ], povided a unique solution X = X t,ϕ,u exists. Notice that the solution pocess X. is defined ove time [,, and the evolution of the system stats at time zeo. The initial time t only appeas in the time agument of the coefficients. The pefomance citeion is usually given in tems of a cost functional. The cost functionals we will conside ae of the fom 1.8 t, ϕ, u. τ E f t +s, X s, us ds + g X τ, whee X = X t,ϕ,u is the solution to Equation 1.7 with initial condition t, ϕ unde stategy u. and τ is the emaining time, which may depend on t and X t,ϕ,u. The functions f, g ae called the cost ate and teminal cost, espectively; they may depend on segments of the solution pocess; in geneal, f is a function [, D Γ R, while g is a function D R. Two vesions of 1.8 will play a ole. The fist vesion gives ise to optimal contol poblems with finite time hoizon. Choose T >, the deteministic time hoizon, and set τ := T t. Fo the second vesion, choose a bounded open set O R d, let ˆτ O be the time of fist exit of X t,ϕ,u fom O and set τ := ˆτ O T t, whee T, ]. This leads to optimal contol poblems with andom time hoizon. Let T > be finite, and let τ in 1.8 be T t. Denote by U the set of admissible stategies, that is, the set of all those contol pocesses u. such that the initial value poblem 1.7 yields a unique solution and the expectation in 1.8 a finite value fo each initial condition t, ϕ [, T ] D. Let the function J : [, T ] D U R be defined accoding to 1.8. Then J is the cost functional of an optimal contol poblem with finite time hoizon. Given an optimal contol poblem, thee is a twofold objective: detemine the minimal costs and find an optimal stategy fo any initial condition. A stategy u is optimal fo a given initial condition t, ϕ iff 1.9 Jt, ϕ, u = inf u U Jt, ϕ, u. Existence of optimal stategies is not always guaanteed. Let us assume that the ight hand side of Equation 1.9 is finite fo all initial conditions which is not necessaily the case. A diect minimisation of Jt, ϕ,. ove the set U is usually not possible. Obseve that initial conditions ae time-state pais; hee, states ae segments, that is, continuous o càdlàg functions on [, ]. A simple, yet fundamental appoach, associated with the wok of R. Bellman, to solving the dynamic optimisation poblem is as follows. Intoduce the function which assigns the minimal costs to each time-state pai. This function is called the value function. The values of the value function ae, of couse, unknown at this stage. If the system, the set of stategies and the cost functional have a cetain additive stuctue in time, then the value function obeys Bellman s Pinciple of Optimality o, as it is also called, the Pinciple of

19 1.1. STOCHASTIC OPTIMAL CONTROL PROBLEMS WITH DELAY 7 Dynamic Pogamming PDP. Let V denote the value function of some optimal contol poblem; thus, V is a function I S R, whee I is a time inteval and S the state space. Bellman s Pinciple then states that V satisfies 1.1 V t, x = T t, V,. t, x fo all x S, t, I, t, whee T t, is a two-paamete semigoup of monotone opeatos, called Bellman opeatos; see Fleming and Sone 26: Sect. II.3 fo this abstact fomulation of the PDP. In the case at hand, the value function is defined by 1.11 V : [, T ] D R, V t, ϕ := inf u U Jt, ϕ, u. The Pinciple of Dynamic Pogamming takes on the fom 1.12 V t, ϕ = inf E f t +s, Xs u, us ds + V t +t, Xt u, t T t, u U whee X u is the solution to Equation 1.7 unde contol pocess u with initial condition t, ϕ. The minimisation on the ight hand side of Equation 1.12 could be esticted to stategies defined on the time inteval [, t]. Obseve that the validity of the PDP has to be veified fo each class of optimal contol poblems. Fo finite hoizon stochastic and deteministic optimal contol poblems with delay, the PDP is indeed valid, see Lassen 22 and also Appendix A.1 fo the pecise statement. The Makov popety of the segment pocesses associated with solutions to Equation 1.7 unde cetain stategies is essential fo the validity of the PDP in the fom of Equation Notice that an optimal contol poblem with delay is, geneally, infinite-dimensional in the sense that the coesponding value function lives on an infinite-dimensional function space, namely the segment space. When the contolled pocesses ae contolled Makov pocesses with finite-dimensional state space and the value function is sufficiently smooth, then the PDP in conjunction with Dynkin s fomula allows to deive a patial diffeential equation PDE which is solved by the value function. Such a PDE, which involves the family of infinitesimal geneatos associated with the contolled Makov pocesses and chaacteises the value function, is called Hamilton-Jacobi-Bellman equation HJB equation. In geneal, the value function need not be sufficiently smooth; consequently, the HJB equation does not necessaily possess classical solutions. Viscosity solutions povide the ight genealisation of the concept of solution fo HJB equations, see Fleming and Sone 26. In pinciple, it is possible to deive an HJB equation and define viscosity solutions also fo contolled Makov pocesses with infinite-dimensional state space. See Chang et al. 26 fo esults in this diection in connection with contolled SDDEs; also cf. Subsection While, in Chapte 3, we will make extensive use of the PDP, we will not need any kind of HJB equation. Let us also mention the fact that knowledge of the value function of an optimal contol poblem enables us to constuct optimal o nealy optimal stategies. When time is discete and the space of contol actions Γ is finite o compact, then optimal stategies can be constucted in feedback fom and fo each initial condition. We will etun to this point in Section 3.4.

20 8 CHAPTER 1. INTRODUCTION A second fundamental appoach to optimal contol poblems is via Pontyagin s Maximum Pinciple, see Yong and Zhou 1999: Chs. 3 & 7 fo the case of finite-dimensional contolled SDEs. Pontyagin s Pinciple povides necessay conditions which an optimal stategy and the associated optimal pocess if such exist have to satisfy in tems of the so-called adjoint equations, which evolve backwads in time. Unde cetain additional assumptions, the necessay conditions become sufficient. Vesions of this pinciple fo the contol of deteministic systems with delay exist, cf. the example in Subsection Fo stochastic contol poblems with delay of a special fom, a vesion of the Pontyagin Maximum Pinciple is deived in Øksendal and Sulem 21. Fo the esults of this thesis, we will not ely on the Maximum Pinciple. We have not made pecise any assumptions on the coefficients of the contol poblems intoduced above. This will be done in Subsection and Section 3.1, espectively, whee we specify the classes of continuous-time contol poblems to be appoximated. 1.2 Examples of optimal contol poblems with delay Some examples of continuous-time optimal contol poblems with delay, mostly fom the liteatue, ae given in this section. Contol poblems with linea dynamics and a quadatic cost citeion ae well-studied in many settings. In Subsection 1.2.1, we cite esults concening the epesentation of optimal stategies fo a class of linea quadatic egulatos with point as well as distibuted delay. In Subsection 1.2.2, a simple deteministic poblem with point delay modelling the optimal allocation of poduction esouces is pesented. Subsection descibes a stochastic optimal contol poblem with delay which may aise in finance when picing deivatives that depend on maket extenal pocesses. Special cases of optimal contol poblems with delay ae eally equivalent to finite-dimensional contol poblems without delay. Subsection contains esults fom the liteatue about those educible poblems. A futhe example is the deteministic infinite hoizon model of optimal economic gowth studied in Boucekkine et al. 25. Optimal contol poblems also aise in finance when the asset pices in a financial maket ae modelled as SDDEs, see Øksendal and Sulem 21, fo instance Linea quadatic contol poblems When the system dynamics ae linea in the state as well as in the contol vaiable, the noise is additive and the cost functional has a quadatic fom ove a finite o infinite time hoizon, then it is possible to deive a epesentation of the optimal stategies of the contol poblem. Such contol poblems ae efeed to as linea quadatic poblems o linea quadatic egulatos. Optimal stategies ae given in feedback fom; the epesentation involves the solution of an associated system of deteministic diffeential equations, the so-called Riccati equations. This is the case not only fo finite-dimensional stochastic and deteministic systems, but also fo systems descibed by abstact evolution equations cf. Bensoussan et al., 27. Hee, we just cite a esult fo finite hoizon linea quadatic systems with one point and one distibuted delay and additive noise, see Kolmanovskiǐ and Shaǐkhet 1996: Ch. 5.

21 1.2. EXAMPLES OF OPTIMAL CONTROL PROBLEMS WITH DELAY 9 We conside the time-homogeneous case. The dynamics of the contol poblem ae given by the affine-linea equation dxt = A Xtdt + A 1 Xt dt + GsXt+sds dt B utdt + σdw t, t >, whee > is the delay length, W. a d 1 -dimensional standad Wiene pocess adapted to the filtation F t t, u. a stategy, σ is a d d 1 -matix, A, A 1 ae d d-matices, G is a bounded continuous function [, ] R d d, and B is a d l-matix. The stategy u. in Equation 1.13 is any R l -valued F t -adapted squae integable pocess. Let U denote the set of all such pocesses. Let D := D[, ], R d denote the space of all R d -valued càdlàg functions on [, ]. Given ϕ D and a stategy u. U, thee is a unique up to indistinguishability d-dimensional càdlàg pocess X. = X ϕ,u. such that Equation 1.13 is satisfied and Xt = ϕt fo all t [, ]. Let T > be the deteministic time hoizon. The quadatic cost functional fo fixed initial time zeo is the function J : D U R given by T 1.14 Jϕ, u := E X T T C XT + X T t C Xt + u T tm ut dt, whee C, C ae positive semi-definite d d-matices and M is a positive definite l l-matix. The associated value function at initial time zeo is defined by V ϕ := inf u U Jϕ, u, ϕ D. Fo the contol poblem detemined by 1.13 and 1.14, a vesion of the Hamilton- Jacobi-Bellman equation 3 allows to deive a epesentation in feedback fom of the optimal stategies. Define the function u : [, T ] D R l by u t, ϕ := M 1 B P T tϕ + Qt, sϕsds, whee P, Q ae matix-valued functions [, T ] R d d and [, T ] [, ] R d d, espectively. The functions P, Q ae detemined by the following system of diffeential equations, which involves, in addition, the unknown functions R: [, T ] [, ] [, ] R d d and g : [, T ] R: 1.15 d dt P t + AT P t + P tat + Qt, + Q T t, + C = P tb M 1 B T P t, t s Qt, s + P tgt, s + A T Qt, s + Rt,, τ = P tb M 1 B T Qt, s, t s τ Rt, s, τ + G T t, sqt, τ + Q T t, sgt, τ = Q T t, sb M 1 B T Qt, τ, d dt gt + tace σ T P tσ =, t [, T ], s, τ [, ], 3 The deivation of the HJB equation in Kolmanovskiǐ and Shaǐkhet 1996: Ch. 5 is not completely igoous; see Chang et al. 26 and the efeences theein fo a moe caeful teatment. The development thee stats fom the expession fo the weak infinitesimal geneato of the segment pocess as deived in Mohammed 1984.

22 1 CHAPTER 1. INTRODUCTION with bounday conditions 1.16 P T = C, RT, s, τ =, QT, s =, gt =, P ta 1 = Qt,, A T 1 Qt, s = Rt,, s, t [, T ], s, τ [, ]. Equations 1.15 can be shown to possess a unique continuously diffeentiable solution P, Q, R, g unde bounday conditions 1.16, see Theoem in Kolmanovskiǐ and Shaǐkhet 1996: p It is also shown that u is indeed an optimal feedback contol. This means the following. Let ϕ D, and let X = X,ϕ be the unique solution to 1.17 X t = ϕ + t A X τ + A 1 X τ + + B u τ, X τ dτ + σ W t if t >, ϕt if t [, ]. GsX τ +sds dτ Recall the notation X τ fo the segment of X. at time τ. Obseve that u is Lipschitz continuous in supemum nom in its segment vaiable, whence stong existence and uniqueness of the solution X ae guaanteed. Indeed, due to the fom of u, Equation 1.17 is an affine-linea uncontolled SDDE, and X can be expessed by a vaiation-of-constants fomula. Set u t := u t, X, t. Then it holds that Jϕ, u = V ϕ, that is, u is an optimal stategy and X is the optimal pocess fo the given initial condition ϕ. In special cases, Equations 1.15 can be solved explicitly. Fo geneal linea quadatic poblems, they may seve as a stating point fo the numeical computation of optimal stategies and minimal costs A simple model of esouce allocation The following finite-hoizon deteministic poblem can be intepeted as a simplified model of optimal esouce allocation; see Betsekas 25: Ex , fo the non-delay case. Let T > be the time hoizon, let [, T be the length of the time delay, and c > a paamete. The dynamics of the model ae given by 1.18 {ẋt = c ut xt, if t >, xt = ϕt, if t [, ], whee the initial path ϕ is in C + := C[, ],, ; if =, then ϕ is just a positive eal numbe. An admissible stategy u. is any element of the set U of all Boel measuable functions [, [, 1]. The initial time will be fixed and equal to zeo. The objective is to maximise, fo each initial segment ϕ C +, the cost functional Jϕ, u := T ove u U. Clealy, this is equivalent to minimising 1.19 Jϕ, u := T 1 ut xt dt ut 1 xt dt

23 1.2. EXAMPLES OF OPTIMAL CONTROL PROBLEMS WITH DELAY 11 ove u U, since sup u U J.,., u = infu U J.,., u. A possible intepetation of the contol poblem detemined by 1.18 and 1.19 is the following cf. Betsekas, 25: Ex The state tajectoy x. = x u. descibes the poduction ate of cetain commodities e. g. wheat. Consequently, the total amount of goods poduced in any time peiod [, τ] is τ xtdt in suitable units. Duing the entie poduction peiod fom time zeo to time T the poduce has the choice between poducing fo einvestment and the poduction of stoable goods. This means that, at any time t [, T ], a potion ut [, 1] of the poduction ate is allocated to einvestment, while the emaining potion 1 ut goes into the poduction of stoable goods. The poduction ate changes in popotion to the level of einvestment. If einvestment is zeo, then the poduction ate will emain constant. In ode to justify Equation 1.18, it is instuctive to conside small time steps. Denote by yt the total amount of goods poduced up to time t, that is, yt = y + xsds, whee x. is the poduction ate. Let h > be the length of a small time step. Clealy, yt+h = yt + +h t xsds. On the othe hand, xt+h xt + c ut yt yt h, whee the paamete c > egulates the effectiveness of einvestment. Letting h tend to zeo and taking into account the initial condition, we obtain The objective is to maximise the total amount of stoed goods, that is, to maximise Jϕ, u ove all stategies u U fo each initial condition ϕ C + on the poduction ate. Equivalently, we can minimise Jϕ, u ove u U fo each ϕ C +. The paamete when positive intoduces a time delay. At time t, instead of allocating a potion ut of the cuent poduction ate xt, the poduce may allocate a potion of the past poduction ate xt. The total amount of stoed goods is measued accodingly, namely by T 1 utxt dt. We may think of as the time it takes to tansfom o sell the goods poduced. The contol poblem descibed above can be solved explicitly, and optimal stategies can be found. In the non-delay case, this is possible by elying on the Pontyagin Maximum Pinciple, see Theoem in Yong and Zhou 1999: p. 13, fo example. Pontyagin s Maximum Pinciple gives a set of necessay conditions an optimal stategy must satisfy if it exists in tems of the so-called adjoint vaiable. Unde additional assumptions, those conditions ae also sufficient fo a stategy to be optimal, cf. Theoem in Yong and Zhou 1999: p In case =, the solution of the above simple contol poblem by means of the Maximum Pinciple is given in Betsekas 25: pp Fo, we may ely on a vesion of Pontyagin s Pinciple fo deteministic systems with delay, cf. Gabasov and Kiillova 1977: p. 84. Given any initial segment ϕ C +, it can be shown that a coesponding optimal stategy satisfies { 1 if pt u 1 c t =, if pt < 1 c, t [, T ], whee p. is the solution to the adjoint teminal-value poblem given, in the case at hand,

24 12 CHAPTER 1. INTRODUCTION by 1.2 pt =, t [T, T ], ṗt = 1, t [T 1 c, T ], ṗt = c pt+, t [, T 1 c ]. Equations 1.2 descibe a deteministic backwad delay diffeential equation with teminal condition. It follows that an optimal stategy is given by 1.21 u t = { 1 if t [, T 1 c ], if t [T 1 c, T ]. Obseve that u depends on the delay length and the effectiveness paamete c >, but not on the initial condition. The minimal costs Jϕ, u, howeve, depend on ϕ C +. If =, we have an explicit solution, if >, we can integate in steps of length. The optimal stategy as given by Equation 1.21 is of bang-bang type. It consists in einvesting as much as possible befoe a citical switching time T 1 c, and then not to einvest any moe, but to poduce and stoe until the final time is eached Picing of weathe deivatives The example poblem of this subsection is based on Ankichne et al. 27, whee picing and hedging of insuance deivatives that depend on extenal physical pocesses is studied. Let X. be a continuous-time stochastic pocess one-dimensional, fo simplicity descibing some physical quantity, e. g. suface tempeatue at a given place o aveaged ove a cetain egion. Suppose X can be modelled as an SDDE of the fom 1.22 dxt = b t, X t dt + σ t, Xt dw t, t >, whee X t is the segment of length > of X. at time t, W. a standad Wiene pocess and b, σ ae appopiate functions; Equation 1.22 should possess a unique solution fo each initial condition ϕ D, whee D = C[, ], fo example. Suppose futhe that an economic agent A e. g. an insuance company intends to sell a financial deivative on the pocess X.. At matuity T >, the deivative yields fom the pespective of A an income F X T, whee F is some deteministic function D R. The income thus may depend on the evolution of X. ove the peiod [T, T ]. Notice that the length of the time delay may be atificially inceased. The question is which pice A should ask fo the deivative coesponding to F. It is assumed that A has the possibility to invest in a financial maket. In this maket, thee is a isky asset with pice pocess S. such that S. and X. ae coelated. We assume that S. is given by the modified Black and Scholes model 1.23 dst = µ t, St Stdt + β 1 StdW t + β 2 Std W t, whee W is a second standad Wiene pocess independent of the fist. The pocesses S. and X. ae coelated though β 1. The financial maket is incomplete, as the physical quantity descibed by X is not taded. The pice p of the deivative that A should ask can be detemined as the utility

25 1.2. EXAMPLES OF OPTIMAL CONTROL PROBLEMS WITH DELAY 13 indiffeence pice, povided a utility function descibing A s attitude towads isk is given. Let Ψ: R R denote such a function. We assume that Ψ is an exponential utility function. Then the pice p is detemined by the equation 1.24 sup u U E Ψ V u T + F X T p = sup E Ψ V u T, u U whee V u. is the value of A s potfolio unde investment stategy u U; see Ankichne et al. 27 fo the details. Fo an exponential utility function Ψ, the unknown p in Equation 1.24 factos out, and, on the left hand side of 1.24, we have a stochastic optimal contol poblem with delay of the type studied in Chapte Delay poblems educible to finite dimension In this subsection, we follow Baue and Riede 25; but also cf. Elsanosi et al. 2 and Lassen and Risebo 23, whee a simila appoach is taken. The value function of an optimal contol poblem with delay lives, fo fixed initial time, on the segment space associated with the system dynamics. The segment space is, apat fom the case when the delay length is equal to zeo, an infinite-dimensional space of functions, say D; fo example, D = C[, ], R d. In geneal, it is not possible to educe the value function to a finite-dimensional object, that is, it is not geneally possible to find a numbe n N and continuous functions Θ: D R n, Ψ: R n R such that V = Ψ Θ. If the contolled SDDE as well as the cost functional have a special fom and cetain additional assumptions ae fulfilled, then the optimal contol poblem with delay is educible to a contol poblem without delay, that is, the poblem is effectively finite-dimensional. Let Γ be a closed subset of Euclidean space of any dimension. Let W be a onedimensional standad Wiene pocess adapted to the filtation F t t. Denote by U the set of all F t -pogessively measuable Γ-valued pocesses. Let >, and let the dynamics of the contol poblem with delay be given by the one-dimensional contolled SDDE 1.25 dxt = µ 1 t, Xt, Y t, ut dt + µ2 Xt, Y t ξtdt + σ t, Xt, Y t, ut dw t, t >, whee u U is a stategy, ξt := wxt and Y t := eλ s wxt+sds fo some continuously diffeentiable function w : R R and a constant λ R. Hee, we only give the one-dimensional esult with initial time set to zeo; see Baue and Riede 25 fo a full account. The coefficients of Equation 1.25 ae measuable functions µ 1 : [, R R Γ R, µ 2 : R R R, σ : [, R R Γ R. Equation 1.25 descibes a system whose evolution depends not only on the cuent state X., but also on a cetain weighted aveage ove the past, namely Y., as well as a point delay, namely ξ.. Let us assume, fo example, that µ 2 is bounded and Lipschitz continuous, thee is a constant K > such that fo all t, γ Γ, x, y R, µ 1 t, x, y, γ + σt, x, y, γ K 1 + x y,

26 14 CHAPTER 1. INTRODUCTION µ 1, σ ae Lipschitz continuous in thei espective second and thid vaiable unifomly in the othe vaiables. Then, fo evey initial segment ϕ C := C[, ] and evey stategy u U, thee is a unique continuous pocess X = X ϕ,u such that Equation 1.25 is satisfied and Xt = ϕt fo all t [, ]. Let T > be the deteministic time hoizon. The cost functional of the optimal contol poblem is the function J : C U R given by T 1.26 Jϕ, u := E f t, Xt, Y t, ut dt + g XT, Y T. The associated value function V is defined by V ϕ:= inf u U Jϕ, u, ϕ C. At this point, an idea could be that V ϕ depends on its agument ϕ C only though ϕ coespondig to Xt and wϕsds coespondig to Y t. Obseve that in Equation 1.25 thee is still the point delay ξt = wxt. Also notice that the pocess Y. is of bounded vaiation. Let Ψ C 2,1 R R. By Itô s fomula, fo any solution X. and the associated pocess Y., dψ Xt, Y t = x Ψ Xt, Y t dxt x x Ψ Xt, Y t d X, X t. y Ψ Xt, Y t dy t While expessions fo dxt and d X, X t now follow fom Equation 1.25, we have 1.27 dy t = w Xt dt e λ ξtdt λy tdt, by constuction of Y. Intoduce the hypothesis that HT thee is Ψ C 2,1 R R such that fo all x, y R, x Ψx, yµ 2x, y e λ y Ψx, y =. If Hypothesis HT holds, then the tansfomed pocess ΨX, Y obeys an equation of the fom 1.28 dψ Xt, Y t = µ t, Xt, Y t, ut dt + σ t, Xt, Y t, ut dw t, whee the coefficients µ, σ can be expessed in tems of the oiginal coefficients. Notice that the point delay ξt has disappeaed. Indeed, Hypothesis HT has been chosen so that the ξt tem stemming fom Equation 1.25 and the ξt tem in Equation 1.27 cancel out. The appeaance of the point delay in Equation 1.27, on the othe hand, is inevitable in view of the fom of Y. If the coefficients µ, σ ae such that they depend on thei x- and y-vaiable only though Ψx, y, then ΨX, Y, the tansfomed pocess, obeys an odinay SDE of the fom dψ Xt, Y t = µ t, ΨXt, Y t, ut dt + σ t, ΨXt, Y t, ut dw t, whee µ, σ ae the new coefficients which can be found by hypothesis. Unde Hypothesis HT and the educibility hypothesis, the tansfomed dynamics can be witten in tems of Zt := ΨXt, Y t. If also the coefficients f, g in 1.26

27 1.3. APPROXIMATION OF CONTINUOUS-TIME CONTROL PROBLEMS 15 ae educible, that is, if the coefficients of the cost functional depend on thei x- and y-vaiable only though Ψx, y, then a finite-dimensional contol poblem without delay aises which is elated to the oiginal contol poblem though the tansfomation Ψ and the coesponding eduction of the coefficients. Notice that the educibility of the coefficients is a second hypothesis. If the Hamilton-Jacobi-Bellman equation associated with the finite-dimensional contol poblem without delay admits a classical solution and if optimal stategies exist, then the finite-dimensional and the delay poblem ae equivalent in that thei value functions ae equivalent, see Theoem 1 in Baue and Riede 25. That all hypotheses can be satisfied at once is shown in Baue and Riede 25: Sects. 4-6 by way of specific examples: a linea quadatic egulato, a model of optimal consumption, and a deteministic model fo congestion contol. 1.3 Appoximation of continuous-time contol poblems Thee ae vaious possible appoaches to appoximating continuous-time optimal contol poblems. We focus on those appoaches which yield an appoximation to the value function of the oiginal poblem. Recall that knowledge of the value function allows to choose optimal o nealy optimal stategies so that an optimal contol poblem is essentially solved once its value function is known. The methods we mention wee mostly developed fo finite-dimensional systems stochastic as well as deteministic. A basic idea is to eplace the oiginal contol poblem by a sequence of contol poblems which ae numeically solvable in such a way that the associated value functions convege to the value function of the oiginal poblem. It is often possible to eintepet a given scheme in tems of appoximating contol poblems even though the scheme itself need not be defined in these tems. A natual ansatz fo constucting a suitable sequence of contol poblems is to deive thei dynamics and cost functionals fom a discetisation of the dynamics and cost functional of the oiginal poblem. This method, known as the Makov chain method, was intoduced by H. J. Kushne and is well-established in the case of finite-dimensional stochastic and deteministic optimal contol poblems, see Kushne and Dupuis 21 and the efeences theein. The method allows to pove convegence of the appoximating value functions to the value function of the oiginal poblem unde vey boad conditions. The most impotant condition to be satisfied is that of local consistency of the discetised dynamics with the oiginal dynamics. Due to its geneal natue, the Makov chain method can also be applied to contol poblems with delay. In Chapte 2, we will study this method in detail and develop an abstact famewok fo the poof of convegence. The famewok may seve as a guide fo using the Makov chain method in the convegence analysis of appoximation schemes fo vaious classes of optimal contol poblems. In Section 2.3, the convegence analysis is caied out fo the discetisation of stochastic optimal contol poblems with delay and a andom time hoizon. We note, howeve, that while the method is well-suited fo establishing convegence of a scheme, it usually povides no infomation about the speed of convegence. The value function of a continuous-time finite-dimensional optimal contol poblem can

28 16 CHAPTER 1. INTRODUCTION often be chaacteised as the unique viscosity solution of an associated patial diffeential equation. Fo classical contol poblems, that equation is the Hamilton-Jacobi-Bellman equation HJB equation associated with the contol poblem, which is a fist ode PDE in the case of a deteministic system and a second ode PDE in the case of a stochastic system diven by a Wiene pocess. Examples show that the value function of a deteministic o degeneate stochastic contol poblem is not necessaily continuously diffeentiable e. g. Fleming and Sone, 26: II.2, whence classical solutions to the HJB equation do not always exist. 4 An appoximation to the value function of a continuous-time optimal contol poblem can be obtained by discetising the associated HJB equation. In paticula, finite diffeence schemes can be used fo the discetisation. In the case of finite-dimensional deteministic optimal contol poblems, convegence as well as ates of convegence fo such schemes wee obtained in the 198s, see, fo instance, Capuzzo Dolcetta and Ishii 1984 o Capuzzo Dolcetta and Falcone Mee convegence of a discetisation scheme fo finite-dimensional deteministic and stochastic equations without eo bounds can be checked by elying on a theoem due to Bales and Souganidis Thei esult is not limited to the analysis of HJB equations aising in contol theoy in that it applies to a wide class of equations possessing a viscosity solution. About ten yeas ago, N. V. Kylov was the fist to obtain ates of convegence fo finite diffeence schemes appoximating finite-dimensional stochastic contol poblems with contolled and possibly degeneate diffusion matix, see Kylov 1999, 2 and the efeences theein. The eo bound obtained thee in the special case of a time discetisation scheme with coefficients that ae Lipschitz continuous in space and 1 2-Hölde continuous in time is of ode h 1/6 with h the length of the time step. Notice that in Kylov 1999 the ode of convegence is given as h 1/3, whee the time step has length h 2. When the space too is discetised, the atio between time and space step is like h 2 against h o, equivalently, h vs. h, which explains why the ode of convegence is expessed in two diffeent ways. In Kylov 25, shap eo bounds ae obtained fo fully discete finite diffeence schemes in a special fom; the bounds ae of ode h 1/2 in the mesh size h of the space discetisation and of ode τ 1/4 in the length τ of the time step. Using puely analytic techniques fom the theoy of viscosity solutions, Bales and Jakobsen 25, 27 obtain eo bounds fo a boad class of finite diffeence schemes fo the appoximation of PDEs of Hamilton-Jacobi-Bellman type. In the case of a simple time discetisation scheme, the estimate fo the speed of convegence they find is of ode h 1/1 in the length h of the time step. A possible ansatz fo extending those esults to the appoximation of contol poblems with delay is to ty to deive a HJB equation fo the value function. Recall that a vesion of the Pinciple of Dynamic Pogamming still holds fo delay systems, cf. Appendix A.1. As in the finite-dimensional setting, such an HJB equation is not guaanteed to admit classical i. e. Féchet-diffeentiable solutions, and viscosity solutions have to be defined. The HJB equation can then be used as a stating point fo constucting finite diffeence 4 Genealised solutions fo the HJB equation can be shown to exist also in the case when thee ae no classical solutions, but uniqueness of genealised solutions does not always hold. Fo viscosity solutions, on the othe hand, existence and uniqueness can be guaanteed; moeove, viscosity solutions ae the ight solutions in the sense that they coincide with the value function of the undelying contol poblem.