Stability of Feedback Equilibrium Solutions for Noncooperative Differential Games

Transcription

1 Stability of Feedback Equilibrium Solutions for Noncooperative Differential Games Alberto Bressan Department of Mathematics, Penn State University Alberto Bressan (Penn State) Noncooperative Games 1 / 33

2 Differential games: the PDE approach The search for equilibrium solutions to noncooperative differential games in feedback form leads to a nonlinear system of Hamilton-Jacobi PDEs for the values functions main focus: existence and stability of the solutions to these PDEs Alberto Bressan (Penn State) Noncooperative Games / 33

3 Any system can be locally approximated by a linear one: ẋ = f (x, u 1, u ), ẋ = Ax + B 1 u 1 + B u Any cost functional can be locally approximated by a quadratic one. Main issue: Assume that an equilibrium solution is found, to an approximating game with linear dynamics and quadratic costs. Does the original nonlinear game also have an equilibrium solution, close to the linear-quadratic one? (i) finite time horizon (ii) infinite time horizon Two cases Alberto Bressan (Penn State) Noncooperative Games 3 / 33

4 An example - approximating a Cauchy problem for a PDE u t = u xx u(0, x) = ϕ(x) Approximate the initial data: ϕ(x) ax + bx + c Conclude: u(t, x) ax + bx + c + at { CORRECT for t 0 WRONG for t < 0 Alberto Bressan (Penn State) Noncooperative Games 4 / 33

5 Noncooperative differential games with finite horizon Finite Horizon Games x R n u 1, u state of the system controls implemented by the players Dynamics: ẋ(t) = f (x, u 1, u ) x(τ) = y Goal of i-th player: maximize: J i (τ, y, u 1, u ). ( ) T = ψ i x(t ) τ L i ( x(t), u1 (t), u (t) ) dt = [terminal payoff] - [integral of a running cost] Alberto Bressan (Penn State) Noncooperative Games 5 / 33

6 Seek: Nash equilibrium solutions in feedback form u i = u i (t, x) Given the strategy u = u (t, x) adopted by the second player, for every initial data (τ, y), the assignment u 1 = u1 (t, x) is a feedback solution to optimal control problem for the first player : subject to max u 1( ) ( ψ 1 (x(t )) T ẋ = f (x, u 1, u (t, x)), τ L 1 (x, u 1, u (t, x)) dt x(τ) = y ) Similarly u = u (t, x) should provide a solution to the optimal control problem for the second player, given that u 1 = u1 (t, x) Alberto Bressan (Penn State) Noncooperative Games 6 / 33

7 The system of PDEs for the value functions V i (τ, y) = value function for the i-th player (= expected payoff, if game starts at τ, y) Assume: f (x, u 1, u ) = f 1 (x, u 1 ) + f (x, u ) L i (x, u 1, u ) = L i1 (x, u 1 ) + L i (x, u ) Optimal feedback controls: u i = u i (t, x, V i ) = argmax ω { } V i (t, x) f i (x, ω) L ii (x, ω) The value functions satisfy a system of PDE s t V i + V i f (x, u 1, u ) = L i (x, u 1, u ) i = 1, with terminal condition: V i (T, x) = ψ i (x) Alberto Bressan (Penn State) Noncooperative Games 7 / 33

8 Systems of Hamilton-Jacobi PDEs Finite horizon game t V 1 = H (1) (x, V 1, V ), t V = H () (x, V 1, V ), V 1 (T, x) = V (T, x) = ψ 1 (x) ψ (x) (backward Cauchy problem, with terminal conditions) Alberto Bressan (Penn State) Noncooperative Games 8 / 33

9 Test well-posedness: by locally linearizing of the equations t V i = H (i) (x, V 1, V ) i = 1, perturbed solution: V (ε) i = V i + ε Z i + o(ε) Differentiating H (i) (x, p 1, p ), obtain a linear equation satisfied by Z i t Z i = H(i) p 1 (x, V 1, V ) Z 1 + H(i) p (x, V 1, V ) Z Alberto Bressan (Penn State) Noncooperative Games 9 / 33

10 Freezing the coefficients at a point ( x, V 1 ( x), V ( x)), one obtains a linear system with constant coefficients ( Z1 Z Each A j is a matrix ) t + n j=1 ( ) Z1 A j = 0 (1) Z x j For a given vector ξ = (ξ 1,..., ξ n ) R n, consider the matrix A(ξ). = j ξ j A j Definition 1. The system (1) is hyperbolic if there exists a constant C such that sup ξ R n exp ia(ξ) C Alberto Bressan (Penn State) Noncooperative Games 10 / 33

11 Computing solutions in terms of Fourier transform, which is an isometry on L, the above definition is motivated by Theorem 1. The system (1) is hyperbolic if and only if the corresponding Cauchy problem is well posed in L (R n ). Z(t) L = Ẑ(t) L sup ξ R n exp( ia(ξ)) Ẑ(0) L = sup ξ R n exp ia(ξ) Z(0) L Lemma 1 (necessary condition). If the system (1) is hyperbolic, then for every ξ R m the matrix A(ξ) has a basis of eigenvectors r 1,..., r n, with real eigenvalues λ 1,..., λ n (not necessarily distinct). Lemma (sufficient condition). Assume that, for ξ = 1, the matrices A(ξ) can be diagonalized in terms of a real, invertible matrix R(ξ) continuously depending on ξ. Then the system (1) is hyperbolic. Alberto Bressan (Penn State) Noncooperative Games 11 / 33

12 A class of differential games dynamics: ẋ = f 1 (x, u 1 ) + f (x, u ) T ( ) payoffs: J i = ψ i (x(t )) L i1 (x, u 1 ) + L i (x, u ) dt, i = 1, 0 Value functions satisfy t V 1 + V 1 f (x, u 1, u ) = L 1(x, u 1, u ) t V + V f (x, u 1, u ) = L (x, u 1, u ) { } u i = u i (x, V i) = argmax V i f i (x, ω) L ii (x, ω), i = 1, ω V 1 (T, x) = ψ 1 (x), V (T, x) = ψ (x) Alberto Bressan (Penn State) Noncooperative Games 1 / 33

13 f = (f 1,..., f n ), V i = p i = (p i1,... p in ) Evolution of a perturbation: n n Z i,t + f k Z i,xk + k=1 k=1 j=1 ( V i f L ) ( i u ) j Z 1,xk + u j Z,xk u j u j p 1k p k = 0 Maximality conditions = V 1 f u 1 L 1 u 1 = 0, V f u L u = 0 Alberto Bressan (Penn State) Noncooperative Games 13 / 33

14 Evolution of a first order perturbation ( Z1,t Z,t ) + n k=1 A k ( Z1,xk Z,xk ) = ( 0 0 ) where the matrices A k are given by A k = ( f k V 1 f L ) 1 u u u p k ( V f L ) u 1 u 1 u 1 p 1k f k Alberto Bressan (Penn State) Noncooperative Games 14 / 33

15 A(ξ) = n k=1 ( f k ξ k V 1 f L ) 1 u ξ k u u p k ( V f L ) u 1 ξ k u 1 u 1 p 1k f k ξ k HYPERBOLICITY = A(ξ) has real eigenvalues for every ξ v = (v 1,..., v n ), v k. = ( V 1 f u L 1 u w = (w 1,..., w n ), w k. = ( V f u 1 L u 1 ) u p k ) u 1 p 1k HYPERBOLICITY = (v ξ)(w ξ) 0 for all ξ R n Alberto Bressan (Penn State) Noncooperative Games 15 / 33

16 HYPERBOLICITY = (v ξ)(w ξ) 0 for all ξ R n TRUE if v, w are linearly dependent, with same orientation. FALSE if v, w are linearly independent. ξ v w ξ w v Alberto Bressan (Penn State) Noncooperative Games 16 / 33

17 In one space dimension, the Cauchy Problem can be well posed for a large set of data. In several space dimensions, generically the system is hyperbolic, and the Cauchy Problem is ill posed A.B., W.Shen, Small BV solutions of hyperbolic non-cooperative differential games, SIAM J. Control Optim. 43 (004), A.B., W.Shen, Semi-cooperative strategies for differential games, Intern. J. Game Theory 3 (004), A.B., Noncooperative differential games. Milan J. Math., 79 (011), Alberto Bressan (Penn State) Noncooperative Games 17 / 33

18 Differential games in infinite time horizon Dynamics: ẋ = f (x, u 1, u ), x(0) = x 0 u 1, u controls implemented by the players Goal of i-th player: +. maximize: J i = e γt ( Ψ i x(t), u1 (t), u (t) ) dt 0 (running payoff, exponentially discounted in time) Alberto Bressan (Penn State) Noncooperative Games 18 / 33

19 A special case Dynamics: ẋ = f (x) + g 1 (x)u 1 + g (x)u Player i seeks to minimize: J i = 0 ( e γt φ i (x(t)) + u i (t) ) dt Alberto Bressan (Penn State) Noncooperative Games 19 / 33

20 A system of PDEs for the value functions The value functions V 1, V for the two players satisfy the system of H-J equations γv 1 = (f V 1 ) 1 (g 1 V 1 ) (g V 1 )(g V ) + φ 1 γv = (f V ) 1 (g V ) (g 1 V 1 )(g 1 V ) + φ Optimal feedback controls: u i (x) = V i(x) g i (x) i = 1, nonlinear, implicit! Alberto Bressan (Penn State) Noncooperative Games 0 / 33

21 Linear - Quadratic games Assume that the dynamics is linear: ẋ = (Ax + b 0 ) + b 1 u 1 + b u, x(0) = y and the cost functions are quadratic: J i = + 0 e γt( ) a i x + x T P i x + u i dt Then the system of PDEs has a special solution of the form V i (x) = k i + β i x + x T Γ i x i = 1, ( ) optimal controls: u i (x) = (β i + x T Γ i ) b i To find this solution, it suffices to determine the coefficients k i, β i, Γ i by solving a system of algebraic equations Alberto Bressan (Penn State) Noncooperative Games 1 / 33

22 Validity of linear-quadratic approximations? Assume the dynamics is almost linear ẋ = f 0 (x)+g 1 (x)u 1 +g (x)u (Ax +b 0 )+b 1 u 1 +b u, x(0) = y and the cost functions are almost quadratic J i = + 0 e γt( φ i (x) + u ) i dt + 0 e γt( a i x + x T P i x + u ) i dt Is it true that the nonlinear game has a feedback solution close to the linear-quadratic game? Alberto Bressan (Penn State) Noncooperative Games / 33

23 One-dimensional, linear-quadratic games in infinite time horizon ẋ = (a 0 x + b 0 ) + b 1 u 1 + b u The ODE for the derivatives of the value functions ξ i = V i takes the form A 11 A 1 A 1 A ξ 1 ξ = ψ 1 (x, ξ 1, ξ ), A ij = A ij (x, ξ 1, ξ ) ψ (x, ξ 1, ξ ) The map (x, ξ 1, ξ ) det A(x, ξ 1, ξ ) is a homogeneous quadratic polynomial An affine solution exists: ξ 1 (x) = k 1x + β 1, ξ (x) = k x + β The map x det A(x, ξ1 (x), ξ (x)) is a quadratic polynomial Alberto Bressan (Penn State) Noncooperative Games 3 / 33

24 Stability w.r.t. perturbations (A.B. - Khai Nguyen, 016) on a bounded interval Ω = [a, b] (positively invariant for the feedback dynamics) on the whole real line Γ _ ξ(x) ξ (x) ξ (x) x x x Γ = { } (x, ξ 1, ξ ) ; det A(x, ξ 1, ξ ) 0 Alberto Bressan (Penn State) Noncooperative Games 4 / 33

25 Stability over a bounded interval Easy case: det A(x, ξ1 (x), ξ (x)) 0 for all x R. ξ 1 ξ ψ 1 (x, ξ 1, ξ ) = A 1 (x, ξ 1, ξ ) ψ (x, ξ 1, ξ ) The linear-quadratic game has a -parameter family of Nash equilibrium solutions in feedback form. One is affine, the other are nonlinear. All of the above solutions are stable w.r.t. small nonlinear perturbations of the dynamics and the cost functions. ξ ξ * (x) ξ 1 0 x Alberto Bressan (Penn State) Noncooperative Games 5 / 33

26 Case : det A vanishes at two points x 1 < x A 11 A 1 A 1 A ξ 1 ξ = ψ 1(x, ξ 1, ξ ) ψ (x, ξ 1, ξ ) Equivalent system: { A11 dξ 1 + A 1 dξ ψ 1 dx = 0 A 1 dξ 1 + A dξ ψ dx = 0 Setting: v. = ψ 1 A 11, w =. A 1 ψ A 1, A We seek continuously differentiable functions x (ξ 1(x), ξ (x)) whose graph is obtained by concatenating trajectories of the system ẋ ξ 1 = v w = ξ A 11A A 1A 1 A ψ 1 1ψ A 11ψ A 1ψ 1. Alberto Bressan (Penn State) Noncooperative Games 6 / 33

27 ( ) A11 A det 1 = 0 on Σ A 1 A 1 Σ Under generic conditions on the coefficients of the linear-quadratic problem, there exists two curves γ 1 Σ 1, γ Σ such that v w = 0 on γ 1 and on γ v w is vertical on Σ 1 \ γ 1 and on Σ \ γ Σ γ ξ (x) Σ 1 γ 1 ξ 1 x ξ Alberto Bressan (Penn State) Noncooperative Games 7 / 33

28 Three generic cases unique solution one parameter family of solutions γ 1 γ γ 1 γ two parameter family of solutions γ 1 γ Alberto Bressan (Penn State) Noncooperative Games 8 / 33

29 The saddle-saddle case Σ Σ 1 γ 1 ξ M 1 M γ P* ξ * (x) P 1 * ξ 1 _ x 0 1 _ x x Under generic assumptions, a unique solution exists Alberto Bressan (Penn State) Noncooperative Games 9 / 33

30 Stability under perturbations, on the entire real line (A.B., K.Nguyen, 016) ẋ = a 0 x + f 0 (x) + (b 1 + h 1 (x))u 1 + (b + h (x))u + ) J i = e (R γt i x + S i x + η i (x) + u i dt 0 Under generic assumptions on the coefficients a 0, b 1, b,..., for any ε > 0 there exists δ > 0 such that the following holds. If the perturbations satisfy f 0 C + η 1 C + η C + h 1 C 1 + h C 1 δ, then the perturbed equations for ξ 1 = V 1, ξ = V admit a solution such that ξ 1 (x) ξ 1 (x) + ξ (x) ξ (x) ε(1 + x ) for all x R Alberto Bressan (Penn State) Noncooperative Games 30 / 33

31 Future work: x R V i = V i (x 1, x ) γv 1 = (f V 1 ) + 1 (g 1 V 1 ) + (g V 1 )(g V ) + φ 1 γv = (f V ) + 1 (g V ) + (g 1 V 1 )(g 1 V ) + φ Linearize around the affine solution of a L-Q game Determine if this linearized PDE is elliptic, hyperbolic, or mixed type Construct solutions to the perturbed nonlinear PDE x hyperbolic elliptic Alberto Bressan (Penn State) Noncooperative Games 31 / 33 x 1

32 1956 # * * * * * * # # # # * # Happy Birthday Jean Michel!! Alberto Bressan (Penn State) Noncooperative Games 3 / 33