Optimal Control. Lecture 3. Palle Andersen, Aalborg University. Opt lecture 3 p. 1/30

Transcription

1 Optimal Control Lecture 3 pa@control.aau.dk Palle Andersen, Aalborg University Opt lecture 3 p. 1/30

2 Stochastic Optimal Control In this lecture we are going to introduce disturbances which are modelled by adding a stochastic input to a state space description: x(k +1) = Φx(k)+Γu(k)+e x (k) Assumptions about noise: Expected value E{e x (k)} = 0 Variance E{e x (k)e T x (k)} = R ex (symmetric, (n n)) Covariance function E{e x (k)e T x(k +l)} = R ex δ(l) Assumptions about the initial state: Expected value E{x(0)} = x m (0) Variance matrix E{(x(0) x m (0))(x(0) x m (0)) T } = R x (0) Opt lecture 3 p. 2/30

3 Performance function Because we can not predict future values of the states we need to change performance function using expectation: N 1 I0 N = E{ (x T (k)q 1 x(k)+u T (k)q 2 u(k))+x T (N)Q N x(n)} = E{ k=0 N H(x(k),u(k))} k=0 As earlier we denote the optimal value J N 0 J N 0 = min u(0),,u(n) E{ N k=0 H(x(k),u(k))} Opt lecture 3 p. 3/30

4 Stochastic Optimal Control Performance in the interval [k;n]. I N k = E{ N H(x(i),u(i))} i=k J N k = min u(k),,u(n) E{ N H(x(i),u(i))} i=k Opt lecture 3 p. 4/30

5 Split up sum J N k (x(k)) = J N k min u(k),,u(n) IN k (x(k)) = min (H(x(k),u(k))+E{J k+1 N (x(k +1))}) u(k) H(x(k),u(k)) = x T (k)q 1 x(k)+u T (k)q 2 u(k) J N k (x(k)) = min u(k) (xt (k)q 1 x(k)+u T (k)q 2 u(k)+ E{Jk+1 N (x(k +1))}) Opt lecture 3 p. 5/30

6 Assumptions onj N k+1 Assumption in deterministic case: J N k (x(k)) = xt (k)s(k)x(k) Assumption in stochastic case J N k (x(k)) = xt (k)s(k)x(k)+w(k) the extra term w(k) is a scalar independent of x(k) We will show how S(k) and w(k) relates to S(k +1) and w(k+1). Since the assumptions holds for k = N, this can be seen as proof by induction of the validity of the assumptions on the structure of the optimal performance. Opt lecture 3 p. 6/30

7 Insert assumption and model : J N k (x(k)) = xt (k)s(k)x(k)+w(k) = min u(k) [xt (k)q 1 x(k)+u T (k)q 2 u(k) +E{x T (k +1)S(k +1)x(k +1)+w(k +1)}] = min u(k) [xt (k)q 1 x(k)+u T (k)q 2 u(k) +E{(Φx(k)+Γu(k)+e x (k)) T S(k +1)(Φx(k) +Γu(k)+e x (k))+w(k +1)}] = min u(k) [xt (k)q 1 x(k)+u T (k)q 2 u(k) +(Φx(k)+Γu(k)) T S(k +1)(Φx(k)+Γu(k)) +E{e T x(k)s(k +1)e x (k)}+w(k +1)] e x (k) is uncorrelated to x(k) og u(k). Opt lecture 3 p. 7/30

8 Stochastic Optimal Control Lemma on Expectation of a Quadratic form E{v T (k)av(k)} where v(k) has stochastic mean v m (k) and variance R E{v T (k)av(k)} = v T m (k)av m(k) +E{(v(k) v m (k)) T A(v(k) v m (k))} = v T m(k)av m (k) +tr[ae{(v(k) v m (k))(v(k) v m (k)) T }] = v T m(k)av m (k)+tr[ar v ] tr[a] is termed the trace of A and is defined as the sum of the diagonal elements of A Opt lecture 3 p. 8/30

9 Stochastic Optimal Control We can using the lemma on expectation of a quadratic form to find: J N k (x(k)) = min u(k) [xt (k)q 1 x(k)+u T (k)q 2 u(k)+ dj N k (x(k)) du(k) (Φx(k)+Γu(k)) T S(k +1)(Φx(k)+Γu(k)) +tr[s(k +1)R ex ]+w(k +1)] = 2Q 2 u(k)+2γ T S(k +1)(Φx(k)+Γu(k)) = 0 u (k) = [Q 2 +Γ T S(k +1)Γ] 1 Γ T S(k +1)Φx(k) Opt lecture 3 p. 9/30

10 Obtain the performance Jk N (x(k)) = xt (k)s(k)x(k)+w(k) = x T (k)[q 1 +L T (k)q 2 L(k) + (Φ ΓL(k)) T S(k +1)(Φ ΓL(k))]x(k) + tr[s(k +1)R ex ]+w(k +1)] From this can be seen: S(k) = Q 1 +L T (k)q 2 L(k)+(Φ ΓL(k)) T S(k +1)(Φ ΓL(k)) w(k) = tr[s(k +1)R ex ]+w(k +1) with S(N) = Q N and w(n) = 0. Opt lecture 3 p. 10/30

11 LQ: stochastic, DT systems with SI x(k +1) = Φx(k)+Γu(k)+e x (k) with E{e x (k)} = 0 E{e x (k)e T x(k)} = R ex E{x(0)} = x m (0) E{(x(0) x m (0))(x(0) x m (0)) T } = R x and a quadratic performance index N 1 I = E{ (x T (k)q 1 x(k)+u T (k)q 2 u(k))+x T (N)Q N x(n)} k=0 where Q 1 and Q N are positive definite and Q 2 is positive semi definite, Opt lecture 3 p. 11/30

12 LQ: stochastic, DT systems with SI The optimal input sequence will be determined by: u(k) = L(k)x(k), with L(k) = [Q 2 +Γ T S(k +1)Γ] 1 Γ T S(k +1)Φ, with S(k) = Q 1 +L T (k)q 2 L(k) +(Φ ΓL(k)) T S(k +1)(Φ ΓL(k)) = Q 1 +Φ T S(k +1)(Φ ΓL(k)), with S(N) = Q N Opt lecture 3 p. 12/30

13 Obtain the performance index: mini N 0 = x T m(0)s(0)x m (0) } {{ } as deterministic +tr[s(0)r x (0)]+ N 1 k=0 tr[s(k +1)R ex ] } {{ } due to state noise Opt lecture 3 p. 13/30

14 Stochastic LQ, incomplete info Now we leave the assumption of full state information and assume a stochastic output equation combined with the state equation x(k +1) = Φx(k)+Γu(k)+e x (k) y(k) = Hx(k)+e y (k) Stochastic properties E{e x (k)} = 0 E{e y (k)} = 0 E{e x (k)e T x(k)} = R ex E{e y (k)e T y(k)} = R ey E{e x (k)e T x(k +l)} = R ex δ(l) E{e y (k)e T y(k +l)} = R ey δ(l) Opt lecture 3 p. 14/30

15 Stochastic LQ, incomplete info Performance function as in the case with complete state information N 1 I = E{ (x T (k)q 1 x(k)+u T (k)q 2 u(k))+x T (N)Q N x(n)} k=0 Opt lecture 3 p. 15/30

16 Stochastic LQ Control, Separation The optimal control law for a linear system with stochastic disturbances and measurements contaminated by stochastic noise can be obtained as a combination of the solutions of two subproblems: 1. An estimator giving the optimal estimate of the system state vector from observations of the system input and output. This is also called an Observer. 2. An optimal feedback law from the estimated states. This feedback law is the same as if complete state information was available. Next we will derive an prediction observer assuming the signals up to k 1 are available to calculate u(k). Similar results can also be obtained with a current observer assuming signals at k are available. Opt lecture 3 p. 16/30

17 Stochastic Control, observation error Observer equation for prediction observer: ˆx(k +1) = Φˆx(k)+Γu(k)+K(k)[y(k) Hˆx(k)] x(k +1) = Φx(k)+Γu(k)+e x (k) Observation error x(k) = x(k) ˆx(k) x(k +1) = (Φ K(k)H) x(k)+e x (k) K(k)e y (k) Mean value of observation error E{ x(k +1)} = (Φ K(k)H)E{ x(k)} The mean value will tend to zero when eigenvalues of (Φ K(k)H) are inside unit circle Opt lecture 3 p. 17/30

18 Variance of estimation error The observer is designed to minimize the variance of the observation error P(k +1) = E{ x(k +1) x(k +1) T } = (Φ K(k)H)P(k)(Φ K(k)H) T + R ex +K(k)R ey K(k) T = K(HPH T +R ey )K T KHPΦ T ΦPH T K T +ΦPΦ T +R ex Minimize by completing the squares: KMK T KN T NK T = (K NM 1 )M(K NM 1 ) T NM 1 N T has minimum where first term is zero: K = NM 1 Opt lecture 3 p. 18/30

19 Observer Riccati equations Substituting for M and N we obtain the minimizing observer gain K(k) = ΦP(k)H T [R ey +HP(k)H T ] 1 P(k +1) = R ex +K(k)R ey K T (k) +(Φ K(k)H)P(k)(Φ K(k)H) T = R ex +(Φ K(k)H)P(k)Φ T, and P(0) = R x (0) Opt lecture 3 p. 19/30

20 Closed loop diagram System e x(k) e y (k) u(k) Γ + z -1 x(k) H y(k) Φ Observer K Γ + z -1 x(k) ^ H Φ Controller -L Opt lecture 3 p. 20/30

21 Closed loop equations A steady state solution of the observer Riccati equation may be found by iterating the time varying equations forward in time. This solution is also the one found by Matlab commands lqe, dlqe or kalman). The closed loop state equations are (steady state L and K) x(k +1) = Φx(k) ΓLˆx(k)+e x (k) ˆx(k +1) = Φˆx(k) ΓLˆx(k)+K(Hx(k)+e y (k) Hˆx(k)) Opt lecture 3 p. 21/30

22 Separation: closed loop poles Closed loop poles are most easily recognized from equations in x = x ˆx and x x(k +1) = Φx(k) ΓL(x(k) x(k))+e x (k) x(k +1) = (Φ KH) x(k)+e x (k) Ke y (k) or [ x(k +1) x(k +1) ] = [ Φ ΓL ΓL 0 Φ KH ][ x(k) x(k) ] + [ e x (k) e x (k) Ke y (k) ] Opt lecture 3 p. 22/30

23 Separation: closed loop poles The poles are the eigenvalues of the state transition matrix or the values of z where ([ ]) (zi Φ+ΓL) ΓL det = 0 0 (zi Φ+KH) or det(zi Φ+ΓL)det(zI Φ+KH) = 0 Opt lecture 3 p. 23/30

24 Duality, Controller & Observer Notice the duality between controllability and observability x(k +1) = Φx(k)+Γu(k) x(k +2) = Φ 2 x(k)+φγu(k)+γu(k +1) x(k +n) = Φ n x(k)+[γ,φγ,...,φ n 1 Γ] u(k +n 1) u(k +n 2)... u(k) The plant is controllable if you can reach the full state space: controllability matrix C has full rank (n) C(Φ,Γ) = [Γ,ΦΓ,...,Φ n 1 Γ] Opt lecture 3 p. 24/30

25 Duality, Controller & Observer In a similar way observability shows if a state vector can be calculated using n subsequent observations of output. Consider the case with no input and no noise y(k n+1) y(k n+2)... y(k) = Hx(k n+1) HΦx(k n+1)... HΦ n 1 x(k n+1) = H HΦ... HΦ n 1 x(k n+1) The plant is observable if the observability matrix O has full rank (n) H HΦ O(Φ,H) =... = [HT,Φ T H T,,(Φ T ) n 1 H] T HΦ n 1 Opt lecture 3 p. 25/30

26 Duality, Controller & Observer Note the duality between controllability and observability: If you construct a plant with system matrix Φ T and input matrix H T this would have a controllabity matrix equal to the transpose of the original obervability matrix. We might write C(Φ T,H T ) = O(Φ,H) T Opt lecture 3 p. 26/30

27 Duality, Controller & Observer Duality between Riccati eqs. for controller and observer Controller : Observer : L = [Q 2 +Γ T SΓ] 1 Γ T SΦ S = Q 1 +L T Q 2 L+(Φ ΓL) T S(Φ ΓL) S(N) = Q N K = ΦPH T [R ey +HPH T ] 1 P = R ex +KR ey K T +(Φ KH)P(Φ KH) T P(0) = R x (0) Immediate you see the following duality: Controller Q 2 Γ T S Φ T Q 1 L T Q N Observer R ey H P Φ R ex K R x (0) Opt lecture 3 p. 27/30

28 How to tune the observer The variance matrices R ex and R ey determine the observer gain K the closed loop poles related to the observer But how can we find the variance of these stochastic quantities. Recognize that R ex and R ey are part of a specification of the performance goal R ex and R ey specifies the disturbance e x and the measurement noise e y Q 1 and Q 2 specify the outputs to weight in performance Separation splits it to a controller problem and an observer problem Opt lecture 3 p. 28/30

29 How to tune the observer Either controller poles or observer poles may determine the bandwidth R ex and R ey specify the classic tradeoff between noise noise immunity and speed of response Large R ex (or small R ey ) results in a fast observer Large R ey (or small R ex ) results in a slow noise immune observer Q 1 and Q 2 specify the classic tradeoff between size of control signal and speed of response Large Q 1 (or small Q 2 ) results in fast controller poles Large Q 2 (or small Q 1 ) results in a slow controller with small control signals Opt lecture 3 p. 29/30

30 How to tune the observer Disturbances can be easy to observe but difficult to reject (close to measurements at output). Fast observer and slow controller. Controller limits the response dificult to observe but easy to reject (close to control inputs). Fast controller and slow observer. Observer limits the response In pole placement design observer poles are often chosen 4 times faster than controller poles. A choice of the controller poles 4 times faster than observer poles can also be well suited Opt lecture 3 p. 30/30