Towards Dual MPC. Tor Aksel N. Heirung B. Erik Ydstie Bjarne Foss

Transcription

1 4th IFAC Nonlinear Model Predictive Control Conference International Federation of Automatic Control Towards Dual MPC Tor Aksel N. Heirung B. Erik Ydstie Bjarne Foss Department of Engineering Cybernetics, Norwegian University of Science and Technology, N-7491 Trondheim, Norway ( Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA. Abstract: We present a novel approach to performing closed-loop system identification with minimal deterioration of output regulation. The adaptive control algorithm excites the system to improve model identification but does not disturb the plant when parameter estimates are good, so that a balance between output regulation and excitation for identification is found without requiring persistent excitation. The formulation of the problem is such that standard state-of-the-art optimization codes can be used in the controller rather than techniques like stochastic dynamic programming, which may lead to a more complicated control design. The approach is based on model predictive control (MPC) and can be implemented with only minor modifications to an existing MPC algorithm. The method is illustrated on an example system where the input gain is unknown, and is shown to intelligently excite the system only when the parameter estimates are uncertain. We also show how a slight increase in system excitation by the controller can greatly enhance parameter identification with minimal effect on the system output. Keywords: MPC, dual control, adaptive control, optimal control, nonlinear programming, parameter estimation, system identification. 1. INTRODUCTION Updating and maintaining models of systems that change over time is important in the process industries but is often hampered by recorded process data that is insufficiently informative for system identification or by the difficulty of finding the informative portion of a large data set [Peretzki et al., 2012]. Furthermore, performing experiments to generate information-rich data may be impractical due to factors such as the time consuming nature of the task, the expertise needed, and expensive operational downtime. The concept of dual control, which addresses online identification, was introduced by Feldbaum in the 1960s [Feldbaum, 1961] and was defined as a control signal whose effects are of a twofold character: directing a system to the desired states, and studying or learning the characteristics of the system. Feldbaum also discussed how the two aspects of the control action conflict and are related. The main message in the theory of dual control is that when there are unknown parameters in the system, an optimal controller must find the best combination of control and excitation. Filatov and Unbehauen [2000] survey many of the most notable results in dual control. Among recent contributions are Lee and Lee [2009] who approach the dual control problem using approximate dynamic programming, and Bayard and Schumitzky [2010] who base their approach on particle filtering and forward dynamic programming. This work was supported by the Center for Integrated Operations in the Petroleum Industry, Trondheim, Norway. One of the most attractive and important features of model predictive control (MPC) is the ability to handle constraints in the open-loop optimization procedure [Mayne et al., 2000]. These constraints can arise from physical limitations in actuators or safety limits, for example, as well as desired path constraints on states or other forms of behavior. The algorithm introduced here takes advantage of the ease of incorporating constraints in the control formulation. MPC is becoming common in the process industries, with more than 4600 MPC applications reported in 1999, the year of most recent count [Qin and Badgwell, 2003]. This widespread use motivates investigation of a solution to the above problems within an MPC framework. We present an MPC-based method for generating control signals that excite the process sufficiently for system identification while maintaining good control. Other researchers have investigated the use of MPC with the goal of actively improving the estimates. A method termed Model Predictive Control and Identification (MCPI) was introduced by Genceli and Nikolaou [1996] and further extended in a number of papers by Nikolaou and coworkers; one notable publication is by Shouche et al. [2002]. The main idea of the MCPI approach is to provide informative data for system identification by adding a constraint to the MPC formulation that ensures persistent excitation. Marafioti [2010] takes a similar approach in developing a persistently exciting MPC. A related problem was investigated by Yan and Bitmead [2002], who incorporated state estimates and their covariances into MPC and presented an analysis of /12/$ IFAC / NL

2 the interaction between model predictive control and state estimation. Our proposed approach leads to a controller that intelligently excites the process when useful, without requiring persistent excitation. The remainder of this article is organized as follows: The control problem is formulated in Section 2, a discussion of the online optimization problem follows in Section 3, the control algorithm is applied to an example system in Section 4, and Section 5 concludes the paper and contains possibilities for future work. 2. PROBLEM FORMULATION The advantage of an MPC with persistent excitation is that parameter estimates converge exponentially when a recursive weighted least-squares algorithm is used [Johnstone et al., 1982]. However, requiring persistent excitation at all times may lead to unnecessary deterioration of the output regulation, unnecessary use of input, or increased wear and tear caused by frequent adjustments (of a valve, for instance). Furthermore, good control is in many cases possible with small parameter estimation errors; unless one has a particular interest in the parameter values, the price of extra excitation may be unreasonably high if the parameter estimates are fairly accurate. The problem of exciting the system for good (not perfect) parameter estimates while keeping deterioration of output regulation small is now formulated. In this paper we consider linear, time-invariant, single input, single output systems in discrete time. This class of systems can be formulated as autoregressive processes with exogenous input (ARX): y(t) + a 1 y(t 1) + + a n y(t n a ) = b 0 u(t 1) + + b n 1 u(t n b ) + v(t) (1) where t is discrete time, y(t) and u(t) are the system output and input respectively at time t, a 1,..., a n and b 0,..., b n 1 are system parameters, and v(t) is a white noise sequence with variance R 2. A more compact system representation is y(t) = ϕ (t 1)θ + v(t) (2) where θ = [b 0,..., b n 1, a 1,..., a n ] is a vector containing all system parameters, and ϕ(t 1) = [ u(t 1),..., u(t n b ), y(t 1),..., y(t n a ) ] (3) is a regression vector containing past inputs and outputs. The system (1) can also be written A(q 1 )y(t) = B(q 1 )u(t) + v(t) (4) where A and B are polynomials in the backwards shift operator q 1, and have no common factors. We control the system (2) with an adaptive certainty equivalence MPC-like controller. The unknown constant parameters in θ are estimated online by a recursive algorithm. The parameter estimates at time t, ˆθ(t), are then used in an optimization problem whose solution is an open-loop optimal input sequence u (t),..., u (t + N), where N is the length of the receding horizon. In contrast to a normal certainty equivalence MPC scheme, our proposed algorithm produces an input sequence that not only attempts to control the output, but also ensures that process data is sufficiently informative for good parameter estimates. We use a recursive weighted least-squares algorithm [Ljung, 1999] for parameter estimation. The algorithm equations are given by ˆθ(t) = ˆθ(t 1) + K(t) ( y(t) ϕ (t 1)ˆθ(t) ) (5a) K(t) = P (t 1)ϕ(t 1) ( λ + ϕ (t 1)P (t 1)ϕ(t 1) ) 1 (5b) P (t) = ( I K(t)ϕ (t 1) ) P (t 1)/λ (5c) where ˆθ(t) is a vector of parameter estimates, K(t) is the injection gain, λ is the forgetting factor, and P (t) is a matrix of parameter estimate covariances or a similar measure of estimate quality. The algorithm equations (5) are evaluated at every time step in the simulation and provide the parameter estimate ˆθ(t) used by the optimization problem for predicting the system response and find the optimal input sequence. Note that in addition to feedback from the output y(t), the MPC uses the estimate ˆθ(t), the covariance measure P (t), and past inputs and outputs in calculating the optimal inputs. 3. THE ONLINE OPTIMIZATION PROBLEM Based on the problem formulation above an online optimization problem will be formulated in this section. 3.1 Objective Function In formulating the cost function to be minimized, we consider the following possible objectives: (1) Quick convergence toward good parameter estimates. (2) Small variance of parameter estimates. (3) Small rate of change in input. (4) Small input magnitude. (5) Small output magnitude. There is generally a conflict between small signals and reliable estimation of parameters. Furthermore, there is a conflict between good regulation and good estimation, as discussed by Feldbaum [1961]. However, a weighted combination of the above objectives can form a cost function in an optimization problem. We formulate this type of cost function as t+n k=t+1 { w 1 trace P (k) + w 2 ( u(k)) 2 } + w 3 u 2 (k) + w 4 y 2 (k) where P (k) is the matrix measuring the quality of the parameter estimates defined by (5), N is the length of the receding time (or prediction) horizon of the optimization problem, and u(k) = u(k) u(k 1). Note that k is discrete time in the optimization horizon of the problem solved at time t, with the receding horizon beginning at t + 1 and ending at t + N. An objective function similar to (6) was investigated by Wittenmark and Elevitch [1985] and minimized using a different technique than the one employed here. Note that the choice of trace P (k) in the (6) 503

3 objective function, as opposed to other functions of P (k), corresponds to the A-optimal performance measure in the theory of optimal input design [Mehra, 1974]. The objective function (6) is fairly standard in an MPC framework (see, for instance, Qin and Badgwell [2003]), except for the term involving trace P (k). If we set w 1 = 0 the resulting controller is in fact a classic adaptive certainty equivalence MPC. (The adaptation is achieved through model updates using the estimates from (5).) In the following, we will think of P (k) as a matrix of parameter estimate covariances, which means that trace P (k) is the sum of the variances. Explicitly minimizing the sum of the parameter estimate variances requires that the optimization problem have a way of calculating or estimating these variances. 3.2 Constraints The main contribution in our proposed algorithm is using a slightly modified version of the recursive least-squares algorithm (5) as a set of equality constraints in the optimization problem. This gives the optimization solver a measure of how the input sequence affects parameter estimation through a prediction of future covariances P (k). Before discussing the necessary modifications to (5), we provide a short discussion on how we incorporate the certainty equivalence principle in the MPC. Since we have unknown parameters θ in our model, the future outputs y(t),..., y(t + N) are also unknown. This means that the process equation (2) cannot be used as a constraint in the optimization problem as in a normal MPC. Instead, we use a predictor ŷ(k) that at time t uses the most recent parameter estimate to predict future process outputs: ŷ(k) = ˆϕ (k 1)ˆθ(t), k {t + 1,..., t + N} (7) where the regressor ˆϕ (k 1) now contains optimization variables, ˆϕ(k 1) = [ u(k 1),..., u(k n b ), ŷ(k 1),..., ŷ(k n a ) ] (8) for k {t + 1,..., t + N}, cf. (3). For k = t + 1, we set ˆϕ(k 1) = ϕ(t), which contains past inputs and outputs and is passed from the simulator to the optimization problem. Note that in (7), ˆθ(t) is the most recent parameter estimate produced by the estimation algorithm (5) during simulation at time t. The predictor (7) is added as an equality constraint to the optimization problem. Further, note that ˆθ(t) is a constant in the optimization problem solved at time t; this means that (7) is a linear equality constraint. We also add upper and lower bounds on both inputs and outputs as inequality constraints ( box constraints ) in the optimization problem. The bounds on the inputs can simply be formulated u min u(k) u max, k {t + 1,..., t + N} (9) where the minimum and maximum values can be chosen based on either hardware limitations or a design preference. Bounds on the output y(t) cannot be directly included in the optimization problem due to the unknown parameters as discussed above. Instead, bounds are added on the predicted outputs: y min ŷ(k) y max, k {t + 1,..., t + N} (10) These bounds on ŷ(k) do not guarantee that the actual output y(t) would respect the bounds if the open-loop optimal input sequence u (t),..., u (t+n) resulting from a single solution were implemented on the real process. However, as the parameter estimates improve, (10) is more likely to ensure that y min y(t) y max A key step in making the MPC excite the system sufficiently for good parameter estimation is to inform the online optimization problem about the quality of the current parameter estimates and give it the means to evaluate the quality of future parameter estimates. We inform the optimization problem about estimate quality by modifying the estimation algorithm (5) and adding the resulting equation set as equality constraints in the optimization problem. The necessary modification is due to y(t + 1),..., y(t + N) not being known at the time the optimization problem is solved, as discussed above. Hence, the equations for the injection gain (5b) and the variance (5c) will in the optimization problem be based on ˆϕ in (8) instead of ϕ in (3). Further, there is no need for a constraint representing future parameter estimates, so no modified version of (5a) is necessary in the optimization problem. We can now state the full online optimization problem: t+n { w 1 trace P (k) + w 2 ( u(k)) 2 min u(k) subject to k=t+1 ŷ(k) = ˆϕ (k 1)ˆθ(t) K(k) = P (k 1) ˆϕ(k 1) } + w 3 u 2 (k) + w 4 ŷ 2 (k) ( λ + ˆϕ (k 1)P (k 1) ˆϕ(k 1) ) 1 P (k) = ( I K(k) ˆϕ (k 1) ) P (k 1)/λ u min u(k) u max y min ŷ(k) y max k = t + 1,..., t + N (11a) (11b) (11c) (11d) (11e) (11f) (11g) It is important to note that (11c) (11d), which represent the estimation algorithm, constitute a set of nonlinear equality constraints. This makes the optimization problem a nonconvex nonlinear programming (NLP) problem. The optimization problem also has a number of variables fixed; the values of ϕ(t + 1), ˆθ(t), and P (t) are all passed from the simulation to the optimization algorithm as fixed parameters. Note that ϕ(t + 1) being fixed implies that y(t),..., y(t n a ) and u(t),..., u(t n b ) are all given and fixed in the NLP problem. 4. EXAMPLE We now demonstrate the algorithm on a test problem. An integrator with unknown gain b, y(t + 1) = y(t) + bu(t) + v(t) (12) is chosen as the test precess. This system is similar to that studied by Åström and Helmersson [1986], with the exception that b here is constant instead of undergoing a random walk. 504

4 The weighted least-squares algorithm (5) is used for estimating b. At time t, the control u(t + 1) is generated by solving the NLP (11). The parameters chosen for the test problem are shown in Table 1. The simulation time is from t = 0 to t = t f. Table 1. Parameters for the example problem. b R 2 y(0) ˆb(0) P (0) λ tf N w 2 w 3 u min u max y min y max y(t) w 1 = 10, w 4 = 0.01 w 1 = 1, w 4 = 1 w 1 = 0, w 4 = Implementation The algorithm was implemented in MATLAB and GAMS [GAMS Development Corporation, 2012] with IPOPT [Cervantes et al., 2000] as the NLP solver. IPOPT is a state-of-the-art interior-point solver, which exploits the sparse structure of the NLP and is capable of solving very large problems. The system is simulated in MATLAB; GAMS is called at every iteration and returns a (locally) optimal input sequence u (t),..., u (t + N). The example was run on a standard laptop computer. Our experience indicates that incorporating the constraints (11e) (11f) into the optimization problem (and thus decreasing the size of the feasible area) decreases the solution time of (11). The improvement is in general only a few percent, except for a small number of times with significant improvement. 4.2 Results Simulation results with the parameters from Table 1 and three different combinations of weights w 1 (the weight on P (k)) and w 4 (the weight on ŷ 2 (k)) are shown in Figure 1. The four plots show, from top to bottom, the process output, the control input, the parameter estimate variance, and the estimate of the unknown parameter b. The combination w 1 = 0, w 4 = 1.0 corresponds to a certainty-equivalence MPC where no explicit excitation takes place. Figure 1 shows that this controller works acceptably well, even initially when the parameter estimate is poor. The control action is moderate and compensates well for the noise in the system. The variance P (t) is reduced to less than 1 after about 20 time steps; the parameter estimate approaches the true value b = 1 around the same time. However, the estimate does not settle in the range of 5 % error until about 40 time steps. Increasing the weight of variance minimization w 1 to 1.0 results in a moderate increase in the magnitude of y(t) for the first few time steps. After approximately 10 time steps, the output is remarkably similar that of the controller with w 1 = 0. The control sequence from this controller has a larger magnitude than that of the controller with w 1 for the first 10 time steps. After 10 time steps the inputs computed by the two controllers are very similar. The initial excitation leads to a significant and quick reduction in the variance. Note also that the controller with w 1 keeps the variance small (around 0.1) for the remainder of the simulation. The parameter estimate is greatly improved with this controller; ˆb(t) b /b is less than 7 % after 4 time steps and less than 6 % after 10 time steps. ˆb(t) P (t) u(t) Fig. 1. Process output, control input, estimate variance, and parameter estimate for the test plant (12) in Section 4 using three different controller tunings. A controller with the weighting w 1 = 10, w 4 = 0.01 produces the third simulation result. Since this weighting gives variance reduction much higher priority than output regulation, the initial excitation is large. The controller settles after about 20 time steps and produces a control signal similar to that of the two other controllers. This initial excitation leads to a rapid reduction of the variance, with P (t = 2) < The period of good output regulation between t = 20 and t = 40 causes P (t) to increase slightly (due to the use of a forgetting factor λ < 1), and the high priority on small variance causes the controller to resume excitation of the system in order to stop the growth in P (t). The excitation leads to an improvement in parameter estimation compared with the controller with w 1 = 1.0, w 4 = 1.0; the estimation error is less than 5 % after only 3 time steps. However, this is t 505

5 t=1 y2 (t))/t f t=1 u2 (t))/t f t=1 b 2 (t))/t f t=1 P (t))/t f w 1 Fig. 2. Illustration of how output regulation, control effort, parameter estimate variance, and estimate quality change with the weight on minimal estimate variance w 1 in the objective function (11a). Note that the x-axes are logarithmic except for the inclusion of w 1 = 0. a very modest improvement and comes at a large increase in excitation. Figure 2 illustrates how different performance measures are affected by the weight w 1 in the proposed control algorithm. The results are obtained with w 4 = 1 and the parameters from Table 1. 1 The first plot shows how the total magnitudes of the output and the control increase as the controller excites the system more as w 1 increases. The second plot shows how the sum of the estimate variance P (t) decreases and how the estimate ˆb(t) improves in sum as w 1 is increased. The principal insight from Figure 2 is that increasing w 1 from 0 to 1 hardly affects average output regulation. This increase in w 1, however, has a major effect on the parameter estimate and its variance, but incurs the cost of increased control action. Our test system (12) therefore has great potential for excitation and enhanced parameter estimation with very little deterioration of output regulation. The effect of adjusting the length of the receding horizon N in (11) is shown in Figure 3. The first two plots show that the myopic controller with horizon length 1 (N = 2) hardly excites the system and hence achieves comparatively poor parameter estimation. Increasing the horizon length improves parameter estimation through more excitation while the output variance hardly increases. The third plot shows how a longer horizon in (11) increases the time GAMS/IPOPT spends solving the optimization problem. Note that the implementation is not optimized 1 For Figures 2 4, all data is obtained with the parameters from Table 1 and w 1 = w 4 = 1 unless otherwise noted. Each data point is the mean of three different simulations with unique seeds for the random noise v(t). The estimation error is b(t) = b ˆb(t). Fig. 3. The sensitivity to N (length of the receding horizon) of output regulation, control effort, average estimation error, average P (t), and average time to solve (11) in CPUs. for execution efficiency so a significant reduction in solution time is possible. The controller s sensitivity to the forgetting factor λ is illustrated in Figure 4. The first plot shows that a small value of λ leads to more control effort than a large value (λ = 1 corresponds to no forgetting). This is a consequence of how the estimation algorithm quickly discards old data when λ is small, which means that the controller has to maintain excitation in order to keep the process data informative. When λ approaches 1, the total excitation needed for successfully estimating the unknown constant b decreases. Note that when λ is low, the extra information needed mainly comes from the control input rather than the process output. The second plot shows that adjusting λ has a small effect on the estimation except for λ Thus, Figure 4 indicates that decreasing λ leads to increased control effort rather than poor estimation. 5. CONCLUSIONS AND FUTURE WORK We present a new approach to the simultaneous control and identification problem. The proposed algorithm does not rely on a persistent excitation condition; instead, a measure of confidence in the parameter estimates is obtained by adding a modified recursive least-squares algorithm as a set of equality constraints to the problem. 506

6 ( t f b t=1 2 (t))/tf t=1 y2 (t))/t f t=1 u2 (t))/t f t=1 b 2 (t))/t f t=1 P (t))/t f λ ( t f t=1 P (t))/t f Fig. 4. The sensitivity to λ (the forgetting factor in the least squares algorithm) of output regulation, control effort, average estimation error, and average P (t). Adding these constraints leads to an MPC that excites the system only when deemed necessary by a large P (t) matrix. The algorithm was demonstrated on a simple example with three differently-weighted objective functions and compared against a classic certainty equivalence MPC. Further, the controller s sensitivity to various tuning parameters was shown. The proposed control algorithm was shown to excite the system in the presence of large estimation uncertainty and prioritize output regulation when enough informative data was gathered from excitation. A slight increase in excitation has the potential to greatly improve parameter estimation without significant disruption of output regulation. The fact that the algorithm relies on a formulation of the online optimization problem as an NLP means that the optimization problems will get large as the number of inputs, outputs, and unknown parameters grows. However, with the recent advances in optimization algorithms and the impressive capabilities of today s solvers, the reliance on an NLP solver may be regarded as an advantage rather than a disadvantage of the proposed algorithm. Immediate extensions of this work include testing the algorithm on larger systems with more unknown parameters, modifying the algorithm to work with time-varying parameters, and extending the approach to work for multipleinput multiple-output (MIMO) systems. The algorithm should also be extended to handle systems with constant disturbances without biased parameter estimates. Additional work includes investigating the convergence and stability properties of the method as well as the potential for its use in pure identification experiments on systems with hard constraints. A possible application of the algorithm is maintaining or updating a low-order model for control of a higher order (possibly nonlinear) plant. We will investigate the extent to which a simple model that is well maintained by our controller could work better than a more accurate and complicated model that does not require updating. REFERENCES K. J. Åström and A. Helmersson. Dual control of an integrator with unknown gain. Computers and Mathematics with Applications, 12(6): , D. S. Bayard and A. Schumitzky. Implicit dual control based on particle filtering and forward dynamic programming. International Journal of Adaptive Control and Signal Processing, 24(3): , A. M. Cervantes, A. Wächter, R. H. Tütüncü, and L. T. Biegler. A reduced space interior point strategy for optimization of differential algebraic systems. Computers and Chemical Engineering, 24(1):39 51, A. A. Feldbaum. Dual-control theory. I. Automation and Remote Control, 21(9): , N. M. Filatov and H. Unbehauen. Survey of adaptive dual control methods. IEE Proceedings - Control Theory and Applications, 147(1): , GAMS Development Corporation. The General Algebraic Modeling System, URL H. Genceli and M. Nikolaou. New approach to constrained predictive control with simultaneous model identification. AIChE Journal, 42(10): , R. M. Johnstone, C. R. Johnson Jr., R. R. Bitmead, and B. D. O. Anderson. Exponential convergence of recursive least squares with exponential forgetting factor. Systems & Control Letters, 2(2):77 82, J. M. Lee and J. H. Lee. An approximate dynamic programming based approach to dual adaptive control. Journal of Process Control, 19(5): , L. Ljung. System Identification. Theory for the User. Prentice-Hall, second edition, G. Marafioti. Enhanced Model Predictive Control: Dual Control Approach and State Estimation Issues. Phd thesis, Norwegian University of Science and Technology, URL nbn:no:ntnu:diva D. Q. Mayne, J. B. Rawlings, C. V. Rao, and P. O. M. Scokaert. Constrained model predictive control: Stability and optimality. Automatica, 36(6): , R. K. Mehra. Optimal input signals for parameter estimation in dynamic systems Survey and new results. IEEE Transactions on Automatic Control, 19(6): , D. Peretzki, A. Isaksson, and A. C. Bittencourt. Data Mining for Process Identification. In 17th Nordic Process Control Workshop, S. J. Qin and T. A. Badgwell. A survey of industrial model predictive control technology. Control Engineering Practice, 11(7): , M. S. Shouche, H. Genceli, and M. Nikolaou. Effect of online optimization techniques on model predictive control and identification (MPCI). Computers & Chemical Engineering, 26(9): , B. Wittenmark and C. Elevitch. An adaptive control algorithm with dual features. In IFAC Symposium on Identification and System Parameter Estimation, pages , J. Yan and R. R Bitmead. Model predictive control and state estimation: A network example. World Congress, pages , In 15th IFAC 507