arxiv: v1 [cs.sy] 22 Jul 2014

Transcription

1 Approxmae Regularzaon Pah for Nuclear Norm Based H Model Reducon Nclas Blomberg, Crsan R. Rojas, and Bo Wahlberg arxv:47.58v [cs.sy] Jul 4 Absrac Ths paper concerns model reducon of dynamcal sysems usng he nuclear norm of he Hankel marx o make a rade-off beween model f and model complexy. Ths resuls n a convex opmzaon problem where hs radeoff s deermned by one crucal desgn parameer. The man conrbuon s a mehodology o approxmaely calculae all soluons up o a ceran olerance o he model reducon problem as a funcon of he desgn parameer. Ths s called he regularzaon pah n sparse esmaon and s a very mporan ool n order o fnd he approprae balance beween f and complexy. We exend hs o he more complcaed nuclear norm case. The key dea s o deermne when o exacly calculae he opmal soluon usng an upper bound based on he so-called dualy gap. Hence, by solvng a fxed number of opmzaon problems he whole regularzaon pah up o a gven olerance can be effcenly compued. We llusrae hs approach on some numercal examples. Index Terms Regularzaon pah, H model reducon, nuclear norm mnmzaon. I. INTRODUCTION The prncple of parsmony saes ha he smples of wo compeng heores s o be preferred. In engneerng and scence hs ranslaes no ha a smple model ha s good enough for he nended applcaon s preferred compared o a more complex one. Model order reducon concerns mehods o fnd an approxmae lower order model of dynamcal sysems and correspondng error bounds. The advanages of workng wh lower order models nclude faser smulaon, easer conrol desgn and more robus mplemenaons. See [] and [] for references. Consder a sable scalar dscree dynamcal sysem wh ransfer funcon G o (z) = g o,k z k, () where {g o,k } s he mpulse response sequence. The model reducon problem s how o fnd a ransfer funcon G(z) = g k z k of lower order n such ha G G o. Many approaches o he model order reducon problem have been aken, e.g. balanced runcaon [9], Hankel-norm Ths work was suppored by he European Research Councl under he advanced gran LEARN, conrac 6738and by he Swedsh Research Councl under conrac The auhors are wh he Deparmen of Auomac Conrol and ACCESS Lnnaeus Cener, School of Elecrcal Engneerng, KTH Royal Insue of Technology, SE- 44 Sockholm, Sweden. (e-mal: {nbl, crro, bo}@kh.se.) model reducon [6], L model reducon [4], and H model reducon [6]. However, a useful way o measure he approxmaon error s o use he H norm G G o = [g k g o,k ]. The correspondng H model reducon problem mn G G G o, subjec o degree G = n s noorously dffcul and many alernave schemes have been proposed. For example, as n Chaper 8 of [3], we can wre he consran as a rank consran. Our problem wll hen be a rank mnmzaon problem. The rank mnmzaon problem s abou fndng a marx wh mnmum rank subjec o a se of convex consrans. I has recenly been gven more and more aenon snce appears n many areas, such as conrol, sysem denfcaon, and machne learnng. However, hese problems are n general NP-hard [8], and many relaxaons of hem have been explored. One popular relaxaon echnque for he rank mnmzaon problem s he nuclear norm mnmzaon heursc, as dscussed n e.g. [], []. I s a convex relaxaon whch uses he fac ha he rank of a marx follows s nuclear norm (whch s defned as he sum of he sngular values) n he sense ha mnmzng he nuclear norm wll correspond o mnmzng he rank. Alhough he nuclear norm mnmzaon problem has been gven much aenon recenly, here are sll aspecs of ha have no ye been undersood. One such aspec s o dsngush cases when he heursc works and when does no. Anoher aspec concerns choosng he regularzaon parameer. Alhough he parameer can ofen be upper bounded (see []), he parcular choce s dffcul and has grea mpac on he rade-off beween godness-of-f and model order. Hence, here s a need o sudy he mpac of he regularzaon parameer over he whole parameer space. Followng up on hs ssue we are here neresed n oulnng he full regularzaon pah n a compuaonally nexpensve way. Inspred by [4] we sugges an ε-guaraneed regularzaon pah for a specfc problem se-up: an H mnmzaon problem wh a nuclear norm consran. The dea s o defne a ceran dualy gap ha s an upper bound on he approxmaon error nsde whch we can confne he rue regularzaon pah wh a olerance level ε.

2 To map ou he full regularzaon pah s useful n many applcaons. For example, provdes he user wh nformaon upon whch he/she can selec model order. Anoher use arses when we sudy erave re-weghng of he nuclear norm mnmzaon problem, as suded n [8]. Then, he parameer choce can be very rcky snce he proper choce can dffer from eraon o eraon. Ths paper s srucured as follows: In Secon II we formulae our problem. In Secon III we nroduce a way o approxmae he soluon o our problem and hen esablsh a bound on he dualy gap, whch makes possble o confne he rue soluon n an ε-approxmae regon below he approxmaon. In Secon III we also sugges an algorhm for mplemenaon of our heory. Fnally, we make smulaon examples and presen our concluson n Secons IV and V, respecvely. II. PROBLEM FORMULATION Consder a sable scalar dscree dynamcal sysem ransfer funcon as n () bu n a runcaed verson k max G o (z) = g o,k z k, () where k max s assumed o be large enough for he runcaed mpulse response coeffcens o be neglgble. Our am s o fnd a low-order approxmaon G of G o : k max G(z) = g k z k. (3) We defne he mpulse response vecors correspondng o () and (3), respecvely: g o = [ g o, g o,... g o,kmax ] T g = [ g g... g kmax ] T. (4) Consder he followng lnear operaor whch creaes a marx wh squared Hankel srucure: g g g n g g 3 g n+ H(g) :=......, (5) g n g n+ g kmax where k max = n s chosen o be odd. Noe ha a generalzaon, whch we do no consder here for smplcy, s o defne an asymmerc n m Hankel srucure. We know from lnear sysem realzaon heory ha for a sysem wh mpulse response vecor g he sysem order s equvalen o he rank of H(g), [7]. Ths sheds some lgh on why Hankel marx rank mnmzaon plays a cenral role n model order reducon. A common and ofen successful surrogae heursc for rank mnmzaon s nuclear norm mnmzaon. Ths s a convex relaxaon of he rank mnmzaon problem. The nuclear norm of a marx X s defned as X = j σ j, where σ j are he sngular values of X. A. Problem Saemen The regularzed nuclear norm mnmzaon problem has been presened n varous forms n he leraure; he reader can compare [3], [], and [5]. The nuclear norm penaly s ofen seen n he objecve funcon bu here we sae an equvalen verson wh a nuclear norm consran. Here we are neresed n an H cos, snce s a useful way o measure he approxmaon error. For g o, g, and H(g) as n (4) and (5) we formulae he followng regularzed H model reducon problem: mnmze g g g o subjec o H(g), where s he regularzaon parameer. A suffcenly large value of wll gve a perfec f, g = g o, whle small gve lower rank of he sysem. We commen here as a movaon for fuure work ha he cos n (6) may be exended o a weghed verson. Wh approprae weghs we could hen use he maxmum lkelhood approach for model reducon defned n [5]. In order o ge rd of he regularzaon parameer n he consran we reformulae he problem n (6) o an equvalen verson. Wh g o, g, and H(g) as n (4) and (5) he reformulaed verson of Problem (6) s mnmze g o subjec o H(), where, agan, s he regularzaon parameer and = g. We also nroduce he followng noon of he objecve funcon: (6) (7) f () = g o. (8) III. METHOD Our approach follows he one n [4], whch we specalze o our problem se-up. Here s an oulne of he dea: We wan o solve Problem (7) only for a sparse se of pons along he regularzaon pah. We call hese pons, =,..., m, for some m. When we have solved Problem (7) n one such pon we decde o approxmae he soluon n some regon >. Evenually, he approxmaon wll dverge oo far from he rue soluon, so we decde o sop and re-solve Problem (7). The followng approxmaon of f () (defned n (8)) s used n he regon > : f ( op ) f ( ), (9) where and op are opmal soluons o Problem (7) n = and >, respecvely. Ths means ha s kep fxed and (8) s evaluaed for >.

3 3 A. The Dualy Gap The nex ssue s o decde a whch pon (when ncreasng ) o sop approxmang and nsead re-solve Problem (7). To do hs wll defne an upper bound on he approxmaon error. When hs upper bound reaches a ceran olerance level ε, we sop and recompue. In resemblance wh [4] we can call he upper bound he dualy gap. I s an upper bound on he approxmaon error. To compue an upper bound on he approxmaon error f ( ) f ( op ), where and op are opmal soluons o Problem (7) n < and, respecvely, we need a lower bound on f ( op ). To hs end, le C = { mn g o s.. H(), whch corresponds o he opmal cos of Problem (7) dvded by. Now, we ry o relax he consran. From he subdfferenal of he nuclear norm (see []) we ge ha H() H( ) + UV T +W, k }, H k ( k k) () where solves Problem (7) for a parcular =, s he sandard nner produc, UΣV T = H( ) s a compac sngular value decomposon, W s any n n-marx obeyng W and U T W = W V =, and H k s he Hankel marx (see (5)) of a vecor wh zeros everywhere excep a he k h elemen whch s one. We rewre UV T + W, k H k ( k k) = h T ( ), where we have defned he vecor h wh elemens h k = r[h k (UV T + W )]. Then, he consran H() can be relaxed o or snce H( ) relaxaon gves C H( ) + h T ( ), h T ( ) = due o he opmaly of. Ths { mn g o s.. h T ( ), where he opmal soluon o he rgh hand sde can be explcly compued by he projecon heorem: ake op = g o + αh, where α has o be chosen such ha Ths gves h T ( g o + αh ) =. } α = ( ) h ht g o h T,, or op = g o ( ) h ht g o h T h so ha he lower bound on C becomes C op g o = h 4 ( hht ) g o We have now esablshed ha. f ( ) f ( op ) g o h 4 ( hht ) g o and we can defne he followng dualy gap: Defnon 3.: Le be he argumen ha solves Problem (7) n. Then, for any he dualy gap s defned as () := g o h 4 hh T ( g o ). () Noce ha he dualy gap equals zero for =, snce, for ha value of, s he opmal soluon o Problem (7), whch mples ha he error vecor g o s orhogonal o he supporng hyperplane { h T = }. In oher words, ( g o ) h, whch mples ha ( ) = g o hh T h 4 ( g o ) = g o hh T h ( g o ) =. Wh he defnon of he dualy gap we have esablshed an upper bound on he approxmaon error. Indeed, we can confne he opmal soluon o Problem (7) a any o le whn he nerval [f ( ) ( ), f ( )]. To confne he dualy gap whn a ceran olerance, we nroduce he noon of an ε-approxmaon as n [4]. Defnon 3.: Le ε >. Consder an argumen ha solves Problem (7) for parameer value. Then, for any parameer value >, we call any g = ha s feasble for Problem (6) an ε-approxmaon f holds for he dualy gap ha ( ) ε. () B. Upper Bound on We here confne he parameer space for o an nerval [, max ]. For a suffcenly large he feasble se of Problem (7) wll conan = go and we ge f ( go ) =. max s he smalles ha sasfes hs,.e. max = H(g o ). (3)

4 4 We noe ha oher bounds occur n oher versons of he nuclear norm mnmzaon problem. There s unforunaely no smple connecon beween hese. In [], where he nuclear norm s a penalze erm n he objecve funcon, he calculaon of he bound nvolves he subdfferenal of he nuclear norm, gvng a more nvolved dervaon of he parameer bound. C. Algorhm The above resuls gve no only explc bounds on he approxmaon error, bu also suggess a sraghforward mplemenaon. Our algorhm s oulned n Algorhm. We skech he procedure as follows: Le =,,..., m, where m s a ye unknown neger represenng he number of mes we solve Problem (7) along he regularzaon pah. Consder a soluon f ( ) ha solves Problem (7) n. For each, we record f ( ) and Σ, whch s he dagonal sngular value marx of H(g ), where g =. We hen calculae he subsequen pon + by solvng (compare ()) + ( ) = ε, where he dualy gap ( ) s defned n (). For hs subsequen pon + we wll agan solve Problem (7). Nex, we approxmae he soluon pah n he regon < < + usng (9). The procedure eraes from = up o max, where max s defned n (3). Algorhm. Inpu: ε, max, funcon f () defned by (8) Oupu: Consder =,..., m. {f ( )}: a se of exac soluons o Problem (7) {f ( )}: approxmae soluons n (, ) {Σ }: he sngular values of H(g ) = = = whle max do = + = soluon o dmax f () = arg mn H() ( ) = ε Σ = sngular values of H(g ) for k = lnspace(, ) do evaluae f k ( ) end for end whle IV. SIMULATION RESULTS In hs secon we show he resuls of mplemenng Algorhm for wo dfferen se-ups {g o, k max, ε}. In he frs case (see Fgure ) we have chosen a 6 h order sysem wh four relavely small sngular values,.e. can be approxmaed by a nd order sysem. In he second case (see Fgure ), we have chosen a h order sysem wh around nney relavely small sngular values,.e. can be approxmaed by a h order sysem. In boh cases, k max s chosen large enough for he runcaed mpulse responses o be neglgble. Throughou he smulaons we have chosen W = n () for convenence. Fgures (a) and (a) show a shaded/green area enclosed by f ( ) (he approxmae pah) from above and f ( ) from below. (The noaon here mgh be confusng snce s dfferen for each subnerval (, ), =,..., m along he regularzaon pah.) For he black, dashed lnes we have solved Problem (7) for a dense grd of and can hence be sad o represen he rue regularzaon pah. The black vercal lnes ndcae he soppng pons where we have re-solved Problem (7). The calculaon of he dualy gap suffers from some numercal errors, whch are mos sgnfcan for very small values of ; hence he -axs does no sar from zero. These numercal errors resul n ha he gap s no always zero n (he pon where we solve Problem (7) exacly). They can o some exen be explaned by he dvson of n (). Furher, hey are ceranly explaned by ha we runcae he marces U and V defned n (). The runcaon s necessary snce oherwse U and V wll always have full rank due o numercal roundng and can be verfed ha () does no make sense. Fgures (b) and (b) plo sngular values of H(g ), where g = and =,..., m. We have ncluded only sngular values ha are of neres n our examples. For example, for he sysem n Fgure (b), where we have used a 6 h order sysem whch can be approxmaed by a nd order sysem, we nclude he 3 rd o 6 h sngular values. In Fgure (b) we have excluded sngular values below σ 6, whch are neglgble. In Fgure (b) we see ha for <.6 we ge a sysem of nd order, snce we have a drop here n σ 3 ; he 3 rd sngular value. In Fgure (b) we see several drops, and he user has o decde on wha model order s desred and when ha s acheved. f log σ j (b) 5 (a) zone nsde gap rue pah Fg.. (a) Black dashed: rue pah, green/shaded area: approxmaon (upper edge), approxmaon mnus dualy gap (lower edge). (ε, k max, max) = (., 3,.87) (b) Some relevan sngular values of H(g ) evaluaed n, =,..., 5. σ 3 σ 4 σ 5 σ 6

5 5 f log σ j 5 (a) zone nsde gap rue pah (b) σ 5 5 σ σ 35 4 Fg.. (a) Black dashed: rue pah, green/shaded area: approxmaon (upper edge), approxmaon mnus dualy gap (lower edge). (ε, k max, max) = (, 5, 35.6) (b) Some relevan sngular values of H(g ) evaluaed n, =,..., 8. σ 5 σ σ 5 σ [8] K. Mohan and M. Fazel. Reweghed nuclear norm mnmzaon wh applcaon o sysem denfcaon. In Proc. Amercan Conrol Conf., volume 8, pages 3 39, Balmore, Maryland, January. [9] B. C. Moore. Prncpal componen analyss n lnear sysems: Conrollably, observably and model reducon. IEEE Trans. on Auomac Conrol, AC-6():7 3, February 98. [] G. Obnaa and B.D.O. Anderson. Model Reducon for Conrol Sysem Desgn. Sprnger,. [] B. Rech, M. Fazel, and P. Parllo. Guaraneed mnmum-rank soluons of lnear marx equaons va nuclear norm mnmzaon. Socey for Indusral and Appled Mahemacs, 5(3):47, 5. [] D. Sadgh, H. Ohlsson, S. Sasry, and S. A. Sesha. Robus subspace sysem denfcaon va weghed nuclear norm opmzaon. arxv:3.3v [cs.sy], 3. [3] R.E. Skelon, T. Iwasak, and K. Grgorads. A Unfed Algebrac Approach o Lnear Conrol Desgn, pages Taylor and Francs, London, 998. [4] F. Tjärnsröm and L. Ljung. L model reducon and varance reducon. Auomaca, 38:57 53,. [5] B. Wahlberg. Model reducon of hgh-order esmaed models: he asympoc ml approach. In. J. Conrol, 49():69 9, 989. [6] K. Zhou, J. C. Doyle, and K. Glover. Robus and Opmal Conrol. Prence Hall, Englewood Clffs, New Jersey, 996. V. CONCLUSION Wh hs paper we have suggesed a mehod o sudy Problem (6) over he whole regularzaon parameer space, nspred by he work n [4]. The smulaon resul s promsng n showng a compuaonally cheap, approxmae regularzaon pah. Ths approxmae pah oulnes he effec of he parameer value. The user can hen make effcen model order selecon. The use of an approxmae pah also arses e.g. when performng eravely re-weghed nuclear norm mnmzaon. Then, he oulned pah makes possble o re-choose parameer value n each eraon. As for fuure scopes, we am o explore oher versons of our cos funcon, possbly weghed versons of. Anoher exenson can be o nclude npu-oupu daa n he problem se-up, urnng he problem no a subspace denfcaon problem. REFERENCES [] A. Anoulas. Approxmaon of large scale dynamcal sysems. Socey for Indusral and Appled Mahemacs, 5. [] M. Fazel, H. Hnd, and S. Boyd. A rank mnmzaon heursc wh applcaon o mnmum order sysem approxmaon. In Proc. of he Amercan Conrol Conf., pages , Arlngon, Texas,. [3] M. Fazel, H. Hnd, and S. P. Boyd. A rank mnmzaon heursc wh applcaon o mnmum order sysem approxmaon. In Proceedngs of he Amercan Conrol Conference (ACC ), volume 6, pages ,. [4] J Gesen, M. Jagg, and S. Laue. Regularzaon pahs wh guaranees for convex semdefne opmzaon. In 5h Inernaonal Conf. on Arfcal Inellgence and Sascs, volume, pages , La Palma, Canary Islands,. [5] D. Glech and L.-K. Lm. Rank aggregaon va nuclear norm mnmzaon. In The 7h ACM SIGKDD Conference on Knowledge Dscovery and Daa Mnng, pages 6 68, San Dego, Calforna,. Assocaon for Compung Machnery. [6] K. Glover. All opmal hankel-norm approxmaons of lnear mulvarable sysems and her l -error bounds. Inernaonal Journal of Conrol, 39(6):5 93, January 984. [7] T. Kalah. Lnear Sysems. Prence Hall, Englewood Clffs, New Jersey, 98.