Sprotego Analysis and the Big Daa

Transcription

1 ONLINE SKETCHING FOR BIG DATA SUBSPACE LEARNING Moreza Mardani and Georgios B. Giannakis Dep. of ECE and Digiial Technology Cener, Universiy of Minnesoa ABSTRACT Skeching a.k.a. subsampling high-dimensional daa is a crucial ask o faciliae daa acquisiion process e.g., in magneic resonance imaging, and o render affordable Big Daa analyics. Mulidimensional naure and he need for realime processing of daa however pose major obsacles. To cope wih hese challenges, he presen paper brings forh a novel real-ime skeching scheme ha explois he correlaions across daa sream o learn a laen subspace based upon ensor PARAFAC decomposiion on he fly. Leveraging he online subspace updaes, we inroduce a noion of imporance score, which is subsequenly adaped ino a randomizaion scheme o predic a minimal subse of imporan feaures o acquire in he nex ime insan. Preliminary ess wih synheic daa corroborae he effeciveness of he novel scheme relaive o uniform sampling. Index Terms Tensor, randomizaion, sreaming daa, subspace learning.. INTRODUCTION Exracing laen low dimensional srucure from high dimensional daases is of paramoun imporance in imely inference asks encounered wih Big Daa analyics. Modern daases are ofenimes indexed by hree or more variables giving rise o a ensor, ha is a daa cube or a muliway array [4]. For insance, in magneic resonance imaging MRI one can form a hree-way ensor by sacking high-resoluion MR snapshos as ensor slices [0]. Unraveling srucure of a large-scale ensor is a dauning ask in nowadays Big Daa applicaions primarily due o: c he sheer volume of daa, and c he need for real-ime processing. Skeching as a mehod of choice o cope wih c ypically needs he enire daase o rank individual each daum according o is level of innovaion owards learning. This is however agains he concern of c. The pas work on skeching of high-dimensional daa predominanly focus on one- and wo-way arrays, where daa samples are ranked according o a cerain imporance or soermed leverage score ha depends on he enire bach of daa; see e.g., [8]. Given a skech of daa, our precursors in [6, 7] have paved he roadblocks associaed wih he growing scale and sreaming naure of daa. In essence, [6,7] inro- Suppored by he MURI Gran No. AFOSR FA duces a novel framework for online ensor subspace learning wih rank regularizaion from incomplee subsampled daa. To dae, real-ime skeching for learning from sreaming daa is sill in infancy. Being compuaionally appealing, his ask is also well moivaed for design of experimens in numerous applicaions such as dynamic cardiac MRI, where he slow acquisiion speed and he moving naure of a paien s hear dicaes acquiring only a small fracion of daa per cardiac snapsho [0]. The presen paper bridges his gap by leveraging he correlaions across he daa sreams. Focusing on hree-way ensors, seen as a sream of correlaed slices marices, PARAFAC decomposiion is adaped o effec low rank for he laen subspace. The skech here refers o a small subse of slice feaures, o be used o inerpolae he missing ones. Broadening he scope of our precursor subspace learning and impuaion schemes in [6, 7], our basic idea is o leverage he inermediae esimaes of he subspace bases revealed by he online scheme o predic a subse of mos imporan feaures o acquire in he nex slice. Towards his end, we adop a measure of imporance based upon he bases, ha is hen adaped ino a randomizaion scheme o collec a subse of imporan feaures. Simulaed ess wih synheic daa confirm he convergence and effeciveness of he novel scheme upon running he algorihm wih a warm iniializaion. Noaion: Operaors, and denoe ransposiion, and ouer produc, respecively; diagx is a diagonal marix wih diagonal enries x; is he cardinaliy of a se; he l -norm of a vecor; F he Frobenius norm of a marix; and he se [K] :={,,...,K}.. PRELIMINARIES ON TENSORS For vecors a R M, b R N, and c R T, he ouer produc a b c is an M N T rank-one hree-way array wih m, n, -h enry given by ambnc. Noe ha his comprises a generalizaion o he wo vecor marix case, where a b := ab is a rank-one marix. The PARAFAC model is arguably he mos basic ensor model because of is connecion o ensor rank [4]. Based on he previous discussion i is naural o form a low-rank approximaion of ensor X R M N T as X R r= a r b r c r. When he decomposiion is exac, his offers he PARAFAC decomposiion of X. Accordingly, he minimum value R for /5/$ IEEE 556

2 which he exac decomposiion is possible is by definiion he rank of X. PARAFAC is he model of choice when one is primarily ineresed in revealing laen srucure. Considering he analysis of dynamic social neworks for insance, each of he rank-one facors could correspond o communiies ha e.g., persis or form and dissolve periodically across ime. Differen from he marix case, here is no sraighforward algorihm o deermine he rank of a given ensor, a problem ha has been shown o be NP-hard [, 3, 4]. Wih reference o low-rank decomposiion, inroduce he facor marix A := [a,...,a R ] R M R, and likewise for B R N R and C R T R. Le X, [T ] denoe he -h slice of X along is hird ime dimension, such ha X m, n =Xm, n,. The following compac marix form of he PARAFAC decomposiion in erms of slice facorizaions will be used in he sequel R X = Adiagγ B = γ,r a r b r, [T ] r= where γ denoes he -h row of C recall ha c r insead denoes he r-h column of C. I is apparen ha each slice X can be represened as a linear combinaion of R rank-one marices {a r b r } R r= forming he bases for he ensor subspace. In pracice, one may have only access o a small fracion of he possibly noisy enries of X, namely y i,j = x i,j, i, j Ω, [T ], where he se Ω [M] [N] v i,j + indexes he available feaures a ime insan. Given he daa {y i,j, i, j Ω } T =, ensor impuaion aims o impue or inerpolae he missing feaures, and possibly denoise he presen ones. In essence, feasibiliy of he impuaion ask relies fundamenally on assuming a low-rank PARAFAC model for he daa, o couple he available and missing enries as well as reduce he effecive degrees of freedom in he problem. Generalizing he nuclear-norm regularizaion echnique from low-rank marices o ensor compleion is no however sraighforward if one also desires o unveil he laen srucure in he daa [4]. Ineresingly, i was argued in [] ha he Frobenius-norm regularizaion of he ensor facors, namely ha, B, C := A F + B F + C F offers a viable opion for bach low-rank ensor compleion under he PARAFAC model, by solving P min {X,A,B,C} T = i,j Ω y i,j x i,j + λ ha, B, C s. o X = Adiagγ B, [T ]. The regularizer encourages low-rank ensor decomposiions, wih conrollable rank by uning he parameer λ []. The presen paper primarily focuses on adapively designing an appropriae subsampling ses {Ω } in real ime. Towards his objecive, he firs sep devises online solvers of P as deailed nex. 3. TENSOR SUBSPACE LEARNING As argued earlier he underlying slice sream {X } lives in a low-dimensional subspace. Wih reference o ensor PARAFAC rank, his low-dimensional subspace is characerized by a small number of R rank-one marices {a r b r } R r=, expressed in he compac marix form {A, B}. Learning hese ime-invarian facor marices comprises an iniial sage owards impuing he absen feaures. In he sreaming scenario, a -h acquisiion ime wih T = parial daa snapshos in hand, one is moivaed o recas P in he empirical form P min {A,B,C} f τ A, B; γ. τ= where he insananeous cos is defined as f τ A, B; γ τ := i,j Ω τ y i,j τ R r= + λ γ τ + λ γ τ,r a i,r b j,r A F + B F Apparenly, finding he opimal soluion o he nonconvex program P is a dauning ask for infinie daa sreams. Hence, one needs o devise valid approximaions ha can afford simple ieraive updaes while approaching he opimal soluion. 3.. Sochasic alernaing minimizaion Towards deriving a real-ime, compuaionally efficien, and recursive solver of P, an alernaing-minimizaion mehod is adoped, in which, ieraions coincide wih he ime-scale of daa acquisiion. In accordance wih P, he ieraive procedure adoped here consiss of wo major seps. The firs sep S relies on recenly updaed subspace, namely {A[ ], B[]} o solve he inner opimizaion which yields ˆγ = arg min f A[], B[]; γ. The second sep S hen adjuss he ensor subspace by moving {A, B} along he opposie direcion of he gradien, namely f A[], B[ ]; ˆγ. Towards S, inroduce Φ := [φ,...,φ Ω ] R Ω R, where [φ i,j ] r := a i,r [ ]b j,r [ ]. The projecion of y ono he ensor subspace esimae so-ermed principle componens hen forms he ridge regressors ˆγ =arg min γ R R y Φ γ + λ γ 3 which admis he closed-form ˆγ =Φ Φ + λi R Φ y. To avoid R R inversion, consider he SVD Φ = U Σ V o end up wih ˆγ = V Σ D U y, wih he diagonal marix D R R R and [D ] i,i = σi /σ i + λ. 557

3 The second sep S primarily aims o refine he subspace facor marices based on he principal componens {ˆγ τ } τ= via {A[], B[]} =argmin {A,B} / τ= f A, B; ˆγ τ. This is a nonconvex program due o he bilinear erms in he LS cos, and hus challenging o solve opimally. To miigae his hurdle, along he lines of our precursor [7], a proper quadraic approximan of f is adoped ha is easy o solve opimally. Bypassing he echnical deails refer o [7], he soluion admis a close form, ha amouns o a sochasic gradien-descen recursion. Ineresingly, he gradien of he ensor facors is separable across columns of A and B, where w.r.. a r i admis ar f A, B; ˆγ =λ/a r i,j Ω ˆγ,r y i,j R r= ˆγ,r a i,r b j,r b j,r e i, 4 and likewise for br f A, B. All in all, he gradien ieraions for learning he ensor subspace proceed in parallel as follows r [R] a r [] =a r [ ] μ ar f A[ ], B[ ]; ˆγ, 5 where μ sands for he sep size; likewise for br f A, B; ˆγ. 4. SUBSPACE LEARNING VIA RANDOMIZED SKETCHING While he prevailing missing daa paradigm perains o lack of valid measuremens, one can purposely skip daa o eiher faciliae he acquisiion process, or o lower he compuaional burden of daa processing algorihms. The former is well moivaed by recen effors owards acceleraing he long MRI scans, creaing a lo of arifacs especially for imaging of moving objecs. Imagine for insance he MR scanner knowing a priori he bes minimal subse of k-space daa o collec in each cardiac snapsho. I has hen sufficien ime o acquire imporan samples before he hear moves o a new sae. In essence, he opimal sampling rajecory demands knowledge of an unseen physical phenomenon, ha is pracically infeasible. Typical sampling sraegies, e.g., in he conex of randomized linear algebra, assume daa are fully acquired o score and subsequenly selec a subse of daa; see e.g., [8]. However, in cerain applicaions such as MRI, daa acquisiion is he main challenge [0], and daa are sreaming. All in all, given he subspace esimaes offered by he online scheme unil ime, our goal is o adapively design/predic he subsampling se Ω for he nex ime insan, wih a small size Ω while aaining a reasonable esimaion accuracy. 4.. Imporance Scores Albei sreaming daa poses an exra challenge o skeching since fuure daa are no given, online learning offers inermediae esimaes of he laen ensor subspace, namely {A[ ], B[ ]}, ha can be leveraged o devise adapive sampling sraegies. In order o predic Ω, our basic idea is o rank he feaures according o heir level of imporance measured by a cerain score along he lines of [8, 9]. Noe again [8, 9] deal wih bach processing of daa. To undersand he idea, i is insrucive o firs focus on he bach scenario, wih he enire ensor daa Y a hand, which can be decomposed o end up wih he facor marices A and B. The m, n-h enry of -h ensor enry can hen be expressed as [Y ] m,n R γ,r a m,r []b n,r [], r= where he principal componens are shared by all feaures in slice. The disincion beween feaures are due o he weighs {a m,r } R r= and {b n,r } R r=, corresponding wo m- and -h rows of A and B, respecively. Broadening he scope of [8], a reasonable meric o score he m, n-h feaure is based on he energy of he corresponding rows in he subspace marices A and B. From he viewpoin of low-rank marix compleion [9], imporan feaures are he ones ha if dropped, hey canno be reconsruced using he presen ones. In essence, he column vecors {a r } R r= resp. {b r } R r= ofa resp. B span he column- resp. row- space of he ensor slices. Albei no necessarily orhonormal, he facor marices A and B play similar role as he orhonormal facors U and V forming he SVD Y = UΣV. For row and column spaces, inroduce he so ermed incoherence measures μ i := U e i and ν j := V e j, denoing projecion of he canonical basis e i ono he column and row spaces, respecively. I is well undersood from he marix compleion conex [9] ha i, j- h enry wih high degree of coherence μ i,ν j are more suscepible o misidenificaion if absen. This is simply because he corresponding row and column are more aligned larger projecion wih he column and row space of he underlying marix, and hence hey can be wrongly esimaed while achieving a small rank. Along his line of hough, consider now he online seup where a ime insan one has access o he subspace esimae A[ ], B[ ], and aims o acquire a few feaures of he nex slice Y indexed by Ω. Suppose ha he slices {Y } change slowly over ime; his is he case for insance in dynamic cardiac MRI where differen slices correspond o differen snapshos of a paien s beaing hear. Assume also he subspace marices A[], B[] updaed a ime provide a decen esimae of he underlying ensor subspace e.g., as a resul of a warm iniializaion. I is hen reasonable o adop he meric A [ ]e m F + B [ ]e n F as he imporance score for he m, n-h feaure a ime. Apparenly if Y conains innovaions, no capured by he subspace A[ ], B[ ] learned from pas daa {Y τ } τ=, he imporan feaures may be misidenified. To 558

4 cope wih his issue, and o avoid any sampling bias due o he iniializaion, he skeching is randomized as discussed nex. 4.. Randomized skeching Normalize columns of he facor marices A[] and B[] o end up wih Ã[ ] and B[ ], respecively. For noaional breviy also inroduce Ã[ ] := [α,...,α M ], and B[ ] := [β,...,β N ]. Then, for he feaure m, n [M] [N] a ime, associae he score ρ m, n := αm + βn 6 RM + N The scores {ρ m, n} are posiive-valued and sum up o uniy, and hus one can inerpre hem as a probabiliy disribuion over he enries. For a prescribed maximum sample coun K, one can hen draw K random rials from he disribuion ρ o collec he imporan feaures in he se Ω.I is more naural o consider random rials drawn wih replacemen so as he resuling sample coun Ω K. This is paricularly effecive when feaures are no equally imporan since i skips naive feaures wih iny scores. Adoping randomized skeching along wih he online ieraes for subspace learning in 5 and 3, he resuling procedure is lised under Algorihm. The algorihm begins wih a warm iniializaion, obained for insance afer firs running he algorihm over a small raining daase. The resulan algorihm a ieraion ime insan comprises hree major seps, where he firs sep S probabilisically decides on he sampling se Ω, which is subsequenly used o acquire he corresponding feaures in Y. Based on he parial feaures in Ω, he second sep finds he principal componens of -h frame across he subspace bases {a r []b r []} R r=. The innovaion of he new impued daum capured hrough } i,j Ω, in he nex sep S3 hen refines he subspace bases. An imporan quesion in his conex perains o he average number of samples acquired per slice by he randomized skecher. This essenially depends on he nonuniformiy of he feaures and iniializaion of he algorihm. In an exreme case wih equally imporan feaures, exacly K samples are acquired per ime, which can be significanly lowered for nonuniform feaures. To see his, inroduce a random variable X m,n denoing he frequency of choosing m, n- he error erm {e i,j h enry afer K rials. The random sample coun is hen Ω = m,n I {X m,n}, where he indicaor I {x} akes value of one if x>0, and zero oherwise. In general, he random variables I {Xm,n} are dependen, which renders he pdf analysis for Ω formidable. The expeced sample coun per each slice however can be expressed as E[ Ω ]= M m= n= N [ ρ m, n] K Algorihm Random subsampling inpu A[ ], B[ ], K,andR Ã[ ] := A[ ]diag a [ ],..., a R[ ] B[ ] := B[ ]diag b [ ],..., b R[ ] Ã[ ] := [α,...,α M ], B[ ] := [β,...,β N ] ρ m, n := α RM+N m + β n Draw K random rials from [M] [N] based on ρ o form Ω oupu Ω Algorihm Randomized ensor subspace learning inpu {μ } =,K,R,λ iniialize {A[0], B[0]} wih a warm sarup for =,... do S Random subsampling Acquire {y m,n } m,n Ω based on Algorihm S Principal-componens updae [Φ ] m,n,r = a m,r[ ]b n,r[ ], Φ = U Σ V D =diag [ σ σ + λ,...,σ RσR + λ ], ˆγ = V Σ D U y e m,n := y m,n φ m,n, γ S3 Parallel subspace updae [r [R]] a r[] = μ λ/a r[ ] +μ ˆγ,r m,n Ω e m,n b n,r[ ]e m b r[] = μ λ/b r[ ] +μ ˆγ,r a m,r[ ]e n end for oupu {A[], B[]} m,n Ω e m,n Finally, he average sample coun across boh ime and enries is given as N := / τ= E[ Ω τ ] Remark Convergence. Regarding he ieraes in Algorihm, our precursor [7] esablishes asympoic convergence when he sampling ses {Ω } are independen of he daa {Y }, and form an i.i.d. sequence over ime. The considered adapive acquisiion sraegy however is daa driven and emporally correlaed, rendering he convergence analysis inricae. In his conex, an inriguing quesion perains o role of sampling rajecory owards convergence of he subspace sequence {A[], B[]}. One may also ponder abou he limi poin of he probabiliy sequence ρ. These sudies go beyond he scope of he presen paper, and will be repored in he journal version of his work. 5. SIMULATED TESTS AND CONCLUSIONS Convergence and effeciveness of he novel adapive skeching scheme is assessed in his secion via compuer simulaions. Tensor slices are generaed according o rilinear model Y = Adiagγ B + V, =,,

5 Running average error adapive, α= uniform, α= adapive, α= uniform, α= adapive, α=0 uniform, α= Ieraion index Fig.. Running average error under various cohernece levels α for adapive and uniform subsampling. The average sampling rae for α =0,, is π =0.08, 0.3, 0.6, respecively. x Fig.. Converged probabiliy disribuion ρ for α =wih he average sampling probabiliy π =0.08. wih Gaussian facors A = Λ M α GΛ R α and B = Λ N α GΛ R α, consruced by he sandard Gaussian marix [G] i,j N0,. The diagonal marices Λ M α := diag, α,...,m α likewise Λ N α, Λ R α are inroduced o make he ensor enries saisically nonuniform. Noe α 0 conrols he incoherence for he column and row subspaces, where small value α induces an incoheren subspace wih uniform feaures. Similarly, he principal componens obey γ,r N0,, and he noise erm v m,n N0,σ. As a warm sarup, we iniialize Algorihm wih A[0] = Λ M α GΛ R α and B[0] = Λ N α GΛ R α, for some independen realizaion G. Throughou he ess we adop M = N =0, ρ = R =5, σ =0 4, and λ =0. To simulae he uniform sampling in he sequel, an enry m, n is assigned o he suppor se Ω wih probabiliy w.p. π, and discarded w.p. π. Fig. plos he evoluion of he running average error e := τ= X τ ˆX τ F / X τ F, wih ˆX τ = A[τ]diagγ τ B[τ] denoing he τ-h inerpolaed slice. The accuracy of our adapive skeching scheme is compared agains uniform sampling sraegy under various subspace incoherence levels α. For α =0, boh uniform and adapive sampling sraegies perform somewha equally well. As α grows, adapive sampling becomes crucial and one observes ha for α =uniform sampling learns almos nohing over ime e, while adapive sampling performs quie well e 0 3. The resuling average sampling raes for α =0,, are respecively π =0.6, 0.3, The converged sampling probabiliy disribuion of feaures, namely ρ is depiced in Fig. when α =. The sparse probabiliy map of Fig. emanaes form he fac ha only a small fracion 8% of feaures, associaed wih he upper lef corner, are imporan, and he res are almos naive feaures. REFERENCES [] E. Acar, D. M. Dunlavy, T. G. Kolda, and M. Mrup, Scalable ensor facorizaions for incomplee daa, Chemo. Inelligence Lab. Sys., vol. 06, pp. 4-56, 0. [] J. A. Bazerque, G. Maeos, and G. B. Giannakis, Rank regularizaion and Bayesian inference for ensor compleion and exrapolaion, IEEE Trans. Signal Process.,vol. 6, pp , Nov. 03. [3] S. Gandy, B. Rech, and I. Yamada, Tensor compleion and low-n-rank ensor recovery via convex opimizaion, Inverse Problems, vol. 7, pp. -9, 0. [4] T. G. Kolda and B. W. Bader, Tensor decomposiions and applicaions, SIAM Review, vol. 5, no. 3, pp , Sep [5] J. Kruskal, Three-way arrays: Rank and uniqueness of rilinear decomposiions, wih applicaion o arihmeic complexiy and saisics, Lin. Alg. Applica., vol. 8, no., pp , 977. [6] M. Mardani, G. Maeos, and G. B. Giannakis, Dynamic anomalography: Tracking nework anomalies via sparsiy and low-rank, IEEE J. Sel. Tpoics in Signal Process., vol. 7, pp , Feb. 03. [7] M. Mardani, G. Maeos, and G. B. Giannakis, Sreaming algorihms for impuaion of Big Daa marices and ensors, IEEE Trans. Signal Process., 05. [8] M. W. Mahoney, Randomized algorihms for marices and daa, Foundaions and Trends R in Machine Learning, vol. 3, no., pp. 3 4, 0. [9] Y. Chen, S. Bhojanapalli, S. Sanghavi, and R. Ward, Coheren marix compleion, in Proc. Inl. Conf. on Machine Learning, Beijing, China, Jun. 04. [0] Lauerbur, Image formaion by induced local ineracions: Examples employing nuclear magneic resonance, Naure, vol. 4, pp. 90 9, Mar