Modeling and Optimization for Big Data Analytics

Size: px
Start display at page:

Download "Modeling and Optimization for Big Data Analytics"


1 [ Konsaninos Slavakis, Georgios B. Giannakis, and Gonzalo Maeos ] Modeling and Opimizaion for Big Daa Analyics [ (Saisical) learning ools for our era of daa deluge ] Wih pervasive sensors coninuously collecing and soring massive amouns of informaion, here is no doub his is an era of daa deluge. Learning from hese large volumes of daa is expeced o bring significan science and engineering advances along wih improvemens in qualiy of life. However, wih such a big blessing come big challenges. Running analyics on voluminous daa ses by cenral processors and sorage unis seems infeasible, and wih he adven of sreaming daa sources, learning mus ofen be performed in real Digial Objec Idenifier 0.09/MSP Dae of publicaion: 9 Augus 04 ime, ypically wihou a chance o revisi pas enries. Workhorse signal processing (SP) and saisical learning ools have o be re-examined in oday s high-dimensional daa regimes. This aricle conribues o he ongoing cross-disciplinary effors in daa science by puing forh encompassing models capuring a wide range of SP-relevan daa analyic asks, such as principal componen analysis (PCA), dicionary learning (DL), compressive sampling (CS), and subspace clusering. I offers scalable archiecures and opimizaion algorihms for decenralized and online learning problems, while revealing fundamenal insighs ino he various analyic and implemenaion radeoffs involved. Exensions of he encompassing models o imely daa-skeching, ensor- and kernel-based learning asks are also provided. Finally, IEEE SIGNAL PROCESSING MAGAZINE [8] SEPTEMBER /4 04IEEE

2 he close connecions of he presened framework wih several big daa asks, such as nework visualizaion, decenralized and dynamic esimaion, predicion, and impuaion of nework link load raffic, as well as impuaion in ensor-based medical imaging are highlighed. Inroducion The informaion explosion propelled by he adven of online social media, Inerne, and global-scale communicaions has rendered daa-driven saisical learning increasingly imporan. A any ime around he globe, large volumes of daa are generaed by oday s ubiquious communicaion, imaging, and mobile devices such as cell phones, surveillance cameras and drones, medical and e-commerce plaforms, as well as social neworking sies. The erm big daa is coined o describe his informaion deluge and, quoing a recen press aricle, heir effec is being fel everywhere, from business o science, and from governmen o he ars [8]. Large economic growh and improvemen in he qualiy of life hinge upon harnessing he poenial benefis of analyzing massive daa [8], [55]. Mining unprecedened volumes of daa promises o limi he spread of epidemics and maximize he odds ha online markeing campaigns go viral [35]; o idenify rends in financial markes, visualize neworks, undersand he dynamics of emergen social-compuaional sysems, as well as proec criical infrasrucure including he Inerne s backbone nework [48], and he power grid [6]. big daa challenges and SP opporuniies While big daa come wih big blessings, here are formidable challenges in dealing wih large-scale daa ses. Firs, he sheer volume and dimensionaliy of daa make i ofen impossible o run analyics and radiional inferenial mehods using sandalone processors, e.g., [8] and [3]. Decenralized learning wih parallelized mulicores is preferred [9], [], while he daa hemselves are sored in he cloud or disribued file sysems as in MapReduce/Hadoop [9]. Thus, here is an urgen need o explicily accoun for he sorage, query, and communicaion burden. In some cases, privacy concerns preven disclosing he full daa se, allowing only preprocessed daa o be communicaed hrough carefully designed inerfaces. Due o heir possibly disparae origins, big daa ses are ofen incomplee and a sizable porion of hem is missing. Large-scale daa ineviably conain corruped measuremens, communicaion errors, and even suffer from cyberaacks as he acquisiion and ransporaion cos per enry is driven o he minimum. Furhermore, as many of he daa sources coninuously generae daa in real ime, analyics mus ofen be performed online subjec o ime consrains so ha a high-qualiy answer obained slowly can be less useful han a medium-qualiy answer ha is obained quickly [46], [48], [75]. Alhough pas research on daabases and informaion rerieval is viewed as having focused on sorage, look-up, and search, he opporuniy now is o comb hrough massive daa ses, o discover new phenomena, and o learn [3]. Big daa challenges offer ample opporuniies for SP research [55], where daa-driven saisical learning algorihms are envisioned o faciliae disribued and real-ime analyics (cf. Figure ). Boh classical and modern SP echniques have already placed significan emphasis on ime/daa adapiviy, e.g., [69], robusness [3], as well as compression and dimensionaliy reducion [43]. Tesamen o his fac is he recen rediscovery of sochasic approximaion and sochasic-gradien algorihms for scalable online convex opimizaion and learning [65], ofenimes neglecing Robbins Monro and Widrow s seminal works ha go back half a cenury [60], [69], [79]. While he principal role of compuer science in big daa research is undeniable, he naure and scope of he emerging daa science field is cerainly mulidisciplinary and welcomes SP experise and is recen advances. For example, Web-colleced daa are ofen replee wih missing enries, which moivaes innovaive SP impuaion echniques ha leverage imely (low-rank) marix decomposiions [39], [5], or, suiable kernel-based inerpolaors [6]. Daa marices gahering raffic values observed in he backbone of large-scale neworks can be modeled as he superposiion of unknown clean raffic, which is usually low-rank due o emporal periodiciies as well as nework opology-induced correlaions, and raffic volume anomalies ha occur sporadically in ime and space, rendering he associaed marix componen sparse across rows and columns [38]. Boh quaniy and richness of high-dimensional daa ses offer he poenial o improve saisical learning performance, requiring however innovaive models ha exploi laen low-dimensional srucure o effecively separae he daa whea from he chaff. To learn hese models however, here is a consequen need o advance online, scalable opimizaion algorihms for informaion processing over graphs (an absracion of boh neworked sources of decenralized daa, and muliprocessor, high-performance compuing archiecures); see, e.g., GraphLab [4] and he alernaing direcion mehod of mulipliers (ADMM) [9], [0], [5] ha enjoy growing populariy for disribued machine learning asks. Encompassing models for succinc big daa represenaions This secion inroduces a versaile model o fi daa marices as a superposiion of a low-rank marix capuring correlaions and periodic rends, plus a linearly compressed sparse marix explaining daa innovaions parsimoniously hrough a se of (possibly laen) facors. The model is rich enough o subsume various saisical learning paradigms wih well-documened meris for high-dimensional daa analysis, including PCA [8], DL [56], compressive sampling CS [], and principal componens pursui (PCP) [], [4], [5], o name a few. The background plus paerns and innovaions model for marix daa Le L! R N# T denoe a low-rank marix ( rank( L ) % min{ NT, }), and S R M #! T a sparse marix wih suppor size considerably smaller han MT. Consider also he large-scale daa se Y R N #! T generically modeled as a superposiion of ) he lowrank marix L ; he daa background or rend, e.g., nominal IEEE SIGNAL PROCESSING MAGAZINE [9] SEPTEMBER 04

3 Massive Scale Parallel, Decenralized Ouliers, Missing Values Time/Daa Adapive Challenges Signal Processing and Learning for Big Daa Models and Opimizaion Real-Time Consrains Robus Cloud Sorage Predicion, Forecasing Dimensionaliy Reducion Succinc, Sparse Cleansing, Impuaion Tasks Regression, Classificaion, Clusering [Fig] SP-relevan big daa hemes. load curves across he power grid or he background scene capured by a surveillance camera, plus, ) he daa paerns, (co) clusers, innovaions, or ouliers expressed by he produc of a (possibly unknown) dicionary D R N #! M imes he sparse marix S, and 3) a marix V R N #! T, which accouns for modeling and measuremen errors; in shor, Y = L+ DS + V. Marix D could be an overcomplee se of bases or a linear compression operaor wih N # M. The aforemenioned model offers a parsimonious descripion of Y, ha is welcomed in big daa analyics where daa ses involve numerous feaures. Such parsimony faciliaes inerpreabiliy, model idenifiabiliy, and i enhances he model s predicive performance by discarding noisy feaures ha bear lile relevance o he phenomenon of ineres [49]. To explicily accoun for missing daa in Y inroduce ) he se X 3 {, f, N} # {, f, T} of index pairs ( n, ), and ) he sampling operaor P X (), $ which nulls enries of is marix argumen no in X, leaving he res unchanged. This way, one can express incomplee and (possibly noise-)corruped daa as PX( Y) = PX ( L+ DS + V). () Given P X ( Y), he challenging goal is o esimae he marix componens L and S (and D if no given), which furher enails denoising he observed enries and impuing he missing ones. An esimaor leveraging he low-rank propery of L and he sparsiy of S will be sough o fi he daa P X ( Y) in he leassquares (LS) error sense, as well as minimize he rank of L, and he number of nonzero enries of S : = [ sm, ] measured by is, 0-(pseudo) norm. Unforunaely, albei naural boh rank and, 0-norm crieria are in general NP-hard o opimize [53]. Wih v k( L) denoing he kh singular value of L, he nuclear norm L * : = / v k( L), and he, k -norm S : = / s m, m, are adoped as surrogaes, as hey are he closes convex approximans o rank ( L) and S 0, respecively, e.g., [4] and [48]. Accordingly, assuming known D for now, one solves min P ( Y-L- DS) + m L + m S, X (P) { LS, } F * * where m*, m $ 0 are rank- and sparsiy-conrolling parameers. Being convex, (P) is compuaionally appealing as elaboraed in he secion Algorihms, in addiion o being widely applicable as i encompasses a gamu of known paradigms. Noice however ha when D is unknown, one obains a bilinear model ha gives rise o nonconvex esimaion crieria. The approaches highlighed nex can in fac accommodae more general models han (P), where daa-fiing erms oher han he Frobenius-norm one and differen regularizers can be uilized o accoun for various ypes of a priori knowledge, e.g., srucured sparsiy or smoohness. Applicaion domains and subsumed paradigms Model () emerges in various applicaions, such as ) nework anomaly deecion oulined in he secion Inference and Impuaion, where Y R N #! T represens raffic volume over N links and T ime slos; L capures he nominal link-level raffic (which IEEE SIGNAL PROCESSING MAGAZINE [0] SEPTEMBER 04

4 is low-rank due o emporal periodiciies and opology-induced correlaions on he underlying flows); D represens a link # flow binary rouing marix; and S sparse anomalous flows [47], [48]; ) medical imaging, where dynamic magneic resonance imaging separaes he background L from he moion componen (e.g., a hear beaing) modeled via sparse dicionary represenaion DS [5] (see also he secion Inference and Impuaion ); 3) face recogniion in he presence of shadows and speculariies []; and 4) acousic SP for singing voice separaion from is music accompanimen [7], o name a few. In he absence of L and missing daa ( L = 0, X = {, f, N} # {, f, T}), model () describes an underdeermined sparse signal recovery problem ypically encounered wih CS []. If in addiion D is unknown, (P) boils down o DL [], [46], [56], [67], or, o nonnegaive marix facorizaion (NNMF) if he enries of D and S are nonnegaive [39]. For L = 0, X = {, f, N} # {, f, T}, and if he columns of Y lie close o a union of a small number of unknown low-dimensional linear subspaces, hen looking for a sparse S in () wih M % T amouns o subspace clusering [78]; see also [70] for oulier-robus varians wih srong performance guaranees. Wihou D and wih V = 0, decomposing Y ino L+ S corresponds o PCP, also referred o as robus PCA (R-PCA) [], [4]. Even when L is nonzero, one could envision a varian where he measuremens are corruped wih correlaed (low-rank) noise [5]. Las bu no leas, when S = 0 and V! 0, recovery of L subjec o a rank consrain is nohing else han PCA arguably, he workhorse of high-dimensional big daa analyics [8]. This same formulaion is adoped for low-rank marix compleion he basic ask carried ou by recommender sysems o impue he missing enries of a low-rank marix observed in noise, i.e., PX( Y) = PX ( L+ V) [3]. Based on he maximum likelihood principle, an alernaive approach for missing value impuaion by expecaion-maximizaion can be found in [73]. Algorihms As (P) is joinly convex wih respec o (w.r..) boh L and S, various ieraive solvers are available, including inerior poin mehods and cenralized online schemes based on (sub)gradienbased recursions [65]. For big daa however, off-he-shelf inerior poin mehods are compuaionally prohibiive, and are no amenable o decenralized or parallel implemenaions. Subgradien-based mehods are srucurally simple bu are ofen hindered by slow convergence due o resricive sep size selecion rules. The desideraa for large-scale problems are lowcomplexiy, real-ime algorihms capable of processing massive daa ses in a parallelizable and/or fully decenralized fashion. The few such algorihms available can be classified as decenralized or parallel schemes, spliing, sequenial, and online or sreaming. Decenralized and parallel algorihms In hese divide-and-conquer schemes, muliple agens operae in parallel on disjoin or randomly subsampled subses of he massive-scale daa, and combine heir oupus as ieraions proceed o accomplish he original learning or inference ask [34], [44]. Unforunaely, he nuclear-norm L * in (P) canno be easily disribued across muliple learners, since he full singular value decomposiion (SVD) of L has o be compued cenrally, prior disribuing is se of singular values o each node. In search of a nuclear-norm surrogae amenable o decenralized processing, i is useful o recall ha minimizing L * is anamoun o minimizing ( P Q F+ F)/, where L= PQ, wih P! R N # and Q! R T #, for some %min{ NT, }, is a bilinear decomposiion of he low-rank componen L [47], [7]. In oher words, each column vecor of L is assumed o lie in a low -dimensional range space spanned by he columns of P. This gives rise o he following problem: min m* P ( Y PQ DS) ( P Q ) S. { } X - - +,, F F+ F + m PQS (P) Unlike (P), he bilinear erm PQ renders (P) nonconvex, even if D is known. Ineresingly, [47, Prop. ] offers a cerificae for saionary poins of (P), qualifying hem as global opima of (P). Thanks o he decomposabiliy of F and across rows, and ignoring for a momen he operaor P X, (P) can be disribued over a number V of nodes or processing cores V wih cardinaliy V = V, where each node o! V learns from a subse of rows R o {, f, N}. In oher words, he N rows of Y are disribued over a pariion of rows { R } V o o =, V where by definiion ' R o = {, f, N}, and Roi+ R oj = Y 0, o = if i! j. Naurally, (P) is equivalen o his (modulo P X ) ask: min V {{ Po} o =, Q, S} V / o = Yo-PoQ -DoS F V m* m* + / Po F+ Q F+ m S, () o = where Yo, Po, and D o are submarices formed by keeping only he R o rows of YP,, and D, respecively. An obsacle in () is he coupling of he daa-fiing erm wih he regularizaion erms via { Po, Q, S}. Direc uilizaion of ieraive subgradien-ype mehods, due o he nonsmooh loss funcion, are able o idenify local minimizers of (), a he cos of slow convergence and meiculous choice of sep sizes. In he convex analysis seing, successful opimizaion approaches o surmoun his obsacle include he ADMM [0] and he more general Douglas Rachford (DR) algorihm [5] ha spli or decouple variables in he nuclear-,, -, and Frobenius-norms. The crux of spliing mehods, such as ADMM and DR, lies on compuing efficienly he proximal mapping of regularizing funcions, which for a (non)differeniable lowersemiconinuous convex funcion g and c 0, is defined as Proxcg( A): = argminal ( /) A- Al F + cg( Al), 6 A [5]. The compuaional cos incurred by Proxc g depends on g. For example, if g is he nuclear-norm, hen Proxc *( A) = USof ( R ) V c, where A = URV is he compuaionally demanding SVD of A, and Sofc ( R) is he sof-hresholding operaor whose (, i j ) h IEEE SIGNAL PROCESSING MAGAZINE [] SEPTEMBER 04

5 enry is [Sofc( R)] ij = sgn ([ R] ij, ) max{ 0, [ R] ij, - c}. On he conrary, if g =, hen Proxc ( A) = Sofc ( A), which is a compuaionally affordable, parallelizable operaion. Even if () is a nonconvex ask, a spliing sraegy mimicking ADMM and DR is promising also in he curren conex. If he nework nodes or cores can also exchange messages, hen () can be decenralized. This is possible if e.g., o! V has a neighborhood No V, where o! No and all members of N o exchange informaion. The decenralized rendiion of (P) becomes min Po, Qo, So ' Pol, Qlo, Slo P X o ( Yo-PoQo -DoSo) F m* + ( Pol F+ Qlo F) + m Slo, 6o! V Qo = Qol, So = Sol 6ol! No :) s.o 6o! V :* Qlo = Qlol, Slo = Slol, Po = Pol (P3) where consensus consrains are enforced per neighborhood N o, and { Pol, Qlo, Sl o} are uilized o spli he LS cos from he Frobenius- and, -norms. Typically, (P3) is expressed in unconsrained form using he (augmened) Lagrangian framework. Decenralized inference algorihms over neworks, implemening he previous spliing mehodology, can been found in [], [47], [5], and [6]. ADMM and DR are convergen for convex coss, bu hey offer no convergence guaranees for he nonconvex (P3). There is, however, ample experimenal evidence in he lieraure ha suppors empirical convergence of ADMM, especially when he nonconvex problem a hand exhibis favorable srucure [0], [47]. Mehods offering convergence guaranees for (P3), afer encapsulaing consensus consrains ino he loss funcion, are sequenial schemes, such as he block coordinae descen mehods (BCDMs) [59], [77]. BCDMs minimize he underlying objecive sequenially over one block of variables per ieraion, while keeping all oher blocks fixed o heir mos up-o-dae values. For example, a BCDM for solving he DL subask of (), ha is when { Po, Q} are absen from he opimizaion problem, is he K-SVD algorihm []. Per ieraion, K-SVD alernaes beween sparse coding of he columns of Y based on he curren dicionary and updaing he dicionary aoms o beer fi he daa. For a consensus-based decenralized implemenaion of K-SVD in he cloud, see [58]. I is worh sressing ha (P3) is convex w.r.. each block among { Po, Qo, So, Pol, Qlo, Sl o}, whenever he res are held consan. Recen parallel schemes wih convergence guaranees ake advanage of his underlying srucure o speed-up decenralized and parallel opimizaion algorihms [33], [64]. Addiional BCDM examples will be given nex in he conex of online learning. Online algorihms for sreaming analyics So far, Y has been decomposed across is rows corresponding o nework agens or processors; in wha follows, Y will be spli across is columns. Aiming a online solvers of (P), wih indexing he columns of Y : = [ y, f, y], and { X } x x = indicaing he locaions of known daa values across ime, consider he analyics engine acquiring a sream of vecors P X ( y), 6. An online counerpar of (P) is he following exponenially weighed LS esimae [48] min / P ' x = { qx, sx} x= d - x = P Xx( yx-pqx-dxsx) m* m* + - P F + q + m s xl x d / xl = x G, (P4) where P R N #!, { qx} x = R, { s } R M x, and d! (, 0] denoes he so-ermed forgeing facor. Wih d, pas daa are exponenially discarded o rack nonsaionary feaures. Clearly, PX can be represened by a marix X, whose rows are a subse of he rows of he N-dimensional ideniy marix. A provably convergen BCDM approach o efficienly solve a simplified version of (P4) was pu forh in [48]. Each ime a new daum is acquired, only q and s are joinly updaed via Lasso for fixed P = P -, and hen (P4) is solved w.r.. P o updae P - using recursive LS (RLS). The laer sep can be efficienly spli across - x rows pn, = argminp / d ~ n, x( yn, x-p qx - dn, xsx) + x = ( m */ ) p an aracive feaure faciliaing parallel processing, which neverheless enails a marix inversion when d. Since firs inroduced in [48], he idea of performing online rank-minimizaion leveraging he separable nuclear-norm regularizaion in (P4) has gained populariy in real-ime NNMF for audio SP [7], and online robus PCA [], o name a few examples. In he case where P,{ q } x x = are absen from (P4), an online DL mehod of he same spiri as in [48] can be found in [46], [67]. Algorihms in [48] are closely relaed o imely robus subspace rackers, which aim a esimaing a low-rank subspace P from grossly corruped and possibly incomplee daa, namely PX( y) = PX ( Pq+ s+ v), =,, f. In he absence of 3 sparse ouliers { s} =, an online algorihm based on incremenal gradien descen on he Grassmannian manifold of subspaces was pu forh in [4]. The second-order RLS-ype algorihm in [6] exends he seminal projecion approximaion subspace racking (PAST) algorihm o handle missing daa; see also [50]. When ouliers are presen, robus counerpars can be found in [5] and [9]. Relaive o all aforemenioned works, he esimaion problem (P4) is more challenging due o he presence of he (compression) dicionary D. Reflecing on (P) (P4), all objecive funcions share a common srucure: hey are convex w.r.. each of heir variable blocks, provided he res are held fixed. Naurally, his calls for BCDMs for minimizaion, as in he previous discussion. However, marix inversions and solving a bach Lasso per slo may prove prohibiive for large-scale opimizaion asks. Projeced or proximal sochasic (sub)gradien mehods are aracive lowcomplexiy online alernaives o BCDMs mainly for opimizing convex objecives [65]. Unforunaely, due o heir diminishing sep-sizes, such firs-order soluions exhibi slow convergence even for convex problems. On he oher hand, acceleraed varians for convex problems offer quadraic convergence of he IEEE SIGNAL PROCESSING MAGAZINE [] SEPTEMBER 04

6 objecive funcion values, meaning hey are opimally fas among firs-order mehods [54], [80]. Alhough quadraic convergence issues for nonconvex and ime-varying coss as in (P4) are largely unexplored, he online, acceleraed, firs-order mehod oulined in Figure offers a promising alernaive for generally nonsmooh and nonconvex minimizaion asks [68]. Le x () i be a block of variables, which in (P4) can be P, or { q } x x =, or { sx} x = ; ha is, i! {, 3, }; and le x (- i) denoe all blocks in x: = ( x (), f, x () I ) excep for x () i. Consider he I sequence of loss funcions F(): x f ( x) g ( x () i = + / ), i = i where f is nonconvex, and Lipschiz coninuously differeniable bu convex w.r.. each x () i, whenever { x () j } j! i are held fixed; { gi} i I = are convex and possibly nondiffereniable; hence, F is nonsmooh. Clearly, he daa fi erm in (P4) corresponds o f, () g( x ): = ( m* /) P F, while g and g3 describe he oher wo regularizaion erms. The acceleraion module Accel of [80], developed originally for offline convex analyic asks, is applied o F in a sequenial, per-block (Gauss Seidel) fashion. Having x (- i) fixed, unless () i (- i) () i minx () i! Hi f( x ; x ) + gi( x ) is easily solvable, Accel is employed for Ri $ imes o updae x () i. The same procedure is carried over o he nex block x ( i + ), unil all blocks are updaed, and subsequenly o he nex ime insan + (Figure ). Unlike ADMM, his firs-order algorihm requires no marix inversions, and can afford inexac soluions of minimizaion subasks. Under several condiions, including (saisical) 3 saionariy of { F} =, i also guaranees quadraic-rae convergence o a saionary poin of E{ F }, where E { } denoes expecaion over noise and inpu daa disribuions [68]. An applicaion of his mehod o he dicionary-learning conex can be found in he Inference and Impuaion secion. Daa Skeching, Tensors, and Kernels The scope of he Algorihms secion can be broadened o include random subsampling schemes on Y (also known as daa skeching), as well as muliway daa arrays (ensors) and nonlinear modeling via kernel funcions. Daa skeching Caering o decenralized or parallel solvers, all variables in (P3) should be updaed in parallel across learners of individual nework nodes. However, here are cases where solving all learning subasks simulaneously may be prohibiive or inefficien for wo main reasons. Firs, he daa size migh be so large ha compuing funcion values or firs-order informaion over all variables is impossible. Second, he naure and srucure of daa may preven a fully parallel operaion; e.g., when daa are no available in heir enirey, bu are acquired eiher in baches over ime or where no all of he nework nodes are equally responsive or funcional. A recen line of research aiming a obaining informaive subses of measuremens for asynchronous and reduced-dimensionaliy processing of big daa ses is based on (random) subsampling or daa Is i I? Yes No i = + x (i) min Is f (x (i) x ( i) + g i (x (i) ) i Easy o Solve? No Yes Run he Acceleraion Module on f (. x ( i ) ), g i for R i Times x (i) min i Solve f (x (i) x ( i) ) + g i (x (i) ) Updae Block x (i) i i + [Fig] The online, acceleraed, sequenial (Gauss Seidel) opimizaion scheme for asympoically minimizing he sequence ( F)! N of nonconvex funcions. IEEE SIGNAL PROCESSING MAGAZINE [3] SEPTEMBER 04

7 skeching (via P X ) of he massive Y [45]. The basic principles of daa skeching will be demonsraed here for he overdeermined ( N & ) LS q* : = y! arg minq! R y- Pq [a ask subsumed by (P) as well], denoes pseudo-inverse, - P = ( P P) P, for P full column-rank. Popular sraegies o obain q * include he expensive SVD; he Cholesky decomposiion if P is full column-rank and well condiioned; and he slower bu more sable QR decomposiion [45]. The basic premise of he subsampling or daa skeching echniques is o largely reduce he number of rows of Y prior o solving he LS ask [45]. A daa-driven mehodology of keeping only he mos informaive rows relies on he so-ermed (saisical) leverage scores and is oulined nex as a hree-sep procedure. Given he (hin) SVD P = URV : (S) find he normalized leverage scores { ln} n N - N =, where ln : = en UU en = en PPen, wih en! R being he nh canonical vecor. Clearly, ln equals he (normalized) nh diagonal elemen and since PP = UU is he orhogonal projecor ono he linear subspace spanned by he columns of P, i follows ha y offers he bes approximaion o y wihin his subspace. Then, (S) for an arbirarily small e 0, and by using { ln} n N = as an imporance sampling disribuion, randomly sample and rescale by ( rln) - a number of O( - r = e log ) rows of P, ogeher wih he corresponding enries of y. Such a sampling and rescaling operaion can be expressed by a marix W! R r # N. Finally, (S3) solve he reduced-size LS problem qu *! argminq! R W( y- Pq). Wih () $ l denoing condiion - number and c : = y UU y, i holds ha [45] y- Pqu * # ( + e) y- Pq* (3a) - q -qu # e l( P) c - q * (3b) * * so ha performance degrades gracefully afer reducing he number of equaions. Similar o he nuclear-norm, a major difficuly is ha leverage scores are no amenable o decenralized compuaion [cf. discussion prior (P)], since he SVD of P is necessary prior o decenralizing he original learning ask. To avoid compuing he saisical leverage scores, he following daa-agnosic sraegy has been advocaed [45]: ) Premuliply P and y wih he N# N random Hadamard ransform H N D, where H N is defined inducively as H N = N H N/ H N/, : + + = G H = = G, H N/ - H N/ + - and D is a diagonal marix whose nonzero enries are drawn independenly and uniformly from {-, + }, ) uniformly sample and rescale a number of r = O( log log N+ e - N log ) rows from HN D P ogeher wih he corresponding componens from HN D y, and 3) find q u *! argminq! R WHN D( y- Pq), where W sands again for he sampling and rescaling operaion. Error bounds similar o hose in () can be also derived for his precondiioning sraegy [45]. Key o deriving such performance bounds is he Johnson Lindensrauss lemma, which loosely assers ha for any e! (, 0), any se of poins in N dimensions can be (linearly) embedded ino r $ 4 ( - e e 3 ) - ln dimensions, while preserving he pairwise Euclidean disances of he original poins up o a muliplicaive facor of (! e ). Besides he previous overdeermined LS ask, daa skeching has been employed o ease he compuaional burden of several large-scale asks ranging from generic marix muliplicaion, SVD compuaion, o k-means clusering and ensor approximaion [0], [45]. In he spiri of H N D, mehods uilizing sparse embedding marices have been also developed for overconsrained LS and, p-norm regression, low-rank and leverage scores approximaion [7]; in paricular, hey exhibi complexiy 3 - l 3 - O( supp( P) ) + O( e log ( e )) for solving he LS ask saisfying (3a), where supp( P ) sands for he cardinaliy of he suppor of P, and l! N *. Viewing he sampling and rescaling operaor W as a special case of a (weighed) PX allows carrying over he algorihms oulined in he Encompassing Models for Succinc Big Daa Represenaions and Algorihms secions o he daa skeching seup as well. big daa ensors Alhough he marix model in () is quie versaile and can subsume a variey of imporan frameworks as special cases, he paricular planar arrangemen of daa poses limiaions in capuring available srucures ha can be crucial for effecive inerpolaion. In he example of movie recommender sysems, marix models can readily handle wo-dimensional srucures of people # movie raings. However, movies are classified in various genres and one could explicily accoun for his informaion by arranging raings in a sparse person # genre # ile hree-way array or ensor. In general, various ensor daa analyic asks for nework raffic, social neworking, or medical daa analysis aim a capuring an underlying laen srucure, which calls for high-order facorizaions even in he presence of missing daa [], [50]. Ia# Ib# Ic A rank-one hree-way array Y = [ yiaibic]! R, where he underline denoes ensors, is he ouer produc a% b% c of hree vecors a! R Ia, b! R Ib, c! R Ic : yiaibic = aia bib cic. One can inerpre aia, bib, and cic as corresponding o he people, genre, and ile componens, respecively, in he previous example. The rank of a ensor is he smalles number of rank-one ensors ha sum up o generae he given ensor. These noions readily generalize o higher-way ensors, depending on he applicaion. Nowihsanding, his is no an incremenal exension from low-rank marices o low-rank ensors, since even compuing he ensor rank is an NP-hard problem in iself [36]. Defining a convex surrogae for he rank penaly such as he nuclear norm for marices is no obvious eiher, since singular values when applicable, e.g., in he Tucker model, are no relaed o he rank [74]. Alhough a hree-way array can be unfolded o obain a marix exhibiing laen Kronecker produc srucure, such an unfolding ypically desroys he srucure ha one looks for. These consideraions, moivae forming a low-rank approximaion of ensor Y as IEEE SIGNAL PROCESSING MAGAZINE [4] SEPTEMBER 04

8 Y. / ar % br% cr. (4) r = Low-rank ensor approximaion is a relaively maure opic in mulilinear algebra and facor analysis, and when exac, he decomposiion (4) is called parallel facor analysis (PARAFAC) or canonical decomposiion (CANDECOMP) [36]. PARAFAC is he model of choice when one is primarily ineresed in revealing laen srucure. Unlike he marix case, low-rank ensor decomposiion can be unique. There is deep heory behind his resul, and algorihms recovering he rank-one facors [37]. However, various compuaional and big daa-relaed challenges remain. Missing daa have been handled in raher ad hoc ways [76]. Parallel and decenralized implemenaions have no been horoughly addressed; see, e.g., ParCube and GigaTensor algorihms for recen scalable approaches [57]. Wih reference o (4), inroduce he facor marix A : = [ a,, a ] R I a # f!, and likewise for B! R Ib # and C R Ic #!. Le Yic, ic =, f, Ic denoe he ic h slice of Y along is hird (ube) dimension, such ha Yic( ia, ib) = yiaibic. I follows ha (4) can be compacly represened in marix form, in erms of slice facorizaions Yic = A diag( eic C) B, 6ic. Capializing on he Frobenius-norm regularizaion (P), decenralized algorihms for low-rank ensor compleion under he PARAFAC model can be based on he opimizaion ask: min Ic / { A,B,C} ic = P X c( Yi - A diag( e C) B ) i c ic + m* 6 A + B + F F F Differen from he marix case, i is unclear wheher he regularizaion in (5) bears any relaion wih he ensor rank. Ineresingly, [7] assers ha (5) provably yields a low-rank Y for sufficienly large m *, while he poenial for scalable BCDMbased inerpolaion algorihms is apparen. For an online algorihm, see also (9) in he secion Big Daa Tasks and [50] for furher deails. Kernel-based learning In impuing random missing enries, predicion of muliway daa can be viewed as a ensor compleion problem, where an enire slice (say, he one orhogonal o he ube direcion represening ime) is missing. Noice ha since (5) does no specify a correlaion srucure, i canno perform his exrapolaion ask. Kernel funcions provide he nonlinear means o infuse correlaions or side informaion (e.g., user age range and educaional background for movie recommendaion sysems) in various big daa asks spanning disciplines such as ) saisics, for inference and predicion [8], ) machine learning, for classificaion, regression, clusering, and dimensionaliy reducion [63], and 3) SP, as well as (non)linear sysem idenificaion, sampling, inerpolaion, noise removal, and impuaion; see, e.g., [6] and [75]. In kernel-based learning, processing is performed in a high-, possibly infinie-dimensional reproducing kernel Hilber space (RKHS) H, where funcion f! H o be learned is expressed as F (5) a superposiion of kernels; i.e., f(): x = / 3 {l i ( x, xi), where i = l : X# X " R is he kernel associaed wih H, {{ i} 3 i = denoe he expansion coefficiens, and xxi,! X, 6 i [63]. Broadening he scope of (5), a kernel-based ensor compleion problem is posed as follows. Wih index ses X a: = {, f, Ia}, X b: = {, f, Ib}, and X c: = {, f, Ic}, and associaed kernels lxa( ia, il a), lxb( ib, il b) and lxc( ic, il c), ensor enry yiaibic is approximaed using funcions from he se F : = { fi ( a, ib, ic) = / ar( i ) b ( i ) c ( i ) a HX, b HX, c HX }, r a r b r c ; r! a r! b r! c where = is an upper bound on he rank. Specifically, wih binary weighs { ~ iaibic} aking value 0 if yiaibic is missing (and oherwise), fiing low-rank ensors is possible using / f = arg min ~ i i i[ yi i i -fi ( a, ib, ic)] f! F ia, ib, ic a b c + m* / 8 ar + br + cr B. (6) r = a b c HX HX HX a b c If all kernels are seleced as Kronecker delas, (6) revers back o (5). The separable srucure of he regularizaion in (6) allows applicaion of Represener s heorem [63], which implies ha ar, br, and cr admi finie dimensional represenaions given Ia Ib by ar( ia) = / arila lxa( i, i ), i a l l a b ( i ) ( i, i ), a = r b = / b i riblxb b l l l b b = Ic and cr( ic) = / crilc lxc( i, i ), i c lc respecively. Coefficiens lc = A : = [ a ria l], B : = [ b ], rib l and C : = [ c ] ric l urn ou o be soluions of [cf. (5)] ( ABC,, c ): = arg min / PX ic( Yi I { ABC,, } ic = - KX A diag( e KX CBK ) X ) a ic c b c + race [ A KX A+ B KX B+ C KX C], F m* a b c (P5) where KXa: = [ lxa( ia, il a)], and likewise for KXb and KXc, sand for kernel marices formed using (cross-)correlaions esimaed from hisorical daa as deailed in, e.g., [7]. Remarkably, he cos in (P5) is convex w.r.. any of { ABC,, }, whenever he res of hem are held fixed. As such, he low-complexiy online acceleraed algorihms of he Algorihms secion carry over o ensors oo. Having A available, he esimae a ria l is obained, and likewise for b rib l and c. rib l The laer yield he desired prediced values as y i i i : a ( i ) b r a r( ib) c r( ic). yi i i. / a b c = r = big daa Tasks The ools and hemes oulined so far will be applied in his secion o a sample of big daa SP-relevan asks. Dimensionaliy reducion Nework visualizaion The rising complexiy and volume of neworked (graph-valued) daa presens new opporuniies and challenges for visualizaion ools ha capure global paerns and srucural informaion such as hierarchy, similariy, and communiies [3], [7]. Mos visualizaion algorihms radeoff he clariy of srucural characerisics of he underlying daa for aesheic requiremens a b c IEEE SIGNAL PROCESSING MAGAZINE [5] SEPTEMBER 04

9 such as minimal edge crossing and fixed inernode disance. Alhough efficien for relaively small neworks or graphs (hundreds of nodes), embeddings Consider an undireced graph G( VE, ), where V denoes he se of verices (nodes, agens, or processing cores) wih cardinaliy V = V, and E sands for for larger graphs using hese echniques are seldom srucurally The rising complexiy nodes ha can communicae. Fol- edges (links) ha represen pairs of informaive. The growing ineres and volume of neworked lowing (P3), node o! V communicaes wih is single- or mulihop in analysis of big daa neworks has (graph-valued) daa presens prioriized he need for effecively new opporuniies and neighboring peers in No V. capuring srucure over aesheics challenges for visualizaion ools Given a se of observed feaure vecors { yo} o! V R, and a pre- P in visualizaion. For insance, layous ha capure global paerns of mero-ransi neworks ha show hierarchically he bulk of raffic and srucural informaion such as hierarchy, similariy, scribed embedding dimension p % P(ypically p! {, 3} for visu- convey a lucid picure abou he and communiies. alizaion), he graph embedding mos criical nodes in he even of a amouns o finding a se of p erroris aack. To his end, [3] capures { zo} o! V R vecors ha preserve hierarchy in neworks or graphs hrough well-defined in he very low-dimensional R p he nework srucure observed measures of node imporance, collecively known as cenraliy via { yo} o! V. The dimensionaliy reducion module of [3] is in he nework science communiy. Examples are he beweenness based on local linear embedding (LLE) principles [6], which cenraliy, which describes he exen o which informaion assume ha he observed { yo} o! V live on a low-dimensional, is roued hrough a specific node by measuring he fracion of all shores pahs raversing i, as well as closeness, eigenvalue, and Markov cenraliy [3]. smooh, bu unknown manifold, wih he objecive of seeking an embedding ha preserves he local srucure of he manifold in he lower dimensional R p. In paricular, LLE accomplishes his by approximaing each daa poin via an affine combinaion (real weighs summing up o ) of is neighbors, followed by.0 consrucion of a lower-dimensional embedding ha bes preserves he weighs. If Y : = [ y, f, y ] R # N o o ol ol No! gahers all he observed daa wihin he neighborhood of node o, and 0.5 along he lines of LLE, he cenraliy consrained (CC-)LLE mehod comprises he following wo seps: (a) (b) [Fig3] The visualizaion of wo snapshos of he large-scale nework Gnuella [40] by means of he CC-LLE mehod. The cenraliy meric is defined by he node degree. Hence, nodes wih low degree are placed far from he cener of he embedding. (a) Gnuella-04 (08/04/0). (b) Gnuella-4 (08/4/0). S: 6o! V, so! arg min yo- Ys o Ys o = h ( co) s. o) s = S: min zo - sool zol / / { zo} o! V o! V ol! V s.o s z = h ( co), 6o! V, o where { co} o! V R are cenraliy merics, h ( ) is a monoone decreasing funcion ha quanifies he cenraliy hierarchy, e.g., hc ( o) = exp( -co), and s = enforces he local affine approximaion of y o by { yol} ol! No. In oher words, and in he spiri of (P3), y o is affinely approximaed by he local dicionary Do: = Yo. I is worh sressing ha boh objecive and consrains in sep of (7) can be compued solely by means of he inner-producs or correlaions { Yo yo, Yo Yo} o! V. Hence, knowledge of { yo} o! V is no needed in CC-LLE, and only a given se of dissimilariy measures { d } ool ( oo, l )! V suffices o formulae (7), where d ool! R$ 0, dool= doo l, and d oo = 0, 6 ( o, ol )! V ; e.g., d : = - y - - ool o yol yo yol in (7). Afer relaxing he nonconvex consrain Ys h o = ( co ) o he convex Ys h o # ( co ) one, a BCDM approach is followed o solve (7) efficienly, wih compuaional complexiy ha scales linearly wih he nework size [3]. Figure 3 depics he validaion of CC-LLE on large-scale degree visualizaions of snapshos of he Gnuella peer-o-peer file-sharing nework ( V = 6, 58, (7) IEEE SIGNAL PROCESSING MAGAZINE [6] SEPTEMBER 04

10 E = 65, 369) [40]. Snapshos of his direced nework were capured on 4 and 4 Augus 00, respecively, wih nodes represening hoss. For convenience, undireced rendiions of he wo neworks were obained by symmerizaion of heir adjacency marices. Noice here ha he mehod can generalize o he direced case oo, a he price of increased compuaional complexiy. The cenraliy meric of ineres was he node degree, and dissimilariies were compued based on he number of shared neighbors beween any pair of hoss. I is clear from Figure 3 ha despie he dramaic growh of he nework over a span of 0 days, mos new nodes had low degree, locaed hus far from he cener of he embedding. The CC-LLE efficiency is manifesed by he low running imes for obaining embeddings in Figure 3;,684 s for Gnuella-04, and 5,639 s for Gnuella-4 [3]. Inference and impuaion Decenralized esimaion of anomalous nework raffic In he backbone of large-scale neworks, origin-o-desinaion (OD) raffic flows experience abrup changes ha can resul in congesion and limi he qualiy of service provisioning of he end users. These raffic anomalies could be due o exernal sources such as nework failures, denial of service aacks, or inruders [38]. Unveiling hem is a crucial ask in engineering nework raffic. This is challenging however, since he available daa are high-dimensional noisy link-load measuremens, which comprise he superposiion of clean and anomalous raffic. Consider as in he secion Dimensionaliy Reducion an undireced, conneced graph G( VE, ). The raffic Y R N #! T, carried over he edges or links E ( E = N) and measured a ime insans! {, f, T} is modeled as he superposiion of unknown clean raffic flows L*, over he ime horizon of ineres, and he raffic volume anomalies S* plus noise V ; Y = L* + S* + V. Common emporal paerns among he raffic flows in addiion o heir periodic behavior render mos rows (respecively columns) of L* linearly dependen, and hus L* ypically has low rank [38]. Anomalies are expeced o occur sporadically over ime, and only las for shor periods relaive o he (possibly long) measuremen inerval. In addiion, only a small fracion of he flows is anomalous a any ime slo. This renders marix S* sparse across rows and columns [48]. In he presen conex, real daa including OD flow raffic levels and end-o-end laencies are colleced from he operaion of he Inerne nework (Inerne backbone nework across he Unied Saes) [30]. OD flow raffic levels were recorded for a hree-week operaion (sampled per 5 min) of Inerne-v during 8 8 December 003 [38]. To beer assess performance, large spikes of ampliude equal o he larges recorded raffic across all flows and ime insans were injeced ino % randomly seleced enries of he ground-ruh marix L*. Along he lines of (P3), where he number of links N =, and T = 504, he rows of he daa marix Y were disribued uniformly over a number of V = nodes. (P3) is solved using ADMM, and a small porion ( 50 # 50) of he esimaed anomaly marix S is depiced in Figure 4(a). Anomaly Ampliude Flow Index (n) 0 0 Real Daa (Inerne-v) Time Index () 50 Relaive Esimaion Error Relaive Esimaion Error Ieraion Index Synheic Daa Time (s) True Esimaed V = RPCA V = V = 4 V = 5 V = 00 V = 65 (a) (b) [Fig4] Decenralized esimaion of nework raffic anomalies measured in bye unis over 5 min ime inervals: (a) only a small porion ( 50 # 50) of he sparse marices S* and S enries are shown; (b) relaive esimaion error versus ADMM ieraion index and cenral processing uni (CPU) ime over neworks wih V number of nodes. The curve obained by he cenralized R-PCA mehod [] is also depiced. IEEE SIGNAL PROCESSING MAGAZINE [7] SEPTEMBER 04

11 As a means of offering addiional design insighs, furher validaion is provided here o reveal he radeoffs ha become relevan as he nework size increases. Specifically, comparisons in erms of running ime are carried ou w.r.. is cenralized counerpar. Throughou, a nework modeled as a square grid (uniform laice) wih agens per row/column is adoped. To gauge running imes as he nework grows, consider a fixed size daa marix 500 Y R, #, 500!. The daa are synhesized according o he previous model of Y = L* + S* + V, deails for which can be found in [47, Sec. V]. Rows of Y are uniformly spli among he nework nodes. Figure 4(b) illusraes he relaive esimaion error S - S* F/ S* F (S sands for he esimae of S* ) versus boh ieraion index of he ADMM and CPU ime over various nework sizes. Dynamic link load raffic predicion and impuaion Consider again he previous undireced graph G( VE, ). Conneciviy and edge srenghs of G are described by he adjacency Link Load Link Load Esimaion Error.5 0.5,500,600,700,800,900,000 Time ,500,600,700,800,900,000 Time True (Missing) Esimaed (Missing) True (Missing) Esimaed (Missing) Link 7 Link ,500,600,700,800,900,000 Time (c) [Fig5] Link load racking (dos and riangles) and impuaion (crosses and circles) on Inerne [30]. The proposed mehod is validaed versus he ADMM-based approach of [3]. (a) (b) True (Observed) Esimaed (Observed) True (Observed) Esimaed (Observed) ADMM Based Proposed marix W R V #! V, where [ W] ool 0 if nodes o and ol are conneced, while [ W ] ool = 0 oherwise. A every! N 0, a variable o! R, which describes a nework-wide dynamical process of ineres, corresponds o a node o! V. All node variables are colleced in : = [, f, V]! R V. A sparse represenaion of he process over G models as a linear combinaion of few aoms in an N# M dicionary D, wih M $ N; and = Ds, M where s! R is sparse. Furher, only a porion of is observed Nl # N per ime slo. Le now X! R, Nl # N, denoe a binary measuremen marix, wih each row of X corresponding o he canonical basis vecor for R N, selecing he measured componens of y! R. In oher words, he observed daa per slo are N y = X + v, where v denoes noise. To impue missing enries of in y, he opology of G will be uilized. The spaial correlaion of he process is capured by he (unnormalized) graph N Laplacian marix K : = diag( W N) - W, where N! R is he all-ones vecor. Following Figure and given a forgeing facor d! (, 0], o gradually diminish he effec of pas daa (and hus accoun for nonsaionariy), define F (, sd): - x mk = / d yx- XxDs + s D KDs D x = g () s g ( D) H F + m s + kd( D), f ( sd, ) where D : = d - x /, and k x = D sands for he indicaor funcion N# M of D : = { D = [ d, f, dm]! R ; dm #, m! {, f, M}}, i.e., k D( D) = 0 if D! D, and k D( D) =+ 3 if D " D (noe ha 6c 0, Prox ckd is he meric projecion ono he closed convex D [5]). The erm including he known K quanifies he a priori informaion on he opology of G, and promoes smooh soluions over srongly conneced nodes of G [3]. This erm is also insrumenal for accommodaing missing enries in ( )! N 0. The algorihm of Figure was validaed on esimaing and racking nework-wide link loads aken from he Inerne measuremen archive [30]. The nework consiss of N = 54 links and nine nodes. Using he nework opology and rouing N informaion, nework-wide link loads ( )! N 0 R become available (in gigabis per second). Per ime slo, only Nl = 30 of he componens, chosen randomly via X, are observed in Nl y! R. Cardinaliy of he ime-varying dicionaries is se o M = 80, 6. To cope wih pronounced emporal variaions of he Inerne link loads, he forgeing facor d in (8) was se equal o 0.5. Figure 5 depics esimaed values of boh observed (dos) and missing (crosses) link loads, for a randomly chosen link of he nework. The normalized squared esimaion error beween he rue and he inferred, specifically - -, is also ploed in Figure 5 versus ime. The acceleraed algorihm was compared wih he sae-of-he-ar scheme in [3] ha relies on ADMM, o minimize a cos closely relaed o (8) w.r.. s, and uses BCD ieraions requiring marix inversion o opimize (8) w.r.. D. On he oher hand, R = and R = 0 in he algorihm of Figure. I is worh noicing here ha ADMM in [3] requires muliple ieraions o achieve a prescribed esimaion accuracy, and ha no marix inversion (8) IEEE SIGNAL PROCESSING MAGAZINE [8] SEPTEMBER 04

12 (a) (b) (c) (d) [Fig6] The impuaion of missing funcional MRI cardiac images by using he PARAFAC ensor model and he online framework of (9). The images were arificially colored o highligh he differences beween he obained recovery resuls. (a) The original image. (b) The degraded image (75% missing values). (c) The recovered image ( = 0) wih relaive esimaion error 0.4. (d) The recovered image ( = 50) wih relaive esimaion error was incorporaed in he realizaion of he proposed scheme. Even if he acceleraed firs-order mehod operaes under lower compuaional complexiy han he ADMM approach, esimaion error performance boh on observed and missing values is almos idenical. Cardiac MRI Cardiac magneic resonance imaging (MRI) is a major imaging ool for noninvasive diagnosis of hear diseases in clinical pracice. However, ime limiaions posed by he paien s breah-holding ime, and hus he need for fas daa acquisiion degrade he qualiy of MRI images, resuling ofen in missing pixel values. In he presen conex, impuaion of he missing pixels uilizes he fac ha cardiac MRI images inrinsically conain low-dimensional componens. The FOURDIX daa se is considered, which conains 63 cardiac scans wih en seps of he enire cardiac cycle [4]. Each scan is an image of size 5 # 5 pixels, which is divided ino 64 ( 3 # 3) -dimensional paches. Placing one afer he oher, 3 # 3 # 67, 38 paches form a sequence of slices of a ensor Y! R. Randomly chosen 75% of he Y enries are dropped o simulae missing daa. Operaing on such a ensor via bach algorihms is compuaionally demanding, due o he ensor s size and he compuer s memory limiaions. Moivaed by he bach formulaion in (5), a weighed LS online counerpar is [50] min / { ABC,, } x = d - x = P X( Yx - A diag( ex C) B ) m* + ( A F+ B F) + m* e C, x x E - d / x = where d 0 is a forgeing facor, and e x is he xh -dimensional canonical vecor. The hird dimension of Y in (9) indicaes he slice number. To solve (9), he variables { ABC,, } are sequenially processed; fixing { AB, }, (9) is minimized w.r.. C, while gradien seepes descen seps are aken w.r.. each one of A and B, having he oher variables held consan. The resulan online learning algorihm is compuaionally ligh, wih 56 operaions (on average) per. The resuls of is applicaion o a randomly chosen scan image, for differen choices of he rank, are depiced in Figure 6 wih relaive esimaion errors, Y - Y x x F/ Yx F, equal o 0.4 and for = 0 and 50, respecively. F (9) IEEE SIGNAL PROCESSING MAGAZINE [9] SEPTEMBER 04

13 Addiional approaches for bach ensor compleion of boh visual and specral daa can be found in [4] and [66], whereas he algorihms in [] and [7] carry ou low-rank ensor decomposiions from incomplee daa and perform impuaion as a by-produc. Acknowledgmens Work in his aricle was suppored by he Naional Science Foundaion grans ECCS and Eager Moreover, i has been cofinanced by he European Union (European Social Fund and Greek naional funds hrough he Operaional Program Educaion and Lifelong Learning of he Naional Sraegic Reference Framework-Research Funding Program: Thalis UoA Secure Wireless Nonlinear Communicaions a he Physical Layer. We wish o hank Moreza Mardani and Brian Baingana, from he Universiy of Minnesoa, for he fruiful discussions and he numerical ess hey provided. AUTHORS Konsaninos Slavakis received his Ph.D. degree from he Tokyo Insiue of Technology (TokyoTech), Japan, in 00. He was a posdocoral fellow wih TokyoTech ( ) and he Deparmen of Informaics and Telecommunicaions, Universiy of Ahens, Greece ( ). He was an assisan professor in he Deparmen of Telecommunicaions and Informaics, Universiy of Peloponnese, Tripolis, Greece (007 0). He is currenly a research associae professor wih he Deparmen of Elecrical and Compuer Engineering and Digial Technology Cener, Universiy of Minnesoa, Unied Saes. His curren research ineress include signal processing, machine learning, and big daa analyics problems. Georgios B. Giannakis received his Ph.D. degree from he Universiy of Souhern California in 986. Since 999, he has been wih he Universiy of Minnesoa, where he holds he ADC chair in wireless elecommunicaions in he Deparmen of Elecrical and Compuer Engineering and serves as direcor of he Digial Technology Cener. His ineress are in he areas of communicaions, neworking, and saisical signal processing subjecs on which he has published more han 360 journal and 60 conference papers, book chapers, wo edied books, and wo research monographs (h-index 08). His curren research focuses on sparsiy and big daa analyics, cogniive neworks, renewables, power grid, and social neworks. He is he (co) invenor of paens and he (co)recipien of eigh bes paper awards from he IEEE Communicaions and Signal Processing Socieies. He is a Fellow of he IEEE and EURASIP and has also received echnical achievemen awards from he IEEE Signal Processing Sociey and EURASIP. Gonzalo Maeos received his B.Sc. degree in elecrical engineering from Universidad de la Republica, Uruguay, in 005 and he M.Sc. and Ph.D. degrees in elecrical engineering from he Universiy of Minnesoa, in 009 and 0, respecively. Since 04, he has been an assisan professor wih he Deparmen of Elecrical and Compuer Engineering, Universiy of Rocheser. During 03, he was a visiing scholar wih he Compuer Science Deparmen, Carnegie Mellon Universiy. From 003 o 006, he worked as a sysems engineer a ABB, Uruguay. His research ineress lie in he areas of saisical learning from big daa, nework science, wireless communicaions, and signal processing. His curren research focuses on algorihms, analysis, and applicaion of saisical signal processing ools o dynamic nework healh monioring, social, power grid, and big daa analyics. References [] E. Acar, D. M. Dunlavy, T. G. Kolda, and M. Mørup, Scalable ensor facorizaions for incomplee daa, Chemome. Inell. Lab. Sys., vol. 06, no., pp. 4 56, 0. [] M. Aharon, M. Elad, and A. Brucksein, K-SVD: An algorihm for designing overcomplee dicionaries for sparse represenaion, IEEE Trans. Signal Processing, vol. 54, no., pp , Nov [3] B. Baingana and G. B. Giannakis, Embedding graphs under cenraliy consrains for nework visualizaion, submied for publicaion. arxiv: [4] L. Balzano, R. Nowak, and B. Rech, Online idenificaion and racking of subspaces from highly incomplee informaion, in Proc. Alleron Conf. Communicaion, Conrol, and Compuing, Monicello, IL, 00, pp [5] H. H. Bauschke and P. L. Combees, Convex Analysis and Monoone Operaor Theory in Hilber Spaces. New York: Springer, 0. [6] J. A. Bazerque and G. B. Giannakis, Nonparameric basis pursui via sparse kernel-based learning, IEEE Signal Process. Mag., vol. 30, no. 4, pp. 5, July 03. [7] J. A. Bazerque, G. Maeos, and G. B. Giannakis, Rank regularizaion in Bayesian inference for ensor compleion and exrapolaion, IEEE Trans. Signal Processing, vol. 6, no., pp , Nov. 03. [8] T. Bengsson, P. Bickel, and B. Li, Curse-of-dimensionaliy revisied: Collapse of he paricle filer in very large scale sysems, in Probabiliy and Saisics: Essays in Honor of David A. Freedman. Beachwood, OH: IMS, 008, vol., pp [9] D. P. Bersekas and J. N. Tsisiklis, Parallel and Disribued Compuaion: Numerical Mehods. Belmon, MA: Ahena Scienific, 999. [0] S. Boyd, N. Parikh, E. Chu, B. Peleao, and J. Ecksein, Disribued opimizaion and saisical learning via he alernaing direcion mehod of mulipliers, Found. Trends Machine Learn., vol. 3, no., pp., 0. [] E. Candès and M. B. Wakin, An inroducion o compressive sampling, IEEE Signal Process. Mag., vol. 5, no., pp. 30, 008. [] E. J. Candès, X. Li, Y. Ma, and J. Wrigh, Robus principal componen analysis?, J. ACM, vol. 58, no., pp. 37, 0. [3] E. J. Candes and Y. Plan, Marix compleion wih noise, Proc. IEEE, vol. 98, no. 6, pp , June 009. [4] V. Chandrasekaran, S. Sanghavi, P. R. Parrilo, and A. S. Willsky, Ranksparsiy incoherence for marix decomposiion, SIAM J. Opim., vol., no., pp , 0. [5] Q. Chenlu and N. Vaswani, Recursive sparse recovery in large bu correlaed noise, in Proc. Alleron Conf. Communicaion, Conrol, and Compuing, Sep. 0, pp [6] Y. Chi, Y. C. Eldar, and R. Calderbank, PETRELS: Parallel subspace esimaion and racking using recursive leas squares from parial observaions, IEEE Trans. Signal Processing, vol. 6, no. 3, pp , 03. [7] K. L. Clarkson and D. P. Woodruff, Low rank approximaion and regression in inpu sparsiy ime, in Proc. Symp. Theory Compuing, June 4, 03, pp arxiv: v4. [8] K. Cukier. (00). Daa, daa everywhere. The Economis. [Online]. Available: hp:// [9] J. Dean and S. Ghemawa, MapReduce: Simplified daa processing on large clusers, in Proc. Symp. Operaing Sysem Design and Implemenaion, San Francisco, CA, 004, vol. 6, p. 0. [0] P. Drineas and M. W. Mahoney, A randomized algorihm for a ensor-based generalizaion of he SVD, Linear Algeb. Appl., vol. 40, no. 3, pp , 007. [] J. Feng, H. Xu, and S. Yan, Online robus PCA via sochasic opimizaion, in Proc. Advances in Neural Informaion Processing Sysems, Lake Tahoe, NV, Dec. 03, pp [] P. Forero, A. Cano, and G. B. Giannakis, Consensus-based disribued suppor vecor machines, J. Mach. Learn. Res., vol., pp , May 00. [3] P. Forero, K. Rajawa, and G. B. Giannakis, Predicion of parially observed dynamical processes over neworks via dicionary learning, IEEE Trans. Signal Processing, o be published. [4] [Online]. Available: hp:// IEEE SIGNAL PROCESSING MAGAZINE [30] SEPTEMBER 04

14 [5] H. Gao, J. Cai, Z. Shen, and H. Zhao, Robus principal componen analysisbased four-dimensional compued omography, Phys. Med. Biol., vol. 56, no., pp , 0. [6] G. B. Giannakis, V. Kekaos, N. Gasis, S. J. Kim, H. Zhu, and B. Wollenberg, Monioring and opimizaion for power grids: A signal processing perspecive, IEEE Signal Process. Mag., vol. 30, no. 5, pp. 07 8, Sep. 03. [7] L. Harrison and A. Lu, The fuure of securiy visualizaion: Lessons from nework visualizaion, IEEE New., vol. 6, pp. 6, Dec. 0. [8] T. Hasie, R. Tibshirani, and J. Friedman, The Elemens of Saisical Learning: Daa Mining, Inference, and Predicion, nd ed. New York: Springer, 009. [9] J. He, L. Balzano, and A. Szlam, Incremenal gradien on he Grassmannian for online foreground and background separaion in subsampled video, in Proc. IEEE Conf. Compuer Vision and Paern Recogniion, Providence, RI, June 0, pp [30] [Online]. Available: hp:// [3] M. I. Jordan, On saisics, compuaion and scalabiliy, Bernoulli, vol. 9, no. 4, pp , 03. [3] S. A. Kassam and H. V. Poor, Robus echniques for signal processing: A survey, Proc. IEEE, vol. 73, no. 3, pp , Mar [33] S.-J. Kim and G. B. Giannakis, Opimal resource allocaion for MIMO ad hoc cogniive radio neworks, IEEE Trans. Info. Theory, vol. 57, no. 5, pp , May 0. [34] A. Kleiner, A. Talwalkar, P. Sarkar, and M. I. Jordan, A scalable boosrap for massive daa, J. Royal Sais. Soc.: Ser. B, o be published. [Online]. Available: hp:// [35] E. D. Kolaczyk, Saisical Analysis of Nework Daa: Mehods and Models. New York: Springer, 009. [36] T. G. Kolda and B. W. Bader, Tensor decomposiions and applicaions, SIAM Rev., vol. 5, no. 3, pp , 009. [37] J. B. Kruskal, Three-way arrays: Rank and uniqueness of rilinear decomposiions, wih applicaions o arihmeic complexiy and saisics, Linear Algeb. Appl., vol. 8, no., pp , 977. [38] A. Lakhina, M. Crovella, and C. Dio, Diagnosing nework-wide raffic anomalies, in Proc. SIGCOMM, Aug. 004, pp [39] D. Lee and H. S. Seung, Learning he pars of objecs by non-negaive marix facorizaion, Naure, vol. 40, pp , Oc [40] J. Leskovec, J. Kleinberg, and C. Falousos, Graph evoluion: Densificaion and shrinking diameers, ACM Trans. Knowl. Discov. Daa, vol., no., Mar [4] J. Liu, P. Musialski, P. Wonka, and J. Ye, Tensor compleion for esimaing missing values in visual daa, IEEE Trans. Paern Anal. Mach. Inell., vol. 35, pp. 08 0, Jan. 03. [4] Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guesrin, and J. Hellersein, GraphLab: A new framework for parallel machine learning, in Proc. 6h Conf. Uncerainy in Arificial Inelligence, Caalina Island: CA, 00. [43] Y. Ma, P. Niyogi, G. Sapiro, and R. Vidal, Dimensionaliy reducion via subspace and submanifold learning [From he Gues Ediors], IEEE Signal Process. Mag., vol. 8, no., pp. 4 6, Mar. 0. [44] L. Mackey, A. Talwalkar, and M. I. Jordan, Disribued marix compleion and robus facorizaion, submied for publicaion. arxiv: v7. [45] M. W. Mahoney, Randomized algorihms for marices and daa, Found. Trends Machine Learn., vol. 3, no., pp. 3 4, 0. [46] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, Online learning for marix facorizaion and sparse coding, J. Machine Learn. Res., vol., pp. 9 60, Mar. 00. [47] M. Mardani, G. Maeos, and G. B. Giannakis, Decenralized sparsiy-regularized rank minimizaion: Algorihms and applicaions, IEEE Trans. Signal Processing, vol. 6, no., pp , Nov. 03. [48] M. Mardani, G. Maeos, and G. B. Giannakis, Dynamic anomalography: Tracking nework anomalies via sparsiy and low rank, IEEE J. Sel. Topics Signal Process., vol. 8, pp , Feb. 03. [49] M. Mardani, G. Maeos, and G. B. Giannakis, Recovery of low-rank plus compressed sparse marices wih applicaion o unveiling raffic anomalies, IEEE Trans. Info. Theory, vol. 59, no. 8, pp , Aug. 03. [50] M. Mardani, G. Maeos, and G. B. Giannakis, Subspace learning and impuaion for sreaming big daa marices and ensors, IEEE Trans. Signal Processing, submied for publicaion. [5] G. Maeos, J. A. Bazerque, and G. B. Giannakis, Disribued sparse linear regression, IEEE Trans. Signal Processing, vol. 58, no. 0, pp , Oc. 00. [5] G. Maeos and G. B. Giannakis, Robus PCA as bilinear decomposiion wih oulier-sparsiy regularizaion, IEEE Trans. Signal Processing, vol. 60, no. 0, pp , Oc. 0. [53] B. K. Naarajan, Sparse approximae soluions o linear sysems, SIAM J. Compu., vol. 4, no., pp. 7 34, Apr [54] Y. Neserov, A mehod for solving he convex programming problem wih convergence rae O(/k ), Dokl. Akad. Nauk SSSR, vol. 69, no. 3, pp , 983. [55] Office of Science and Technology Policy. (0). Big daa research and developmen iniiaive. Execuive Office of he Presiden. [Online]. Available: hp:// final_.pdf [56] B. A. Olshausen and D. J. Field, Sparse coding wih an overcomplee basis se: A sraegy employed by V? Vision Res., vol. 37, no. 3, pp , 997. [57] E. E. Papalexakis, U. Kang, C. Falousos, N. D. Sidiropoulos, and A. Harpale, Large scale ensor decomposiions: Algorihmic developmens and applicaions, IEEE Daa Eng. Bull., vol. 36, no. 3, pp , Sep. 03. [58] H. Raja and W. U. Bajwa, Cloud K-SVD: Compuing daa-adapive represenaions in he cloud, in Proc. Alleron Conf. Communicaion, Conrol, and Compuing, Oc. 03, pp [59] M. Razaviyayn, M. Hong, and Z.-Q. Luo, A unified convergence analysis of block successive minimizaion mehods for nonsmooh opimizaion, SIAM J. Opim., vol. 3, no., pp. 6 53, 03. [60] H. Robbins and S. Monro, A sochasic approximaion mehod, Ann. Mah. Sais., vol., pp , Sep. 95. [6] L. K. Saul and S. T. Roweis, Think globally, fi locally: Unsupervised learning of low dimensional manifolds, J. Mach. Learn. Res., vol. 4, pp. 9 55, Dec [6] I. D. Schizas, A. Ribeiro, and G. B. Giannakis, Consensus in ad hoc WSNs wih noisy links Par I: Disribued esimaion of deerminisic signals, IEEE Trans. Signal Processing, vol. 56, no., pp , Jan [63] B. Schölkopf and A. J. Smola, Learning wih Kernels. Cambridge, MA: MIT Press, 00. [64] G. Scuari, F. Facchinei, P. Song, D. P. Palomar, and J.-S. Pang, Decomposiion by parial linearizaion: Parallel opimizaion of muli-agen sysems, IEEE Trans. Signal Processing, vol. 6, no. 3, pp [65] S. Shalev-Shwarz, Online learning and online convex opimizaion, Found. Trends Mach. Learn., vol. 4, no., pp , 0. [66] M. Signoreo, R. V. Plas, B. D. Moor, and J. A. K. Suykens, Tensor versus marix compleion: A comparison wih applicaion o specral daa, IEEE Signal Process. Le., vol. 8, pp , July 0. [67] K. Skreing and K. Engan, Recursive leas squares dicionary learning algorihm, IEEE Trans. Signal Processing, vol. 58, no. 4, pp. 30, Apr. 00. [68] K. Slavakis and G. B. Giannakis, Online dicionary learning from big daa using acceleraed sochasic approximaion algorihms, in Proc. ICASSP, Florence, Ialy, 04, pp [69] V. Solo and X. Kong, Adapive Signal Processing Algorihms: Sabiliy and Performance. Englewood Cliffs, NJ: Prenice Hall, 995. [70] M. Solanolkoabi and E. J. Candès, A geomeric analysis of subspace clusering wih ouliers, Ann. Sais., vol. 40, no. 4, pp , Dec. 0. [7] P. Sprechmann, A. M. Bronsein, and G. Sapiro, Real-ime online singing voice separaion from monaural recordings using robus low-rank modeling, in Proc. Conf. In. Sociey for Music Informaion Rerieval, Oc. 0, pp [7] N. Srebro and A. Shraibman, Rank, race-norm and max-norm, in Learning Theory. Berlin/Heidelberg: Germany: Springer, 005, pp [73] N. Sädler, D. J. Sekhoven, and P. Bühlmann, Paern alernaing maximizaion algorihm for missing daa in large p small n problems, J. Mach. Learn. Res., o be published. arxiv: v3. [74] J. M. F. en Berge and N. D. Sidiropoulos, On uniqueness in CANDECOMP/ PARAFAC, Psychomerika, vol. 67, no. 3, pp , 00. [75] S. Theodoridis, K. Slavakis, and I. Yamada, Adapive learning in a world of projecions: A unifying framework for linear and nonlinear classificaion and regression asks, IEEE Signal Process. Mag., vol. 8, no., pp. 97 3, Jan. 0. [76] G. Tomasi and R. Bro, PARAFAC and missing values, Chemom. Inell. Lab. Sys., vol. 75, no., pp , 005. [77] P. Tseng, Convergence of block coordinae decen mehod for nondiffereniable minimizaion, J. Opim. Theory Appl., vol. 09, pp , June 00. [78] R. Vidal, Subspace clusering, IEEE Signal Process. Mag., vol. 8, no., pp. 5 68, Mar. 0. [79] B. Widrow and J. M. E. Hoff, Adapive swiching circuis, IRE WESCON Conv. Rec., vol. 4, pp , Aug [80] M. Yamagishi and I. Yamada, Over-relaxaion of he fas ieraive shrinkagehresholding algorihm wih variable sepsize, Inverse Probl., vol. 7, no. 0, p , 0. [SP] IEEE SIGNAL PROCESSING MAGAZINE [3] SEPTEMBER 04

Optimal demand response: problem formulation and deterministic case

Optimal demand response: problem formulation and deterministic case Opimal demand response: problem formulaion and deerminisic case Lijun Chen, Na Li, Libin Jiang, and Seven H. Low Absrac We consider a se of users served by a single load-serving eniy (LSE. The LSE procures

More information

A Working Solution to the Question of Nominal GDP Targeting

A Working Solution to the Question of Nominal GDP Targeting A Working Soluion o he Quesion of Nominal GDP Targeing Michael T. Belongia Oho Smih Professor of Economics Universiy of Mississippi Box 1848 Universiy, MS 38677 and Peer N. Ireland Deparmen

More information

Improved Techniques for Grid Mapping with Rao-Blackwellized Particle Filters

Improved Techniques for Grid Mapping with Rao-Blackwellized Particle Filters 1 Improved Techniques for Grid Mapping wih Rao-Blackwellized Paricle Filers Giorgio Grisei Cyrill Sachniss Wolfram Burgard Universiy of Freiburg, Dep. of Compuer Science, Georges-Köhler-Allee 79, D-79110

More information

Cost-Sensitive Learning by Cost-Proportionate Example Weighting

Cost-Sensitive Learning by Cost-Proportionate Example Weighting Cos-Sensiive Learning by Cos-Proporionae Example Weighing Bianca Zadrozny, John Langford, Naoki Abe Mahemaical Sciences Deparmen IBM T. J. Wason Research Cener Yorkown Heighs, NY 0598 Absrac We propose

More information

Towards Optimal Capacity Segmentation with Hybrid Cloud Pricing

Towards Optimal Capacity Segmentation with Hybrid Cloud Pricing Towards Opimal Capaciy Segmenaion wih Hybrid Cloud Pricing Wei Wang, Baochun Li, and Ben Liang Deparmen of Elecrical and Compuer Engineering Universiy of Torono Torono, ON M5S 3G4, Canada,

More information

Forecasting Electricity Prices

Forecasting Electricity Prices Forecasing Elecriciy Prices Derek W. Bunn 1 and Nekaria Karakasani London Business School 2003 v1 Absrac This is a review paper documening he main issues and recen research on modeling and forecasing elecriciy

More information

2009 / 2 Review of Business and Economics. Federico Etro 1

2009 / 2 Review of Business and Economics. Federico Etro 1 The Economic Impac of Cloud Compuing on Business Creaion, Employmen and Oupu in Europe An applicaion of he Endogenous Marke Srucures Approach o a GPT innovaion Federico Ero ABSTRACT Cloud compuing is a

More information

World-trade web: Topological properties, dynamics, and evolution

World-trade web: Topological properties, dynamics, and evolution PHYSICAL REVIEW E 79, 365 29 World-rade web: Topological properies, dynamics, and evoluion Giorgio Fagiolo* Laboraory of Economics and Managemen, San Anna School of Advanced Sudies, Piazza Mariri della

More information

Follow the Leader If You Can, Hedge If You Must

Follow the Leader If You Can, Hedge If You Must Journal of Machine Learning Research 15 (2014) 1281-1316 Submied 1/13; Revised 1/14; Published 4/14 Follow he Leader If You Can, Hedge If You Mus Seven de Rooij VU Universiy and

More information

EMBARGO: December 4th, 2014, 11am Pacific/2pm Eastern/7pm UK. The Social Bayesian Brain: Does Mentalizing Make a Difference When We Learn?

EMBARGO: December 4th, 2014, 11am Pacific/2pm Eastern/7pm UK. The Social Bayesian Brain: Does Mentalizing Make a Difference When We Learn? EMBARGO: December 4h, 2014, 11am Pacific/2pm Easern/7pm UK The Social Bayesian Brain: Does Menalizing Make a Difference When We Learn? Marie Devaine 1,2, Guillaume Hollard 3,4, Jean Daunizeau 1,2,5 * 1

More information

Basic Life Insurance Mathematics. Ragnar Norberg

Basic Life Insurance Mathematics. Ragnar Norberg Basic Life Insurance Mahemaics Ragnar Norberg Version: Sepember 22 Conens 1 Inroducion 5 1.1 Banking versus insurance...................... 5 1.2 Moraliy............................... 7 1.3 Banking................................

More information

The U.S. Treasury Yield Curve: 1961 to the Present

The U.S. Treasury Yield Curve: 1961 to the Present Finance and Economics Discussion Series Divisions of Research & Saisics and Moneary Affairs Federal Reserve Board, Washingon, D.C. The U.S. Treasury Yield Curve: 1961 o he Presen Refe S. Gurkaynak, Brian

More information

Anchoring Bias in Consensus Forecasts and its Effect on Market Prices

Anchoring Bias in Consensus Forecasts and its Effect on Market Prices Finance and Economics Discussion Series Divisions of Research & Saisics and Moneary Affairs Federal Reserve Board, Washingon, D.C. Anchoring Bias in Consensus Forecass and is Effec on Marke Prices Sean

More information


EDUCATION POLICIES AND STRATEGIES EDUCATION POLICIES AND STRATEGIES Naional Educaion Secor Developmen Plan: A resul-based planning handbook 13 Educaion Policies and Sraegies 13 Educaion Policies and Sraegies 13 Naional Educaion Secor Developmen

More information

Making a Faster Cryptanalytic Time-Memory Trade-Off

Making a Faster Cryptanalytic Time-Memory Trade-Off Making a Faser Crypanalyic Time-Memory Trade-Off Philippe Oechslin Laboraoire de Securié e de Crypographie (LASEC) Ecole Polyechnique Fédérale de Lausanne Faculé I&C, 1015 Lausanne, Swizerland

More information

Today s managers are very interested in predicting the future purchasing patterns of their customers, which

Today s managers are very interested in predicting the future purchasing patterns of their customers, which Vol. 24, No. 2, Spring 25, pp. 275 284 issn 732-2399 eissn 1526-548X 5 242 275 informs doi 1.1287/mksc.14.98 25 INFORMS Couning Your Cusomers he Easy Way: An Alernaive o he Pareo/NBD Model Peer S. Fader

More information

The power and size of mean reversion tests

The power and size of mean reversion tests Journal of Empirical Finance 8 493 535 www.elsevier.comrlocaereconbase he power and size of mean reversion ess Ken Daniel ) Kellogg School of Managemen, Norhwesern UniÕersiy, Sheridan Road, EÕanson, IL

More information

The concept of potential output plays a

The concept of potential output plays a Wha Do We Know (And No Know) Abou Poenial Oupu? Susano Basu and John G. Fernald Poenial oupu is an imporan concep in economics. Policymakers ofen use a one-secor neoclassical model o hink abou long-run

More information

Are Under- and Over-reaction the Same Matter? A Price Inertia based Account

Are Under- and Over-reaction the Same Matter? A Price Inertia based Account Are Under- and Over-reacion he Same Maer? A Price Ineria based Accoun Shengle Lin and Sephen Rasseni Economic Science Insiue, Chapman Universiy, Orange, CA 92866, USA Laes Version: Nov, 2008 Absrac. Theories

More information

On Valuing Equity-Linked Insurance and Reinsurance Contracts

On Valuing Equity-Linked Insurance and Reinsurance Contracts On Valuing Equiy-Linked Insurance and Reinsurance Conracs Sebasian Jaimungal a and Suhas Nayak b a Deparmen of Saisics, Universiy of Torono, 100 S. George Sree, Torono, Canada M5S 3G3 b Deparmen of Mahemaics,

More information

Dynamic Portfolio Choice with Deferred Annuities

Dynamic Portfolio Choice with Deferred Annuities 1 Dynamic Porfolio Choice wih Deferred Annuiies Wolfram Horneff * Raimond Maurer ** Ralph Rogalla *** 200_final_Horneff, e al Track E Financial Risk (AFIR) Absrac We derive he opimal porfolio choice and

More information

Bid-Price Control for Energy-Aware Pricing of Cloud Services

Bid-Price Control for Energy-Aware Pricing of Cloud Services Bid-Price Conrol for Energy-Aware Pricing of Cloud Services Marc Premm Universiy of Hohenheim, Deparmen of Informaion Sysems 2, Sugar, Germany Absrac. The amoun of elecrical

More information

Preliminary. Comments welcome. Equity Valuation Using Multiples

Preliminary. Comments welcome. Equity Valuation Using Multiples Preliminary. Commens welcome. Equy Valuaion Using Muliples Jing Liu Anderson Graduae School of Managemen Universy of California a Los Angeles (310) 206-5861 Doron Nissim Columbia

More information

When Is Growth Pro-Poor? Evidence from a Panel of Countries

When Is Growth Pro-Poor? Evidence from a Panel of Countries Forhcoming, Journal of Developmen Economics When Is Growh Pro-Poor? Evidence from a Panel of Counries Aar Kraay The World Bank Firs Draf: December 2003 Revised: December 2004 Absrac: Growh is pro-poor

More information

Firms as Buyers of Last Resort

Firms as Buyers of Last Resort Firms as Buyers of Las Resor Harrison Hong Princeon Universiy Jiang Wang MIT and CCFR Jialin Yu Columbia Universiy Firs Draf: May 005 This Draf: April 007 Absrac: We develop a model o explore he asse pricing

More information

Research. Michigan. Center. Retirement

Research. Michigan. Center. Retirement Michigan Universiy of Reiremen Research Cener Working Paper WP 2006-124 Opimizing he Reiremen Porfolio: Asse Allocaion, Annuiizaion, and Risk Aversion Wolfram J. Horneff, Raimond Maurer, Olivia S. Michell,

More information

Is China Over-Investing and Does it Matter?

Is China Over-Investing and Does it Matter? WP/12/277 Is China Over-Invesing and Does i Maer? Il Houng Lee, Muraza Syed, and Liu Xueyan 2012 Inernaional Moneary Fund WP/12/277 IMF Working Paper Asia and Pacific Deparmen Is China Over-Invesing and

More information

Central Bank Communication: Different Strategies, Same Effectiveness?

Central Bank Communication: Different Strategies, Same Effectiveness? Cenral Bank Communicaion: Differen Sraegies, Same Effeciveness? Michael Ehrmann and Marcel Frazscher * European Cenral Bank, November 2004 Absrac The paper

More information

Uncertainty and International Banking *

Uncertainty and International Banking * Uncerainy and Inernaional Banking * Claudia M. Buch (Deusche Bundesbank) Manuel Buchholz (Halle Insiue for Economic Research) Lena Tonzer (Halle Insiue for Economic Research) July 2014 Absrac We develop

More information

The Pasts and Futures of Private Health Insurance in Australia

The Pasts and Futures of Private Health Insurance in Australia The Pass and Fuures of Privae Healh Insurance in Ausralia Casey Quinn NCEPH Working Paper Number 47 W O R K I N G P A P E R S NATIONAL CENTRE FOR EPIDEMIOLOGY AND POPULATION HEALTH Naional Cenre for Epidemiology

More information