A Common Neural Nework Model for Unsupervsed Exploraory Daa Analyss and Independen Componen Analyss Keywords: Unsupervsed Learnng, Independen Componen Analyss, Daa Cluserng, Daa Vsualsaon, Blnd Source Separaon. Mark Grolam, Andrzej Cchock, and Shun-Ich Amar RIKEN Bran Scence Insue Laboraory for Open Informaon Sysems & Laboraory for Informaon Synhess Hrosawa -1, Wako-sh, Saama 351-01, Japan. el (+81) 48 467-9669 Fax (+81) 48 467-9687 {ca, amar}@bran.rken.go.jp gro@open.bran.rken.go.jp Currenly on Secondmen from Deparmen of Compung and Informaon Sysems he Unversy of Pasley Pasley, PA1 BE, Scoland el (+44) 141 848 3963 Fax (+44) 848 3404 gro0c@pasley.ac.uk IEEE Membershp Number: 4016669 Manuscrp Number NN#333 Acceped for I.E.E.E ransacons on Neural Neworks BRIEF PAPER Submed :13 08-97 Acceped : 8 07-98 Correspondng Auhor 1
Absrac hs paper presens he dervaon of an unsupervsed learnng algorhm, whch enables he denfcaon and vsualsaon of laen srucure whn ensembles of hgh dmensonal daa. hs provdes a lnear projecon of he daa ono a lower dmensonal subspace o denfy he characersc srucure of he observaons ndependen laen causes. he algorhm s shown o be a very promsng ool for unsupervsed exploraory daa analyss and daa vsualsaon. Expermenal resuls confrm he aracveness of hs echnque for exploraory daa analyss and an emprcal comparson s made wh he recenly proposed Generave opographc Mappng (GM) and sandard prncpal componen analyss (PCA). Based on sandard probably densy models a generc nonlneary s developed whch allows boh; 1) denfcaon and vsualsaon of dchoomsed clusers nheren n he observed daa and, ) separaon of sources wh arbrary dsrbuons from mxures, whose dmensonaly may be greaer han ha of number of sources. he resulng algorhm s herefore also a generalsed neural approach o ndependen componen analyss (ICA) and s consdered o be a promsng mehod for analyss of real world daa ha wll conss of sub and super-gaussan componens such as bomedcal sgnals. 1. Inroducon hs paper develops a generalsaon of he adapve neural forms of he ndependen componen analyss (ICA) algorhm prmarly as a mehod for neracve unsupervsed daa exploraon, cluserng and vsualsaon. he ICA ransformaon has araced a grea deal of research focus n an aemp o solve he sgnal-processng problem of blnd source separaon (BSS) [1,, 3, 6, 7, 8, 10, 14, 15, 16, 18]. However, he work repored n hs paper has been movaed by unsupervsed daa exploraon and daa vsualsaon. Unsupervsed sascal analyss for classfcaon or cluserng of daa s a subjec of grea neres when no classes are defned a pror. he projecon pursu (PP) mehodology as dealed n [13], was developed o seek or pursue P dmensonal projecons of muldmensonal daa, whch would maxmse some measure of sascal neres, where P = 1 or o enable vsualsaon. Projecon pursu herefore provdes a means of laen daa srucure denfcaon hrough vsualsaon [13].
he lnk wh projecon pursu and ndependen componen analyss (ICA) s dscussed n [18], and a neural mplemenaon of projecon pursu s developed and ulsed for BSS n [15]. I s argued [18] ha he maxmal ndependence ndex of projecon offered by ICA bes descrbes he fundamenal naure of he daa. hs s of course n accordance wh he laen varable model of daa [1], whch nvarably assumes ha he laen varables are orhogonal.e. ndependen. A sochasc graden-based algorhm s developed whch s shown, ulmaely, o be an exenson of he naural graden famly of algorhms proposed n [1, ]. he paper has he followng srucure; Secon nroduces he ICA or laen varable model of daa and brefly revews he ICA daa ransformaon. Secon 3 presens he dervaon of he algorhm for daa vsualsaon and ICA. In Secon 4 he classcal Pearson Mxure of Gaussans (MOG) densy model [19] for cluser denfcaon s ulsed as he non-lnear erm for he developed algorhm. I s found ha hs provdes an elegan closed form generc acvaon funcon, whch also provdes a mehod of separang arbrary mxures of non-gaussan sources. Secon 5 repors on a daa exploraon smulaon and a source separaon expermen. he concludng secon dscusses he poenal exensons of he proposed approach.. he Independen Componen Analyss Daa Model he parcular ICA daa model consdered n hs paper s defned as follows () As() n() x = + (1) N he observed zero mean daa vecor s real valued such ha x () R he vecor of M underlyng ndependen sources or facors s gven as s () R such ha M N, and due o source ndependence he mulvarae jon densy s facorable M () p ( s ) p s = =. he nose vecor n() s assumed o be Gaussan wh a dagonal 1 covarance marx E{ nn } R nn =, E denoes expecaon, where he varance of each nose componen s usually assumed as consan such ha R nn σ n = I. he unknown real valued marx N M A R s desgnaed he mxng marx n ICA leraure [6] or n facor analyss, he facor loadng marx [1]. Our objecve s hen o fnd a lnear ransformaon y = Wx () 3
P N wh W R where P << N ypcally wh P = for vsualsaon purposes, such ha he elemens of y are as non-gaussan and muually ndependen as possble. he objecve of sandard ICA s o recover all or some of he orgnal source sgnals s ( ), or, ndeed, o exrac one specfc source, when only he observaon vecor x ( ) s avalable [16]. Alernavely he objecve may be o esmae he mxng marx A. In conras o hese objecves n hs paper our prmary ask s no o esmae or exrac any specfc source sgnals s ( ) bu raher o cluser he daa no logcal groupngs and allow her vsualsaon. he non-gaussan naure of he margnal componens of s, n erms of exploraory daa analyss, s ndcave of neresng srucure such as b-modales.e clusers and nrnsc classes. hs ndcaes he poenal for ICA o be appled o he cluserng of daa and hs shall be explored furher heren. he followng secon derves an unsupervsed learnng algorhm, whch wll be capable of denfyng laen srucure whn daa. 3. A Graden Algorhm Based on Maxmal Margnal Negenropy Creron he projecon pursu mehodology, whch seeks lnear projecons of he observed daa, can be consdered as a means of seekng laen non-gaussan srucure whn he observaons. In many ways he ICA model whch assumes non-gaussan sources can be a more realsc daa model han he ndependen Gaussan generaed FA model. In [13] he maxmsaon of hgher order momens s ulsed o pursue projecons ha wll denfy srucure assocaed wh he maxmsed momen.e. mulple modes or skewness. However, f he resulng momens are small hus descrbng a mesokurc (slghly devaed from Gaussan) srucure hen momen based ndces may no be suable. Marro n [13] argues ha nformaon heorec crera for maxmsaon may requre o be consdered n hs case. he mos obvous choce of an nformaon heorec measure sgnfyng deparure from Gaussan wll be negenropy, [9]. Negenropy [9] s defned as he Kullback-Lebler dvergence of a dsrbuon p y ( y ) from a Gaussan dsrbuon wh dencal mean and varance p ( ) unvarae case hs s, KL ( p p ) p ( y ) G G ( y ) ( y ) G y. In he p y = y log dy (3) p 4
where he subscrp denoes he h margnal densy of a daa vecor y. Negenropy wll always be greaer han zero and only vanshes when he dsrbuon p y ( y) s normally dsrbued [9]. hs wll hen lend self o sochasc graden ascen echnques. he dervaon of a learnng rule for a smple sngle layer srucure, whch wll drve he oupu of each neuron maxmally from normaly, s he goal of hs parcular secon. Each oupu neuron wll be parameersed ndvdually as s he nenon for each neuron o respond opmally o dfferng ndependen feaures whn he daa. hs s also n accordance wh he facoral represenaon of he densy of he underlyng sources (or laen varables) n boh he FA and ICA daa models. We hen use he facoral paramerc form for he densy of he nework oupu where y = Wx and P N W R a measure of non-gaussany for he P oupus KL ( p p ) p () y F G, P N. he followng creron s proposed as P = 1 = y G p ( y ) () y log dy (4) p where he subscrp F denoes he facored form. hs creron can hen be wren as P ( ) + p () y log p ( y ) 1 P ( p p ) = log ( e) de( R ) KL F G yy y = 1 π dy (5) he covarance marces of he observed daa and he ransformed daa R xx = E{ xx } = E yy = WR W are posve defne marces respecvely. By and R yy { } xx consderng he maxmsaon of hs creron we noe ha he wo ndvdual erms ( ) play a sgnfcan role n he emergen properes of he learnng. Consder he lefhand erm, 1 log ( π e) P de( ) Where = E { } y y R yy. he Haddamard nequaly [9] s gven as P ( ) yy de R σ (6) σ s he varance of he daa from he -h oupu. hs ndcaes ha he maxmal value of he erm wll be aaned when he covarance marx of he nework oupu s dagonalsed. hs s precsely he effec of he sphereng daa ransformaon ofen dscussed n he projecon pursu leraure [1]. Lkewse, he second erm wll be maxmal once he ransformed daa conforms o a facoral = 1 y 5
represenaon hus ensurng ha each neuron wll ndeed be respondng o dsnc ndependen underlyng characerscs of he daa. In order o derve he learnng algorhm le us compue he gradens of he proposed creron over he weghs wj of he ransformaon marx. P ( ) + E{ log p ( y )} P 1 log( e) + log de( WR ) xxw W = 1 I s neresng o noe ha he erm log( de( W ) π (7) WR xx ensures ha he rank of W s equal o P and so each of he rows of W wll be lnearly ndependen. hs s a naurally occurrng erm, whch ensures ha each oupu neuron wll seek muually ndependen drecons. Now s sraghforward o show ha 1 ( de( WR ) ( ) xxw = WRxxW WRxx log W akng he graden of he oupu enropy gves P E W = 1 { log p ( y )} = E f() y { x } he funcon f(y) operaes elemen-wse on he vecor y, such ha f () y = [ f ( y ), f ( y )] and f ( y ), M M p '( y ) ( y ) =. he fnal graden erm s hen p 1 ( p p ) = ( WR W ) WR E{ f() y x } KL F G xx xx + (10) W he sandard nsananeous graden ascen echnque can now be used n a sochasc updae equaon, however we consder ulsng he more effcen naural graden for he wegh updae [1, ]. In he parcular case under consderaon where P << N he (8) (9) square symmerc marx erm naural graden formula: W W wll be posve sem-defne so he sandard ΔW = η KL( pf pg ) W W W (11) can no be used drecly. We propose he modfed formula Δ W = η KL F G + W (1) W W + ε I s ( p p ) ( W W ε I) Where ε s a small posve consan whch ensures ha he erm ( ) always posve defne. hs approach s smlar o ha used n many opmsaon mehods o keep he nverse of he Hessan posve defne. 6
Usng (10) and (1) yelds he followng sochasc wegh updae. For small values of ( x ) xx xx + 1 ( I + f ( y ) y ) W + η ε ( WR W ) WR f () y ΔW = η (13) ε hen O( η ) O( η ε ) >> and he rghmos erm wll have a neglgble effec on he wegh updae so he learnng equaon can be approxmaed by ( I + f( y ) y ) W ΔW = η (14) he wegh updae (13) and (14) wll seek maxmally non-gaussan projecons ono lower dmensonal sub-spaces for unsupervsed exploraory daa analyss. However (14) can also be seen o be a generalsaon of he orgnal equvaran ICA algorhm [1,, 3, 6, 7, 8] for ICA, as s capable of fndng P ndependen componens n an N-dmensonal subspace. An alernave approach o hs problem s proposed n, for example, [16] where he noon of exracng sources sequenally from he observed mxure s ulsed. 4. A Mxure Model o Idenfy Laen Clusered Daa Srucure Dsrbuons ha are b-modal exhb one form of laen srucure whch s of neres n denfyng. Mulple modes may ndcae specfc clusers and classes nheren n he daa. Maxmum lkelhood esmaon (MLE) approaches o daa cluserng [11] employ Mxures of Gaussan (MOG) models o seek ranson regons of hgh and low densy and so denfy poenal daa clusers. One parcular unvarae MOG model whch s of parcular neres n hs sudy was orgnally proposed n [19]. he generc form of he Pearson model s gven below p ( y) ( 1 a) p ( y μ, σ ) ap ( y μ, ) = (15) G 1 1 + G σ where 0 a 1(see Fgure 1). I s clear ha he dsrbuon s symmerc possesng wo dsnc modes when he mxng coeffcen a = ½. For he srcly symmerc case where a = ½, 1 σ σ σ = = he above MOG densy model (15) can be wren as μ 1 = μ = μ and 1 μ y μy p ( y) = exp exp cosh (16) σ π σ σ σ 7
Employng (16) o compue he ndvdual nonlnear erms n (14) produces he followng '( y) ( y) p y μ μ y f ( y) = = + anh (17) p σ σ σ hs s a parcularly neresng form of nonlneary as he boh he lnear and hyperbolc angen erms have been suded as he acvaons for sngle layer unsupervsed neural neworks. A densy model has now been denfed wh hese parcular acvaons and now allows a probablsc nerpreaon of he hyperbolc angen acvaon funcon. o gan some nsgh regardng he sascal naure of he proposed MOG model he assocaed cumulan generang funcon (CGF) s employed [14]. he explc form of he CGF for he generc Pearson model (15) n he case where 1 σ μ 1 = μ = μ and σ σ = = s smply φ( w) = log (( 1 a) exp( A) + a exp( B) ) where A = μw σ w, B = μ w + σ w [14]. he relaed cumulans of he dsrbuon can be now be compued and afer some edous algebrac manpulaon he kuross for he dsrbuon under consderaon s ( 1 a)( 6a 6a + 1) ( 4aμ ( 1 a) σ ) 4 16a μ + (18) For he symmerc case where a = ½ hen he expresson for kuross reduces o ( μ ) 4 μ + σ (19) whch, neresngly, akes on srcly negave values for all μ > 0 [14]. hs s of parcular sgnfcance as hs dsrbuon and nonlnear funcon can also be ulsed for BSS of srcly sub-gaussan sources. For he case where wo dsnc modes are defned such ha μ = and σ = 1 (see Fgure 1) he nonlneary akes he very smple form of f ( y) = anh( y) y (0) he followng wegh adapaon wll hen seek projecons, whch denfy maxmally dchoomsed clusered srucure nheren n he daa. [ I + anh( y ) y y y ] W ΔW = η (1) An alernave densy model, whch defnes a unmodal super-gaussan densy, s gven as p( y) exp( y ) sech( y), where he normalsng consan s negleced 8
[14]. he assocaed dervave of log-densy s hen f ( y) = anh ( y) y. hs can be combned wh he nonlnear erm based on he symmerc Pearson model when μ =1 and σ = 1, yeldng, n vecor forma () y = K anh() y y f () 4 he square dagonal marx, whch conans each ndvdual oupu kuross sgn, s defned as 1 M [ sgn( κ ), sgn( κ ), ( κ )] K 4 = dag 4 4 sgn 4 (3) he kuross of each oupu can be esmaed onlne usng he followng movng average esmaor p p [ y ] = [ 1 μ ] mˆ [ y ] + y m ˆ + 1 μ (4) [ y ] mˆ [ y ] mˆ [ y ] p ( ) 3 κ ˆ (5) 4 = 4 he sample momens of order p are esmaed usng (4); n hs case he second and fourh order momens are requred. he sample kuross esmae s hen gven by (5). he generc erm () can be subsued no (14) fnally gvng ( I K ( y ) y y y ) W ΔW = η 4, anh (6) hs updae equaon can hen also be appled o general ICA where he number of oupus s less han he number of sensors. From he form of (6) s clear ha hs adapaon rule can also be ulsed o separae scalar mxures whch may conan arbrary numbers of boh sub and super-gaussan sources. he use of (1) wll specfcally seek lnear projecons denfyng b-modal and dchoomsed clusered srucure whn he daa. 5. Expermenal Resuls 5.1 Daa Vsualsaon he daase 1 used n hs expermen arses from synhec daa modellng of a nonnvasve monorng sysem whch s used o measure he relave quany of ol whn a mul-phase ppelne carryng ol, waer and gas. he daa consss of welve dmensons, whch correspond o he measuremens from sx dual powered gamma 1 hs daa se s avalable from hp://www.ncrg.ason.ac.uk/gm/3phasedaa.hml 9
ray densomeers [4]. here are hree parcular flow regmes whch may occur whn he ppelne namely lamnar, annular and homogenous. he Generave opographc mappng (GM) [4] has been appled successfully o he problem of vsualsng he laen srucure whn hs daa se and s used here as a means of comparson wh he derved adapaon rule (1). he daa s frs made zero-mean and hen sequenally presened o he nework unl he weghs acheve a seady value. A fxed learnng rae of value 0.001 was used n hs smulaon. he resuls usng he nonlnear GM mappng under he condons repored n [4] are gven n Fgure a. I s clear ha he hree clusers correspondng o he dfferen phases have been clearly denfed and separaed. In comparson o prncpal componen analyss (PCA) he resuls from GM provde consderably more dsnc separaon of he clusers correspondng o he hree flow regmes. Fgure b shows he resuls usng he adapaon rule (1), agan s clear ha he pons relang o he lamnar, annular and homogenous flow regmes have been dsncly clusered ogeher. However, s of sgnfcance o noe ha here exs wo clusers correspondng o he lamnar flow. As he proporons of each phase changes whn he lamnar flow over me here wll be a change n he physcal boundary beween he phases whch wll rgger a sep change n he across ppe beams. I s hs physcal effec whch gves rse o he dsnc clusers whn he lamnar flow. hs denfcaon of he addonal clusered srucure whn he lamnar flow requres he use of a lnear herarchcal approach o daa vsualsaon and s demonsraed n [5]. 5. Blnd Source Separaon hs smulaon focuses on mage enhancemen and s used here o demonsrae he algorhm performance when appled o ICA. he problem consss of hree orgnal source mages whch are mxed by a non-square lnear memoryless mxng marx such ha he number of mxures s greaer han he number of sources. he pxel dsrbuon of each mage s such ha wo of hem are sub-gaussan wh negave values of kuross and he oher s super-gaussan wh a posve value of kuross. he values of kuross for each mage are compued as 0.3068, -1.3753 and 0.415. I s neresng o noe ha wo of he mages (Fgure 3) have relavely small absolue values of kuross and as such are approxmaely Gaussan. hs s a parcularly dffcul problem due o he non-square mxng and he presence of boh 10
sub and super Gaussan sources whn he mxure. hs dffculy s also compounded wh he small absolue values of kuross of wo sources. he frs problem ha has o be addressed s denfyng he number of sources. Smply compung he rank of he covarance marx of he mxure can do hs. Hsorcally he nex problem would be wo-fold as he mxure consss of a number of sources whch are sub-gaussan and some whch are super-gaussan. hs of course affecs he choce of he nonlneary requred o successfully separae he sources. However, from (6) all ha s requred s o learn he dagonal erms of he K4 marx. Fgure 3, shows he observaons and he fnal separaed sources. Each value s drawn randomly from he mxure and (6) s used o updae he nework weghs. he learnng rae s kep a a fxed value of 0.0001. I should be sressed ha s no requred o make any assumpons on he ype of non-gaussan sources presen n he mxure, nor s choosng anoher form of nonlneary and changng he smple form of he algorhm requred. Fgure 3 shows he fnal separaed mages ndcang he good performance of he algorhm. 6. Conclusons By consderng an nformaon heorec ndex of projecon based on negenropy a generalsed learnng algorhm has been derved and hs may be appled o boh unsupervsed exploraory daa analyss and ndependen componen analyss wh an arbrary number of oupus. he powerful capably of hs approach for unsupervsed exploraory daa analyss has been demonsraed usng he ol ppelne daa and compared wh he probablsc (GM). hs echnque has been appled o oher classcal daa-ses such as he Irs, Crab and Swss Banknoes. In each case he nrnsc clusered naure of he daa s revealed by he use of he proposed learnng algorhm (6). In erms of ICA a parcularly dffcul mage enhancemen problem has been used o demonsrae he algorhm performance for blnd source separaon. Curren work, whch s beng addressed from he perspecve of daa analyss, s a means by whch hs echnque can be exended o a herarchcal mehod of dchoomsng and cluserng daa. Acknowledgemens M Grolam s suppored by a gran from he Knowledge Laboraory Advanced echnology Cenre, NCR Fnancal Sysems Lmed, Dundee, Scoland. We are 11
ndebed o Dr. J.F. Cardoso for helpful dscussons regardng hs work. Mark Grolam s graeful o Dr. Mchael ppng and Prof. Chrs Bshop for provdng he ol ppelne daa and gvng helpful nsghs regardng he physcal nerpreaon of he daa analyss. hs work was compleed whls Mark Grolam was an nved vsng researcher a he Laboraory for Open Informaon Sysems, Bran Scence Insue, Rken, Insue of Chemcal and Physcal Research, Wako-sh, Japan. References [1] Amar, S., Chen,, P., and Cchock, A., Sably Analyss of Learnng Algorhms for Blnd Source Separaon, Neural Neworks, Vol.10, No8, 1345-1351, 1997. [] Amar, S., Cchock, A, and Yang, H, A New Learnng Algorhm for Blnd Sgnal Separaon. Neural Informaon Processng, Vol 8, pp. 757-763. M.I. Press, 1995. [3] Bell, A and Sejnowsk,, An Informaon Maxmsaon Approach o Blnd Separaon and Blnd Deconvoluon. Neural Compuaon 7, 119 1159, 1995. [4] Bshop, C., Svensen, M., Wllams, C., GM: he Generave opographc Mappng., Neural Compuaon, Vol. 10, Number 1, pp15-34. [5] Bshop, C., and ppng, M., A Herarchcal Laen Varable Model for Daa Vsualsaon, echncal Repor NCRG/96/08, Ason Unversy, 1997. [6] Cardoso, J, F. and Laheld, B, H, Equvaren Adapve Source Separaon I.E.E.E ransacons on Sgnal Processng, SP-43, pp 3017 309, 1997. [7] Cchock, A., Unbehauen, R. and Rummer, E, Robus Learnng Algorhm for Blnd Separaon of Sgnals, Elecroncs Leers,.30, No.17, pp 1386-1387, 1994. [8] Cchock, A. and Unbehauen, R., Robus Neural Neworks Wh On-Lne learnng For Blnd Idenfcaon and Blnd Separaon of Sources, IEEE ransacons on Crcus and Sysems I: Fundamenal heory and Applcaons, Vol.43, pp 894-906. [9] Cover,. and homas, J, A, Elemens of Informaon heory, Wley Seres n elecommuncaons, 1991. [10] Douglas, S, C., Cchock, A., and Amar, S, Mulchannel Blnd Separaon and Deconvoluon of Sources wh Arbrary Dsrbuons. Proc. I.E.E.E Workshop Neural Neworks for Sgnal Processng, pp.436-444, 1997. [11] Ever, B, S., Cluser Analyss, Henemann Educaonal Books, 1993. 1
[1] Ever, B, S, An Inroducon o Laen Varable Models, London: Chapman and Hall, 1984. [13] Fredman, J. H, Exploraory Projecon Pursu. Journal of he Amercan Sascal Assocaon, 8 (397):pp 49-66, 1987. [14] Grolam, M. An Alernave Perspecve on Adapve Independen Componen Analyss Algorhms. Neural Compuaon, Vol. 10, No. 8, pp 103-114,1998. [15] Grolam, M and Fyfe, C. Exracon of Independen Sgnal Sources usng a Deflaonary Exploraory Projecon Pursu Nework wh Laeral Inhbon. I.E.E Proceedngs on Vson, Image and Sgnal Processng, Vol 14, No 5, pp 99-306, 1997. [16] Hyvarnen, A., and Oja, E, A Fxed-Pon Algorhm for Independen Componen Analyss. Neural Compuaon, Vol. 9, No. 7, pp. 1483-149, 1997. [17] Jones, M. C. and Sbson, R, Wha s Projecon Pursu. he Royal Sascal Socey. 150(1), pp. 1 36, 1987. [18] Karhunen, J., Oja, E., Wang, L., Vgaro, R., Jousensalo J, A Class of Neural Neworks for Independen Componen Analyss. IEEE ransacons on Neural Neworks, 8, pp 487 504, 1997. [19] Pearson, K., Conrbuons o he Mahemacal Sudy of Evoluon. Phl. rans. Roy. Soc. A 185, 71, 1894. 13
Fgure 1 14
Lamnar Annular Homogenous Fgure a 15
Lamnar Annular Homogenous Fgure b 16
Lamnar Annular Homogenous Fgure c 17
Orgnal Source Images Image No 1 : Kuross = -1.37 Image No : Kuross = +0.31 Image No 3 : Kuross = -0.5 A R N M Observed Mxed Images W R M N Fgure 3 18
Fgure Capons Fgure 1: Examples of he un-varae Pearson mxure model for μ = and σ = 1 and varous parameer values a. Fgure a: Plo of he poseror mean for each pon n laen space usng he GM, clearly he hree flow regmes responsble for generang he welve dmensonal measuremens have been clusered successfully. Fgure b: Plo of he welve dmensonal daa projeced ono a wo dmensonal subspace whch maxmses he negenropy of he daa whn he subspace. he hree flow regmes responsble for he measuremens have been clearly defned and n addon he clusered naure of he lamnar flow has been denfed. Fgure c: Plo of he welve dmensonal daa projeced ono he wo dmensonal subspace whose bass s he frs wo prncpal componens. he hree flow regmes responsble for he measuremens have no been clearly defned or separaed no dsnc clusers. Fgure 3 : Independen Componen Analyss performed on a 5 x 3 mxure of hree mages. One s super gaussan wh a kuross value of + 1.37 anoher has a very small value of posve kuross +0.307, wh he hrd havng a negave kuross of -0.5. he wo mages wh he small absolue values of kuross could be consdered as approxmaely mesokurc. 19