Very Sparse Random Projections

Transcription

1 Very Sare Random Projection Ping i eartment of Statitic Stanford Univerity Stanford CA 9435 USA ingli@tattanfordedu Trevor J Hatie eartment of Statitic Stanford Univerity Stanford CA 9435 USA hatie@tanfordedu Kenneth W Church Microoft Reearch Microoft Cororation Redmond WA 985 USA church@microoftcom ABSTRACT There ha been coniderable interet in random rojection an aroximate algorithm for etimating ditance between air of oint in a high-dimenional vector ace et A R n be our n oint in dimenion The method multilie A by a random matrix R R reducing the dimenion down to jut for eeding u the comutation R tyically conit of entrie of tandard normal N( ) It i well nown that random rojection reerve airwie ditance (in the exectation) Achliota rooed are random rojection by relacing the N( ) entrie in R with entrie in { } with robabilitie { } achieving a threefold eedu in roceing time We recommend uing R of entrie in { } with robabilitie { } for achieving a ignificant - fold eedu with little lo in accuracy Categorie and Subject ecritor H8 [atabae Alication]: ata Mining General Term Algorithm Performance Theory Keyword Random rojection Samling Rate of convergence INTROUCTION Random rojection [ 43] have been ued in Machine earning [4534] VSI layout [4] analyi of atent Semantic Indexing (SI) [35] et interection [7 36] finding motif in bio-equence [6 7] face recognition [6] rivacy reerving ditributed data mining [3] to name a few The AMS etching algorithm [3] i alo one form of random rojection We define a data matrix A of ize n to be a collection of n data oint {u i} n i= R All airwie ditance can Permiion to mae digital or hard coie of all or art of thi wor for eronal or claroom ue i granted without fee rovided that coie are not made or ditributed for rofit or commercial advantage and that coie bear thi notice and the full citation on the firt age To coy otherwie to reublih to ot on erver or to reditribute to lit require rior ecific ermiion and/or a fee K 6 Augut 3 6 Philadelhia Pennylvania USA Coyright 6 ACM /6/8 $5 be comuted a AA T at the cot of time O(n ) which i often rohibitive for large n and in modern data mining and information retrieval alication To eed u the comutation one can generate a random rojection matrix R R and multily it with the original matrix A R n to obtain a rojected data matrix B = AR R n min(n ) () The (much maller) matrix B reerve all airwie ditance of A in exectation rovided that R conit of iid entrie with zero mean and contant variance Thu we can achieve a ubtantial cot reduction for comuting AA T from O(n ) to O(n + n ) In information retrieval we often do not have to materialize AA T Intead databae and earch engine are intereted in toring the rojected data B in main memory for efficiently reonding to inut querie While the original data matrix A i often too large the rojected data matrix B can be mall enough to reide in the main memory The entrie of R (denoted by {r ji} i=) hould be iid with zero mean In fact thi i the only neceary condition for reerving airwie ditance [4] However different choice of r ji can change the variance (average error) and error tail bound It i often convenient to let r ji follow a ymmetric ditribution about zero with unit variance A imle ditribution i the tandard normal ie r ji N( ) E (r ji) = E `r ji = E `r4 ji = 3 It i imle in term of theoretical analyi but not in term of random number generation For examle a uniform ditribution i eaier to generate than normal but the analyi i more difficult In thi aer when R conit of normal entrie we call thi ecial cae a the conventional random rojection about which many theoretical reult are nown See the monograh by Vemala [43] for further reference We derive ome theoretical reult when R i not retricted to normal In articular our reult lead to ignificant imrovement over the o-called are random rojection Sare Random Projection In hi novel wor Achliota [] rooed uing the ro- The normal ditribution i -table It i one of the few table ditribution that have cloed-form denity [9]

2 jection matrix R with iid entrie in 8 r ji = < with rob with rob : with rob () where Achliota ued = or = 3 With = 3 one can achieve a threefold eedu becaue only of the data need 3 to be roceed (hence the name are random rojection) Since the multilication with can be delayed no floating oint arithmetic i needed and all comutation amount to highly otimized databae aggregation oeration Thi method of are random rojection ha gained it oularity It wa firt exerimentally teted on image and text data by [5] in SIGK ater many more ublication alo adoted thi method eg [ ] Very Sare Random Projection We how that one can ue 3 (eg = or even = ) to ignificantly eed u the comutation log Examining () we can ee that are random rojection are random amling at a rate of ie when = 3 one-third of the data are amled Statitical reult tell u that one doe not have to amle one-third (/3) of the data to obtain good etimate In fact when the data are aroximately normal log of the data robably uffice (ie = ) becaue of the exonential error tail log bound common in normal-lie ditribution uch a binomial gamma etc For better robutne we recommend chooing le aggreively (eg = ) To better undertand are and very are random rojection we firt give a ummary of relevant reult on conventional random rojection in the next ection CONVENTIONA RANOM PROJECTIONS: R N( ) Conventional random rojection multily the original data matrix A R n with a random matrix R R coniting of iid N( ) entrie enote by {u i} n i= R the row in A and by {v i} n i= R the row of the rojected data ie v i = R T u i We focu on the leading two row: u u and v v For convenience we denote m = u = u j m = u = u j a = u T u = u ju j d = u u = m + m a Moment It i eay to how that (eg emma 3 of [43]) E ` v = u = m Var ` v N = m (3) E ` v v = d Var ` v v N = d (4) where the ubcrit N indicate that a normal rojection matrix i ued From our later reult in emma 3 (or [8 emma ]) we can derive E v T v = a Var v T v = `mm + a (5) N Therefore one can comute both airwie -norm ditance and inner roduct in (intead of ) dimenion achieving a huge cot reduction when min(n ) itribution It i eay to how that (eg emma 3 of [43]) v i N( ) v m/ m / χ (6) v i v i d/ N( )» vi» N v i Σ = v v χ (7) d/» «m a (8) a m where χ denote a chi-quared random variable with degree of freedom v i iid i any entry in v R Knowing the ditribution of the rojected data enable u to derive (har) error tail bound For examle variou Johnon and indentrau (J) embedding theorem [49 5] have been roved for reciely determining given ome ecified level of accuracy for etimating the -norm ditance According to the bet nown reult []: If = 4+γ log n then with robability at leat ɛ / ɛ 3 /3 n γ for any two row u i u j we have ( ɛ) u i u j v i v j ( + ɛ) u i u j (9) Remar: (a) The J lemma i conervative in many alication becaue it wa derived baed on Bonferroni correction for multile comarion (b) It i only for the l ditance while many alication care more about the inner roduct A hown in (5) the variance of the inner roduct etimator Var `v T v N i dominated by the margin (ie m m ) even when the data are uncorrelated Thi i robably the weane of random rojection 3 Sign Random Projection A oular variant of conventional random rojection i to tore only the ign of the rojected data from which one can etimate the vector coine angle θ = co a m m by the following reult [7 7]: Pr (ign(v i) = ign(v i)) = θ π () One can alo etimate a by auming that m m are nown from a = co(θ) m m at the cot of ome bia The advantage of ign random rojection i the aving in toring the rojected data becaue only one bit i needed for the ign With ign random rojection we can comare vector uing hamming ditance for which efficient algorithm are available [736] See [8] for more comment on ign random rojection 3 OUR CONTRIBUTIONS We rooe very are random rojection to eed u the (roceing) comutation by a factor of or more We derive exact variance formula for v v v and v T v a function of Under reaonable regularity condition they converge to the correonding variance when r ji N( ) i ued a long a = o() [] roved the uer bound for the variance of v and v v for = and = 3

3 (eg = or even = ) When = the log rate of convergence i O which i fat ince /4 ha to be large otherwie there would be no need of eeing aroximate anwer Thi mean we can achieve a -fold eedu with little lo in accuracy We how that v i v i v i and (v i v i) converge to normal at the rate O when = Thi /4 allow u to aly with a high level of accuracy reult of conventional random rojection eg the Jembedding theorem in (9) and the ign random rojection in () In articular we ugget uing a maximum lielihood etimator of the aymtotic (normal) ditribution to etimate the inner roduct a = u T u taing advantage of the marginal norm m m Our reult eentially hold for any other ditribution of r ji When r ji i choen to have negative urtoi we can achieve trictly maller variance (error) than conventional random rojection 4 MAIN RESUTS Main reult of our wor are reented in thi ection with detailed roof in Aendix A For convenience we alway let = o() (eg = ) and aume all fourth moment are bounded eg E(u 4 j) < E(u 4 j) < and E(u ju j) < In fact analyzing the rate of convergence of aymtotic normality only require bounded third moment and an even much weaer aumtion i needed for enuring aymtotic normality ater we will dicu the oibility of relaxing thi aumtion of bounded moment 4 Moment The firt three lemma concern the moment (mean and variance) of v v v and v T v reectively emma E ` v = u = m () Var ` v = m + ( 3) u 4 j () A ie ( 3) P (uj)4 m 3 E (u j) 4 E (3) (u j) Var ` v `m (4) denote aymtotically equivalent for large Note that m P P = = u4 j+ P j j u ju j u j with diagonal term and ( ) cro-term When all dimenion of u are roughly equally imortant the croterm dominate Since i very large the diagonal term are negligible However if a few entrie are extremely large comared to the majority of the entrie the cro-term may be of the ame order a the diagonal term Auming bounded fourth moment revent thi from haening The next emma i trictly analogou to emma We reent them earately becaue emma i more convenient to reent and analyze while emma contain the reult on the -norm ditance which we will ue emma E ` v v = u u = d (5) Var ` v v = d + ( 3) (u j u j) 4 (6) `d (7) The third lemma concern the inner roduct emma 3 E v T v = u T u = a (8) Var v T v = m m + a + ( 3) u ju j (9) `mm + a () Therefore very are random rojection reerve airwie ditance in exectation with variance a function of Comared with Var( v ) N Var( v v ) N and Var(v T v ) N in (3) (4) and (5) reectively the extra term all involve ( 3) and are aymtotically negligible The rate q of convergence i O 3 in term of the tandard error (quare root of variance) When = the rate of convergence i O /4 When < 3 are random rojection can actually achieve lightly maller variance 4 Aymtotic itribution The aymtotic analyi rovide a feaible method to tudy ditribution of the rojected data The ta of analyzing the ditribution i eay when a normal random matrix R i ued The analyi for other tye of random rojection ditribution i much more difficult (in fact intractable) To ee thi each entry v i = R T i u = P rjiuj Other than the cae rji N( ) analyzing v i and v exactly i baically imoible although in ome imle cae [] we can tudy the bound of the moment and moment generating function emma 4 and emma 5 reent the aymtotic ditribution of v and v v reectively Again emma 5 i trictly analogou to emma 4 emma 4 A v i m/ = N( ) with the rate of convergence F vi (y) Φ(y) 8 r 8 v m / P uj 3 m 3/ = χ () E u j 3 `E `u j3/ () where = denote convergence in ditribution; F vi (y) i the emirical cumulative denity function (CF) of v i and Φ(y) i the tandard normal N( ) CF

4 emma 5 A v i v i d/ = N( ) with the rate of convergence F vi v i (y) Φ(y) 8 v v d/ = χ (3) P uj uj 3 d 3/ (4) The above two lemma how that both v i and v i v i are aroximately normal with the rate of convergence determined by / which i O when = and /4 The next lemma concern the joint ditribution of (v i v i) emma 6 A»» Σ vi = N v i» «(5) Pr (ign(v i) = ign(v i)) θ π (6) where Σ =» «m a θ = co a a m mm The aymtotic normality how that we can ue other random rojection matrix R to achieve aymtotically the ame erformance a conventional random rojection which are the eaiet to analyze Since the convergence rate i o fat we can imly aly reult on conventional random rojection uch a the J lemma and ign random rojection when a non-normal rojection matrix i ued 3 43 A Margin-free Etimator Recall that becaue E(v T v ) = u T u one can etimate a = u T u without bia a â = v T v with the variance Var (â ) = m m + a + ( 3) u ju j (7) Var (â ) = `mm + a (8) where the ubcrit indicate Margin-free ie an etimator of a without uing margin Var (â ) i the variance of v T v in (9) Ignoring the aymtotically negligible art involving 3 lead to Var (â ) We will comare â with an aymtotic maximum lielihood etimator baed on the aymtotic normality 44 An Aymtotic ME Uing Margin The tractable aymtotic ditribution of the rojected data allow u to derive more accurate etimator uing maximum lielihood In many ituation we can aume that the marginal norm m = P u j and m = P u j are nown 3 In the roof of the aymtotic normality we ued E( r ji 3 ) and E( r ji +δ ) They hould be relaced by the correonding moment when other rojection ditribution are ued a m and m can often be eaily either exactly calculated or accurately etimated 4 The author very recent wor [8] on conventional random rojection how that if we now the margin m and m we can etimate a = u T u often more accurately uing a maximum lielihood etimator (ME) The following lemma etimate a = u T u taing advantage of nowing the margin emma 7 When the margin m and m are nown we can ue a maximum lielihood etimator (ME) to etimate a by maximizing the joint denity function of (v v ) Since (v i v i) converge to a bivariate normal an aymtotic ME i the olution to a cubic equation a 3 a v T v + a ` m m + m v + m v m m v T v = (9) The aymtotic variance of thi etimator denoted by â ME i Var (â ME) = `mm a Var (â m m + a ) (3) The ratio Var(â ME) Var(â ) = (m m a ) = ( co (θ)) (m m +a ) (+co (θ)) range from to indicating oibly ubtantial imrovement For examle when co(θ) (ie a m m ) the imrovement will be huge When co(θ) (ie a ) we do not benefit from â ME Note that ome tudie (eg dulicate detection) are mainly intereted in data oint that are quite imilar (ie co(θ) cloe to ) 45 The Kurtoi of r ji : ( 3) We have een that the arameter lay an imortant role in the erformance of very are random rojection It i intereting that 3 i exactly the urtoi of r ji: γ (r ji) = E((rji E(rji))4 ) E 3 = 3 (3) ((r ji E(r ji)) ) a r ji ha zero mean and unit variance 5 The urtoi for r ji N( ) i zero If one i only intereted in maller etimation variance (ignoring the benefit of arity) one may chooe the ditribution of r ji with negative urtoi A coule of examle are A continuou uniform ditribution in [ l l] for any l > It urtoi = 6 5 A dicrete uniform ditribution ymmetric about zero with N oint It urtoi = 6 N + ranging between - (when N = ) and 6 (when N ) The 5 N 5 cae with N = i the ame a () with = icrete and continuou U-haed ditribution 4 Comuting all marginal norm of A cot O(n) which i often negligible A imortant ummary tatitic the marginal norm may be already comuted during variou tage of roceing eg normalization and term weighting 5 Note that the urtoi can not be maller than becaue of the Cauchy-Schwarz inequality: E (rji) E(rji) 4 One may conult htt://enwiiediaorg/wii/kurtoi for reference to urtoi of variou ditribution

5 5 HEAVY-TAI AN TERM WEIGHTING The very are random rojection are ueful even for heavy-tailed data mainly becaue of term weighting We have een that bounded forth and third moment are needed for analyzing the convergence of moment (variance) and the convergence to normality reectively The roof of aymtotic normality in Aendix A ugget that we only need tronger than bounded econd moment to enure aymtotic normality In heavy-tailed data however even the econd moment may not exit Heavy-tailed data are ubiquitou in large-cale data mining alication (eecially Internet data) [534] The airwie ditance comuted from heavy-tailed data are uually dominated by outlier ie excetionally large entrie Pairwie vector ditance are meaningful only when all dimenion of the data are more or le equally imortant For heavy-tailed data uch a the (unweighted) term-bydocument matrix airwie ditance may be mileading Therefore in ractice variou term weighting cheme are rooed eg [33 Chater 5] [ ] to weight the entrie intead of uing the original data It i well-nown that chooing an aroriate term weighting method i vital For examle a hown in [3 6] in text categorization uing uort vector machine (SVM) chooing an aroriate term weighting cheme i far more imortant than tuning ernel function of SVM See imilar comment in [37] for the wor on Naive Baye text claifier We lit two oular and imle weighting cheme One variant of the logarithmic weighting ee zero entrie and relace any non-zero count with +log(original count) Another cheme i the quare root weighting In the ame irit of the Box-Cox tranformation [44 Chater 68] thee variou weighting cheme ignificantly reduce the urtoi (and ewne) of the data and mae the data reemble normal Therefore it i fair to ay that auming finite moment (third or fourth) i reaonable whenever the comuted ditance are meaningful However there are alo alication in which airwie ditance do not have to bear any clear meaning For examle P u 4 j uing random rojection to etimate the joint ize (et interection) If we exect the original data are everely heavy-tailed and no term weighting will be alied we recommend uing = O() Finally we hall oint out that very are random rojection can be fairly robut againt heavy-tailed data when = For examle intead of auming finite fourth moment a long a ( P grow lower than O( ) u j) we can till achieve the convergence of variance if = in emma Similarly analyzing the rate of converge to P u j 3 normality only require that ( P grow lower u j) 3/ than O( /4 ) An even weaer condition i needed to only enure aymtotic normality We rovide ome additional analyi on heavy-tailed data in Aendix B 6 EXPERIMENTA RESUTS Some exerimental reult are reented a a anity chec uing one air of word THIS and HAVE from two row of a term-by-document matrix rovided by MSN = 6 = That i u j (u j) i the number of occurrence of word THIS (word HAVE) in the jth document log (Web age) j = to Some ummary tatitic are lited in Table The data are certainly heavy-tailed a the urtoe for u j and u j are 95 and 5 reectively far above zero Therefore we do not exect that very are random rojection with = 6 wor well though the reult are actually not diatrou a hown in Figure (d) Table : Some ummary tatitic of the word air THIS (u ) and HAVE (u ) γ denote the E(u urtoi η(u j u j) = j u j ) affect the convergence of Var `v T v E(u j )E(u j )+E (u j u j ) (ee the roof of emma 3) Thee exectation are comuted emirically from the data Two oular term weighting cheme are alied The quare root weighting relace u j with u j and the logarithmic weighting relace any non-zero u j with + log u j Unweighted Square root ogarithmic γ (u j) γ (u j) E(u 4 j ) E (u j ) E(u 4 j ) E (u j ) η(u j u j) co(θ(u u )) We firt tet random rojection on the original (unweighted heavy-tailed) data for = 3 56 = and 6 log reented in Figure We then aly quare root weighting and logarithmic weighting before random rojection The reult are reented in Figure for = 56 and = 6 Thee reult are conitent with what we would exect: When i mall ie O() are random rojection erform very imilarly to conventional random rojection a hown in anel (a) and (b) of Figure With increaing the variance of are random rojection increae With = the error are large log (but not diatrou) becaue the data are heavy-tailed With = are random rojection are robut Since co(θ(u u )) 7 8 in thi cae marginal information can imrove the etimation accuracy quite ubtantially The aymtotic variance of â ME match the emirical variance of the aymtotic ME etimator quite well even for = After alying term weighting on the original data are random rojection are almot a accurate a conventional random rojection even for a hown in Figure log 7 CONCUSION We rovide ome new theoretical reult on random rojection a randomized aroximate algorithm widely ued in machine learning and data mining In articular our theoretical reult ugget that we can achieve a ignificant -fold eedu in roceing time with little lo in accuracy where i the original data dimenion When the data

6 Standard error Standard error ME Theor Theor (a) = ME Theor Theor (c) = 56 Standard error Standard error ME Theor Theor 5 5 (b) = 3 ME Theor Theor (d) = 6 Figure : Two word THIS (u ) and HAVE (u ) from the MSN Web crawl data are teted = 6 Sare random rojection are alied to etimated a = u T u with four value of : 3 56 = and 6 in anel (a) (b) (c) and (d) log reectively reented in term of the normalized Var(â) tandard error 4 imulation are conducted for each ranging from to There a are five curve in each anel The two labeled a and Theor overla tand for the emirical variance of the Margin-free etimator â ; while Theor for the theoretical variance of â ie (7) The olid curve labeled a ME reent the emirical variance of â ME the etimator uing margin a formulated in emma 7 There are two curve both labeled a Theor for the aymtotic theoretical variance of â (the higher curve (8)) and â ME (the lower curve (3)) are free of outlier (eg after careful term weighting) a cot reduction by a factor of i alo oible log Our roof of the aymtotic normality jutifie the ue of an aymtotic maximum lielihood etimator for imroving the etimate when the marginal information i available 8 ACKNOWEGMENT We than imitri Achliota for very inightful comment We than Xavier Gabaix and avid Maon for ointer to ueful reference Ping i than the enjoyable and helful converation with Tze eung ai Joeh P Romano and Yiyuan She Finally we than the four anonymou reviewer for contructive uggetion 9 REFERENCES [] imitri Achliota atabae-friendly random rojection: Johnon-indentrau with binary coin Journal of Comuter and Sytem Science 66(4): Standard error Standard error ME Theor Theor (a) Square root ( = 56) ME Theor Theor (c) Square root ( = 6) Standard error Standard error ME Theor Theor (b) ogarithmic ( = 56) ME Theor Theor (d) ogarithmic ( = 6) Figure : After alying term weighting on the original data are random rojection are almot a accurate a conventional random rojection even for = 6 Note that the legend are the log ame a in Figure [] imitri Achliota Fran McSherry and Bernhard Schölof Samling technique for ernel method In Proc of NIPS age Vancouver BC Canada [3] Noga Alon Yoi Matia and Mario Szegedy The ace comlexity of aroximating the frequency moment In Proc of STOC age 9 PhiladelhiaPA 996 [4] Roa Arriaga and Santoh Vemala An algorithmic theory of learning: Robut concet and random rojection In Proc of FOCS (Alo to aear in Machine earning) age New Yor 999 [5] Ella Bingham and Heii Mannila Random rojection in dimenionality reduction: Alication to image and text data In Proc of K age 45 5 San Francico CA [6] Jeremy Buhler and Martin Toma Finding motif uing random rojection Journal of Comutational Biology 9():5 4 [7] Moe S Chariar Similarity etimation technique from rounding algorithm In Proc of STOC age Montreal Quebec Canada [8] G P Chityaov and F Götze imit ditribution of tudentized mean The Annal of Probability 3(A): [9] Sanjoy aguta and Anuam Guta An elementary roof of a theorem of Johnon and indentrau Random Structure and Algorithm (): [] Suan T umai Imroving the retrieval of information from external ource Behavior Reearch Method Intrument and Comuter 3(): [] Richard urrett Probability: Theory and Examle uxbury Pre Belmont CA econd edition 995 [] William Feller An Introduction to Probability Theory and It Alication (Volume II) John Wiley & Son New Yor NY econd edition 97 [3] Xiaoli Zhang Fern and Carla E Brodley Random

7 rojection for high dimenional data clutering: A cluter enemble aroach In Proc of ICM age Wahington C 3 [4] mitriy Fradin and avid Madigan Exeriment with random rojection for machine learning In Proc of K age 57 5 Wahington C 3 [5] P Franl and H Maehara The Johnon-indentrau lemma and the hericity of ome grah Journal of Combinatorial Theory A 44(3): [6] Navin Goel George Bebi and Ara Nefian Face recognition exeriment with random rojection In Proc of SPIE age Bellingham WA 5 [7] Michel X Goeman and avid P Williamon Imroved aroximation algorithm for maximum cut and atifiability roblem uing emidefinite rogramming Journal of ACM 4(6): [8] F Götze On the rate of convergence in the multivariate CT The Annal of Probability 9(): [9] Piotr Indy Stable ditribution eudorandom generator embedding and data tream comutation In FOCS age Redondo BeachCA [] Piotr Indy and Rajeev Motwani Aroximate nearet neighbor: Toward removing the cure of dimenionality In Proc of STOC age alla TX 998 [] W B Johnon and J indentrau Extenion of ichitz maing into Hilbert ace Contemorary Mathematic 6: [] Samuel Kai imenionality reduction by random maing: Fat imilarity comutation for clutering In Proc of IJCNN age Picataway NJ 998 [3] Man an Chew im Tan Hwee-Boon ow and Sam Yuan Sung A comrehenive comarative tudy on term weighting cheme for text categorization with uort vector machine In Proc of WWW age 3 33 Chiba Jaan 5 [4] Erich ehmann and George Caella Theory of Point Etimation Sringer New Yor NY econd edition 998 [5] Will E eland Murad S Taqqu Walter Willinger and aniel V Wilon On the elf-imilar nature of Ethernet traffic IEEE/ACM Tran Networing (): [6] Edda eoold and Jorg Kindermann Text categorization with uort vector machine how to rereent text in inut ace? Machine earning 46(-3): [7] Henry CM eung Franci Y Chin SM Yiu Roni Roenfeld and WW Tang Finding motif with inufficient number of trong binding ite Journal of Comutational Biology (6): [8] Ping i Trevor J Hatie and Kenneth W Church Imroving random rojection uing marginal information In Proc of COT Pittburgh PA 6 [9] Jeica in and imitrio Gunoulo imenionality reduction by random rojection and latent emantic indexing In Proc of SM San Francico CA 3 [3] Bing iu Yiming Ma and Phili S Yu icovering unexected information from your cometitor web ite In Proc of K age San Francico CA [3] Kun iu Hillol Karguta and Jeica Ryan Random rojection-baed multilicative data erturbation for rivacy reerving ditributed data mining IEEE Tranaction on Knowledge and ata Engineering 8():9 6 6 [3] B F ogan C Mallow S O Rice and A She imit ditribution of elf-normalized um The Annal of Probability (5): [33] Chri Manning and Hinrich Schutze Foundation of Statitical Natural anguage Proceing The MIT Pre Cambridge MA 999 [34] M E J Newman Power law areto ditribution and zif law Contemorary Phyic 46(5): [35] Chrito H Paadimitriou Prabhaar Raghavan Hiao Tamai and Santoh Vemala atent emantic indexing: A robabilitic analyi In Proc of POS age SeattleWA 998 [36] eea Ravichandran Patric Pantel and Eduard Hovy Randomized algorithm and NP: Uing locality enitive hah function for high eed noun clutering In Proc of AC age 6 69 Ann Arbor MI 5 [37] Jaon Rennie awrence Shih Jaime Teevan and avid R Karger Tacling the oor aumtion of naive Baye text claifier In Proc of ICM age Wahington C 3 [38] Ozgur Sahin Aziz Gulbeden Fatih Emeçi ivyaant Agrawal and Amr El Abbadi Prim: indexing multi-dimenional data in networ uing reference vector In Proc of ACM Multimedia age Singaore 5 [39] Gerard Salton and Chri Bucley Term-weighting aroache in automatic text retrieval Inf Proce Manage 4(5): [4] I S Shiganov Refinement of the uer bound of the contant in the central limit theorem Journal of Mathematical Science 35(3): [4] Chunqiang Tang Sandhya warada and Zhichen Xu On caling latent emantic indexing for large eer-to-eer ytem In Proc of SIGIR age Sheffield UK 4 [4] Santoh Vemala Random rojection: A new aroach to VSI layout In Proc of FOCS age Palo Alto CA 998 [43] Santoh Vemala The Random Projection Method American Mathematical Society Providence RI 4 [44] William N Venable and Brian Riley Modern Alied Statitic with S Sringer-Verlag New Yor NY fourth edition [45] Clement T Yu K am and Gerard Salton Term weighting in information retrieval uing the term reciion model Journal of ACM 9(): APPENIX A PROOFS et {u i} n i= denote the row of the data matrix A R n A rojection matrix R R conit of iid entrie r ji: Pr(r ji = ) = Pr(r ji = ) = Pr(rji = ) = E(r ji) = E(r ji) = E(r 4 ji) = E( r 3 ji ) = E (r ji r j i ) = E `r ji r j i = when i i or j j We denote the rojected data vector by v i = R T u i For convenience we denote m = u = u j m = u = a = u T u = u j u ju j d = u u = m + m a We will alway aume = o() E(u 4 j) < E(u 4 j) < ( E(u ju j) < ) By the trong law of large number P ui j P (ujuj)j E u I j P (uj uj)i E (u j u j) I E (u ju j) J a I = 4 J =

8 A Moment The following exanion are ueful for roving the next three lemma m m = m = a = u j u j = u ju j + u ju j j j uj = u 4 j + X u ju j u ju j = u ju j + X u ju ju j u j emma A E ` v = u = m Var ` v = ( 3) P (uj)4 m m + ( 3) 3 u 4 j E (u j) 4 E (u j) Proof of emma v = R T u et R i be the i th column of R i We can write the i th element of v to be v i = R T P i u = (rji) uj Therefore vi `r ji u j + X (r ji) u j (r j i) u j A from which it follow that E `vi = u j E ` v = u j = m vi 4 `r ji u j + X (r ji) u j (r j i) u j A `r4 ji u4 j + P j<j `r ji u j `r = P +4 P (r ji) u j (r j i) u j +4 P j i u j `r ji u j P (r ji) u j (r j i) u j from which it follow that E `v i 4 u 4 j + 6 X u ju A j Var `v i = Var ` v u 4 j + 6 X u ju j X u j ) u 4 j + 4 X u ju A j = m + ( 3) m + ( 3) u 4 j u 4 j C A A A ( 3) P (uj)4 = 3 P (uj)4 / m m / o() E (u j) 4 E (u j) emma E ` v v = u u = d Var ` v v = A d + ( 3) ( 3) P (uj uj)4 3 d (u j u j) 4 E (u j u j) 4 E (u j u j) Proof of emma The roof i analogou to the roof of emma emma 3 E v T v = u T u = a Var v T v = A ( 3) P u ju j m m + a 3 m m + a + ( 3) u ju j E `u ju j E `u j E `u j + E (u ju j) Proof of emma 3 v iv i `r ji uju j + X (r ji) u j (r j i) u j A j j v iv i = E (v iv i) = u ju j E v T v = a `r ji uju j + X (r ji) u j (r j i) u j A j j P `r4 ji u j u j+ = P j<j `r ji uju j `r j i uj u j + P j j (r ji) u j (r j i) u j + P `r ji uju j P j j (r ji) u j (r j i) u j C A

9 = E `v iv i u ju j + 4 X u ju ju j u j + X u ju A j j j ) u ju j + X u ju j + a A j j = m m + ( 3) u ju j + a Var (v iv i) = m m + a + ( 3) Var v T v = m m + a + ( 3) A Aymtotic itribution emma 4 A v i m/ = N( ) with the rate of convergence F vi (y) Φ(y) 8 r 8 v m / P uj 3 m 3/ u ju j u ju j = χ E u j 3 `E `u j3/ where = denote convergence in ditribution F vi (y) i the emirical cumulative denity function (CF) of v i and Φ(y) i the tandard normal N( ) CF Proof of emma 4 The indeberg central limit theorem (CT) and the Berry-Eeen theorem are needed for the roof [ Theorem VIII43 and XVI5] 6 Write v i = R T i u with z j = (r ji) u j Then E(z j) = Var(z j) = u j = P et = P P Var(zj) = u j indeberg condition Then (r ji) u j = P zj E( zj +δ ) = δ uj +δ δ > (+δ)/ = m E `zj ; z j ɛ for any ɛ > P zj = vi m/ = N( ) Aume the 6 The bet Berry-Eeen contant 795 ( 8) i from [4] which immediately lead to v i m / = χ v m / = X i= v «i = χ m / We need to go bac and chec the indeberg condition E `zj «zj ; z j ɛ +δ E (ɛ ) δ δ = ɛ δ P uj +δ / P u j / (+δ)/ «δ o() E u j +δ ɛ δ `E(u j )(+δ)/ rovided E u j +δ < for ome δ > which i much weaer than our aumtion that E(u 4 j) < It remain to how the rate of convergence uing the Berry- Eeen theorem et ρ = P P E zj 3 = / 3/ uj 3 F vi (y) Φ(y) 8 ρ = 8 emma 5 A v i v i d/ 3 r 8 = N( ) with the rate of convergence F vi v i (y) Φ(y) 8 r 8 P uj 3 m 3/ E u j 3 `E `u j3/ v v d/ = χ P uj uj 3 d 3/ E u j u j 3 E 3 (u j u j) Proof of emma 5 The roof i analogou to the roof of emma 4 The next lemma concern the joint ditribution of (v i v i) emma 6 A»» vi = N v i Σ and» «Σ =» m a a m Pr (ign(v i) = ign(v i)) θ «π θ = a co mm Proof of emma 6 We have een that Var (v i) = m Var (v i) = m E (vivi) = a ie» «vi cov =» m a = Σ v i a m The indeberg multivariate central limit theorem [8] ay»»» «Σ vi = N v j

10 The multivariate indeberg condition i automatically atified by auming bounded third moment of u j and u j A trivial conequence of the aymtotic normality yield Pr (ign(v i) = ign(v i)) θ π Strictly eaing we hould write θ = co E(u j u q j) E(u j)e(u j) A3 An Aymtotic ME Uing Margin emma 7 Auming that the margin m and m are nown and uing the aymtotic normality of (v i v i) we can derive an aymtotic maximum lielihood etimator (ME) which i the olution to a cubic equation a 3 a v T v + a ` m m + m v + m v m m v T v = enoted by â ME the aymtotic variance of thi etimator i Var (â ME) = `mm a m m + a Proof of emma 7 For notational convenience we treat (v i v i) a exactly normally ditributed o that we do not need to ee trac of the convergence notation The lielihood function of {v i v i} i= i then li {v i v i} i= = (π) Σ ex X» ˆ vi vi v i Σ v i where Σ = i=» m a a m We can then exre the log lielihood function l(a) a log li {v i v i} i= l(a) = log `m m a m m a X `v i m v iv ia + vim i= The ME equation i the olution to l (a) = which i a 3 a v T v + a ` m m + m v + m v m m v T v = The large amle theory [4 Theorem 63] ay that â ME i aymtotically unbiaed and converge in ditribu- a where I(a) tion to a normal random variable N the exected Fiher Information i I(a) mm + a I(a) = E `l (a) = (m m a ) after ome algebra Therefore the aymtotic variance of â ME would be `mm a Var (â ME) = m m + a (3) «B HEAVY-TAIE ATA We illutrate that very are random rojection are fairly robut againt heavy-tailed data by a Pareto ditribution The aumtion of finite moment ha imlified the analyi of convergence a great deal For examle auming (δ + )th moment < δ and = o() we have () δ/ P uj +δ P δ/ P +δ/ = uj +δ / (u j ) P +δ/ (u j )/ +δ δ/ E `uj (33) `E `u j+δ/ Note that δ = correond to the rate of convergence for the variance in emma and δ = correond to the rate of convergence for aymtotic normality in emma 4 From the roof of emma 4 in Aendix A we can ee that the convergence of (33) (to zero) with any δ > uffice for achieving aymtotic normality For heavy-tailed data the fourth moment (or even the econd moment) may not exit The mot common model for heavy-tailed data i the Pareto ditribution with the denity function 7 f(x; α) = α whoe mth moment = α only x α+ α m defined if α > m The meaurement of α for many tye of data are available in [34] For examle α = for the word frequency α = 4 for the citation to aer α = 5 for the coie of boo old in the US etc For imlicity we aume that < α + δ 4 Under thi aumtion the aymtotic normality i guaranteed and it remain to how the rate of convergence of moment and ditribution In thi cae the econd moment E `u j exit The um P uj +δ grow a O (+δ)/α a hown in [ Examle 74] 8 Thu we can write P δ/ uj +δ P «δ/ +δ/ = O (u j ) +δ +δ/ α 8 < O δ = = 4/α / (34) : O δ = 3 6/α from which we can chooe uing rior nowledge of α For examle uoe α = 3 and = (34) indicate that the rate of convergence for variance would be O( / ) in term of the tandard error (34) alo verifie that the rate of convergence to normality i O( /4 ) a exected Of coure we could alway chooe more conervatively eg = /4 if we now the data are everely heavy-tailed Since i large a factor of /4 i till coniderable What if α <? The econd moment no longer exit The analyi will involve the o-called elf-normalizing um [8 3]; but we will not delve into thi toic In fact it i not really meaningful to comute the l ditance when the data do not even have bounded econd moment 7 Note that in general a Pareto ditribution ha an addition arameter x min and f(x; α x min) = αx min with x x x min Since we are only intereted in the relative α+ ratio of moment we can without lo of generality aume x min = Alo note that in [34] their α i equal to our α + 8 Note that if x Pareto(α) then x t Pareto(α/t)