Speeding up k-means Clustering by Bootstrap Averaging

Transcription

1 Speedg up -meas Clusterg by Bootstrap Averagg Ia Davdso ad Ashw Satyaarayaa Computer Scece Dept, SUNY Albay, NY, USA,. {davdso, Abstract K-meas clusterg s oe of the most popular clusterg algorthms used data mg. However, clusterg s a tme cosumg tas, partcularly wth the large data sets foud data mg. I ths paper we show how bootstrap averagg wth -meas ca produce results comparable to clusterg all of the data but much less tme. The approach of bootstrap (samplg wth replacemet) averagg cossts of rug -meas clusterg to covergece o small bootstrap samples of the trag data ad averagg smlar cluster cetrods to obta a sgle model. We show why our approach should tae less computato tme ad emprcally llustrate ts beefts. We show that the performace of our approach s a mootoc fucto of the sze of the bootstrap sample. However, owg the sze of the bootstrap sample that yelds as good results as clusterg the etre data set remas a ope ad mportat questo.. Itroducto ad Motvato Clusterg s a popular data mg tas [] wth - meas clusterg beg a commo algorthm. However, sce the algorthm s ow to coverge to local optma of ts loss/objectve fucto ad s sestve to tal startg postos [8] t s typcally restarted from may tal startg postos. Ths results a very tme cosumg process ad may techques are avalable to speed up the -meas clusterg algorthm cludg preprocessg the data [], parallelzato [3] ad tellgetly settg the tal cluster postos [8]. I ths paper we propose a alteratve approach to speedg up -meas clusterg ow as bootstrap averagg. Ths approach s complmetary to other speed-up techques such as parallelzato. Our approach bulds multple models by creatg small bootstrap samples of the trag set ad buldg a model from each, but rather tha aggregatg le baggg [4], we average smlar cluster ceters to produce a sgle model that cotas clusters. I ths paper we shall focus o bootstrap samples that are smaller tha the trag data sze. Ths produces results that are comparable wth multple radom restartg of -meas clusterg usg all of the trag data, but taes far less computato tme. For example, whe we tae T bootstrap samples of sze 5% of the trag data set the the techque taes at least four tmes less computato tme but yelds as good as results f we had radomly restarted -meas T tmes usg all of the trag data. To test the effectveess of bootstrap averagg, we apply clusterg two popular settgs: fdg represetatve clusters of the populato ad predcto. Our approach yelds a speedup for two reasos. Frstly, we are clusterg less data ad secodly because the -meas algorthm coverges (usg stadard tests) more qucly for smaller data sets tha larger data sets from the same source/populato. It s mportat to ote that we do ot eed to re-start our algorthm may tmes for each bootstrap sample. Our approach s superfcally smlar to Bradley ad Fayyad s tal pot refemet (IPR) [8] approach that: ) sub-samples the trag data, ) clusters each sub-sample, 3) clusters the resultat cluster ceters may tmes to geerate refed tal startg postos for -meas. However, we shall show there are ey dffereces to ther clever alteratve to radomly choosg startg postos. We beg ths paper by troducg the -meas algorthm ad explore ts computatoal behavor. I partcular we show ad emprcally demostrate why clusterg smaller sets of data leads to faster covergece tha clusterg larger sets of data from the same data source/populato. We the troduce our bootstrap averagg algorthm after whch we dscuss our expermetal methodology ad results. We show that for bootstrap samples of less sze tha the orgal trag data set our approach performs as well as stadard techques but far less tme. We the dscuss the related Bradley ad Fayyad techque IPR ad dscuss dffereces to our ow wor. Fally, we coclude ad dscuss future wor.. Bacgroud to -meas Clusterg Cosder a set of data cotag staces/observatos each descrbed by m attrbutes.

2 The -meas clusterg problem s to dvde the staces to clusters wth the clusters parttog the staces (x x ) to the subsets Q. The subsets ca be summarzed as pots (C ) the m dmesoal space, commoly ow as cetrods or cluster ceters, whose co-ordates are the average of all pots belogg to the subset. We shall refer to ths collecto of cetrods obtaed from a applcato of the clusterg algorthm as a model. K-meas clusterg ca also be thought of as vector quatzato wth the am beg to mmze the vector quatzato error (also ow as the dstorto) show equato ( ). The mathematcal trval soluto s to have a cluster for each stace but typcally <<. VQ = = D( x, C f ( x ) ),where D s a dstace fucto ad f(x) returs the closet cluster dex tostace ( ) What s ow today as the -meas clusterg algorthm was postulated a umber of papers the etee sxtes [5][6]. The algorthm s extremely popular appearg leadg commercal data mg sutes offered by SAS, SPSS, SGI, ANGOSS [7]. Typcally the tal cetrod locatos are determed by assgg staces to a radomly chose cluster though alteratves exst [8]. After tal cluster cetrod placemet the algorthm cossts of the two followg steps that are repeated utl covergece. As the soluto coverged to s sestve to the startg posto the algorthm s typcally restarted may tmes. ) The assgmet step: staces are placed the closest cluster as defed by the dstace fucto. f ( x ) = arg m D( x, C j ) j ) The re-estmato step: the cluster cetrods are recalculated from the staces assged to the cluster. x C j = f ( x ) = j Q j These two steps repeat utl the re-estmato step leads to mmal chages cetrod values. Throughout ths paper we use the verso of the - meas clusterg algorthm commoly foud data mg applcatos. We use the Eucldea dstace, radomly assg staces to clusters to obta tal cluster locatos ad ormalze each attrbutes value to be betwee 0 ad by dvdg by the attrbutes rage. The -meas algorthm performs gradet descet o the dstorto error surface ad hece coverges to a local optmum of ts loss fucto (the dstorto). Covergece ca be measured a umber of ways such as the sum of the chages the cluster cetrods betwee adjacet teratos does ot exceed some very small umber epslo ( ths paper 0-5 ). 3. Computatoal Complexty of -meas Clusterg As before, let be the umber of staces to cluster, m be the umber of attrbutes/colums for each stace ad the umber of clusters. The t s well ow that f the algorthm performs teratos the the algorthm complexty s O(m) [9]. Whle, m ad are ow before the algorthm begs executo, as stated earler the algorthm s typcally ru utl covergece occurs, meag that, s, at vocato, uow. Of course ca be predetermed f the test for covergece s abadoed, but ths s computatoally effcet. For the rest of ths secto we derve results showg that (umber of teratos) s drectly proportoal to (the umber of staces) for a gve data source ad stadard tests of covergece. That s, smaller data sets wll coverge less teratos tha larger data sets draw from the same populato. Wthout loss of geeralty, we assume a uvarate data set ad Eucldea dstace. We frst state the dstorto whch -meas tres to mmze, tae the dervatve of ths expresso wth respect to the cluster cetrod value, set to 0 ad solve to derve the expresso that mmzes the dstorto. We fd that ths s as expected, the cluster cetrod update expresso as show below. VQ = VQ 0 δ ( f ( x ), ).( C = K, f ( x ) = x ) [( C ) ] t C t x x tae frst order dervatve, set to zero ad solve VQ = [ ] C t x t C, f ( x ) = C = = t = K t ( C, f ( x ) = = Q VQ = C t x, f ( x ) = Q x,, f ( x ) = t + x ) ( )

3 We ow derve a expresso to calculate the chage a cluster cetrod betwee tme t- ad t below: t C = C C = ( x ) C Q, ft ( x ) = = Q, ft ( x ) = as the summato t t t ( x C ) occurs Q tmes. ( 3 ) A smlar aalyss to our ow has bee performed whle llustratg that -meas s performg Newto s gradet descet wth a learg rate versely related to the cluster sze [0] as llustrated equato ( 3 ). Furthermore, from equato ( 4 ) we see that the sze of cluster cetrod chage s depedet o the chage the umber of staces assged to a cluster adjacet teratos, that s the umber of staces assged to at tme t but ot t- plus those ot assged to at tme t- but assged at tme t. For a gve data source, the larger the data set the more lely the codto assocated wth the frst summato equato ( 4 ) wll occur towards the ed of the - meas ru as our expermets wll llustrate later ths secto. Whether mproved tests of covergece that cosder the data set sze may remove ths pheomeo remas a ope questo., f ft ( x ) = ft ( x ) = ad f ft ( x ) = ft ( x ) = the C therefore : ( x ), f t ( x ) f t ( x ), f t ( x ) = or f t ( x ) = C = δ ( ft ( x ), ) + δ ( ft ( x ), ) ( 4 ), f t ( x ) f t ( x ), f t ( x ) f t ( x ) where δ ( a, b) =, whe a = b, zero otherwse = 0, We ow emprcally llustrate our earler clam that the umber of teratos utl covergece s related to data set sze o a umber of real world data sets as show Table, Table ad Table 3. These tables measure the average umber of teratos utl covergece occurs agast creasg data set szes. Covergece occurs f the chage (betwee adjacet teratos) clusters cetrod postos whe summed across all attrbutes ad clusters s less tha 0-5. Other covergece tests were appled such as: the chage cluster ceters s less tha a percetage of the smallest dstace betwee two clusters, that oe cetrod s chages s below epslo or that o chages cluster ceters locatos t t occurred, that s, Q = Q but our fdgs dd ot dffer sgfcatly. We report average results for 00 expermets (radom restarts). It s mportat to ote that for a partcular data set the 00 uque radom startg postos are detcal regardless of data set sze. That s, we start the algorthm from the same 00 startg postos regardless of dataset sze. Each smaller data set s a subset of the larger data sets. Average Iteratos No. 0% 5% 50% 75% 00% Dgt Pma Image Table. The average umber of teratos for -meas to coverge for a varety of data sets. Note that = ad results are average over 00 expermets (radom restarts). Average Iteratos No. 0% 5% 50% 75% 00% Dgt Pma Image Table. The average umber of teratos for -meas to coverge for a varety of data sets. Note that =4 ad results are average over 00 expermets (radom restarts). Average Iteratos No. 0% 5% 50% 75% 00% Dgt Pma Image Table 3. The average umber of teratos for -meas to coverge for a varety of data sets. Note that =6 ad results are average over 00 expermets (radom restarts). Fgure dagrammatcally llustrates our expermets showg that for larger values of that the umber of teratos mootocally creases as the data set sze creases.

4 Bag Sze Agast Number of Iteratos Utl Covergece - = Bag Sze Agast Number of Iteratos Utl Covergece - =4 5 Average Number of Iteratos Dgt Pma Image Average Number of Iteratos Dgt Pma Image 4 0% 5% 50% 75% 00% 7 0% 5% 50% 75% 00% Sze of Bag (% of Trag Data Set) Sze of Bag (% of Trag Data Set) Bag Sze Agast Number of Iteratos Utl Covergece - =6 Average Number of Iteratos % 5% 50% 75% 00% Sze of Bag (% of Trag Data Set) Dgt Pma Image Fgure. The average umber of teratos for -meas to coverge for a varety of data sets for =, =4 ad =6. Results are average over 00 expermets (radom restarts). To llustrate the pot that for a gve source of data, clusterg smaller data sets leads to faster covergece we map the trajectory the cluster algorthm taes through the stace space for dfferet sze data sets startg from the same startg posto. To vsualze these trajectores we wll reduce the dgt data set to two dmesos. Ths correspods to the data set represetg the startg posto where the pe wrtg the dgt was placed oto the tablet (a excellet predctor for the dgt type). Some example trajectores are show Fgure ad Fgure 3. I the 00 trals the average umber of teratos for the 500 stace data set s 4.33 ad for 000 staces 9.. Over the trals oly 6 trals was the umber of teratos for the larger data set less tha for the smaller data set. The fgures llustrate that for both data sets the cetrods qucly move to approxmately the same locato but the larger data set taes loger to coverge. Ths s so because for larger data sets the codto assocated wth the frst summato equato ( 4 ) occurs more ofte towards the ed of the -meas ru (as see the rght-had fgures above, t s more crowded ear the fal cluster cetrod postos). Ths result s cosstet wth the results of Mee et al [] whch foud that whe calculatg a learg curve for mxture models that allowg the algorthm to reach covergece was ot requred, the results obtaed after a few teratos was suffcet to determe the shape of the learg curve Cluster Cluster Cluster 3 Cluster 4 Cluster Cluster Cluster 3 Cluster 4 Fgure. The trajectory of four cluster cetrods through the stace space. The top fgure s for 500 staces (7 teratos), the bottom for 000 staces (3 teratos). Startg postos are crcled.

5 Cluster Cluster Cluster 3 Cluster 4 Cluster Cluster Cluster 3 Cluster 4 Fgure 3. The trajectory of four cluster cetrods through the stace space. The top fgure s for 500 staces (5 teratos), the bottom for 000 staces (8 teratos). Startg postos are crcled. 4. Descrpto of Our Approach We ow dscuss our algorthm. Note oly a sgle model s bult from each bootstrap sample as -meas s oly ru oce o each sample. Algorthm: Bootstrap Averagg Iput: D:Trag Data,T: Number of bags, K: Number of clusters Output: A: The averaged cetrods. // Geerate ad cluster each bag For = to T X = BootStrap(D) C = -meas-cluster(x,k) // Note C s the set of cluster // cetrods ad C = {c, c c K ) EdFor // Group smlar clusters to bs // wth the b averages stored B // B ther szes are S S For = to T For j = to K Idex = AssgToB(c j ) //See secto o sgature based comparso B Idex += c j EdFor EdFor For = to K B /= S A = B EdFor Our approach volves averagg smlar cluster cetrods. We propose the followg geeral-purpose method that s used throughout the paper, however, practce problem specfc approaches may be better. 4. Sgature Based Cluster Groupg For each cluster cetrod we create a sgature that ca be geerated qucly ad group clusters accordg to the sgature. We use the postos of the attrbutes ad ther values to create a sgature of the form: Sgature(c j )=Σ l c jl * l, where c jl s the l th attrbute for the j th cluster of the th model. As each attrbute s scaled to be betwee zero ad oe ths creates a sgature wth the rage 0 tll m+ as there are m attrbutes. After the sgature from each cluster s derved we sort them ascedg order ad dvde them to equally sze tervals to form the groups. Throughout ths paper we use ths method. I the future we pla to explore the feasblty of usg the approaches proposed by Strehl ad Ghosh [] that volves combg multple parttos/clusters. 5. How Much Our Approach Speeds up Clusterg If we average T bootstrap samples of sze /s of the trag data the we expect our techque to obta as good results as performg T radom restarts but less computato tme. As descrbed earler we expect a speedup for two reasos. The speedup due to clusterg less data wll be of magtude s sce the relatve complexty of the clusterg process wll be O(TmI) versus O(Tm/sI) assumg the same umber of teratos utl covergece. However, as dscussed earler for dfferet szed data sets from the same populato/source the tme to covergece s ot the same, typcally I /s <I, ths provdes a eve greater speed up whch the ext set of expermets quatfy. 6. Expermetal Results The frst use of our approach s ts ablty to fd more accurate estmates of the geeratg mechasm. To test ths ablty we eed to artfcally create data to ow the actual geeratg mechasm. We created a artfcal data set of sx clusters wth sx attrbutes. All attrbutes are Gaussas wth a mea of zero ad a stadard devato of 0.5 except the th attrbute of the th cluster whch has a mea of oe ad a stadard devato of 0.5. Formally, C Ge ={c c 6 }, c = {c j =, f =j, otherwse 0, j = to 6}. We ca measure the goodess of a clusterg soluto by ts Eucldea

6 dstace to these geeratg mechasms. We geerated 3000 staces from ths dstrbuto ffty tmes. For each sample we too 0 bootstrap samples, bult a sgle model from each ad averaged the cluster cetrods to produce a sgle model. We group smlar cluster cetrods usg the method descrbed earler. The bootstrap averagg approach taes at least fve tmes less computato tme tha clusterg all of the data after performg 0 radom restarts. If the umber of radom restarts s creased to 50 ad eve 00 the best model (mmum dstorto) from the radom restart approach stll does ot yeld better results tha bootstrap averagg. The dstace to the true cluster ceters s show Fgure 4. Ths fgure shows that for 0% bootstrap sample szes that the averaged model s further away tha the best model from radom restarts. However, for 0% bootstrap sample sze the averaged model s closer to the geeratg mechasm. Determg the precse sze of the bootstrap sample whe performace s as good as clusterg the etre data set remas a ope questo. KL Dstace to Geeratg Mechasm KL Dstace to Geeratg Mechasm Dstace to Geeratg Mechasm Versus Trag Set Sze Bags of 0% of Orgal Trag Set Sze Trag Set Sze Dstace to Geeratg Mechasm Versus Trag Set Sze Bags of 0% of Orgal Trag Set Sze Trag Set Sze Averaged Model Best Model Averaged Model Best Model Fgure 4. Trag data sze agast mea dstace (over 50 samples) to the geeratg mechasm for averaged model (0 bags) ad best sgle model foud from 0 radom restarts for the etre data set. The top graph s for bootstrap samples of sze 0% ad the bottom graph 0% of the etre data set. The decrease computato tme by usg the bootstrap averagg approach s ever less tha fve tmes. We ow show results for the sze of the bootstrap sample agast predctve accuracy. Though predcto s ot a commo applcato of clusterg, t allows us to show the performace of our approach o real world problem where we do ot ow the geeratg mechasm or true model. We fd as before that averagg smaller bootstraps yelds as accurate results as radom restarts o all of the data but less computato tme. I all expermets we drew 50 radom samples from the data, dvded ths data to a trag set (70%) ad test set (30%). For each data set, we radomly restarted -meas 50 tmes ad selected the model that mmzed the dstorto. We compare ths model agast the model obtaed by averagg over 50 bootstrap samples. Dgt Data Set: The accuracy of the averaged model ad computato tme for the dgt data set (predctg 3 or 8, the most dffcult dgt predcto decso ths data set) for dfferet szed bootstrap samples are show Fgure 5. The best model foud from all of the data after the radom restarts has a mea accuracy of 3.70% ad the total computato tme s CPU mutes compared to the averaged model s accuracy 30.8% foud approxmately 5 CPU mutes whe usg 5% of data sze bootstrap sample. Note that all our results we report the user tme reported by the Lux tme commad. Ths correspods to the amout of CPU tme that was dedcated to the process apart from system erel calls whch was eglgble all case (uder 0.0secods). We fd that smlar speedups hold for other data sets. However, the sze of the reduced bags whe the accuracy of the averaged model s the same as obtaed from radom restarts from all of the data vares betwee data sets. Image ad Letter Datasets: We see Fgure 6 ad Fgure 7 that for the Image ad Letter data sets that the computato speedup s approxmately a factor of 3.98 ad 3.54 respectvely. Ths s so as the bootstrap sample sze must crease to 40% to obta a acceptable accuracy. Determg the correct sze of the reduce bag remas a ope questo ad we hope to explore lterature from the learg curve area [] to address ths questo our future wor. Our results dcate that the sze of the bootstrap sample sze s a mootoc fucto of the averaged model s dstace to the geeratg mechasm (true model). Ths s a advatage over the wor by Bradley ad Fayyad where the sze of the sub-samples leads to dfferet results [secto 3., 8].

7 Error Versus Bag Sze For Averaged M odel CPU M utes Versus Bag Sze for Averaged M odel 50% 48% 46% 44% 4% 40% 38% 36% 34% 3% 30% % 0% 5% 0% 5% % 0% 5% 0% 5% Fgure 5. Dgt Data Set. Trag data sze agast predctve error (over 50 samples) (left graph) ad computato tme (rght graph) for the averaged model. Note: Usg the etre data set the computato tme s CPU mutes ad accuracy s 3.7% as compared to 30.8% approxmately 5 CPU mutes whe usg 5% of data sze bootstrap sample Error Versus Bag Sze For Averaged Model CPU Mutes Versus Bag Sze for Averaged Model 50% 48% 46% 44% 4% 40% 38% 36% 34% 3% 30% % 0% 5% 30% 40% % 0% 5% 30% 40% Fgure 6. Image Data Set. Trag data sze agast predctve error (over 50 samples) (left graph) ad computato tme (rght graph) for the averaged model. Note: Usg the etre data set the computato tme s 3.5 CPU mutes ad accuracy s 3.7%, as compared to accuracy of aroud 3.9% ad 5 CPU mutes whe usg 40% of data sze bootstrap sample. Error Versus Bag Sze For Averaged M odel CPU M utes Versus Bag Sze for Averaged M odel 40% 38% 36% 34% 3% 30% 8% 6% 4% % 0% % 0% 5% 30% 40% % 0% 5% 30% 40% Fgure 7. Letter Data Set. Trag data sze agast predctve error (over 50 samples) (left graph) ad computato tme (rght graph) for the averaged model. Note: Usg the etre data set the computato tme s 8. CPU mutes ad accuracy s 0.6%, as compared to accuracy of aroud.0% ad 5. CPU mutes whe usg 40% of data sze bootstrap sample. Table 4 shows a summary table of the stuato whe the bootstrap averaged accuracy equals the accuracy of clusterg f all the data s used. Ths table shows the expected speed up ([ / colum ] * [colum 3 / colum 4], see secto 5 for detals) ad the actual speedup. The two umbers dffer as the expected speed up does ot cosder the extra overhead such as the tme requred to geerate the bootstrap samples. We dd ot attempt to optmze our code ad hece expect the real dfferece betwee the expected ad actual fgures to be closer.

8 Bootstrap Sample Sze Requred Table 4. A summary of the statstcs where the bootstrap averaged accuracy approxmately equals the accuracy f all the data s clustered. 7. Dscusso We have show that: Radom Restarts: Ave. Number of Iteratos Bootstrap Averagg: Ave. Number of Iteratos Expected (Actual) Speed Up Dgt 5% (4.63) Image 40% (3.98) Letter 40% (3.54). Bootstrap averagg T subsets of the data set wll typcally be more computatoally effcet as radomly restartg the algorthm T tmes o the etre data set. (Secto 3, Table, Table ad Table 3). Bootstrap averagg o a proporto of the dataset ca yeld as accurate results as clusterg the etre data set (see Fgure 4): Ths produces results that are comparable wth radom restartg of -meas clusterg usg all of the trag data, but taes far less computato tme. 3. The results of bootstrap averagg mootocally mprove as a fucto of the bootstrap sample sze (see Fgure 5, Fgure 6 ad Fgure 7): As we crease the sze of the bootstrap sample, the accuracy mproves, utl at some pot the accuracy s comparable to that whe the etre dataset s used. Determg the sze of the bags at whch the averaged model performs as well as clusterg the etre data set vares from data set to data set hece remas a ope questo. A vald questo s: how s our approach related to the IPR approach. For the rest of ths secto we emprcally show that the two approaches obta qute dfferet results. 7. Is Bootstrap Averagg a Geeralzato of IPR We beg by showg that the bootstrap averagg approach heretly does ot produce a model that mmzes the dstorto. That s, we are ot compesatg for some search effcecy of the - meas algorthm as the IPR approach s effectvely dog, stead we are mmzg aother loss fucto as Table 5 dcates. The IPR approach attempts to fd a good set of tal postos ad the apples the -meas clusterg algorthm to further mmze the dstorto. The secod colum ths table s the result of applyg bootstrap averagg. The frst colum was geerated by talzg the clusterg algorthm to the averaged model ad clusterg the etre trag data. We fd that the -meas algorthm that performs a gradet descet o the dstorto error surface fds aother model that further mmzes the dstorto but yelds worse performace. The averaged models are statstcally sgfcatly better at the 95% cofdece level. Startg from Averaged Averaged Model Model Dgt 7.9% (5.6) 3.6% (.3) Image 3.3% (4.6) 4.3% (4.) Pma 3.3% (3.) 7.9% (3.3) Letter 3.7% (4.) 7.5% (.3) Abaloe 5.6% (5.) 9.9% (3.) Adult 33.4% (7.) 5.% (4.5) Table 5. Collecto of data sets. The average ad paretheses stadard devato test set error statstcs for the predctve ablty of models foud by startg -meas from the averaged model ad the averaged model for 50 radom dvsos of trag (70%) ad test (30%) sets. I our ext set of expermets we show that bootstrap averagg performs qute dfferetly to IPR. To llustrate ths pot clearly, we show that for bootstrap samples of equal sze to the trag data set that the results that -meas wth IPR coverges to s qute dfferet from the bootstrap averaged model. I Table 6 the averaged model s sgfcatly better (at the 95% cofdece level) tha -meas wth IPR. The frst colum refers to the predcto ablty of the model mmzg the dstorto from 00 radom restarts of the algorthm o the orgal trag data. The secod colum refers to the predcto ablty of the model mmzg dstorto from 00 IPR selected startg solutos o the orgal trag data. The fal colum (our approach) refers to the sgle model that s the average of all 0 bags. Predctg sex of abaloe

9 Radom Restart IPR Restart Averaged Model Dgt 30.6% (6.7) 9.5% (4.4) 3.6% (.3) Image 35.5% (9.8) 9.3% (5.) 4.3% (4.) Pma 33.5% (4.5) 3.% (3.) 7.9% (3.3) Letter.0% (6.4).3% (3.) 7.5% (.3) Abaloe 5.3% (7.9).4% (4.5) 9.9% (3.) Adult 34.3% (0.3) 8.9% (5.4) 5.% (4.5) Table 6. Collecto of data sets. The average ad paretheses stadard devato test set error statstcs for the predctve ablty of models foud by applyg -meas a varety of stuatos over 50 radom dvsos of trag (70%) ad test (30%) sets. 8. Cocluso K-Meas clusterg s a popular but tme cosumg algorthm used data mg. It s tme cosumg as t coverges to a local optmum of ts loss fucto (the dstorto) ad the soluto coverged to s partcularly sestve to the tal startg postos. As a result ts typcal use practce volves applyg the algorthm from may radomly chose tal startg postos. I ths paper we explore a approach we term bootstrap averagg. Bootstrap averagg bulds multple models by creatg small bootstrap samples of the trag set ad buldg a sgle model from each, smlar cluster ceters are the averaged to produce a sgle model that cotas clusters. If we average T bags of sze /s of the etre data set the our approach taes less tme tha radomly restartg the algorthm T tmes by a factor of at least s. Kowg the value of s where the averaged model performs as well as clusterg the etre data set vares betwee data sets. The speedup s because the computatoal complexty of -meas s lear wth respect to the umber of data pots. I practce we fd that the speedup our approach provdes s fact greater tha s sce the stadard test for covergece (the chage cluster cetrods s less tha some small umber, epslo) do ot cosder the sze of the trag data set. Our results dcate that the umber of teratos of the algorthm utl covergece s proportoal to the sze of the data set. I future wor we wll explore developg tests of covergece that factor the data set sze. Predctg dgt 3 or 8 Our emprcal results show that bootstrap samplg ca acheve comparable results as clusterg all of the data but less computato tme. We perform expermets to measure a clusterg model s results two ways: ) dstace to the geeratg mechasm ad ) predctve ablty. However, owg the sze of the sample that performs as well as clusterg the etre data set remas a ope questo. We hope to explore usg the learg curve lterature to determe potetal ways to address ths questo. Our research emprcally shows that clusterg bgger data sets s ot always desrable. No doubt that clusterg large data sets offer addtoal beefts amely producg better results, but our expermets dcate that bootstrappg smaller portos of the dataset ca produce ths beeft as well but at a reducto computato tme. 9. Refereces [] Ha J., ad Kamber M., Data Mg: Cocepts ad Techques, Morga Kauffma, 000. [] Pelleg, D. ad Moore, A, "Acceleratg Exact - meas Algorthms wth Geometrc Reasog", KDD-99, Proc. of the Ffth ACM SIGKDD Iter. Cof. O Kowledge Dscovery ad Data Mg, page [3] Dhllo, I. S. ad Modha, D. M., A Data Clusterg Algorthm o Dstrbuted Memory Multprocessors, Large-Scale Parallel Data Mg, Lecture Notes Artfcal Itellgece, Volume 759, pages 45-60, 000. [4] L. Brema. Baggg predctors. Mache Learg, 6():3-40, 996. [5] MacQuee, J., Some Methods for classfcato ad aalyss of multattrbute staces, Ffteth Bereley Symposum o Mathematcs, Statstcs ad Probablty, vol, 967. [6] Max, J., Quatzg for Mmum Dstorto, IEEE Trasactos o Iformato Theory, 6, pages 7-, 960 [7] H. Edelste, The Two Crows Report: 999. Avalable at [8] P. Bradley, U. Fayyad, Refg Ital Pots for -meas Clusterg. ICML 998. [9] Hartga, J., Clusterg Algorthms, Wley Publshg, 975. [0] Bottou, L., ad Bego, Y., Covergece propertes of the -meas algorthm. I G. Tesauro ad D. Touretzy, edtors, Adv. Neural Ifo. Proc.

10 Systems, volume 7, pages MIT Press, Cambrdge MA, 995. [] A. Strehl ad J. Ghosh. Cluster esembles - a owledge reuse framewor for combg multple parttos. Joural o Mache Learg Research (JMLR), 3:583-67, December 00. [] C. Mee, B. Thesso, ad D. Hecerma. The learg-curve method appled to model-based clusterg. Joural of Mache Learg Research, : , 00.