AT&Tsalesdataset:theneedistostoreamultiGiga-bytematrixon-line,withcustomersforrows,daysforcolumns,andamountspentineachcellofthematrix.

Transcription

1 DataMiningatCALD-CMU:Tools,Experiencesand C.Faloutsos,G.Gibson,T.Mitchell,A.Moore,S.Thrun CenterforAutomatedLearningandDiscovery(CALD) ResearchDirections inthecenterforautomatedlearninganddiscovery(cald)atcmu.specically, Wedescribethedataminingproblemsandsolutionsthatwehaveencountered CarnegieMellonUniversity wedescribethesesettingsandtheiroperationalcharacteristics,describeourproposed solutions,listtheperformanceresults,andnallyoutlinefutureresearchdirections. Abstract 1Introduction TheCenterforAutomatedLearningandDiscovery(CALD)isacross-disciplinarycenter atcmu,focusingontheresearchquestion\howcanhistoricaldatabebestusedtoimprovefuturedecisions?"participantsincaldaredrawnfromdiversebackgrounds,such ascomputerscience(andspecically,articialintelligence,databases,theory),robotics, mationretrieval,andlanguageprocessing.thecenterinvolvesindustrialpartnerswith challengingdata-miningproblems. Statistics,Neurology,Philosophy,Engineering(Electrical,Civil,andMechanical),Infor- 2SomeRealApplications andnallylistfutureresearchdirections. Nextwedescribetwosettingsthatwehaveencounteredsofar,andwhichseemtypicalin Inthispaperwedescribesomeofthesesettings,recentresearchprogressinourcenter, dataminingenvironments: TheAUTONproject( 1

2 AT&Tsalesdataset:theneedistostoreamultiGiga-bytematrixon-line,withcustomersforrows,daysforcolumns,andamountspentineachcellofthematrix.In wantisalossycompressionmethod,carefullydesignedtoallowfast(andasaccurate learningcontrolleranactivelearningcapabilityinwhichitautonomouslydesignsits withsubstantialeconomicsavings.currentworkisextendingautontogivethe ownconservativeexperiments. cessesinafoodmanufacturingindustry(bagging,packaging,coolingandcooking) system(usingalgorithmsdescribedin[2,3])hasbeendeployedinanumberofpro- moredetail,thespecicationsareasfollows:thenumberofrowsisverylarge:105; thenumberofcolumnsismuchsmaller(103columns);randomaccesstoeachcell isessential.ontheotherhand,approximateanswersareacceptable;thus,whatwe 3RecentResearchAdvancesinCALD ValueDecomposition(SVD). lution,basedonapowerfultechniquefromstatisticsandmatrixalgebra,thesingular aspossible)reconstructionofanycellorcells.inthenextsectionwedescribeourso- ThissectiondescribesseveralrecentresearchdevelopmentswithinCALD.Specically,the AD-treemethodforfastmanipulationofcontingencytables;aprobabilisticreasoningapproachanditsperformanceonaroboticssetting;andalossycompressionmethodforlarge 3.1AD-tree datamatrices,whichsupportsrandomaccesstoarbitrarycells,withsmallerrorandhigh compressionratio.eachtopicisdiscussedindetailbelow. ProblemManydataminingandmachinelearningalgorithmsneedtodovastnumbersof bersofcontingencytables.acontingencytable(alsoknownasa\datacube"[18]inthe countingqueries(e.g."howmanyrecordshavecolor=blue,nationality=british,smoker=false, couldtakevaluesfred,green,bluegandifanattributecalled\smoker"couldtake Databasecommunity)isdenedbyasetofattributes.Acontingencytablehasonerowfor valuesfyes,nogthenthecontingencytableforattribute-setfcolor,smokergwouldbe Status=Married,Nose=Big?").Similarly,manyalgorithmsneedtobuildandtesthugenum- eachpossiblesetofvaluesthatthesetofattributesmaytake.ifanattributecalled\color" 2

3 ColorSmokerNumberRecordsMatching GreenNo GreenYes BlueNo RedNo RedYes n1 n2 n3 n4 tableson-linewhileanalyzingthedataset[10].interactivevisualizationtoolssimilarlyneed applicationsindatamining.adatabaseusermaywishtobringupcountsorcontingency wherepini=thetotalnumberofrecords. Whydowewishtocomputecountsandcontingencytablesquickly?Therearemany BlueYes n5 n6 tocomputethesestatisticsquickly.moreimportantly,manymachinelearning,statistics anddataminingalgorithms(e.g.bayesnetbuilders,featureselectors,rulelearners, InductiveLogicProgramLearners,DecisionTreeLearners)spendmostoftheireorton countingcomputations. sizeddatastructurescanbeproducedforlargereal-worlddatasetsby(a)usingasparse that,subjecttocertainassumptions,thecostsbecomeindependentofthenumberofrecords dimensionstree)cachessucientinformationtoreconstructanycountingquery.tractably- Proposedsolution-ResultsTheADtreeapproachempiricallygivesusatwotofour orderofmagnitudespeedup(40to2000-fold)indoingcounting.analytically,wecanshow fullynearitsleaves. treestructurethatneverallocatesmemoryforcountsofzero,(b)neverallocatingmemory forcountsthatcanbededucedfromothercounts,and(c)notbotheringtoexpandthetree andloglinearinthenumberofnon-zeroentriesinthecontingencytable.theadtree(all- Net(andlearningrules)foraCensusDatasetinvolving17verynon-sparseattributes,the averagespeed-upoverdirectcountingwasapproximately1000-fold.whenbuildingabayes eightattribute-valuepaironamedicaldatabasewith10,000recordsand100attributesthe structurendingalgorithms,rulelearningalgorithms,andfeatureselectionalgorithmsona numberoflargereal-worlddatasets.forexample,forarbitrarycountingqueriesinvolving InworksofarwehaveshownhowtheADTreecanbeusedtoaccelerateBayesnet bynsfand3mcorporation)weareusingad-treestopermittractablefeature-generation tationssuchaskd-trees[5,13],r-trees[9]andfrequentsets[12].incurrentwork(funded speedupwas50-fold. algorithms(whichinventnewattributesusefulforpredictionascomplexfunctionsofthe originalattributes).wearealsoactivelyseekingcollaborationswithpeoplewithlargenance,medicineormanufacturingdatasetstowhichwemayattempttoapplyadtree-based Furtherresultsaregivenin[14],whichalsocomparesAD-treeswithalternativerepresen- 3

4 sensor-actuatorsystems.systemsequippedwithsensors(suchasrobots)areinherentlyuncertainastowhatisthecaseintheworld.thisuncertaintyusuallyarisesfromperceptual learning.furtherad-treeinformationmaybefound(shortly)atwww.cs.cmu.edu/auton. ProblemAnotherCALDprojectfocusesonstateestimationanddecisionmakingin 3.2ProbabilisticReasoning waystodealwiththisuncertaintyandtomakeoptimaldecisionsunderuncertainty. limitationsofsensorsandfromthedynamicsoftheworld.ourresearchseeksfundamental ProposedsolutionWerecentlyhavedevelopedafamilyofprobabilisticapproachesfor perceptionanddecisionmaking,whichhasbeenappliedtoavarietyofdicultrobotics problems.thekeyideaofthesemethodsisthatthesystemreasonsprobabilistically:instead theyproviderobustandmathematicallyelegantwaysfordealingwithambiguities,sensor noise,anddynamics. ofjustconsideringasingleinterpretationofwhatmightbethecaseintheworld,these ResultsThesealgorithmswereappliedtoproblemssuchasmobilerobotlocalization, methodsconsideranentirecollectionofinterpretations,annotatedbyanumericalplausibility landmarkdetectionandrecognition,mappingoflarge-scaleenvironments,andothers[15].in factor(aconditionalprobabilityasaresult,thesemethodscanrepresentuncertainty,and someofthesedomains,theprobabilisticapproachledtocompletelynewinsights,thatmade robotinthe"deutschesmuseumbonn".here,amobilerobotgaveinteractivetoursto peopleinadenselypopulatedmuseum.therobotnavigatedalmostawlesslyatatotal distanceof18.6kmandatanaveragespeedof36cm/sec,entertainingmorethan3,000visitors algorithmshasbeendemonstratedtoenablerobotstobuildmapsofunprecedentedlylarge environments[17,16].otheralgorithmswereessentialforarecentinstallationofamobile possiblesolutionsforpreviouslyunsolvedroboticsproblems.forexample,ourprobabilistic (realvisitorsandwebusers)[4].theprobabilisticalgorithmswerecriticalforposition trackingandmodelacquisition. 3.3LossyCompressionforDataMining ProblemAdhocqueryingisdicultonverylargedatasets,sinceitisusuallynotpossible thedataset,compresseddataisnotoriouslydiculttoindexoraccess. tohavetheentiredatasetondisk.whilecompressioncanbeusedtodecreasethesizeof applicationwastheat&tsalesdataset,describedearlier.eachpointinthesequenceisa Weconsideraverylargedatasetcomprisingmultipledistincttimesequences.Ourdriving 4

5 theapproximationerror,byexplicitlystoringthosedatapointsthatwere'outliers'.the ProposedmethodTheideabehindourmethod[11]istousetheso-calledSingular querying,providedthatasmallerrorcanbetoleratedwhenthedataisuncompressed. ValueDecomposition(SVD)toapproximatethedatamatrix;wewentfurthertoreduce numericalvalue.ourgoalistocompresssuchadatasetintoaformatthatsupportsadhoc resultingmethod,called`svdd'(for\svdwithdeltas")achievesallthespeciedgoals. ResultsExperimentsonlarge,realworlddatasets(AT&Tcustomercallingpatterns)show thattheproposedmethodachievesanaverageoflessthan5%errorinanydatavalueafter compressingtoamere2.5%oftheoriginalspace(i.e.,a40:1compressionratio),withthese numbersnotverysensitivetodatasetsize.experimentsonaggregatequeriesachieveda 4FutureDirections TheprimarygoalofCALDresearchistoextendthestateoftheartinusinghistoricaldata toimprovefuturedecisions.ourroleistoinventnewapproachesthatwillbecomethebasis 0.5%reconstructionerrorwithunder2%spacerequirement. forfuturecommercialsoftware.wewilldevelopthesenewapproachesbystudyingproblems immediatelyavailabletocaldcorporatemembersandpartners.thus,ourpartnerswill haveaccesstonewmethodslongbeforetheybecomecommerciallyavailable. anddatacontributedbyourindustrialandgovernmentpartners,andwillmakeourresults orintermsofapplications,asillustratedinfigure1.theexactlistofapplicationsis byfacultyresearchinterestsandexpertise. beingdeterminedbytheneedsofourindustrialandgovernmentpartners.thelistofbasic researchtopicswillbedeterminedprimarilybytheneedsoftheseapplicationproblems,and CALDresearchcanbeviewedeitherintermsofbasicscienticissuestobeaddressed, mostimportantscienticissueswillhavesignicantimpactacrossmanydierentapplication areas.thisallowscaldtospreadthecostofthisbasicresearchovermultipleproblem domainsandmultiplefundingsources.examplesofsuchbasicscienticissuesinclude: ThethesisunderlyingtheCALDmatrixresearchorganizationinFigure1isthatthe Learningfrommixedmediadata.Inmanyapplicationdomains,historicaldatawill includeavarietyoftypesofmedia.forexample,whenlearningtopredictmedical outcomesbasedonhistoricaldata,patientrecordsoftenincludeacombinationof andaudio(dictationsofphysiciansastheymakehospitalrounds).unfortunately, symbolicdata(e.g.,gender),numericaldata(e.g.,temperature),images(e.g.,x-rays currentlearningmethodscanmakeuseofonlyafractionofthispatientrecord,because andcatscans),othersensordata(e.g.,ekg),text(e.g.,notesonthepatientchart), 5

6 Scientific Issues, Basic Technologies Applications Learning from mixed media data, e.g., Medicine potentialpayoacrossmultipleapplicationareas(right). Figure1:CALDresearchemphasizesfundamentalscienticissues(left)withsignicant numeric, text, image, voice, sensor,... Manufacturing Active experimentation, exploration Financial forsymbolic/numericdata,neuralnetworksforimageanalysis,bayesianmethodsfor theyaretypicallyspecictoasingletypeofmedia(e.g.,decisiontreelearningmethods Optimizing decisions rather than predictions Intelligence analysis rangeofdataavailableinthehistoricalrecord.ifsuccessful,thislineofresearchwill Inventing new features to improve Public policy accuracy theirabilitytoutilizetheentiremultiple-mediahistoricalrecord. producemoreaccuratelearningmethodsusefulinavarietyofapplications,dueto textclassication).weneednewlearningmethodscapableoflearningfromthefull Learning from databases and Marketing world wide web Activeexperimentation.Mostcurrentdataminingsystemspassivelyacceptapredetermineddataset.Weneednewcomputermethodsthatactivelygenerateoptimal experimentstoobtaintheadditionalneededinformation.forexample,inmodelinga manufacturingprocessitisrelativelyeasytocapturedatawhiletheprocessrunsunder Optimizingdecisionsratherthanpredictions.Thegoalhereistousehistoricaldatato normalconditions.however,thisdatamaylackinformationabouthowtheprocess optimalexperimentstocollectthemostinformativedata,takingintoaccountprecise willperformundernon-standardconditions.werequirealgorithmsthatwillpropose modelsoftheexpectedcostsandbenetsoftheexperiment. Forexample,considertheproblemofcustomerretention.Givenhistoricaldataon improvethechoiceofactionsinadditiontothemoreusualgoalofpredictingoutcomes. customerpurchasesovertime,onecommondataminingproblemistopredictwhich customersarelikelytoremainloyal,andwhicharenot.whilethisisuseful,aneven moreusefultaskistolearnwhichactionscanincreasetheprobabilityofretainingthese 6

7 customers.thepointhereisthatweseeknewalgorithmsthatgobeyondpredicting Inventingnewfeaturestoimprovepredictionaccuracy.Inmanycases,theaccuracyof outcome.thisproblemraisesdicultbasicissuessuchaslearningfrombiaseddata samples,andhowtoincorporateconjecturesbyhumanexpertsabouttheeectiveness theoutcomeofsometimeseries,andinsteadlearnwhichactionsachievethedesired ofvariousinterventionactions.ifsuccessful,thisresearchwillallowapplyinghistorical datamuchmoredirectlytothedecision-makingproblemathand. theavailabledata.forexample,considertheproblemofdetectingtheimminent predictionscanbeimprovedbyinventingamoreappropriatesetoffeaturestodescribe theequipment.itiseasytogeneratemillionsoffeaturesthatdescribethistime seriesbytakingdierences,sums,ratios,averages,etcofprimitivesensorreadings long-durationdatasetitshouldbefeasibletoautomaticallyexplorethislargespaceof possibledenedfeaturesinordertoidentifythesmallfractionofthesefeaturesmost andpreviouslydenedfeatures.ourconjectureisthatgivenasucientlylargeand failureofapieceofequipmentbasedonthetimeseriesofsensordatacollectedfrom Storageissues-RAID,striping,andbeyond.Severaldataminingalgorithmsarevery usefulforfuturelearning.ifsuccessful,thisworkwouldleadtoincreasedaccuracyin creditrepayment,medicaloutcomes,etc. manypredictionproblems,suchaspredictingequipmentfailure,customerattrition, suitableforintelligentstoragedevices,suchasthoseadvocatedbycmu'snetwork AttachedSecureDisks(NASD)project[6,7,8].ANASDdevicecandosomecomputationallysimple,butdata-intensiveprocessing,reducingtheamountofdatato executioninintelligentstoragedevices: {Bandwidthreduction:Diskdrivessustain15MB/snowandthisdatarateis besenttothecpus-thisisappropriateforminingassociationrules[1]aswellas forthetrainingofneuralnetworks.specically,therearetwoprinciplebenetsfrom {Innerloopcomputationalparallelism:Forcomputationallysimple,data-intensive growingat40%peryear.networkinterfacesandclientmachinescannotcosteectivelymoveandconsumemultipledrives'bandwidthandhaveprocessing inthedriveprovidescomputationalparallelisminproportiontodatacapacityi.e.,100driveswith100mhzprocessoroptimizedfordatastreamingislikelyto bebetterforsimpleinnerloopsthan4cpusat500mhz. innerloops(wheregeneralpurposeprocessorsgainlittlefromcaches),execution morescalabledatamining. sizebyafactoroftenormore.thus,device-embeddedprocessingenablesfaster, resourcesleftovertolterthatdata,wheresimpleltersreducetransfereddata CALDfacultyresearchinterestsincludemanyadditionaltopicsaswell,suchasproblemsin automaticdatacapture,visualizationoflargedatasets,learningacrossmultipledatabases, 7

8 protectingdataprivacywhileutilizinghistoricaldata,methodsforinformationltering,and usingtheinternetasalargepublicdatasourcetoaugmentapplication-specicdatasets. ods(r-treesetc);svddusesstatisticalmethodstosolvelarge-scaledatabaseproblems; 5Conclusions probabilisticreasoningledtosuccessfulrobotnavigation;advancesisstoragetechnology AD-treesaremotivatedbyBayesianNetworks,butuseorcompeteagainstdatabasemeth- Inthispaperwehaveprovidedalistofsometypicalsettings,somerecentresearchachievementsandtheirperformancebenets,andnallyweoutlinedfutureresearchdirections. (\NetworkAttachedSecureDevices")seemespeciallybenecialfordataminingtasks.Furthercross-fertilizationofideasandtoolsamongtheaboveareas,aswellasInformation Inouropinion,themostsignicantpointistheneedforcross-disciplinarycollaboration: Retrieval,LanguageProcessing,RoboticsandNeuro-physiologyseemsverypromising,and probablynecessary,topushthestateoftheartindatamining. Acknowledgements: WewouldliketothanktheremainingmembersofCALDfortheirfeedback: (Statistics);RaviKannan(CS);RobertKass(Statistics);JohnLaerty(CS);TaiSingLee phergenovese(statistics);alexhauptmann(cs);brianjunker(statistics);jaykadane (CNBC);JohnLehoczky(Statistics);MarshaCLovett(CIL);RoyMaxion(CS);James (CS);StephenFienberg(Statistics);AlanFrieze(Math);JimGarrett(CivilEng.);Christo- (MedicalInf.,U.Pitt.);MarkDerthick(Robotics);BillEddy(Statistics);ScottFahlman AvrimBlum(CS);JaimeCarbonell(CS);HowieChoset(Mech.Eng.);GregCooper (PSC);YimingYang(LTI). PantelisVlachos(Statistics);AlexWaibel(CS);LarryWasserman(Statistics);JoelWelling RichardScheines(Phil);MarkSchervish(Statistics);TeddySeidenfeld(Statistics);Reid Simmons(CS);PeterSpirtes(Phil);KannanSrinivasan(GSIA);SaroshTalukdar(Elec. McClelland(Psych);MikeMeyer(Statistics);RoniRosenfeld(CS);SteveRoth(Robotics); References Comp.Eng.);RaulValdes-Perez(CS);ManuelaVeloso(CS);IsabellaVerdinelli(Statistics); [1]RakeshAgrawal,TomaszImielinski,andArunSwami.Miningassociationrulesbetween setsofitemsinlargedatabases.proc.acmsigmod,pages207{216,may

9 [2]C.G.Atkeson,A.W.Moore,andS.A.Schaal.LocallyWeightedLearning.AIReview, [3]C.G.Atkeson,A.W.Moore,andS.A.Schaal.LocallyWeightedLearningforControl. [4]W.Burgard,D.Fox,G.Lakemeyer,D.Hahnel,D.Schulz,W.Steiner,S.Thrun,and 11:11{73,April1997. AIReview,11:75{113,April1997. A.B.Cremers.Realrobotsfortherealworld therhinomuseumtour-guideproject. [6]GarthA.Gibson,DavidF.Nagle,KhalilAmiri,FayW.Chang,EugeneM.Feinberg,HowardGobio,ChenLee,BerendOzceri,ErikRiedel,DavidRochberg,andJim Zelenka.Fileserverscalingwithnetwork-attachedsecuredisks.ACMInternational [5]J.H.Friedman,J.L.Bentley,andR.A.Finkel.AnAlgorithmforFindingBestMatches September1977. inlogarithmicexpectedtime.acmtrans.onmathematicalsoftware,3(3):209{226, (submittedforpublication),october1997. [7]GarthA.Gibson,J.S.Vitter,andJ.Wilkes.Workinggrouponstoragei/oissuesin [8]GarthA.GibsonandJ.Wilkes.Self-managingnetwork-attachedstorage,strategic ConferenceonMeasurementandModelingofComputerSystems(Sigmetrics'97),June large-scalecomputing.computingsurveys,28(4),december1996. [10]VenkyHarinarayan,AnandRajaraman,andJereyD.Ullman.ImplementingData [9]A.Guttman.R-trees:Adynamicindexstructureforspatialsearching.InProceedings ofsigmod84,1984. directionsincomputingresearch:workinggrouponstoragei/oissuesinlarge-scale computing.computingsurveys,28a(online)(4),december1996. [12]HeikkiMannilaandHannuToivonen.Multipleusesoffrequentsetsandcondensed [11]FlipKorn,H.V.Jagadish,andChristosFaloutsos.Ecientlysupportingadhocqueries CubesEciently.InProc.ACMSIGMOD,pages205{216,May1996. [13]A.W.Moore,J.Schneider,andK.Deng.EcientLocallyWeightedPolynomialRegressionPredictions.InProceedingsofthe1997InternationalMachineLearningConference. MorganKaufmann,1997. inlargedatasetsoftimesequences.acmsigmod,pages289{300,may1997. representations.inproceedingsofthesecondinternationalconferenceonknowledge DiscoveryandDataMining,

10 [14]AndrewW.MooreandMarySoonLee.CachedSucientStatisticsforEcientMachineLearningwithLargeDatasets.TRCMU-RI-TR-97-27:Seealso [15]S.Thrun.Abayesianapproachtolandmarkdiscoveryinmobilerobotnavigation. [16]S.Thrun.Learningmapsforindoormobilerobotnavigation.ArticialIntelligence,to [17]SebastianThrun,DieterFox,andWolframBurgard.Aprobabilisticapproachforconcurrentmapacquisitionandlocalization.TechnicalReportCMU-CS ,Carnegie MachineLearning,toappear. [18]J.D.Ullman.DatabaseandKnowledgeBaseSystems.ComputerSciencePress,1988. appear. MellonUniversity,SchoolofComputerScience,Pittsburgh,PA15213,October

11 Contents 1Introduction 2SomeRealApplications 3RecentResearchAdvancesinCALD 3.1AD-tree::::::::::::::::::::::::::::::::::::::2 3.2ProbabilisticReasoning:::::::::::::::::::::::::::::: LossyCompressionforDataMining:::::::::::::::::::::::4 1 5Conclusions 4FutureDirections 85 11