María del Rosar io Bruer a. I BM Scholar s Pr ogram. Pregunt as y respuest as

Transcription

1 Dat a Mining María del Rosar io Bruer a I BM Scholar s Pr ogram Pregunt as y respuest as Pr egunt as: &XiOHVHOYDORUGHORVFOLHQWHV" &XiOHV VRQ ORV FOLHQWHV TXH WLHQHQ PD\RU SUREDELOLGDGGHGHVHUWDU" &XiOHV VRQ ORV SURGXFWRV TXH VH YHQGHQ HQ IRUPDFRQMXQWD"«Respuest as: (VWiQHQORVGDWRVGHOXVXDULR 6HQHFHVLWDQKHUUDPLHQWDVHVSHFLDOHVSDUD HQFRQWUDUODV

2 Business Int elligenc e S (V XQ SDUDJXDV EDMR HO TXH VH LQFOX\H XQ FRQMXQWR GH FRQFHSWRV \ PHWRGRORJtDVFX\DPLVLyQFRQVLVWHHQ PHMRUDU HO SURFHVR GH WRPD GH GHFLVLRQHVHQORVQHJRFLRVEDViQGRVH HQ KHFKRV \ VLVWHPDV TXH WUDEDMDQ FRQKHFKRVµ +RZDUG'UHVQHU *DUWQHU*URXS B.I.: rec ursos y herram ient as S)XHQWHV GH GDWRV ZDUHKRXVHV GDWDPDUWVHWF S+HUUDPLHQWDV GH DGPLQLVWUDFLyQ GH GDWRV S+HUUDPLHQWDV GH H[WUDFFLyQ \ FRQVXOWD S+HUUDPLHQWDVGHPRGHOL]DFLyQ'DWD 0LQLQJ

3 Qué es Dat a Mining? (1997) 'DWD 0LQLQJ es el pr oceso de explor ación y análisis - de maner a aut omát ica o semiaut omát ica - de los dat os par a obt ener pat r ones signif icat ivos y r eglas de negocio. 0LFKDHO%HUU\*RUGRQ/LQRII 'DWD0LQLQJIRUPDUNHWLQJVDOHV DQGFXVWRPHUVXSSRUW :LOH\86$ Reflex iones (2000) S QRV JXVWD OD QRFLyQ GH TXH ORV SDWURQHV GHEHQVHUVLJQLILFDWLYRV«S 6L KD\DOJR TXH UHFKD]DPRV HV OD IUDVH SRU PHGLRV DXWRPiWLFRV R VHPLDXWRPiWLFRVµ QR SRUTXH QR VHD FLHUWR VLQ DXWRPDWL]DFLyQ HV LPSRVLEOH PLQDU JUDQGHV FDQWLGDGHV GH GDWRV VLQR SRUTXH HQWHQGHPRV TXH VH KD SXHVWR GHPDVLDGR pqidvlv HQ OD DXWRPDWL]DFLyQ \ QR VXILFLHQWH HQ ODV HWDSDV GH H[SORUDFLyQ \ DQiOLVLV S 'DWD0LQLQJHVXQSURFHVR! "$# %'&( # ) * %+,- /

4 Qué NO es Dat a Mining S1R HV XQ SURGXFWR TXH VH FRPSUD HQODWDGR VLQR XQD GLVFLSOLQD TXH GHEHVHUGRPLQDGD S1RHVXQDVROXFLyQLQVWDQWiQHDDORV SUREOHPDVGHQHJRFLR S1R HV XQ ILQ HQ Vt PLVPR VLQR XQ SURFHVR TXH D\XGD D HQFRQWUDU VROXFLRQHVDSUREOHPDVGHQHJRFLR Pilares del proc eso de Dat a Mining S S S 'DWRV $OJRULWPRV\WpFQLFDV 3UiFWLFDVGHPRGHOL]DFLyQ

5 Disc iplinas que se int egran S,QWHOLJHQFLD$UWLILFLDO S(VWDGtVWLFD S7HFQRORJtDV GH VRSRUWH GH GHFLVLRQHV2/73 S7HFQRORJtDVGHKDUGZDUH\VRIWZDUH Perspec t iva hist óric a Fuente: Mining your Own Business Data Using DB2 Intelligent Miner for Data

6 Fuente: Mining your Own Business Data Using DB2 Intelligent Miner for Data Et apas en el proc eso de Dat a Mining,GHQWLILFDUHOSUREOHPDGHQHJRFLR 7UDQVIRUPDU ORV GDWRV HQ LQIRUPDFLyQ $FWXDUDSDUWLUGHORVUHVXOWDGRV 0HGLUORVUHVXOWDGRVGHODVDFFLRQHV

7 The Mining Pr ocess Fuente: Mining your Own Business Data Using DB2 Intelligent Miner for Data El Analist a de Dat os S(V HO YtQFXOR HQWUH ODV iuhdv GH WHFQRORJtD LQIRUPiWLFD \ ODV iuhdv GH QHJRFLRV S7UDGXFH ORV UHTXHULPLHQWRV GH LQIRUPDFLyQHQSUHJXQWDVDSURSLDGDVSDUD VX DQiOLVLV FRQ ODV KHUUDPLHQWDV GH PLQHUtD S5HDOLPHQWDHO'DWD:DUHKRXVHGHOD FRPSDxtDFRQQXHYRVFULWHULRVGHGDWD FOHDQLQJ\GDWDYDOLGDWLRQ

8 El Analist a de Dat os 7HFQRORJtD LQIRUPiWLFD 8VXDULRV GHQHJRFLR El Analist a de Dat os 78 96:;< 9 9>=?@A 9 B6@ 96C D: 9E?F<G H ];?^MA <[6=< D: 9E?F<G H \ ; 96:=K I 9 D@ D: 9E?F<G H H 96: 9 D@ A:;= D@ J@G <K?8 9< D@ ;= 9 ST?UGTV>@ IW:?U; LM:;6: ;=ND@

9 Habilidades requeridas S'DWDPDQLSXODWLRQ64/ S&RQRFLPLHQWRGHODVWpFQLFDVGH PLQHUtD\DQiOLVLVH[SORUDWRULR S+DELOLGDG GH FRPXQLFDFLyQ LQWHUSUHWDFLyQGHORVSUREOHPDVGH QHJRFLR S&UHDWLYLGDG Dat a Mining Team Fuente: Mining your Own Business Data Using DB2 Intelligent Miner for Data

10 Cost os de proyec t o Fuente: Mining your Own Business Data Using DB2 Intelligent Miner for Data Origen de los dat os S%DVHVGH'DWRV5HODFLRQDOHV S'DWD:DUHKRXVHV S'DWD0DUWVDQG2/$3 S2WURVIRUPDWRV([FHODUFKLYRV $6&,,HQFXHVWDVGDWRVFHQVDOHV HWF

11 Tipos de fuent es de dat os S7UDQVDFFLRQDOHVHMODVRSHUDFLRQHV UHDOL]DGDVFRQWDUMHWDGHFUpGLWR S5HODFLRQDOHVHMODHVWUXFWXUDGH ORVSURGXFWRVTXHRIUHFHHO%DQFR S'HPRJUiILFRVHMFDUDFWHUtVWLFDV GHOJUXSRIDPLOLDU La form a de los dat os para Dat a Mining S6HRUJDQL]DQHQIRUPDGHXQDWDEOD SODQD FRPSXHVWD SRU ILODV \ FROXPQDV S/DV )LODV XQLGDG GH DQiOLVLV3RU HMHPSORXQDFXHQWDXQWLFNHW S/DV FROXPQDV ORV DWULEXWRV GH FDGD XQLGDG GH DQiOLVLV3RU HMHPSOR IUHFXHQFLDGHXVRGHODWDUMHWDGH FUpGLWR

12 Carac t eríst ic as de las t ablas de dat os para Dat a Mining S7RGRVORVGDWRVGHEHQHVWDUHQXQDVROD WDEODR YLVWDµGHOD%DVHGH'DWRV S&DGD ILOD GHEH FRUUHVSRQGHU D XQD LQVWDQFLDUHOHYDQWHDOQHJRFLR S/DV &ROXPQDV VLQ YDULDELOLGDG GHEHQ VHU LJQRUDGDV S/DV &ROXPQDV FRQ YDORUHV ~QLFRV SDUD FDGD FDVR GHEHQ VHU LJQRUDGDV 1UR GH FXHQWD La c alidad de los dat os (O p[lwr GH ODV DFWLYLGDGHV GH Dat a Mining VH UHODFLRQD GLUHFWDPHQWH FRQ OD CALI DADGHORVGDWRV 6H GHEH LGHQWLILFDU ORV GDWRV IDOWDQWHV missings RIXHUDGHUDQJR out lier s

13 La c alidad de los dat os 0XFKDVYHFHVUHVXOWDQHFHVDULRSUHSURFHVDUORV GDWRVDQWHVGHGHULYDUORVDOPRGHORGHDQiOLVLV (OSUHSURFHVDPLHQWRSXHGHLQFOXLU WUDQVIRUPDFLRQHVUHGXFFLRQHVRFRPELQDFLRQHV GHORVGDWRV /DVHPiQWLFDGHORVGDWRVGHEHD\XGDUSDUDOD VHOHFFLyQGHXQDFRQYHQLHQWHr epr esent ación \ ODVERQGDGHVGHODUHSUHVHQWDFLyQHOHJLGD JUDYLWDQGLUHFWDPHQWHVREUHODFDOLGDGGHO PRGHOR\GHORVUHVXOWDGRVSRVWHULRUHV Problem as c on los dat os 'HPDVLDGRVGDWRV _ GDWRVFRUUXSWRVRFRQUXLGR _ GDWRVUHGXQGDQWHVUHTXLHUHQIDFWRUL]DFLyQ _ GDWRVLUUHOHYDQWHV _ H[FHVLYDFDQWLGDGGHGDWRVPXHVWUHR 3RFRVGDWRV _ DWULEXWRVSHUGLGRVPLVVLQJV _ YDORUHVSHUGLGRV _ SRFDFDQWLGDGGHGDWRV 'DWRVIUDFWXUDGRV _ GDWRVLQFRPSDWLEOHV _ P~OWLSOHVIXHQWHVGHGDWRV

14 Preparac ión de los dat os `ba cta dea f$ghi jklg m gn go$ctgp `ba cta 7UDQVIRUPHG'DWD ([WUDFWHG,QIRUPDWLRQ $VVLPLODWHG,QIRUPDWLRQ 6HOHFW 7UDQVIRUP 0LQH $VVLPLODWH Dat a Warehouse S'DWD :DUHKRXVH LV D VXEMHFWRULHQWHG LQWHJUDWHG WLPHYDULDQW QRQ YRODWLOH FROOHFWLRQ RI GDWD LQ VXSSRUW RI PDQDJHPHQWGHFLVLRQV %LOO,QPRQ S$FRS\RIWUDQVDFWLRQGDWDVSHFLILFDOO\ VWUXFWXUHGIRUTXHU\DQGDQDO\VLV 5DOSK.LPEDOO

15 Dat a Mart s S7pFQLFDPHQWHHVXQVXEFRQMXQWRGHO ': RULHQWDGR D XQD ILQDOLGDG HVSHFtILFD GH QHJRFLR PDUNHWLQJ ILQDQ]DVSURGXFFLyQHWF S(O WpUPLQR VH XWLOL]D WDPELpQ SDUD LGHQWLILFDU VROXFLRQHV DOWHUQDWLYDV D XQ': FRUSRUDWLYR PiV UHGXFLGDV\ GH PHQRU FRVWR \ WLHPSR GH LPSODQWDFLyQ Arquit ec t ura del Dat aw arehouse qrts uwvtwwwtx ylz{$wt }Wvlz ~ wwv$ut tvr } vls ƒt Metadata DW ReportQ uery, EIS OLAP Datos operacionales y externos Data Mining

16 Herram ient as de ex plot ac ión del DW S+HUUDPLHQWDVGHYLVXDOL]DFLyQ S5HSRUWLQJ S2/$3 S'DWD0LQLQJ OLAP S2Q/LQH$QDO\WLFDO3URFHVVLQJ S3HUPLWHQ OD HODERUDFLyQ GH YLVWDV PXOWLGLPHQVLRQDOHV GHO ': SDUD RSWLPL]DUSHUIRUPDQFH S(VWiQ VRSRUWDGDV SRU PRWRUHV GH DGPLQLVWUDFLyQ GHO ': TXH DGPLWHQ ODFRQVWUXFFLyQGHHVWRV FXERVµ

17 OLAP S+HUUDPLHQWDV~WLOHV\SRGHURVDV SDUDDFFHGHUD%DVHVGH'DWRV\ 'DWD:DUHKRXVHV\REWHQHU UHSRUWHVµGHLQIRUPDFLyQ S/D WHFQRORJtD 2/$3 FRPPSOHPHQWD ODV DFWLYLGDGHV GH 'DWD 0LQLQJ \ VXSHUDODVSRVLELOLGDGHVGHO64/ Dat a Mining y OLAP S/DV KHUUDPLHQWDV GH UHSRUWLQJ 2/$3 \ FRQVXOWD UHVSRQGHQ HIHFWLYDPHQWH SDUD OD FRQVWUXFFLyQ GH PRGHORV GHVFULSWLYRV \ UHWURVSHFWLYRV SDUD FRQILUPDU R UHFKD]DU KLSyWHVLV SUHYLDV GHO XVXDULR

18 Dat a Mining y OLAP S/DV KHUUDPLHQWDV GH 'DWD 0LQLQJ SHUPLWHQ HQFRQWUDU SDWURQHV QR HYLGHQWHV HQ ORV JUDQGHV YRO~PHQHV GH LQIRUPDFLyQ GHO ': \ SURSRQHU PRGHORVSUHGLFWLYRV Qué es la Est adíst ic a S(V OD GLVFLSOLQD TXH H[WUDH LQIRUPDFLyQ JHQHUDO D SDUWLU GH GDWRVHVSHFtILFRV S(VHOHVWXGLRGHODHVWDELOLGDGHQOD YDULDFLyQ S(V HO DUWH GH H[DPLQDU VXPDUL]DU \ H[WUDHU FRQFOXVLRQHV D SDUWLU GH ORVGDWRV

19 Dat a Mining y Est adíst ic a S/RV PpWRGRV HVWDGtVWLFRV VRQ HO FRUD]yQ GH PXFKDV GH ODV WpFQLFDV GHPLQHUtDGHGDWRV S2ULJLQDOPHQWH PXFKDV GH HVWDV WpFQLFDV IXHURQ GLVHxDGDV FRQ SURSyVLWRVFRQILUPDWRULRV S/D HVWDGtVWLFD H[SORUDWRULD DSDUHFH HQ ORV FRQ ORV DSRUWHV GH -7XFNH\ Dat a Mining y Est adíst ic a S(Q OD 0LQHUtD GH 'DWRV QR VH KDFHQ VXSXHVWRV D SULRUL VREUH OD QDWXUDOH]D GH ODV YDULDEOHV \ GH ODV UHODFLRQHV HQWUH HOODV QRUPDOLGDG OLQHDOLGDG HWF S/RVDOJRULWPRVHVWDGtVWLFRVVHDGDSWDQ SDUD 0LQHUtD GH 'DWRV DO SURFHVDPLHQWR GH JUDQGHV YRO~PHQHV GHGDWRV

20 Dat a Mining e IA S/D,QWHOLJHQFLD$UWLILFLDO VHLQWHJUD D OD 0LQHUtD GH 'DWRV D SDUWLU GH ODVUHGHVQHXURQDOHVDUWLILFLDOHV S6H XWLOL]DQ SDUD FRQVWUXLU PRGHORV SUHGLFWLYRV QR OLQHDOHV TXH DSUHQGHQ DWUDYpVGHHQWUHQDPLHQWR\TXHVH DVLPLODQDORVPRGHORVGHUHGHVGH QHXURQDVELROyJLFDV Redes neuronales S/DVUHGHVQHXURQDOHVVRQDGHFXDGDV SDUDSUREOHPDVGHWLSRSUHGLFWLYR S8QSUREOHPDDSURSLDGRSDUDXQDUHG QHXURQDOWLHQHWUHVFDUDFWHUtVWLFDV Se compr enden clar ament e los I NPUTS Se compr ende clar ament e el OUTPUT Exist en ej emplos (exper iencia) suf icient es par a ent r enar a la r ed

21 Los m odelos neuronales S/D UHG QHXURQDO QR SURGXFH UHJODV H[SOtFLWDVTXHGHVFULEDQHOPRGHOR S8QPRGHORQHXURQDOHVWDQEXHQRFRPROR HV HO VHW GH GDWRV XVDGR SDUD HQWUHQDU ODUHG S(O PRGHOR HV HVWiWLFR \ GHEH VHU H[SOtFLWDPHQWH DFWXDOL]DGR DJUHJDQGR HMHPSORV UHFLHQWHV \ UHHQWUHQDQGR OD UHGSDUDDVHJXUDUVXYLJHQFLD\XWLOLGDG Los m odelos neuronales S&RQ PRGHORV QHXURQDOHV VH SXHGH DWDFDU XQD JUDQ YDULHGDG GH SUREOHPDV \ SURGXFLU EXHQRV UHVXOWDGRV D~Q HQ GRPLQLRV FRPSOHMRV FRQ YDULDEOHV FRQWLQXDV\FDWHJyULFDV S6RQ DSURSLDGRV SDUD WDUHDV GH FODVLILFDFLyQ \ SUHGLFFLyQ FXDQGR ORV UHVXOWDGRV GHO PRGHOR VRQ PiV LPSRUWDQWHV TXH FRPSUHQGHU FyPR IXQFLRQDHOPRGHOR

22 Cust om er Relat ionship Managem ent S(V HO SURFHVR TXH DGPLQLVWUD OD UHODFLyQ HQWUH OD FRPSDxtD \ VXV FOLHQWHV S3DUD TXH UHVXOWH H[LWRVR UHVXOWD QHFHVDULR LGHQWLILFDU ORV SDWURQHV GHFRQVXPR\FRPSRUWDPLHQWRGHORV FOLHQWHV Dat a Mining - CRM S'DWD 0LQLQJ VH XWLOL]D SDUD VLVWHPDWL]DU ORV SURFHVRV GH E~VTXHGD GH ORV SUHGLFWRUHV GH FRPSRUWDPLHQWR GH ORV FOLHQWHV HQ ODVHWDSDVGHGLVHxRGHFDPSDxDV S7DPELpQ VH DSOLFD SDUD OD PHGLFLyQ GHORVUHVXOWDGRVGHODFDPSDxD\OD UHDOLPHQWDFLyQGHO&50

23 Problem as t ípic os de Dat a Mining S&ODVLILFDFLyQ S(VWLPDFLyQ S3UHGLFFLyQ S$JUXSDPLHQWRDSDUWLUGH UHJODVGH DVRFLDFLyQ S&OXVWHULQJ S'HVFULSFLyQ\YLVXDOL]DFLyQHWF Problem a de Clust ering $JUXSDUDORVFOLHQWHVVHJ~QVXVLQGLFDGRUHV 55HFHQF\ ))UHFXHQFLD 0 0RQWR HWF HQVHJPHQWRVGHFRPSRUWDPLHQWRKRPRJpQHR 5HVXOWDGR &OLHQWHV +HDY\ 0HGLXP /LJKW HWF (OGHODIDFWXUDFLyQVHFRQFHQWUDHQHO FOXVWHU+HDY\GHORVFOLHQWHV /RV FOLHQWHV +HDY\ VRQ FDVDGRV FRQ KLMRV WUDEDMDGRUHV DXWyQRPRV FRQ XQ LQJUHVR VXSHULRUD

24 Problem a de Clasific ac ión &ODVLILFDU XQ QXHYR FOLHQWH GH DFXHUGR D VX SHUILO VRFLRGHPRJUiILFR FRPR SRWHQFLDO FOLHQWH+HDY\0HGLXP/LJKW Problem a de Est im ac ión (VWLPDU HO FRQVXPR GH XQ GHWHUPLQDGR UXEUR GH DUWtFXORV GH XQ JUXSR FOLHQWHV HQ HO SUy[LPR WULPHVWUH (VWLPDU HO /79 /LIH 7LPH 9DOXH SRWHQFLDOGHXQQXHYRFOLHQWH

25 Problem a de Predic c ión 3UHGHFLUHODEDQGRQRGHXQFOLHQWH FKXUQLQJDWULWWLRQ 3DUDXQDFRPSDxtDGHWHOHIRQtD FHOXODU 3DUDXQD$)-3 3DUDXQDWDUMHWDGHFUpGLWR Problem a de Asoc iac ión (QFRQWUDUODVUHJODVTXHGHWHUPLQDQ HOFURVVWUDIILFHQWUHSURGXFWRV SDUDORVFOLHQWHVGHXQ%DQFR3RU HMHPSOR &XDQGRXQFOLHQWHVHDFWLYDHQ&DMD GH$KRUURVHOVLJXLHQWHSURGXFWR HQGRQGHVHDFWLYDHV3UpVWDPRV SHUVRQDOHV(VWHSDWUyQRFXUUHHQ HOGHORVFDVRVµ

26 Problem a de visualizac ión 5HSUHVHQWDU PHGLDQWH XQ VRIWZDUH GH JHRORFDOL]DFLyQ *,6 OD GLVWULEXFLyQ GH ORV FOLHQWHV HQ OD ]RQDGH LQIOXHQFLD GH ODVVXFXUVDOHV GHXQFRPHUFLR Problem as usuales S&DUDFWHUL]DFLyQ GH SHUILOHV GH FOLHQWHVSDUDGHILQLUDFFLRQHVGH8S VHOOLQJ\&URVVVHOOLQJ S7UDFNLQJ GH FDPSDxDV \ SUHGLFFLyQ GHUHVSXHVWDQRUHVSXHVWD S&DQDVWDGHFRQVXPRGHWDUMHWDVGH FUpGLWR\SUHYHQFLyQGHIUDXGHV S0RGHORVGHSUHGLFFLyQGHDEDQGRQR

27 Problem as usuales S3URJUDPDVGHPLOODMH\ILGHOL]DFLyQ S&RQVROLGDFLyQGH%DVHVGH'DWRV SURSLDVFRQIXHQWHVH[WHUQDV S:HEPLQLQJ\DQiOLVLVGHWUiILFR\ XVRGHUHFXUVRVGHHEXVLQHVV S'HILQLFLyQ GH PDUFRV PXHVWUDOHV SDUD LQYHVWLJDFLRQHV GH PHUFDGR \ HQFXHVWDVGHFXVWRPHUVDWLVIDFWLRQ La elec c ión del m odelo para Dat a Mining 3ULQFLSDOHVREMHWLYRVGHOSURFHVRGH'DWD 0LQLQJ pr edicción descr ipción (O PpWRGR D XWLOL]DU GHSHQGH GH ORV REMHWLYRVSHUVHJXLGRVSRUHODQiOLVLVSHUR WDPELpQ GH OD FDOLGDG \ FDQWLGDG GH ORV GDWRVGLVSRQLEOHV

28 Fuente: Mining your Own Business Data Using DB2 Intelligent Miner for Data Cóm o selec c ionar una pot enc ial aplic ac ión de DM &RQVLGHUDFLRQHVSUiFWLFDV 3RWHQFLDOLPSDFWRVLJQLILFDWLYR5HODFLyQ FRVWREHQHILFLR 1RKD\RWUDDOWHUQDWLYD ([LVWHVRSRUWHLQVWLWXFLRQDO 1RH[LVWHQLPSHGLPHQWRVOHJDOHVGHXVR GHODLQIRUPDFLyQ

29 Cóm o selec c ionar una pot enc ial aplic ac ión de DM Consider aciones t écnicas: 'LVSRQLELOLGDGVXILFLHQWHGHGDWRV 5HOHYDQFLDGHDWULEXWRV %DMRVQLYHOHVGHUXLGRHQORVGDWRV 3UHFLVDUHOQLYHOGHFRQILDQ]DSDUDORV UHVXOWDGRV &RQRFLPLHQWRDQWHULRUH[LVWHQWH La evaluac ión de los m odelos &XiQDMXVWDGRHVHOPRGHOR" (VFRUUHFWDVXGHVFULSFLyQGHORV GDWRVREVHUYDGRV" &XDQWDFRQILDQ]DVHSXHGHWHQHUHQ VXVSUHGLFFLRQHV" &XiQFRPSUHQVLEOHHVHOPRGHOR"

30 Las m edidas /D FRQFRUGDQFLD GH XQ PRGHOR SUHGLFWLYR FRQODUHDOLGDGVHPLGHFRQUHODFLyQDOD WDVDGHHUURUHVGHFLUHOSRUFHQWDMHGH FDVRV FODVLILFDGRV R FX\D SUHGLFFLyQ IXH LQFRUUHFWD 3DUD HOOR VH GLVSRQH GH GDWRV GH YDOLGDFLyQ \ WHVWLQJ VREUH ORV TXH GHEH DSOLFDUVH SHULyGLFDPHQWH HO PRGHOR D PRGRGHFRQWURO Las m edidas (Q HO FDVR GH ORV PRGHORV GHVFULSWLYRV XQDEXHQDUHJODHVODTXHSURSRUFLRQDOD LQIRUPDFLyQ PiV FRPSUHQVLEOH FRQ OD PHQRU ORQJLWXGµ GH H[SUHVLyQ GH OD UHJOD (Q GHILQLWLYD OD PHGLGD PiV LPSRUWDQWH GH HIHFWLYLGDG HV HO UHWRUQR GH OD LQYHUVLyQ

31 Un proyec t o ex it oso S8Q~QLFRSURMHFWOHDGHU S8QHTXLSR PXOWLGLVFLSOLQDULR LQWHJUDGR SRU SHUVRQDVGHODViUHDVGH,7\GHQHJRFLR S/DV XQLGDGHV GH QHJRFLR HVWiQ LQYROXFUDGDVGHVGHHOFRPLHQ]R S(O iuhd GH,7 HVWi LQYROXFUDGD GHVGH HO FRPLHQ]R S8Q SHTXHxR SUR\HFWR SLORWR TXH PXHVWUH ODVYHQWDMDVGH'DWD0LQLQJ Las nuevas t ec nologías

32 Web Mining S(V HO GHVFXEULPLHQWR GH SDWURQHV VLJQLILFDWLYRVDSDUWLUGHODQiOLVLVGH OD HVWUXFWXUD FRQWHQLGRV\ XVR GH OD:HE Web Mining Tax onom y :HE0LQLQJ :HEFRQWHQW :HE6WUXFWXUH :HEXVDJH

33 Result ados Web m ining S(O GH ORV YLVLWDQWHV TXH DFFHGHQ D ZZZLEPFRPUHGERRNV DFFHGHQD ZZZLEPFRPVRIWZDUHGDWDLPLQHU IRUGDWD S(QWU\DQG([LWSRLQWV Result ados Web m ining S/LQN DQDO\VLV \ SDWURQHV VHFXHQFLDOHVGHHQODFHVGHSiJLQDV S6HJPHQWDFLyQGHFOLHQWHVGHH FRPPHUFH S&DQDVWDGHSURGXFWRV SHWFHWFHWF

34 Tex t Mining S6RQQXHYDVKHUUDPLHQWDVGHVWLQDGDV D H[WUDHU LQIRUPDFLyQ GH GRFXPHQWRV QR HVWUXFWXUDGRVµ RUJDQL]DUORV VHJPHQWDUORV LQGH[DUORV Problem as de Tex t Mining S'LUHFFLRQDPLHQWR DXWRPiWLFR GH HPDLOVVHJ~QVXFRQWHQLGR S&ODVLILFDFLyQ DXWRPiWLFD GH GRFXPHQWRVGHXQDLQWUDQHW S%~VTXHGD GH LQIRUPDFLyQ HQ GRFXPHQWRV GH GLVWLQWRV LGLRPDV VLPXOWiQHDPHQWH

35 Problem as de Tex t Mining S$QiOLVLV GH FRQWHQLGRV GH SiJLQDV :HE S2UJDQL]DFLyQ GH VHUYLFLRV GH E~VTXHGDHQOD:HE S([WUDFFLyQGHFRQFHSWRVGHVtQWHVLV HQ GRFXPHQWRV UHIHULGRV DO PLVPR DVXQWR Conc lusiones

36 Para qué Minería de Dat os S/D 0LQHUtD GH 'DWRV HV XQD KHUUDPLHQWDHILFD]SDUDGDUUHVSXHVWD SUHJXQWDVFRPSOHMDVGH,QWHOLJHQFLDGH 1HJRFLRV S/DV KHUUDPLHQWDV GLVSRQLEOHV SHUPLWHQ DXWRPDWL]DU SDUWH GH OD WDUHD GH HQFRQWUDU ORV SDWURQHV GH FRPSRUWDPLHQWRRFXOWRVHQORVGDWRV S3HUR«Qué no puede aut om at izarse (t odavía) S/D HOHFFLyQ GH ORV SUREOHPDV GH QHJRFLR FDQGLGDWRVSDUDWDUHDVGH'DWD0LQLQJ S/D LGHQWLILFDFLyQ \ UHFROHFFLyQ GH ORV GDWRV TXH FRQWLHQHQ OD LQIRUPDFLyQ EXVFDGD S(O PDVDMHR \ WUDWDPLHQWR GH ORV GDWRV TXHSRVLELOLWDODE~VTXHGDGHSDWURQHV S(OGLVHxR\FiOFXORGHYDULDEOHVGHULYDGDV

37 Qué no puede aut om at izarse (t odavía) S(OSODQGHDFFLRQHVTXHDSR\iQGRVHHQORV UHVXOWDGRVGHOPRGHORSURGX]FDHO52, S/D PHGLFLyQ GHO p[lwr GH ODV DFFLRQHV UHDOL]DGDV D SDUWLU GH ORV UHVXOWDGRV SURSRUFLRQDGRVSRU'DWD0LQLQJ Conc lusiones S&RQYLHUWD D 'DWD 0LQLQJ HQ XQD SDUWHGHVXSUR\HFWRGHQHJRFLR S,QFOX\D D 'DWD 0LQLQJ HQ OD FXOWXUDµGHVXRUJDQL]DFLyQ

38 Ejem plos c on DB2 Int elligent Miner for Dat a Téc nic as ut ilizadas S&OXVWHULQJVHJPHQWDFLyQ S&DQDVWDGHSURGXFWRV S$UEROGHGHFLVLyQ S5HGQHXURQDOFRPRPRGHOR SUHGLFWLYR

39 Qué es c lust ering? S(V OD SDUWLFLyQ GHO FRQMXQWR GH LQGLYLGXRV HQ VXEFRQMXQWRV OR PiV KRPRJpQHRVSRVLEOHV S(OREMHWLYRHVPD[LPL]DUODVLPLOLWXG GH ORV LQGLYLGXRV GHO FOXVWHU \ PD[LPL]DU ODV GLIHUHQFLDV HQWUH FOXVWHUV Aplic ac iones de la t éc nic a S6HJPHQWDFLyQGHODEDVHGHGDWRV S'HWHFFLyQGHIUDXGHV S'HWHFFLyQGHGHIHFWRV

40 Objet ivos S'HWHUPLQDUHOQ~PHURySWLPRGH FOXVWHUV S$VLJQDUDFDGDLQGLYLGXRDXQ~QLFR FOXVWHU S(YDOXDUHOLPSDFWRGHODVYDULDEOHV HQODIRUPDFLyQGHOFOXVWHU S&RPSUHQGHU HO SHUILOµ GH FDGD FOXVWHU Medidas de sim ilaridad S9DULDEOHV FDWHJyULFDV HVFDODV QRPLQDOHV \ RUGLQDOHV VRQ VLPLODUHVVLVRQLJXDOHV S9DULDEOHV QXPpULFDV HVFDODV PpWULFDV HO DOJRULWPR GHWHUPLQD VXGLIHUHQFLDH[SUHVDGDHQXQLGDGHV GHGHVYLDFLRQHVVWDQGDUG

41 Ejem plo sim ilaridad 1RPEUH 6H[R (VW&LYLO /XJDU 6LPLODULGDG Juan M C Cap.Fed 0.33 Maria F C GBA 0.33 No evaluado Diferente Igual Diferente Crit erio Condorc et S(VXQDPHGLGDGHVLPLODULGDGTXHYDUtD HQWUH\ S9DOH ORV LQGLYLGXRV HVWiQ XELFDGRV DOHDWRULDPHQWHHQORVFOXVWHUV S9DOH 7RGRV ORV LQGLYLGXRV GH ORV FOXVWHUVVRQLGpQWLFRV\QRKD\LQGLYLGXRV FRQ HVDV FDUDFWHUtVWLFDV IXHUD GH FDGD FOXVWHU S&RQGRUFHWPtQLPRXVXDO

42 El problem a 6HWUDWDGHVHJPHQWDUOD%DVHGH 'DWRVGHORVFOLHQWHVGHXQDWDUMHWD GH FUpGLWR D SDUWLU GH VXV LQGLFDGRUHV GH FRQVXPR SDUD LGHQWLILFDU DO VHJPHQWR GH PD\RU YDORU Los dat os disponibles S $ SDUWLU GH OD %DVH GH 'DWRV GH WUDQVDFFLRQHV GHO ~OWLPR DxR GH ORV FOLHQWHV VH REWLHQHQ FRPR YDULDEOHV )UHFXHQFLDGHXVRGHODWDUMHWD : calculada como media de días ent re t r ansacciones. 6DOGRSURPHGLRPHQVXDOGHWUDQVDFFLRQHVHQ 0RQWRSURPHGLRSRUWUDQVDFFLyQ &DQWLGDGGHVHUYLFLRVSRUGpELWRDXWRPiWLFR 'DWRVVRFLRGHPRJUiILFRVVH[RHGDG HVWDGRFLYLORFXSDFLyQKLMRV

43 La preparac ión de dat os S'HILQLUODXQLGDGGHDQiOLVLV FXHQWDRWDUMHWD" S'HILQLUTXpHVXQDWUDQVDFFLyQ HM FyPRVHFRQVLGHUDQORVDMXVWHV PRQWRVQHJDWLYRV" S'HILQLUYDULDEOHVGHULYDGDVHQOD IUHFXHQFLD FyPRLQWHUYLHQHQORV GpELWRVDXWRPiWLFRV" La preparac ión de dat os S'HVFULELUODVYDULDEOHVDLQFOXLUHQ HOPRGHORSDUD ˆ Calcular medidas de posición y disper sión ˆ I dent if icar dist r ibuciones asimét r icas ˆ I dent if icar missings ˆ I dent if icar valor es incor r ect os o f uer a de r ango ˆ I dent if icar out lier s

44 E s ta d is tic a s Clu s te r ,0 0 % d e p o b la c ió n 'HVFULSWLYRVJHQHUDOHV S o lte Dro ivo rc ia d o /Viud o Ca s a d o s c io s e d a d e s ta d o _civil N o tra b acja ue nta P ro p ia R e la c io n d e p e n d e nc ia Ma s c ulino F e m e nin o S i N o o c up s e xo hijo s a vg tc kt fre c u p e s o s Crit erios de segm ent ac ión S6H WRPDQ FRPR YDULDEOHV DFWLYDVµ ODV TXH FRUUHVSRQGHQ DO FRPSRUWDPLHQWRGHFRQVXPR S6H WRPDQ FRPR YDULDEOHV VXSOHPHQWDULDV ORV DWULEXWRV VRFLRGHPRJUiILFRV

45 Credit Card 1 Masculino Femenino Divo So ltero rciado /Viud o Casado No Cuenta trabajapropia Si Relacio n d ep end encia No scios [se xo] [estad o_ civil] [ocup] [hijos ] fre cu pesos avgtckt [edad] 55 2 Divo So ltero rciado /Viud o Casado No Cuenta trabajapropia Masculino Femenino Relacio n d epend encia Si No s cios [estad o_ civil] [ocup] [se xo] pe s os [hijos ] fre cu [e da d] a vgtckt 27 0 Divo So ltero rciado /Viud o Casado No Cuenta trabajapropia Si Relacio n d ep end encia No Masculino Femenino 18 scios fre cu pesos [estad o_ civil] [ocup] [hijos ] [se xo] a vgtckt [e da d] Credit Ca rd Clus ter 2 Tienen 4 o más débitos automáticos Casados S o ltedivorcia ro do /Viudo Ca s a d o 27,21% de pobla ción Trabajo Cta Propia No tra b acue ja nta P ro p ia R e la cio n d e p e nd e nc ia s cios [e s tado_civil] [ocup] Ma s c ulino Fe me nino Saldo >>> S i No Varones Con hijos [s e xo] Uso frecuente Edad pe s os [hijos ] Ticket >>> fre cu [e dad] a vgtckt

46 Paret o Cluster 0 Cluster 1 Cluster % Cuentas % Suma Saldo Arboles de dec isión S6RQ WpFQLFDV TXH VH XWLOL]DQ FRQ ILQDOLGDGSUHGLFWLYD\GHFODVLILFDFLyQ S6H REWLHQH FRPR UHVXOWDGR UHJODVµ TXHH[SOLFDQHOFRPSRUWDPLHQWRGHXQD YDULDEOH 7$5*(7 FRQ UHODFLyQ D RWUDV35(',&725$6 S(Q HVWH HMHPSOR VH XWLOL]DQ SDUD H[SOLFDUµORVFOXVWHUV

47 Algorit m os S&+$,'&KL6TXDUHG$XWRPDWLF 'HWHFWLRQ S& 57 &ODVVLILFDWLRQ DQG 5HJUHVVLRQ7UHH S&4XHVW\RWURV S,QWHOOLJHQW 0LQHU XWLOL]D XQD YDULDQWHGH& 57 Arbol de c om port am ient o Si tiene 4 o más débitos automáticos y un saldo > $ 727 entonces su probabilidad de pertenecer al cluster 2 es del 99%

48 Arbol soc iodem ográfic o Mark et Bask et Analysis S(OSUREOHPD6HWUDWDGHHQFRQWUDU ODV UHJODV GH DVRFLDFLyQ TXH RUJDQL]DQ ORV SHGLGRV GH WRSSLQJVµ H[WUD GH XQD SL]]HUtD D SDUWLU GHO DQiOLVLV GH XQ FRQMXQWR GH WLFNHWVGHYHQWD

49 La t abla de Dat a Mining S,GWLFNHW S&yGLJRGHSURGXFWR +RQJRV 3HSSHURQL 4XHVR &HUYH]D *DVHRVD 2WUDEHELGD Propósit o de MBA S*HQHUDUUHJODVGHOWLSR Š I F (SI ) condición ENTONCES (THEN) r esult ado S(MHPSOR 6Lpr oduct o A y pr oduct o C ENTONCES pr oduct o B

50 Tipos de reglas S8WLOHV DSOLFDEOHV UHJODV TXH FRQWLHQHQ EXHQD FDOLGDG GH LQIRUPDFLyQ TXH SXHGHQ WUDGXFLUVH HQDFFLRQHVGHQHJRFLR S7ULYLDOHVUHJODV\DFRQRFLGDVHQHO QHJRFLRSRUVXIUHFXHQWHRFXUUHQFLD S,QH[SOLFDEOHV FXULRVLGDGHV DUELWUDULDVVLQDSOLFDFLyQSUiFWLFD Problem as del MBA S/DH[LVWHQFLDGHPXFKRVLWHPVHQHO VHW GH DQiOLVLV FRPSOLFD H[SRQHQFLDOPHQWH HO WLHPSR GH FiOFXOR S5HVXOWD QHFHVDULR GHILQLU FULWHULRV SDUDVHOHFFLRQDUODVPHMRUHVUHJODV S(V LPSRUWDQWH OD FRQVWUXFFLyQ GH XQDWD[RQRPtDGHSURGXFWRV

51 Cuán buena es una regla? S0HGLGDVTXHFDOLILFDQDXQDUHJOD Š Sopor t e Š Conf ianza Š Lif t (I mpr ovement ) Soport e S(VODFDQWLGDGGHWUDQVDFFLRQHV HQGRQGHVHHQFXHQWUDODUHJOD Š Ej : Si A ent onces B est á pr esent e en 4000 de t r ansacciones. Š Sopor t e (A/ B) : 40%

52 Confianza S&DQWLGDGGHWUDQVDFFLRQHVTXH FRQWLHQHQODUHJODUHIHULGDDOD FDQWLGDGGHWUDQVDFFLRQHVTXH FRQWLHQHQODFOiXVXODFRQGLFLRQDO Š Ej : Par a el caso ant er ior, si A est á pr esent e en 6000 t r ansacciones (60%) Š Conf ianza (A/ B) = 40% / 60% = 66% Mejora (Im provem ent ) S&DSDFLGDGSUHGLFWLYDGHODUHJOD Š Mej or a = p(a/ B) / p(a) * p(b) Š Ej : p(a/ B) = 40% ; p(a) = 60%; p(b) = 30% I mpr ov (A/ B) = 40% / (60% * 30%) = 2.22 Mayor a 1 : la r egla t iene valor pr edict ivo

53 Ejem plo de c álc ulo Dat os básic os +RQJRV 3HSSHURQL 4XHVR &DQWLGDG Si Si Si 100 Si Si No 400 Si No Si 300 Si No No 100 No Si Si 200 No Si No 150 No No Si 200 No No No 550 TOTAL 2000

54 Reglas ŒU 6Ž R Œ Ž UŽ šœ W 6 Hongos Pepperoni Queso Hongos --> Pepperoni Hongos --> Queso Queso --> Pepperoni Hongos + Pepperoni --> Queso Hongos + Queso --> Pepperoni Queso + Pepperoni --> Hongos Pueden descartarse por bajo soporte Reglas significativas Ot ro ejem plo de MBA S/D DVRFLDFLyQ VH SODQWHD HQWUH ORV WRSSLQJV GH ODV SL]]DV \ ODV EHELGDV S/RV JUiILFRV GH UHJODV SHUPLWHQ YLVXDOPHQWH LGHQWLILFDU UHJODV FRQ EXHQVRSRUWHFRQILDQ]D\OLIW

55 Reglas Soporte (%)Confianza(%) Tipo Elevación Cuerpo de regla Cabecera de regla [Hongos]+[Otra bebida] ==> [Pepperoni] [Cerveza]+[Pepperoni] ==> [Hongos] [Cerveza]+[Queso] ==> [Hongos] [Hongos]+[Pepperoni] ==> [Cerveza] [Cerveza] ==> [Hongos] [Hongos] ==> [Cerveza] [Hongos]+[Queso] ==> [Cerveza] [Pepperoni]+[Queso] ==> [Gaseosa] [Hongos]+[Pepperoni]+[Queso] ==> [Cerveza] [Hongos]+[Gaseosa] ==> [Queso] [Gaseosa]+[Pepperoni] ==> [Queso]

56 Web Mining S(O SUREOHPD VH WUDWD GH DQDOL]DU ODV WUDQVDFFLRQHV \ HO SHUILO GH ORV XVXDULRV GH XQ :HE VLWH GH XQ FRPHUFLRGHYHQWDSRULQWHUQHW Modelos aplic ados S$VRFLDFLyQGHSiJLQDVYLVLWDGDV FDQDVWDGHSURGXFWRV S3HUILOGHXVXDULRVFOXVWHULQJ GHPRJUiILFR S3RWHQFLDOHVFRPSUDGRUHViUEROGH GHFLVLyQ

57 Asoc iac ión de páginas žÿ RŸ Ÿ U ª «6 ±6 ²U«³³ ẗ µr ¹ 6 ºµœ ³ 6»¼ ŸŸ½b T¾( R«RÀÂÁÁUûl ¾ U Q T¾( R«RÀ ³³ ẗ µr >Ä œ µ6 6 ºµœ ³ 6»l ¾ U Q T¾( R«RÀÂÁÁUû¼ ŸŸ½b T¾( R«RÀ ³ Q ³³ 6Å> Æ $ µ6 6 ³ ẗ µ6»wç6ÿ U RÈUÉ UȾR Ÿ Z T¾( R«RÀÁÁUÃ»Ê ËÈ œẅ¾ R«RÀ ÆẗÄÄ6 ÌÅÅQ $¹Å 6 ³ ẗ 6Í»l ¾ U Q T¾( R«RÀÂÁÁUÃb»WÇ6Ÿ U RÈ6É UȾR Ÿ Z T¾ R«RÀ ÆẗÄÄ6 Î 6¹Q $Å 6 6 ³ ẗ 6Í»WÇ6Ÿ U RÈUÉ UȾR Ÿ Z T¾( R«RÀÁÁUûl ¾ U Q T¾( R«RÀ High r evenue Low COMMUNI CATI ON Most ar e male Low FUN High AGE High r at e in REGI ON 6 = Fr ankf ur t Clust er ing r esult : Business clust er

58 Most ar e f emale 10% of all user s High COMMUNI CATI ON Low r evenue High r at e in REGI ON 5 = Cologne High FUN Low AGE Clust er ing r esult : Fun clust er,) t he int er est in I NFORMATI ON is ver y low (near ly 0) $1' in COMMUNI CATI ON high (wit h at least an access rat e of 5) 7+(1 visit or will pr obably not buy (95.5%). Classif icat ion r esult

59 Sec uenc ia de c lic k s Ï6ÐÑ ÒÓÔÓÒÑ ÕÑ Ö ÐØ in 17.2% (of all t ransact ions) t he user goes t o GOURMET.ht ml ; he t hen sends t wo s out. Ï6ÐÑ ÒÓÔÓÒÑ ÕÑ Ö ÐØ in 56.9% (of all t r ansact ions) t he user goes f irst SPORTS.ht ml ; he t hen uses t he chat as a communicat ion medium; f inally, he f ocus his at t ent ion t o Fashion. Ï6ÐÑ ÒÓÔÓÒÑ ÕÑ Ö ÐØ I n 25.9% (of all t r ansact ions) t he user goes f ir st t o womens-f ashion.ht ml ; he t hen sends a post car d, and goes t o womens-f ashion.ht ml back again. Det ec c ión t em prana de m ora S(OSUREOHPD6HWUDWDGHLGHQWLILFDU DQWLFLSDGDPHQWH ORV FOLHQWHV FRQ PD\RUSRVLELOLGDGGHHQWUDUHQPRUD SDUD DQWLFLSDU ODV DFFLRQHV SUHYHQWLYDVGHFREUDQ]D\UHFXSHUR

60 Las soluc iones posibles S5HJODV SDUD LGHQWLILFDU D ORV VHJPHQWRV GH FOLHQWHV FRQ PD\RU SURSHQVLyQDPRUD S6FRULQJGHULHVJRGHPRURVLGDG Modelos aplic ables S3DUD ODV UHJODV iuero GH FODVLILFDFLyQ S3DUDHOVFRULQJPRGHORQHXURQDO

61 Arbol m uest ra 50/50 Morosos Mo ra 6 0 d ia s R e g ió n ,3 9 % d e p o b la c ió n N Y 1 Y N MO R A 6 0 VIP C U S TO ME R LATE F E E S P AID 3 0 D AYS Y N O VE R C R E D IT LIMIT C R E D IT S C O R E C U S TO ME R AG E C R E D IT LIMIT IN C O ME ME M B E R (MO N T H S ) # P U R CH AS E S / W E E K C AS H LIMIT

62 No Morosos Mo ra 6 0 dia s Re g ió n 0-2 6,8 1% de p obla ció n 0 Y N Y MO RA 60 LATE F E E S P AID 30 DAYS VIP CUS TOMER Y N OVE R CR E DIT LIMIT CUS TOME R AG E CR E DIT S COR E ME MBE R (MO NT HS ) INCO ME CAS H LIMIT C R E DIT LIMIT # P UR CHAS ES / W EEK Verific ac ión Scoring predicho N = NO SI Mora real El scoring que predice la red está netamente diferenciado para morosos y pagadores

63 Referenc ias 'DWD0LQLQJ7HFKQLTXHVIRU0DUNHWLQJ6DOHV DQG&XVWRPHU6XSSRUW0LFKDHO%HUU\*RUGRQ /LQRII:LOH\86$ 'DWD0LQLQJZLWK1HXUDO1HWZRUNV-RVHSK %LJXV0F*UDZ+LOO86$ 'DWD0LQLQJDKDQGVRQDSSURDFKIRUEXVLQHVV SURIHVVLRQDOV5REHUW*URWK3UHQWLFH+DOO 86$ 0DVWHULQJ'DWD0LQLQJ0LFKDHO%HUU\*RUGRQ /LQRII:LOH\86$ Referenc ias S'DWDSUHSDUDWLRQIRU'DWD0LQLQJ'RULDQ3\OH 0RUJDQ.DXIPDQQ3XEOLVKHUV,QF6DQ)UDQFLVFR 86$ S $QiOLVLV0XOWLYDULDQWH+DLU$QGHUVRQ7DWKDP %ODFN3UHQWLFH+DOO0DGULG S%XLOGLQJ'DWD0LQLQJDSSOLFDWLRQVIRU&50$ %HUVRQ66PLWK.7KHDUOLQJ0F*UDZ+LOO

64 Referenc ias,%0 Ù ZZZLEPFRPVRIWZDUHGDWDLPLQHUIRUGDWD Ù ZZZGPJRUJ Ù ZZZLEPFRPUHGERRNV 7KH'DWD0LQHZZZWKHGDWDPLQHFRP.''0LQHZZZNGQXJJHWVFRP FKE#FHQVXVFRPDU