Thank you! NetMine Data mining on networks IIS AWSOM. Outline. Proposed method. Goals

Transcription

1 NetMine Data mining on networks IIS Christos Faloutsos (CMU) Michalis Faloutsos (UCR) Peggy Agouris George Kollios Fillia Makedon Betty Salzberg Anthony Stefanidis Thank you! NSF-IDM 04 C. Faloutsos & M. Faloutsos 1 NSF-IDM 04 C. Faloutsos & M. Faloutsos 2 Single-link traffic characterization AWSOM Given: a semi-infinite time sequence x 1, x 2, x t, Find: patterns, outliers, and forecast (Traffic matrix) NSF-IDM 04 C. Faloutsos & M. Faloutsos 3 Periodicity? (daily) NSF-IDM 04 C. Faloutsos & M. Faloutsos 4 Goals AUTOMATIC Adapt and handle arbitrary periodic components No human intervention/tuning STREAMING Single pass over the data Limited memory (logarithmic) Any-time frequency Proposed method W l,t W l,t-2 W l,t = b l,1 W l,t-1 + b l,2 W l,t-2 + W l,t-1 W W W l,t -2 l,t -1 l,t W l,t = b l,1 W l,t -1 + b l,2 W l,t -2 + time [VLDB 03] Math NSF-IDM 04 C. Faloutsos & M. Faloutsos 5 NSF-IDM 04 C. Faloutsos & M. Faloutsos 6 1

2 Results Real data Automobile Results Real data Sunspot Automobile traffic Daily periodicity with rush-hour peaks AR fails to capture any trend (average) Seasonal AR estimation fails AWSOM captures periodicity automatically Sunspot intensity Slightly time-varying period AR captures wrong trend (average) Seasonal ARIMA Captures immediate wrong downward trend Requires human to determine seasonal component period (fixed) NSF-IDM 04 C. Faloutsos & M. Faloutsos 7 NSF-IDM 04 C. Faloutsos & M. Faloutsos 8 The SELFIS tool Traffic characterization Is it Poisson? Is there long range dependency (~ self-similarity)? Which of the 4+ methods for LRD should we use? NSF-IDM 04 C. Faloutsos & M. Faloutsos 9 NSF-IDM 04 C. Faloutsos & M. Faloutsos 10 The SELFIS tool Traffic characterization LRD analysis tool by Karagiannis at UCR downloads, from around the world NSF-IDM 04 C. Faloutsos & M. Faloutsos 11 NSF-IDM 04 C. Faloutsos & M. Faloutsos 12 2

3 Introduction Framework Given a graph, (computer/social/ etc network) Will a virus/rumor/fad take over? Needle exchange networks of drug users [Weeks et al. 2002] Susceptible-Infected-Susceptible (SIS) model (like, say, flu ) Cured nodes immediately become susceptible β : (virus) birth rate = prob. an infected neighbor attacks me δ : (virus) death rate = prob. I heal NSF-IDM 04 C. Faloutsos & M. Faloutsos 13 NSF-IDM 04 C. Faloutsos & M. Faloutsos 14 t Defined as the value of τ, such that if β / δ < τ an epidemic can not happen Thus, given a graph compute its epidemic threshold t What should τ depend on? avg. degree? and/or highest degree? and/or variance of degree? and/or something else? NSF-IDM 04 C. Faloutsos & M. Faloutsos 15 NSF-IDM 04 C. Faloutsos & M. Faloutsos 16 Homogeneous graphs: 1/<k> BA (γ=3) <k> / <k 2 > more complicated graphs? arbitrary, REAL graphs? [Theorem] We have no epidemic, if ß/d <t = 1/? 1,A how many parameters?? NSF-IDM 04 C. Faloutsos & M. Faloutsos 17 NSF-IDM 04 C. Faloutsos & M. Faloutsos 18 3

4 [Theorem] We have no epidemic, if recovery prob. attack prob. Proof: [Wang+03] epidemic threshold ß/d <t = 1/? 1,A largest eigenvalue of adj. matrix A Experiments 2 graphs Star network: one hub + 99 spokes Oregon Internet AS graph: 10,900 nodes, edges topology.eecs.umich.edu/data.html NSF-IDM 04 C. Faloutsos & M. Faloutsos 19 NSF-IDM 04 C. Faloutsos & M. Faloutsos 20 Number of Infected Nodes Experiments (Star) Star ß= Time d: ß/d > t (above threshold) ß/d = t (at the threshold) ß/d < t (below threshold) NSF-IDM 04 C. Faloutsos & M. Faloutsos 21 Number of Infected Nodes Experiments (Oregon) Oregon ß = Time d: ß/d > t (above threshold) ß/d = t (at the threshold) ß/d < t (below threshold) NSF-IDM 04 C. Faloutsos & M. Faloutsos 22 Overall conclusions Automatic stream mining: AWSOM; Tool for Long Range Dependencies (LRD): SELFIS graphs and virus propagation: eigenvalue Related work Network traffic network tomography [w/ Airoldi +] Graphs and topology graph partitioning [w/ Deepay+] important subgraphs [w/ Tomkins + McCurley] graph generators [RMAT, w/ Deepay] NSF-IDM 04 C. Faloutsos & M. Faloutsos 23 NSF-IDM 04 C. Faloutsos & M. Faloutsos 24 4

5 SpirosPapadimitriou, Anthony Brockwell and Christos Faloutsos Adaptive, Hands-Off Stream Mining VLDB 2003, Berlin, Germany, Sept (invited for fast track publication to VLDB-Journal) Yasushi Sakurai, Spiros Papadimitriou, Christos Faloutsos AutoLag: Automatic Discovery of Lag Correlations in Stream Data. ICDE 2005 Thomas Karagiannis, Mart Molle, Michalis Faloutsos, Andre Broido A Nonstationary Poisson View of Internet Traffic IEEE INFOCOM, Hong Kong, March Thomas Karagiannis, Michalis Faloutsos, Mart Molle A User-Friendly Self-Similarity Analysis Tool Special Section on Tools and Technologies for Networking Research and Education, ACM SIGCOMM Computer Communication Review, NSF-IDM 04 C. Faloutsos & M. Faloutsos 25 NSF-IDM 04 C. Faloutsos & M. Faloutsos 26 [Wang+03] Yang Wang, Deepayan Chakrabarti, Chenxi Wang and Christos Faloutsos: Epidemic Spreading in Real Networks: an Eigenvalue Viewpoint, SRDS 2003, Florence, Italy. Additional References Christos Faloutsos, Kevin McCurley and Andrew Tomkins, Fast Discovery of 'Connection Subgraphs' KDD 2004, Seattle, WA, Aug Deepayan Chakrabarti, Spiros Papadimitriou, Dharmendra Modha and Christos Faloutsos Fully Automatic Cross-Assocations KDD 2004, Seattle, WA, Aug Edoardo Airoldiand Christos Faloutsos Recovering Latent Time- Series from their Observed Sums: Network Tomography with Particle Filters KDD 2004, Seattle, WA, Aug RMAT: A recursive graph generator, D. Chakrabarti, Y. Zhan, C. Faloutsos, SIAM-DM 2004 NSF-IDM 04 C. Faloutsos & M. Faloutsos 27 NSF-IDM 04 C. Faloutsos & M. Faloutsos 28 Thank you! Contact info: google christos faloutsos cs.cmu.edu NSF-IDM 04 C. Faloutsos & M. Faloutsos 29 5