Discovering Roles and Anomalies in Graphs: Theory and Applications
|
|
|
- Aldous Hall
- 7 years ago
- Views:
Transcription
1 Discovering Roles and Anomalies in Graphs: Theory and Applications Part 2: patterns, anomalies and applications Tina Eliassi-Rad (Rutgers) Christos Faloutsos (CMU)
2 OVERVIEW - high level: Features Roles Anomalies = rare roles Patterns T. Eliassi-Rad & C. Faloutsos 2
3 Resource: Open source system for mining huge graphs: PEGASUS project (PEta GrAph mining System) code and papers T. Eliassi-Rad & C. Faloutsos 3
4 Patterns in graphs Roadmap overview Static graphs Weighted graphs Time-evolving graphs Anomaly Detection Application: ebay fraud Conclusions T. Eliassi-Rad & C. Faloutsos 4
5 Graphs - why should we care? Food Web [Martinez 91] Friendship Network [Moody 01] Internet Map [lumeta.com] T. Eliassi-Rad & C. Faloutsos 5
6 Graphs - why should we care? IR: bi-partite graphs (doc-terms) D T 1 web: hyper-text graph D N T M... and more: T. Eliassi-Rad & C. Faloutsos 6
7 Graphs - why should we care? viral marketing web-log ( blog ) news propagation computer network security: /ip traffic and anomaly detection... T. Eliassi-Rad & C. Faloutsos 7
8 Problem #1 - network and graph mining What does the Internet look like? What does FaceBook look like? What is normal / abnormal? which patterns/laws hold? T. Eliassi-Rad & C. Faloutsos 8
9 Problem #1 - network and graph mining What does the Internet look like? What does FaceBook look like? What is normal / abnormal? which patterns/laws hold? To spot anomalies (rarities), we have to discover patterns T. Eliassi-Rad & C. Faloutsos 9
10 Problem #1 - network and graph mining What does the Internet look like? What does FaceBook look like? What is normal / abnormal? which patterns/laws hold? To spot anomalies (rarities), we have to discover patterns Large datasets reveal patterns/anomalies that may be invisible otherwise T. Eliassi-Rad & C. Faloutsos 10
11 Graph mining Are real graphs random? T. Eliassi-Rad & C. Faloutsos 11
12 Laws and patterns Are real graphs random? A: NO!! Diameter in- and out- degree distributions other (surprising) patterns So, let s look at the data T. Eliassi-Rad & C. Faloutsos 12
13 Real Graph Patterns static dynamic unweighted P01. Power-law degree distribution [Faloutsos et. al.`99, Kleinberg et. al.`99, Chakrabarti et. al. `04, Newman`04] P02. Triangle Power Law [Tsourakakis `08] P03. Eigenvalue Power Law [Siganos et. al. `03] P04. Community structure [Flake et. al.`02, Girvan and Newman `02] P05. Clique Power Laws [Du et. al. 09] P06. Densification Power Law [Leskovec et. al.`05] P07. Small and shrinking diameter [Albert and Barabási `99, Leskovec et. al. 05, McGlohon et. al. 08] P08. Gelling point [McGlohon et. al. `08] P09. Constant size 2 nd and 3 rd connected components [McGlohon et. al. `08] P10. Principal Eigenvalue Power Law [Akoglu et. al. `08] P11. Bursty/self-similar edge/weight additions [Gomez and Santonja `98, Gribble et. al. `98, Crovella and Bestavros `99, McGlohon et.al. `08] weighted P12. Snapshot Power Law [McGlohon et. al. `08] P13. Weight Power Law [McGlohon et. al. `08] P14. Skewed call duration distributions [Vaz de Melo et. al. `10] SDM'12 RTG: Tutorial A Recursive Realistic T. Eliassi-Rad Graph & Generator C. Faloutsos using Random Typing 13 Leman Akoglu and Christos Faloutsos. ECML PKDD 09.
14 Patterns in graphs Roadmap overview Static graphs Weighted graphs Time-evolving graphs Anomaly Detection Application: ebay fraud Conclusions T. Eliassi-Rad & C. Faloutsos 14
15 Solution# S.1 Power law in the degree distribution [SIGCOMM99] internet domains log(degree) att.com ibm.com log(rank) T. Eliassi-Rad & C. Faloutsos 15
16 Solution# S.1 Power law in the degree distribution [SIGCOMM99] internet domains log(degree) att.com ibm.com log(rank) T. Eliassi-Rad & C. Faloutsos 16
17 Solution# S.2: Eigen Exponent E Eigenvalue Exponent = slope E = May 2001 Rank of decreasing eigenvalue A2: power law in the eigenvalues of the adjacency matrix T. Eliassi-Rad & C. Faloutsos 17
18 Solution# S.2: Eigen Exponent E Eigenvalue Exponent = slope E = May 2001 Rank of decreasing eigenvalue [Mihail, Papadimitriou 02]: slope is ½ of rank exponent T. Eliassi-Rad & C. Faloutsos 18
19 But: How about graphs from other domains? T. Eliassi-Rad & C. Faloutsos 19
20 More power laws: web hit counts [w/ A. Montgomery] Web Site Traffic Count (log scale) Zipf ``ebay users sites in-degree (log scale) T. Eliassi-Rad & C. Faloutsos 20
21 epinions.com count who-trusts-whom [Richardson + Domingos, KDD 2001] trusts-2000-people user (out) degree T. Eliassi-Rad & C. Faloutsos 21
22 And numerous more # of sexual contacts Income [Pareto] distribution Duration of downloads [Bestavros+] Duration of UNIX jobs ( mice and elephants ) Size of files of a user Black swans T. Eliassi-Rad & C. Faloutsos 22
23 Patterns in graphs overview Static graphs Roadmap S1: Degree, S2: eigenvalues S3-4: Triangles, S5: cliques Radius plot Other observations ( eigenspokes ) Weighted graphs Time-evolving graphs T. Eliassi-Rad & C. Faloutsos 23
24 Solution# S.3: Triangle Laws Real social networks have a lot of triangles T. Eliassi-Rad & C. Faloutsos 24
25 Solution# S.3: Triangle Laws Real social networks have a lot of triangles Friends of friends are friends Any patterns? T. Eliassi-Rad & C. Faloutsos 25
26 Triangle Law: #S.3 [Tsourakakis ICDM 2008] HEP-TH ASN Epinions X-axis: # of participating triangles Y: count (~ pdf) T. Eliassi-Rad & C. Faloutsos 26
27 Triangle Law: #S.3 [Tsourakakis ICDM 2008] HEP-TH ASN Epinions X-axis: # of participating triangles Y: count (~ pdf) T. Eliassi-Rad & C. Faloutsos 27
28 Triangle Law: #S.4 [Tsourakakis ICDM 2008] Reuters SN Epinions X-axis: degree Y-axis: mean # triangles n friends -> ~n 1.6 triangles T. Eliassi-Rad & C. Faloutsos 28
29 Triangle Law: Computations [Tsourakakis ICDM 2008] details But: triangles are expensive to compute (3-way join; several approx. algos) Q: Can we do that quickly? T. Eliassi-Rad & C. Faloutsos 29
30 Triangle Law: Computations [Tsourakakis ICDM 2008] details But: triangles are expensive to compute (3-way join; several approx. algos) Q: Can we do that quickly? A: Yes! #triangles = 1/6 Sum ( λ i 3 ) (and, because of skewness (S2), we only need the top few eigenvalues! T. Eliassi-Rad & C. Faloutsos 30
31 Triangle Law: Computations [Tsourakakis ICDM 2008] details 1000x+ speed-up, >90% accuracy T. Eliassi-Rad & C. Faloutsos 31
32 Triangle counting for large graphs? Anomalous nodes in Twitter(~ 3 billion edges) [U Kang, Brendan Meeder, +, PAKDD 11] T. Eliassi-Rad & C. Faloutsos 32
33 Triangle counting for large graphs? Anomalous nodes in Twitter(~ 3 billion edges) [U Kang, Brendan Meeder, +, PAKDD 11] T. Eliassi-Rad & C. Faloutsos 33
34 Triangle counting for large graphs? Anomalous nodes in Twitter(~ 3 billion edges) [U Kang, Brendan Meeder, +, PAKDD 11] T. Eliassi-Rad & C. Faloutsos 34
35 Triangle counting for large graphs? Anomalous nodes in Twitter(~ 3 billion edges) [U Kang, Brendan Meeder, +, PAKDD 11] T. Eliassi-Rad & C. Faloutsos 35
36 Triangle counting for large graphs? Anomalous nodes in Twitter(~ 3 billion edges) [U Kang, Brendan Meeder, +, PAKDD 11] T. Eliassi-Rad & C. Faloutsos 36
37 Triangle counting for large graphs? Q: How to compute # triangles in B-node graph? (O(d max ** 2) )? T. Eliassi-Rad & C. Faloutsos 37
38 Triangle counting for large graphs? Q: How to compute # triangles in B-node graph? (O(d max ** 2) )? A: cubes of eigvals T. Eliassi-Rad & C. Faloutsos 38
39 Patterns in graphs overview Static graphs Roadmap S1: Degree, S2: eigenvalues S3-4: Triangles, S5: cliques Radius plot Other observations ( eigenspokes ) Weighted graphs Time-evolving graphs T. Eliassi-Rad & C. Faloutsos 39
40 How about cliques? T. Eliassi-Rad & C. Faloutsos 40
41 Large Human Communication Networks Patterns and a Utility-Driven Generator Nan Du, Christos Faloutsos, Bai Wang, Leman Akoglu KDD 2009
42 Cliques Clique is a complete subgraph. If a clique can not be contained by any larger clique, it is called the maximal clique T. Eliassi-Rad & C. Faloutsos 42
43 Clique Clique is a complete subgraph. If a clique can not be contained by any larger clique, it is called the maximal clique T. Eliassi-Rad & C. Faloutsos 43
44 Clique Clique is a complete subgraph. If a clique can not be contained by any larger clique, it is called the maximal clique T. Eliassi-Rad & C. Faloutsos 44
45 Clique Clique is a complete subgraph. If a clique can not be contained by any larger clique, it is called the maximal clique. {0,1,2}, {0,1,3}, {1,2,3} {2,3,4}, {0,1,2,3} are cliques; {0,1,2,3} and {2,3,4} are the maximal cliques T. Eliassi-Rad & C. Faloutsos 45
46 S5: Clique-Degree Power-Law Power law: d i C d avg i α # maximal cliques of node i degree of node i α is the power law exponent α [., 1822.] for S1~S3 More friends, even more social circles! T. Eliassi-Rad & C. Faloutsos 46
47 S5: Clique-Degree Power-Law Outlier Detection T. Eliassi-Rad & C. Faloutsos 47
48 S5: Clique-Degree Power-Law Outlier Detection T. Eliassi-Rad & C. Faloutsos 48
49 Patterns in graphs overview Static graphs Roadmap S1: Degree, S2: eigenvalues S3-4: Triangles, S5: cliques Radius plot Other observations ( eigenspokes ) Weighted graphs Time-evolving graphs T. Eliassi-Rad & C. Faloutsos 49
50 HADI for diameter estimation Radius Plots for Mining Tera-byte Scale Graphs U Kang, Charalampos Tsourakakis, Ana Paula Appel, Christos Faloutsos, Jure Leskovec, SDM 10 Naively: diameter needs O(N**2) space and up to O(N**3) time prohibitive (N~1B) Our HADI: linear on E (~10B) Near-linear scalability wrt # machines Several optimizations -> 5x faster T. Eliassi-Rad & C. Faloutsos 50
51 Count???? 19+ [Barabasi+] ~1999, ~1M nodes Radius T. Eliassi-Rad & C. Faloutsos 51
52 Count?????? 19+ [Barabasi+] ~1999, ~1M nodes Radius YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) Largest publicly available graph ever studied. T. Eliassi-Rad & C. Faloutsos 52
53 Count 14 (dir.)???? ~7 (undir.) 19+? [Barabasi+] Radius YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) Largest publicly available graph ever studied. T. Eliassi-Rad & C. Faloutsos 53
54 Count 14 (dir.)???? ~7 (undir.) 19+? [Barabasi+] YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) 7 degrees of separation (!) Diameter: shrunk T. Eliassi-Rad & C. Faloutsos Radius 54
55 Count???? ~7 (undir.) Radius YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) Q: Shape? T. Eliassi-Rad & C. Faloutsos 55
56 YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) effective diameter: surprisingly small. Multi-modality (?!) T. Eliassi-Rad & C. Faloutsos 56
57 Radius Plot of GCC of YahooWeb. T. Eliassi-Rad & C. Faloutsos 57
58 YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) effective diameter: surprisingly small. Multi-modality: probably mixture of cores. T. Eliassi-Rad & C. Faloutsos 58
59 Conjecture: EN DE ~7 BR YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) effective diameter: surprisingly small. Multi-modality: probably mixture of cores. T. Eliassi-Rad & C. Faloutsos 59
60 Conjecture: ~7 YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) effective diameter: surprisingly small. Multi-modality: probably mixture of cores. T. Eliassi-Rad & C. Faloutsos 60
61 Patterns in graphs overview Static graphs Roadmap S1: Degree, S2: eigenvalues S3-4: Triangles, S5: cliques Radius plot Other observations ( eigenspokes ) Weighted graphs Time-evolving graphs T. Eliassi-Rad & C. Faloutsos 61
62 S6: EigenSpokes B. Aditya Prakash, Mukund Seshadri, Ashwin Sridharan, Sridhar Machiraju and Christos Faloutsos: EigenSpokes: Surprising Patterns and Scalable Community Chipping in Large Graphs, PAKDD 2010, Hyderabad, India, June T. Eliassi-Rad & C. Faloutsos 62
63 EigenSpokes Eigenvectors of adjacency matrix equivalent to singular vectors (symmetric, undirected graph) A = U U T T. Eliassi-Rad & C. Faloutsos 63
64 EigenSpokes details Eigenvectors of adjacency matrix equivalent to singular vectors (symmetric, undirected graph) N A = U U T N u 1 u i 64 T. Eliassi-Rad & C. Faloutsos
65 EigenSpokes details Eigenvectors of adjacency matrix equivalent to singular vectors (symmetric, undirected graph) N A = U U T N u 1 u i 65 T. Eliassi-Rad & C. Faloutsos
66 EigenSpokes details Eigenvectors of adjacency matrix equivalent to singular vectors (symmetric, undirected graph) N A = U U T N u 1 u i 66 T. Eliassi-Rad & C. Faloutsos
67 EigenSpokes details Eigenvectors of adjacency matrix equivalent to singular vectors (symmetric, undirected graph) N A = U U T N u 1 u i 67 T. Eliassi-Rad & C. Faloutsos
68 EE plot: Scatter plot of scores of u1 vs u2 One would expect Many origin A few scattered ~randomly EigenSpokes 2 nd Principal component u2 u1 1 st Principal component T. Eliassi-Rad & C. Faloutsos 68
69 EigenSpokes EE plot: Scatter plot of scores of u1 vs u2 One would expect Many origin A few scattered ~randomly u2 90 o u1 T. Eliassi-Rad & C. Faloutsos 69
70 EigenSpokes - pervasiveness Present in mobile social graph across time and space Patent citation graph T. Eliassi-Rad & C. Faloutsos 70
71 EigenSpokes - explanation Near-cliques, or nearbipartite-cores, loosely connected T. Eliassi-Rad & C. Faloutsos 71
72 EigenSpokes - explanation Near-cliques, or nearbipartite-cores, loosely connected T. Eliassi-Rad & C. Faloutsos 72
73 EigenSpokes - explanation Near-cliques, or nearbipartite-cores, loosely connected T. Eliassi-Rad & C. Faloutsos 73
74 EigenSpokes - explanation Near-cliques, or nearbipartite-cores, loosely connected spy plot of top 20 nodes So what? Extract nodes with high scores high connectivity Good communities T. Eliassi-Rad & C. Faloutsos 74
75 Bipartite Communities! patents from same inventor(s) `cut-and-paste bibliography! magnified bipartite community T. Eliassi-Rad & C. Faloutsos 75
76 Patterns in graphs Roadmap overview Static graphs Weighted graphs Time-evolving graphs Anomaly Detection Application: ebay fraud Conclusions T. Eliassi-Rad & C. Faloutsos 76
77 Observations on weighted graphs? A: yes - even more laws! M. McGlohon, L. Akoglu, and C. Faloutsos Weighted Graphs and Disconnected Components: Patterns and a Generator. SIG-KDD 2008 T. Eliassi-Rad & C. Faloutsos 77
78 Observation W.1: Fortification Q: How do the weights of nodes relate to degree? T. Eliassi-Rad & C. Faloutsos 78
79 Observation W.1: Fortification More donors, more $? $10 Reagan $5 $7 Clinton T. Eliassi-Rad & C. Faloutsos 79
80 Observation W.1: fortification: Snapshot Power Law Weight: super-linear on in-degree exponent iw : 1.01 < iw < 1.26 More donors, even more $ $10 In-weights ($) Orgs-Candidates e.g. John Kerry, $10M received, from 1K donors $5 Edges (# donors) T. Eliassi-Rad & C. Faloutsos 80
81 Patterns in graphs Roadmap overview Static graphs Weighted graphs Time-evolving graphs Anomaly Detection Application: ebay fraud Conclusions T. Eliassi-Rad & C. Faloutsos 81
82 Problem: Time evolution with Jure Leskovec (CMU -> Stanford) and Jon Kleinberg (Cornell CMU) T. Eliassi-Rad & C. Faloutsos 82
83 T.1 Evolution of the Diameter Prior work on Power Law graphs hints at slowly growing diameter: diameter ~ O(log N) diameter ~ O(log log N) What is happening in real data? T. Eliassi-Rad & C. Faloutsos 83
84 T.1 Evolution of the Diameter Prior work on Power Law graphs hints at slowly growing diameter: diameter ~ O(log N) diameter ~ O(log log N) What is happening in real data? Diameter shrinks over time T. Eliassi-Rad & C. Faloutsos 84
85 T.1 Diameter Patents Patent citation network 25 years of 2.9 M nodes 16.5 M edges diameter time [years] T. Eliassi-Rad & C. Faloutsos 85
86 T.2 Temporal Evolution of the Graphs N(t) nodes at time t E(t) edges at time t Suppose that N(t+1) = 2 * N(t) Q: what is your guess for E(t+1) =? 2 * E(t) T. Eliassi-Rad & C. Faloutsos 86
87 T.2 Temporal Evolution of the Graphs N(t) nodes at time t E(t) edges at time t Suppose that N(t+1) = 2 * N(t) Q: what is your guess for E(t+1) =? 2 * E(t) A: over-doubled! But obeying the ``Densification Power Law T. Eliassi-Rad & C. Faloutsos 87
88 T.2 Densification Patent Citations Citations among patents 2.9 M nodes 16.5 M edges Each year is a datapoint E(t) 1.66 N(t) T. Eliassi-Rad & C. Faloutsos 88
89 Patterns in graphs Roadmap Time-evolving graphs T1: shrinking diameter; T2: densification T3: connected components T4: popularity over time T5: phonecall patterns T. Eliassi-Rad & C. Faloutsos 89
90 More on Time-evolving graphs M. McGlohon, L. Akoglu, and C. Faloutsos Weighted Graphs and Disconnected Components: Patterns and a Generator. SIG-KDD 2008 T. Eliassi-Rad & C. Faloutsos 90
91 Observation T.3: NLCC behavior Q: How do NLCC s emerge and join with the GCC? (``NLCC = non-largest conn. components) Do they continue to grow in size? or do they shrink? or stabilize? T. Eliassi-Rad & C. Faloutsos 91
92 Observation T.3: NLCC behavior Q: How do NLCC s emerge and join with the GCC? (``NLCC = non-largest conn. components) Do they continue to grow in size? or do they shrink? or stabilize? T. Eliassi-Rad & C. Faloutsos 92
93 Observation T.3: NLCC behavior Q: How do NLCC s emerge and join with the GCC? YES YES YES (``NLCC = non-largest conn. components) Do they continue to grow in size? or do they shrink? or stabilize? T. Eliassi-Rad & C. Faloutsos 93
94 Observation T.3: NLCC behavior After the gelling point, the GCC takes off, but NLCC s remain ~constant (actually, oscillate). CC size IMDB Time-stamp T. Eliassi-Rad & C. Faloutsos 94
95 (Computation scalability?) Q: How to handle billion node graphs? A: hadoop + Pegasus Most operations -> matrix-vector multiplications T. Eliassi-Rad & C. Faloutsos 95
96 Generalized Iterated Matrix Vector Multiplication (GIMV) PEGASUS: A Peta-Scale Graph Mining System - Implementation and Observations. U Kang, Charalampos E. Tsourakakis, and Christos Faloutsos. (ICDM) 2009, Miami, Florida, USA. Best Application Paper (runner-up). T. Eliassi-Rad & C. Faloutsos 96
97 Generalized Iterated Matrix details Vector Multiplication (GIMV) PageRank proximity (RWR) Diameter Connected components (eigenvectors, Belief Prop. ) Matrix vector Multiplication (iterated) T. Eliassi-Rad & C. Faloutsos 97
98 Example: GIM-V At Work Connected Components 4 observations: Count Size T. Eliassi-Rad & C. Faloutsos 98
99 Example: GIM-V At Work Connected Components Count 1) 10K x larger than next Size T. Eliassi-Rad & C. Faloutsos 99
100 Example: GIM-V At Work Connected Components Count 2) ~0.7B singleton nodes Size T. Eliassi-Rad & C. Faloutsos 100
101 Example: GIM-V At Work Connected Components Count 3) SLOPE! Size T. Eliassi-Rad & C. Faloutsos 101
102 Example: GIM-V At Work Connected Components Count 300-size cmpt X size cmpt Why? X 65. Why? 4) Spikes! Size T. Eliassi-Rad & C. Faloutsos 102
103 Example: GIM-V At Work Connected Components Count suspicious financial-advice sites (not existing now) Size T. Eliassi-Rad & C. Faloutsos 103
104 GIM-V At Work Connected Components over Time LinkedIn: 7.5M nodes and 58M edges Stable tail slope after the gelling point T. Eliassi-Rad & C. Faloutsos 104
105 Patterns in graphs Roadmap Time-evolving graphs T1: shrinking diameter; T2: densification T3: connected components T4: popularity over time T5: phonecall patterns T. Eliassi-Rad & C. Faloutsos 105
106 Timing for Blogs with Mary McGlohon (CMU->Google) Jure Leskovec (CMU->Stanford) Natalie Glance (now at Google) Mat Hurst (now at MSR) [SDM 07] T. Eliassi-Rad & C. Faloutsos 106
107 T.4 : popularity over time # in links lag: days after post Post popularity + lag T. Eliassi-Rad & C. Faloutsos 107
108 T.4 : popularity over time # in links (log) Post popularity drops-off exponentially? POWER LAW! Exponent? days after post (log) T. Eliassi-Rad & C. Faloutsos 108
109 T.4 : popularity over time # in links (log) -1.6 Post popularity drops-off exponentially? POWER LAW! Exponent? -1.6 close to -1.5: Barabasi s stack model and like the zero-crossings of a random walk days after post (log) T. Eliassi-Rad & C. Faloutsos 109
110 -1.5 slope J. G. Oliveira & A.-L. Barabási Human Dynamics: The Correspondence Patterns of Darwin and Einstein. Nature 437, 1251 (2005). [PDF] Log Prob() -1.5 Log # days to respond T. Eliassi-Rad & C. Faloutsos 110
111 Patterns in graphs Roadmap Time-evolving graphs T1: shrinking diameter; T2: densification T3: connected components T4: popularity over time T5: phonecall patterns T. Eliassi-Rad & C. Faloutsos 111
112 T.5: duration of phonecalls Surprising Patterns for the Call Duration Distribution of Mobile Phone Users Pedro O. S. Vaz de Melo, Leman Akoglu, Christos Faloutsos, Antonio A. F. Loureiro PKDD 2010 T. Eliassi-Rad & C. Faloutsos 112
113 Probably, power law (?)?? T. Eliassi-Rad & C. Faloutsos 113
114 No Power Law! T. Eliassi-Rad & C. Faloutsos 114
115 TLaC: Lazy Contractor The longer a task (phonecall) has taken, The even longer it will take Odds ratio= Casualties(<x): Survivors(>=x) == power law T. Eliassi-Rad & C. Faloutsos 115
116 Data Description n Data from a private mobile operator of a large city n n n 4 months of data 3.1 million users more than 1 billion phone records n Over 96% of talkative users obeyed a TLAC distribution ( talkative : >30 calls) T. Eliassi-Rad & C. Faloutsos 116
117 Outliers: T. Eliassi-Rad & C. Faloutsos 117
118 Real Graph Patterns static dynamic unweighted P01. Power-law degree distribution [Faloutsos et. al.`99, Kleinberg et. al.`99, Chakrabarti et. al. `04, Newman`04] P02. Triangle Power Law [Tsourakakis `08] P03. Eigenvalue Power Law [Siganos et. al. `03] P04. Community structure [Flake et. al.`02, Girvan and Newman `02] P05. Clique Power Laws [Du et. al. 09] P06. Densification Power Law [Leskovec et. al.`05] P07. Small and shrinking diameter [Albert and Barabási `99, Leskovec et. al. 05, McGlohon et. al. 08] P08. Gelling point [McGlohon et. al. `08] P09. Constant size 2 nd and 3 rd connected components [McGlohon et. al. `08] P10. Principal Eigenvalue Power Law [Akoglu et. al. `08] P11. Bursty/self-similar edge/weight additions [Gomez and Santonja `98, Gribble et. al. `98, Crovella and Bestavros `99, McGlohon et.al. `08] weighted P12. Snapshot Power Law [McGlohon et. al. `08] P13. Weight Power Law [McGlohon et. al. `08] P14. Skewed call duration distributions [Vaz de Melo et. al. `10] SDM'12 RTG: Tutorial A Recursive Realistic T. Eliassi-Rad Graph & Generator C. Faloutsos using Random Typing 118 Leman Akoglu and Christos Faloutsos. ECML PKDD 09.
119 Patterns in graphs Roadmap overview Static graphs Weighted graphs Time-evolving graphs Anomaly Detection Application: ebay fraud Conclusions T. Eliassi-Rad & C. Faloutsos 119
120 OddBall: Spotting Anomalies in Weighted Graphs Leman Akoglu, Mary McGlohon, Christos Faloutsos Carnegie Mellon University School of Computer Science PAKDD 2010, Hyderabad, India
121 For each node, Main idea extract ego-net (=1-step-away neighbors) Extract features (#edges, total weight, etc etc) Compare with the rest of the population T. Eliassi-Rad & C. Faloutsos 121
122 What is an egonet? ego egonet T. Eliassi-Rad & C. Faloutsos 122
123 Selected Features N i : number of neighbors (degree) of ego i E i : number of edges in egonet i W i : total weight of egonet i λ w,i : principal eigenvalue of the weighted adjacency matrix of egonet I T. Eliassi-Rad & C. Faloutsos 123
124 Near-Clique/Star T. Eliassi-Rad & C. Faloutsos 124
125 Near-Clique/Star T. Eliassi-Rad & C. Faloutsos 125
126 Near-Clique/Star T. Eliassi-Rad & C. Faloutsos 126
127 Near-Clique/Star Andrew Lewis (director) T. Eliassi-Rad & C. Faloutsos 127
128 Dominant Heavy Link T. Eliassi-Rad & C. Faloutsos
129 Patterns in graphs Roadmap overview Static graphs Weighted graphs Time-evolving graphs Anomaly Detection Application: ebay fraud Conclusions T. Eliassi-Rad & C. Faloutsos 129
130 NetProbe: The Problem Find bad sellers (fraudsters) on ebay who don t deliver their (expensive) items $$$ X T. Eliassi-Rad & C. Faloutsos 130
131 E-bay Fraud detection w/ Polo Chau & Shashank Pandit, CMU [www 07] T. Eliassi-Rad & C. Faloutsos 131
132 E-bay Fraud detection T. Eliassi-Rad & C. Faloutsos 132
133 E-bay Fraud detection T. Eliassi-Rad & C. Faloutsos 133
134 E-bay Fraud detection - NetProbe T. Eliassi-Rad & C. Faloutsos 134
135 NetProbe: Key Ideas Fraudsters fabricate their reputation by trading with their accomplices Transactions form near bipartite cores How to detect them? T. Eliassi-Rad & C. Faloutsos 135
136 NetProbe: Key Ideas Use Belief Propagation and ~heterophily F A H Fraudster Accomplice Honest Darker means more likely T. Eliassi-Rad & C. Faloutsos 136
137 NetProbe: Main Results T. Eliassi-Rad & C. Faloutsos 137
138 Roadmap Patterns in graphs Anomaly Detection Application: ebay fraud How-to: Belief Propagation Conclusions T. Eliassi-Rad & C. Faloutsos 138
139 Guilt-by-Association Techniques details red green Given: graph and few labeled nodes Find: class (red/green) for rest nodes Assuming: network effects (homophily/ heterophily, etc) F A H T. Eliassi-Rad & C. Faloutsos 139
140 Correspondence of Methods details Random Walk with Restarts (RWR) Google Semi-supervised Learning (SSL) Belief Propagation (BP) Bayesian T. Eliassi-Rad & C. Faloutsos 140
141 Correspondence of Methods details Random Walk with Restarts (RWR) Semi-supervised Learning (SSL) Belief Propagation (BP) Method Matrix unknown known RWR [I c AD -1 ] x = (1-c)y SSL [I + a(d - A)] x = y FABP [I + a D - c A] b h = φ h ? 0 1 Unifying Guilt-by-Association Approaches: Theorems T. Eliassi-Rad & C. Faloutsos 141 and Fast Algorithms. Danai Koutra, et al PKDD 11
142 Roadmap Patterns in graphs Anomaly Detection Application: ebay fraud Conclusions T. Eliassi-Rad & C. Faloutsos 142
143 Roles: Overall conclusions Past work in social networks ( regular, structural etc) Scalable algo s to find such roles Anomalies & patterns Static (power-laws, six degrees ) Weighted (super-linearity) Time-evolving (densification, -1.5 exponent) T. Eliassi-Rad & C. Faloutsos 143
144 OVERALL CONCLUSIONS high level: Features Roles Anomalies = rare roles Patterns T. Eliassi-Rad & C. Faloutsos 144
145 OVERALL CONCLUSIONS high level BIG DATA: -> roles/patterns/outliers that are invisible otherwise T. Eliassi-Rad & C. Faloutsos 145
146 References Leman Akoglu, Christos Faloutsos: RTG: A Recursive Realistic Graph Generator Using Random Typing. ECML/PKDD (1) 2009: Deepayan Chakrabarti, Christos Faloutsos: Graph mining: Laws, generators, and algorithms. ACM Comput. Surv. 38(1): (2006) T. Eliassi-Rad & C. Faloutsos 146
147 References Deepayan Chakrabarti, Yang Wang, Chenxi Wang, Jure Leskovec, Christos Faloutsos: Epidemic thresholds in real networks. ACM Trans. Inf. Syst. Secur. 10(4): (2008) Deepayan Chakrabarti, Jure Leskovec, Christos Faloutsos, Samuel Madden, Carlos Guestrin, Michalis Faloutsos: Information Survival Threshold in Sensor and P2P Networks. INFOCOM 2007: T. Eliassi-Rad & C. Faloutsos 147
148 References Christos Faloutsos, Tamara G. Kolda, Jimeng Sun: Mining large graphs and streams using matrix and tensor tools. Tutorial, SIGMOD Conference 2007: 1174 T. Eliassi-Rad & C. Faloutsos 148
149 References T. G. Kolda and J. Sun. Scalable Tensor Decompositions for Multi-aspect Data Mining. In: ICDM 2008, pp , December T. Eliassi-Rad & C. Faloutsos 149
150 References Jure Leskovec, Jon Kleinberg and Christos Faloutsos Graphs over Time: Densification Laws, Shrinking Diameters and Possible Explanations, KDD 2005 (Best Research paper award). Jure Leskovec, Deepayan Chakrabarti, Jon M. Kleinberg, Christos Faloutsos: Realistic, Mathematically Tractable Graph Generation and Evolution, Using Kronecker Multiplication. PKDD 2005: T. Eliassi-Rad & C. Faloutsos 150
151 References Jimeng Sun, Yinglian Xie, Hui Zhang, Christos Faloutsos. Less is More: Compact Matrix Decomposition for Large Sparse Graphs, SDM, Minneapolis, Minnesota, Apr Jimeng Sun, Spiros Papadimitriou, Philip S. Yu, and Christos Faloutsos, GraphScope: Parameterfree Mining of Large Time-evolving Graphs ACM SIGKDD Conference, San Jose, CA, August 2007 T. Eliassi-Rad & C. Faloutsos 151
152 References Jimeng Sun, Dacheng Tao, Christos Faloutsos: Beyond streams and graphs: dynamic tensor analysis. KDD 2006: T. Eliassi-Rad & C. Faloutsos 152
153 References Hanghang Tong, Christos Faloutsos, and Jia-Yu Pan, Fast Random Walk with Restart and Its Applications, ICDM 2006, Hong Kong. Hanghang Tong, Christos Faloutsos, Center-Piece Subgraphs: Problem Definition and Fast Solutions, KDD 2006, Philadelphia, PA T. Eliassi-Rad & C. Faloutsos 153
154 References Hanghang Tong, Christos Faloutsos, Brian Gallagher, Tina Eliassi-Rad: Fast best-effort pattern matching in large attributed graphs. KDD 2007: T. Eliassi-Rad & C. Faloutsos 154
155 Project info Chau, Polo Koutra, Danai Prakash, Aditya Akoglu, Leman Kang, U McGlohon, Mary Tong, Hanghang Thanks to: NSF IIS , IIS , CTA-INARC; ADAMS-DARPA; Yahoo (M45), T. Eliassi-Rad & C. Faloutsos 155 LLNL, IBM, SPRINT, Google, INTEL, HP, ilab
CMU SCS Large Graph Mining Patterns, Tools and Cascade analysis
Large Graph Mining Patterns, Tools and Cascade analysis Christos Faloutsos CMU Roadmap Introduction Motivation Why big data Why (big) graphs? Patterns in graphs Tools: fraud detection on e-bay Conclusions
CMU SCS Mining Billion-Node Graphs - Patterns and Algorithms
Mining Billion-Node Graphs - Patterns and Algorithms Christos Faloutsos CMU Thank you! Panos Chrysanthis Ling Liu Vladimir Zadorozhny Prashant Krishnamurthy C. Faloutsos (CMU) 2 Resource Open source system
CMU SCS Mining Large Graphs and Tensors - Patterns, Tools and Discoveries.
Mining Large Graphs and Tensors - Patterns, Tools and Discoveries. Christos Faloutsos CMU Thank you! Nikos Sidiropoulos Kuo-Chu Chang Zhi (Gerry) Tian C. Faloutsos (CMU) 2 Roadmap Introduction Motivation
Graph Mining Techniques for Social Media Analysis
Graph Mining Techniques for Social Media Analysis Mary McGlohon Christos Faloutsos 1 1-1 What is graph mining? Extracting useful knowledge (patterns, outliers, etc.) from structured data that can be represented
School of Computer Science Carnegie Mellon Graph Mining, self-similarity and power laws
Graph Mining, self-similarity and power laws Christos Faloutsos University Overview Achievements global patterns and laws (static/dynamic) generators influence propagation communities; graph partitioning
Graphs over Time Densification Laws, Shrinking Diameters and Possible Explanations
Graphs over Time Densification Laws, Shrinking Diameters and Possible Explanations Jurij Leskovec, CMU Jon Kleinberg, Cornell Christos Faloutsos, CMU 1 Introduction What can we do with graphs? What patterns
Scaling Up HBase, Hive, Pegasus
CSE 6242 A / CS 4803 DVA Mar 7, 2013 Scaling Up HBase, Hive, Pegasus Duen Horng (Polo) Chau Georgia Tech Some lectures are partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John Stasko,
The ebay Graph: How Do Online Auction Users Interact?
The ebay Graph: How Do Online Auction Users Interact? Yordanos Beyene, Michalis Faloutsos University of California, Riverside {yordanos, michalis}@cs.ucr.edu Duen Horng (Polo) Chau, Christos Faloutsos
Big Graph Mining: Algorithms and Discoveries
Big Graph Mining: Algorithms and Discoveries U Kang and Christos Faloutsos Carnegie Mellon University {ukang, christos}@cs.cmu.edu ABSTRACT How do we find patterns and anomalies in very large graphs with
Introduction to Graph Mining
Introduction to Graph Mining What is a graph? A graph G = (V,E) is a set of vertices V and a set (possibly empty) E of pairs of vertices e 1 = (v 1, v 2 ), where e 1 E and v 1, v 2 V. Edges may contain
The Shape of the Network. The Shape of the Internet. Why study topology? Internet topologies. Early work. More on topologies..
The Shape of the Internet Slides assembled by Jeff Chase Duke University (thanks to and ) The Shape of the Network Characterizing shape : AS-level topology: who connects to whom Router-level topology:
The Internet Is Like A Jellyfish
The Internet Is Like A Jellyfish Michalis Faloutsos UC Riverside Joint work with: Leslie Tauro, Georgos Siganos (UCR) Chris Palmer(CMU) Big Picture: Modeling the Internet Topology Traffic Protocols Routing,
Jure Leskovec (@jure) Stanford University
Jure Leskovec (@jure) Stanford University KDD Summer School, Beijing, August 2012 8/10/2012 Jure Leskovec (@jure), KDD Summer School 2012 2 Graph: Kronecker graphs Graph Node attributes: MAG model Graph
Graph Processing and Social Networks
Graph Processing and Social Networks Presented by Shu Jiayu, Yang Ji Department of Computer Science and Engineering The Hong Kong University of Science and Technology 2015/4/20 1 Outline Background Graph
Big Data Analytics Process & Building Blocks
Big Data Analytics Process & Building Blocks Duen Horng (Polo) Chau Georgia Tech CSE 6242 A / CS 4803 DVA Jan 10, 2013 Partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John Stasko, Christos
Thank you! NetMine Data mining on networks IIS -0209107 AWSOM. Outline. Proposed method. Goals
NetMine Data mining on networks IIS -0209107 Christos Faloutsos (CMU) Michalis Faloutsos (UCR) Peggy Agouris George Kollios Fillia Makedon Betty Salzberg Anthony Stefanidis Thank you! NSF-IDM 04 C. Faloutsos
Guilt-by-Constellation: Fraud Detection by Suspicious Clique Memberships
Guilt-by-Constellation: Fraud Detection by Suspicious Clique Memberships Véronique Van Vlasselaer KU Leuven Veronique.VanVlasselaer @kuleuven.be Leman Akoglu Stony Brook University leman@cs.stonybrook.edu
Cost effective Outbreak Detection in Networks
Cost effective Outbreak Detection in Networks Jure Leskovec Joint work with Andreas Krause, Carlos Guestrin, Christos Faloutsos, Jeanne VanBriesen, and Natalie Glance Diffusion in Social Networks One of
Expansion Properties of Large Social Graphs
Expansion Properties of Large Social Graphs Fragkiskos D. Malliaros 1 and Vasileios Megalooikonomou 1,2 1 Computer Engineering and Informatics Department University of Patras, 26500 Rio, Greece 2 Data
Social Network Mining
Social Network Mining Data Mining November 11, 2013 Frank Takes (ftakes@liacs.nl) LIACS, Universiteit Leiden Overview Social Network Analysis Graph Mining Online Social Networks Friendship Graph Semantics
Graph-Based User Behavior Modeling: From Prediction to Fraud Detection
Graph-Based User Behavior Modeling: From Prediction to Fraud Detection Alex Beutel Carnegie Mellon University abeutel@cs.cmu.edu Christos Faloutsos Carnegie Mellon University christos@cs.cmu.edu March
Social Prediction in Mobile Networks: Can we infer users emotions and social ties?
Social Prediction in Mobile Networks: Can we infer users emotions and social ties? Jie Tang Tsinghua University, China 1 Collaborate with John Hopcroft, Jon Kleinberg (Cornell) Jinghai Rao (Nokia), Jimeng
Hadoop Based Link Prediction Performance Analysis
Hadoop Based Link Prediction Performance Analysis Yuxiao Dong, Casey Robinson, Jian Xu Department of Computer Science and Engineering University of Notre Dame Notre Dame, IN 46556, USA Email: ydong1@nd.edu,
Big Data Analytics of Multi-Relationship Online Social Network Based on Multi-Subnet Composited Complex Network
, pp.273-284 http://dx.doi.org/10.14257/ijdta.2015.8.5.24 Big Data Analytics of Multi-Relationship Online Social Network Based on Multi-Subnet Composited Complex Network Gengxin Sun 1, Sheng Bin 2 and
Big Data Analytics Building Blocks. Simple Data Storage (SQLite)
http://poloclub.gatech.edu/cse6242 CSE6242 / CX4242: Data & Visual Analytics Big Data Analytics Building Blocks. Simple Data Storage (SQLite) Duen Horng (Polo) Chau Georgia Tech Partly based on materials
Procedia Computer Science
Procedia Computer Science 00 (2015) 1 8 Procedia Computer Science Scalable Tensor Mining Lee Sael a, Inah Jeon b, U Kang b, a Department of Computer Science, State University of New York Korea, Republic
Big Data Analytics Building Blocks; Simple Data Storage (SQLite)
Big Data Analytics Building Blocks; Simple Data Storage (SQLite) Duen Horng (Polo) Chau Georgia Tech CSE6242 / CX4242 Jan 9, 2014 Partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John
Practical Graph Mining with R. 5. Link Analysis
Practical Graph Mining with R 5. Link Analysis Outline Link Analysis Concepts Metrics for Analyzing Networks PageRank HITS Link Prediction 2 Link Analysis Concepts Link A relationship between two entities
Big Data Analytics Building Blocks; Simple Data Storage (SQLite)
Big Data Analytics Building Blocks; Simple Data Storage (SQLite) Duen Horng (Polo) Chau Georgia Tech CSE6242 / CX4242 Aug 21, 2014 Partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John
NetProbe: A Fast and Scalable System for Fraud Detection in Online Auction Networks
NetProbe: A Fast and Scalable System for Fraud Detection in Online Auction Networks Shashank Pandit, Duen Horng Chau, Samuel Wang, Christos Faloutsos Carnegie Mellon University Pittsburgh, PA 15213, USA
Part 2: Community Detection
Chapter 8: Graph Data Part 2: Community Detection Based on Leskovec, Rajaraman, Ullman 2014: Mining of Massive Datasets Big Data Management and Analytics Outline Community Detection - Social networks -
Two Step Graph-based Semi-supervised Learning for Online Auction Fraud Detection
Two Step Graph-based Semi-supervised Learning for Online Auction Fraud Detection Phiradet Bangcharoensap 1, Hayato Kobayashi 2, Nobuyuki Shimizu 2, Satoshi Yamauchi 2, and Tsuyoshi Murata 1 1 Tokyo Institute
Social Networks and Social Media
Social Networks and Social Media Social Media: Many-to-Many Social Networking Content Sharing Social Media Blogs Microblogging Wiki Forum 2 Characteristics of Social Media Consumers become Producers Rich
Mining Large Graphs. ECML/PKDD 2007 tutorial. Part 2: Diffusion and cascading behavior
Mining Large Graphs ECML/PKDD 2007 tutorial Part 2: Diffusion and cascading behavior Jure Leskovec and Christos Faloutsos Machine Learning Department Joint work with: Lada Adamic, Deepay Chakrabarti, Natalie
Big Data Analytics Building Blocks. Simple Data Storage (SQLite)
http://poloclub.gatech.edu/cse6242 CSE6242 / CX4242: Data & Visual Analytics Big Data Analytics Building Blocks. Simple Data Storage (SQLite) Duen Horng (Polo) Chau Georgia Tech Partly based on materials
LINEAR-ALGEBRAIC GRAPH MINING
UCSF QB3 Seminar 5/28/215 Linear Algebraic Graph Mining, sanders29@llnl.gov 1/22 LLNL-PRES-671587 New Applications of Computer Analysis to Biomedical Data Sets QB3 Seminar, UCSF Medical School, May 28
Analyzing the Facebook graph?
Logistics Big Data Algorithmic Introduction Prof. Yuval Shavitt Contact: shavitt@eng.tau.ac.il Final grade: 4 6 home assignments (will try to include programing assignments as well): 2% Exam 8% Big Data
Graph models for the Web and the Internet. Elias Koutsoupias University of Athens and UCLA. Crete, July 2003
Graph models for the Web and the Internet Elias Koutsoupias University of Athens and UCLA Crete, July 2003 Outline of the lecture Small world phenomenon The shape of the Web graph Searching and navigation
With the increasing availability of large social network data, there is
Learning and Predicting the Evolution of Social Networks Björn Bringmann, Katholieke Universiteit Leuven Michele Berlingerio, ISTI-CNR Francesco Bonchi and Aristides Gionis, Yahoo Research Graph Evolution
HADI: Mining Radii of Large Graphs
HADI: Mining Radii of Large Graphs U KANG Carnegie Mellon University CHARALAMPOS E. TSOURAKAKIS Carnegie Mellon University ANA PAULA APPEL USP at São Carlos CHRISTOS FALOUTSOS Carnegie Mellon University
NETZCOPE - a tool to analyze and display complex R&D collaboration networks
The Task Concepts from Spectral Graph Theory EU R&D Network Analysis Netzcope Screenshots NETZCOPE - a tool to analyze and display complex R&D collaboration networks L. Streit & O. Strogan BiBoS, Univ.
Discovering Social Media Experts by Integrating Social Networks and Contents
Proceedings of the Twenty-Third Australasian Database Conference (ADC 2012), Melbourne, Australia Discovering Social Media Experts by Integrating Social Networks and Contents Zhao Zhang Bin Zhao Weining
Analysis of Internet Topologies: A Historical View
Analysis of Internet Topologies: A Historical View Mohamadreza Najiminaini, Laxmi Subedi, and Ljiljana Trajković Communication Networks Laboratory http://www.ensc.sfu.ca/cnl Simon Fraser University Vancouver,
Studying E-mail Graphs for Intelligence Monitoring and Analysis in the Absence of Semantic Information
Studying E-mail Graphs for Intelligence Monitoring and Analysis in the Absence of Semantic Information Petros Drineas, Mukkai S. Krishnamoorthy, Michael D. Sofka Bülent Yener Department of Computer Science,
Information Management course
Università degli Studi di Milano Master Degree in Computer Science Information Management course Teacher: Alberto Ceselli Lecture 01 : 06/10/2015 Practical informations: Teacher: Alberto Ceselli (alberto.ceselli@unimi.it)
Analysis of Internet Topologies
Analysis of Internet Topologies Ljiljana Trajković ljilja@cs.sfu.ca Communication Networks Laboratory http://www.ensc.sfu.ca/cnl School of Engineering Science Simon Fraser University, Vancouver, British
Big Graph Processing: Some Background
Big Graph Processing: Some Background Bo Wu Colorado School of Mines Part of slides from: Paul Burkhardt (National Security Agency) and Carlos Guestrin (Washington University) Mines CSCI-580, Bo Wu Graphs
Evaluating Online Payment Transaction Reliability using Rules Set Technique and Graph Model
Evaluating Online Payment Transaction Reliability using Rules Set Technique and Graph Model Trung Le 1, Ba Quy Tran 2, Hanh Dang Thi My 3, Thanh Hung Ngo 4 1 GSR, Information System Lab., University of
SOCIAL NETWORK DATA ANALYTICS
SOCIAL NETWORK DATA ANALYTICS SOCIAL NETWORK DATA ANALYTICS Edited by CHARU C. AGGARWAL IBM T. J. Watson Research Center, Yorktown Heights, NY 10598, USA Kluwer Academic Publishers Boston/Dordrecht/London
An Analysis of Verifications in Microblogging Social Networks - Sina Weibo
An Analysis of Verifications in Microblogging Social Networks - Sina Weibo Junting Chen and James She HKUST-NIE Social Media Lab Dept. of Electronic and Computer Engineering The Hong Kong University of
DATA ANALYSIS II. Matrix Algorithms
DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where
Network Monitoring Using Traffic Dispersion Graphs (TDGs)
Network Monitoring Using Traffic Dispersion Graphs (TDGs) Marios Iliofotou Joint work with: Prashanth Pappu (Cisco), Michalis Faloutsos (UCR), M. Mitzenmacher (Harvard), Sumeet Singh(Cisco) and George
Large-scale Data Mining: MapReduce and Beyond Part 2: Algorithms. Spiros Papadimitriou, IBM Research Jimeng Sun, IBM Research Rong Yan, Facebook
Large-scale Data Mining: MapReduce and Beyond Part 2: Algorithms Spiros Papadimitriou, IBM Research Jimeng Sun, IBM Research Rong Yan, Facebook Part 2:Mining using MapReduce Mining algorithms using MapReduce
Graph Mining and Social Network Analysis
Graph Mining and Social Network Analysis Data Mining and Text Mining (UIC 583 @ Politecnico di Milano) References Jiawei Han and Micheline Kamber, "Data Mining: Concepts and Techniques", The Morgan Kaufmann
Part 1: Link Analysis & Page Rank
Chapter 8: Graph Data Part 1: Link Analysis & Page Rank Based on Leskovec, Rajaraman, Ullman 214: Mining of Massive Datasets 1 Exam on the 5th of February, 216, 14. to 16. If you wish to attend, please
AN INTRODUCTION TO SOCIAL NETWORK DATA ANALYTICS
Chapter 1 AN INTRODUCTION TO SOCIAL NETWORK DATA ANALYTICS Charu C. Aggarwal IBM T. J. Watson Research Center Hawthorne, NY 10532 charu@us.ibm.com Abstract The advent of online social networks has been
MapReduce Algorithms. Sergei Vassilvitskii. Saturday, August 25, 12
MapReduce Algorithms A Sense of Scale At web scales... Mail: Billions of messages per day Search: Billions of searches per day Social: Billions of relationships 2 A Sense of Scale At web scales... Mail:
Graphic Analysis of Volatile Graphs
Metric Forensics: A Multi-Level Approach for Mining Volatile Graphs Keith Henderson Lawrence Livermore Lab keith@llnl.gov Leman Akoglu Carnegie Mellon University lakoglu@cs.cmu.edu Tina Eliassi-Rad Lawrence
A SURVEY OF PROPERTIES AND MODELS OF ON- LINE SOCIAL NETWORKS
International Conference on Mathematical and Computational Models PSG College of Technology, Coimbatore Copyright 2009, Narosa Publishing House, New Delhi, India A SURVEY OF PROPERTIES AND MODELS OF ON-
MapReduce and Distributed Data Analysis. Sergei Vassilvitskii Google Research
MapReduce and Distributed Data Analysis Google Research 1 Dealing With Massive Data 2 2 Dealing With Massive Data Polynomial Memory Sublinear RAM Sketches External Memory Property Testing 3 3 Dealing With
Search in BigData2 - When Big Text meets Big Graph 1. Introduction State of the Art on Big Data
Search in BigData 2 - When Big Text meets Big Graph Christos Giatsidis, Fragkiskos D. Malliaros, François Rousseau, Michalis Vazirgiannis Computer Science Laboratory, École Polytechnique, France {giatsidis,
How To Find Out How A Graph Densifies
Graphs over Time: Densification Laws, Shrinking Diameters and Possible Explanations Jure Leskovec Carnegie Mellon University jure@cs.cmu.edu Jon Kleinberg Cornell University kleinber@cs.cornell.edu Christos
Strong and Weak Ties
Strong and Weak Ties Web Science (VU) (707.000) Elisabeth Lex KTI, TU Graz April 11, 2016 Elisabeth Lex (KTI, TU Graz) Networks April 11, 2016 1 / 66 Outline 1 Repetition 2 Strong and Weak Ties 3 General
Online Estimating the k Central Nodes of a Network
Online Estimating the k Central Nodes of a Network Yeon-sup Lim, Daniel S. Menasché, Bruno Ribeiro, Don Towsley, and Prithwish Basu Department of Computer Science UMass Amherst, Raytheon BBN Technologies
Social Network Analysis
Social Network Analysis Challenges in Computer Science April 1, 2014 Frank Takes (ftakes@liacs.nl) LIACS, Leiden University Overview Context Social Network Analysis Online Social Networks Friendship Graph
Exploring Big Data in Social Networks
Exploring Big Data in Social Networks virgilio@dcc.ufmg.br (meira@dcc.ufmg.br) INWEB National Science and Technology Institute for Web Federal University of Minas Gerais - UFMG May 2013 Some thoughts about
Tutorial, IEEE SERVICE 2014 Anchorage, Alaska
Tutorial, IEEE SERVICE 2014 Anchorage, Alaska Big Data Science: Fundamental, Techniques, and Challenges (Data Mining on Big Data) 2014. 6. 27. By Neil Y. Yen Presented by Incheon Paik University of Aizu
Learning Gaussian process models from big data. Alan Qi Purdue University Joint work with Z. Xu, F. Yan, B. Dai, and Y. Zhu
Learning Gaussian process models from big data Alan Qi Purdue University Joint work with Z. Xu, F. Yan, B. Dai, and Y. Zhu Machine learning seminar at University of Cambridge, July 4 2012 Data A lot of
(B ) Empirical observation: Shrinking diameters: The effective diameter is, in many cases, actually decreasing as the network grows.
Graphs over Time: Densification Laws, Shrinking Diameters and Possible Explanations Jure Leskovec Carnegie Mellon University jure@cs.cmu.edu Jon Kleinberg Cornell University kleinber@cs.cornell.edu Christos
HADI: Fast Diameter Estimation and Mining in Massive Graphs with Hadoop
HADI: Fast Diameter Estimation and Mining in Massive Graphs with Hadoop U Kang, Charalampos Tsourakakis, Ana Paula Appel, Christos Faloutsos, Jure Leskovec December 2008 CMU-ML-08-117 HADI: Fast Diameter
Fast counting of triangles in real-world networks: proofs, algorithms and observations
Fast counting of triangles in real-world networks: proofs, algorithms and observations Charalampos E. Tsourakakis May 2008 CMU-ML-08-103 Fast counting of triangles in real-world networks: proofs, algorithms
Influence Propagation in Social Networks: a Data Mining Perspective
Influence Propagation in Social Networks: a Data Mining Perspective Francesco Bonchi Yahoo! Research Barcelona - Spain bonchi@yahoo-inc.com http://francescobonchi.com/ Acknowledgments Amit Goyal (University
Large Scale Social Network Analysis
Large Scale Social Network Analysis DATA ANALYTICS 2013 TUTORIAL Rui Sarmento email@ruisarmento.com João Gama jgama@fep.up.pt Outline PART I 1. Introduction & Motivation Overview & Contributions 2. Software
AFRAID: Fraud Detection via Active Inference in Time-evolving Social Networks
AFRAID: Fraud Detection via Active Inference in Time-evolving Social Networks Véronique Van Vlasselaer Tina Eliassi-Rad Leman Akoglu Monique Snoeck Bart Baesens Department of Decision Sciences and Information
A Survey on Outlier Detection Techniques for Credit Card Fraud Detection
IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 2278-0661, p- ISSN: 2278-8727Volume 16, Issue 2, Ver. VI (Mar-Apr. 2014), PP 44-48 A Survey on Outlier Detection Techniques for Credit Card Fraud
New Metrics for Reputation Management in P2P Networks
New for Reputation in P2P Networks D. Donato, M. Paniccia 2, M. Selis 2, C. Castillo, G. Cortesi 3, S. Leonardi 2. Yahoo!Research Barcelona Catalunya, Spain 2. Università di Roma La Sapienza Rome, Italy
Exploring and Making Sense of Large Graphs
Exploring and Making Sense of Large Graphs Danai Koutra CMU-CS-15-126 August 2015 Computer Science Department Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee: Christos Faloutsos, Chair
Mining the Temporal Dimension of the Information Propagation
Mining the Temporal Dimension of the Information Propagation Michele Berlingerio, Michele Coscia 2, and Fosca Giannotti 3 IMT-Lucca, Lucca, Italy 2 Dipartimento di Informatica, Pisa, Italy {name.surname}@isti.cnr.it
Dynamics of information spread on networks. Kristina Lerman USC Information Sciences Institute
Dynamics of information spread on networks Kristina Lerman USC Information Sciences Institute Information spread in online social networks Diffusion of activation on a graph, where each infected (activated)
Understanding Graph Sampling Algorithms for Social Network Analysis
Understanding Graph Sampling Algorithms for Social Network Analysis Tianyi Wang, Yang Chen 2, Zengbin Zhang 3, Tianyin Xu 2 Long Jin, Pan Hui 4, Beixing Deng, Xing Li Department of Electronic Engineering,
Top 10 Algorithms in Data Mining
Top 10 Algorithms in Data Mining Xindong Wu ( 吴 信 东 ) Department of Computer Science University of Vermont, USA; 合 肥 工 业 大 学 计 算 机 与 信 息 学 院 1 Top 10 Algorithms in Data Mining by the IEEE ICDM Conference
Top Top 10 Algorithms in Data Mining
ICDM 06 Panel on Top Top 10 Algorithms in Data Mining 1. The 3-step identification process 2. The 18 identified candidates 3. Algorithm presentations 4. Top 10 algorithms: summary 5. Open discussions ICDM
CS 5614: (Big) Data Management Systems. B. Aditya Prakash Lecture #18: Dimensionality Reduc7on
CS 5614: (Big) Data Management Systems B. Aditya Prakash Lecture #18: Dimensionality Reduc7on Dimensionality Reduc=on Assump=on: Data lies on or near a low d- dimensional subspace Axes of this subspace
CHRISTOS FALOUTSOS PERSONAL
CHRISTOS FALOUTSOS PERSONAL University Address: Computer Science Department Carnegie Mellon University GHC, room 8019 tel: 412-268.14.57 5000 Forbes Avenue FAX: 412-268.55.76 Pittsburgh, PA 15213-3891
MMap: Fast Billion-Scale Graph Computation on a PC via Memory Mapping
: Fast Billion-Scale Graph Computation on a PC via Memory Mapping Zhiyuan Lin, Minsuk Kahng, Kaeser Md. Sabrin, Duen Horng (Polo) Chau Georgia Tech Atlanta, Georgia {zlin48, kahng, kmsabrin, polo}@gatech.edu
Reconstruction and Analysis of Twitter Conversation Graphs
Reconstruction and Analysis of Twitter Conversation Graphs Peter Cogan peter.cogan@alcatellucent.com Gabriel Tucci gabriel.tucci@alcatellucent.com Matthew Andrews andrews@research.belllabs.com W. Sean
Online Fraud Detection Model Based on Social Network Analysis
Journal of Information & Computational Science :7 (05) 553 5 May, 05 Available at http://www.joics.com Online Fraud Detection Model Based on Social Network Analysis Peng Wang a,b,, Ji Li a,b, Bigui Ji
On the Bursty Evolution of Online Social Networks
On the Bursty Evolution of Online Social Networks Sabrina Gaito, Matteo Zignani, Gian Paolo Rossi, Alessandra Sala Xiaohan Zhao, Haitao Zheng, Ben Y. Zhao Computer Science, Universitá degli Studi di Milano
Parallel Data Mining. Team 2 Flash Coders Team Research Investigation Presentation 2. Foundations of Parallel Computing Oct 2014
Parallel Data Mining Team 2 Flash Coders Team Research Investigation Presentation 2 Foundations of Parallel Computing Oct 2014 Agenda Overview of topic Analysis of research papers Software design Overview
Netzcope - A Tool to Display and Analyze Complex Networks
NEMO Working Paper #16 Netzcope - A Tool to Display and Analyze Complex Networks Oleg Strogan and Ludwig Streit (CCM, University of Madeira) Supported by the EU FP6-NEST-Adventure Programme Contract n
APPM4720/5720: Fast algorithms for big data. Gunnar Martinsson The University of Colorado at Boulder
APPM4720/5720: Fast algorithms for big data Gunnar Martinsson The University of Colorado at Boulder Course objectives: The purpose of this course is to teach efficient algorithms for processing very large
How To Understand The Network Of A Network
Roles in Networks Roles in Networks Motivation for work: Let topology define network roles. Work by Kleinberg on directed graphs, used topology to define two types of roles: authorities and hubs. (Each
Clustering Big Data. Anil K. Jain. (with Radha Chitta and Rong Jin) Department of Computer Science Michigan State University November 29, 2012
Clustering Big Data Anil K. Jain (with Radha Chitta and Rong Jin) Department of Computer Science Michigan State University November 29, 2012 Outline Big Data How to extract information? Data clustering
Complex Networks Analysis: Clustering Methods
Complex Networks Analysis: Clustering Methods Nikolai Nefedov Spring 2013 ISI ETH Zurich nefedov@isi.ee.ethz.ch 1 Outline Purpose to give an overview of modern graph-clustering methods and their applications
Extracting Information from Social Networks
Extracting Information from Social Networks Aggregating site information to get trends 1 Not limited to social networks Examples Google search logs: flu outbreaks We Feel Fine Bullying 2 Bullying Xu, Jun,
arxiv:cs.dm/0204001 v1 30 Mar 2002
A Steady State Model for Graph Power Laws David Eppstein Joseph Wang arxiv:cs.dm/0000 v 0 Mar 00 Abstract Power law distribution seems to be an important characteristic of web graphs. Several existing
Distance Metric Learning in Data Mining (Part I) Fei Wang and Jimeng Sun IBM TJ Watson Research Center
Distance Metric Learning in Data Mining (Part I) Fei Wang and Jimeng Sun IBM TJ Watson Research Center 1 Outline Part I - Applications Motivation and Introduction Patient similarity application Part II
Mobile Call Graphs: Beyond Power-Law and Lognormal Distributions
Mobile Call Graphs: Beyond Power-Law and Lognormal Distributions Mukund Seshadri Sprint Burlingame, California, USA mukund.seshadri@sprint.com Jean Bolot Sprint Burlingame, California, USA bolot@sprint.com
Systems and Algorithms for Big Data Analytics
Systems and Algorithms for Big Data Analytics YAN, Da Email: yanda@cse.cuhk.edu.hk My Research Graph Data Distributed Graph Processing Spatial Data Spatial Query Processing Uncertain Data Querying & Mining
So, how do you pronounce. Jilles Vreeken. Okay, now we can talk. So, what kind of data? binary. * multi-relational
Simply Mining Data Jilles Vreeken So, how do you pronounce Exploratory Data Analysis Jilles Vreeken Jilles Yill less Vreeken Fray can 17 August 2015 Okay, now we can talk. 17 August 2015 The goal So, what