CMU SCS Large Graph Mining Patterns, Tools and Cascade analysis

Size: px
Start display at page:

Download "CMU SCS Large Graph Mining Patterns, Tools and Cascade analysis"

Transcription

1 Large Graph Mining Patterns, Tools and Cascade analysis Christos Faloutsos CMU

2 Roadmap Introduction Motivation Why big data Why (big) graphs? Patterns in graphs Tools: fraud detection on e-bay Conclusions C. Faloutsos (CMU) 2

3 Why big data Why big data? What is the problem definition? What are the major research challenges? C. Faloutsos (CMU) 3

4 Main message: Big data: often > experts Super Crunchers Why Thinking-By-Numbers is the New Way To Be Smart by Ian Ayres, 2008 Google won the machine translation competition mt05eval_official_results_release_ _v3.html C. Faloutsos (CMU) 4

5 Problem definition big picture Tera/Peta-byte data Analytics Insights, outliers C. Faloutsos (CMU) 5

6 Roadmap Introduction Motivation Why big data Why (big) graphs? Patterns in graphs Tools: fraud detection on e-bay Conclusions C. Faloutsos (CMU) 6

7 Graphs - why should we care? >$10B revenue Food Web [Martinez 91] >0.5B users Internet Map [lumeta.com] C. Faloutsos (CMU) 7

8 Graphs - why should we care? IR: bi-partite graphs (doc-terms) D T 1 web: hyper-text graph D N T M... and more: C. Faloutsos (CMU) 8

9 Graphs - why should we care? viral marketing web-log ( blog ) news propagation computer network security: /ip traffic and anomaly detection... Subject-verb-object -> graph Many-to-many db relationship -> graph C. Faloutsos (CMU) 9

10 Outline Introduction Motivation Patterns in graphs Static graphs Time evolving graphs Radius, conn components Tools: fraud detection on e-bay Conclusions C. Faloutsos (CMU) 10

11 Problem - network and graph mining What does the Internet look like? What does FaceBook look like? What is normal / abnormal? which patterns/laws hold? C. Faloutsos (CMU) 11

12 Problem - network and graph mining What does the Internet look like? What does FaceBook look like? What is normal / abnormal? which patterns/laws hold? To spot anomalies (rarities), we have to discover patterns C. Faloutsos (CMU) 12

13 Problem - network and graph mining What does the Internet look like? What does FaceBook look like? What is normal / abnormal? which patterns/laws hold? To spot anomalies (rarities), we have to discover patterns Large datasets reveal patterns/anomalies that may be invisible otherwise C. Faloutsos (CMU) 13

14 Graph mining Are real graphs random? C. Faloutsos (CMU) 14

15 Laws and patterns Are real graphs random? A: NO!! Diameter in- and out- degree distributions other (surprising) patterns So, let s look at the data C. Faloutsos (CMU) 15

16 Solution# S.1 Power law in the degree distribution [SIGCOMM99] internet domains log(degree) att.com ibm.com log(rank) C. Faloutsos (CMU) 16

17 Solution# S.1 Power law in the degree distribution [SIGCOMM99] internet domains log(degree) att.com ibm.com log(rank) C. Faloutsos (CMU) 17

18 But: How about graphs from other domains? C. Faloutsos (CMU) 18

19 More power laws: web hit counts [w/ A. Montgomery] Web Site Traffic Count (log scale) Zipf ``ebay users sites in-degree (log scale) C. Faloutsos (CMU) 19

20 epinions.com count who-trusts-whom [Richardson + Domingos, KDD 2001] trusts-2000-people user (out) degree C. Faloutsos (CMU) 20

21 And numerous more # of sexual contacts Income [Pareto] distribution Duration of downloads [Bestavros+] Duration of UNIX jobs ( mice and elephants ) Size of files of a user Black swans C. Faloutsos (CMU) 21

22 Roadmap Introduction Motivation Problem#1: Patterns in graphs Static graphs degree, diameter, triangles Cliques Time evolving graphs C. Faloutsos (CMU) 22

23 Solution# S.2: Triangle Laws Real social networks have a lot of triangles C. Faloutsos (CMU) 23

24 Solution# S.2: Triangle Laws Real social networks have a lot of triangles Friends of friends are friends Any patterns? C. Faloutsos (CMU) 24

25 Triangle Law: #S.2 [Tsourakakis ICDM 2008] Reuters SN Epinions X-axis: degree Y-axis: mean # triangles n friends -> ~n 1.6 triangles C. Faloutsos (CMU) 25

26 Triangle Law: #S.2 [Tsourakakis ICDM 2008] Reuters SN Epinions X-axis: degree Y-axis: mean # triangles n friends -> ~n 1.6 triangles C. Faloutsos (CMU) 26

27 Triangle counting for large graphs???? Anomalous nodes in Twitter(~ 3 billion edges) [U Kang, Brendan Meeder, +, PAKDD 11] C. Faloutsos (CMU) 27

28 Triangle counting for large graphs? Anomalous nodes in Twitter(~ 3 billion edges) [U Kang, Brendan Meeder, +, PAKDD 11] C. Faloutsos (CMU) 28

29 Triangle counting for large graphs? Anomalous nodes in Twitter(~ 3 billion edges) [U Kang, Brendan Meeder, +, PAKDD 11] C. Faloutsos (CMU) 29

30 Triangle counting for large graphs? Anomalous nodes in Twitter(~ 3 billion edges) [U Kang, Brendan Meeder, +, PAKDD 11] C. Faloutsos (CMU) 30

31 Roadmap Introduction Motivation Patterns in graphs Static graphs Time evolving graphs Tools C. Faloutsos (CMU) 31

32 Problem: Time evolution with Jure Leskovec (CMU -> Stanford) and Jon Kleinberg (Cornell CMU) C. Faloutsos (CMU) 32

33 T.1 Evolution of the Diameter Prior work on Power Law graphs hints at slowly growing diameter: diameter ~ O(log N) diameter ~ O(log log N) What is happening in real data? C. Faloutsos (CMU) 33

34 T.1 Evolution of the Diameter Prior work on Power Law graphs hints at slowly growing diameter: diameter ~ O(log N) diameter ~ O(log log N) What is happening in real data? Diameter shrinks over time C. Faloutsos (CMU) 34

35 T.1 Diameter Patents Patent citation network 25 years of 2.9 M nodes 16.5 M edges diameter time [years] C. Faloutsos (CMU) 35

36 T.2 Temporal Evolution of the Graphs N(t) nodes at time t E(t) edges at time t Suppose that N(t+1) = 2 * N(t) Q: what is your guess for E(t+1) =? 2 * E(t) C. Faloutsos (CMU) 36

37 T.2 Temporal Evolution of the Graphs N(t) nodes at time t E(t) edges at time t Suppose that N(t+1) = 2 * N(t) Q: what is your guess for E(t+1) =? 2 * E(t) A: over-doubled! But obeying the ``Densification Power Law C. Faloutsos (CMU) 37

38 T.2 Densification Patent Citations Citations among patents 2.9 M nodes 16.5 M edges Each year is a datapoint E(t) 1.66 N(t) C. Faloutsos (CMU) 38

39 T.3 : popularity over time # in links lag: days after post Post popularity + lag C. Faloutsos (CMU) 39

40 T.3 : popularity over time # in links (log) Post popularity drops-off exponentially? POWER LAW! Exponent? days after post (log) C. Faloutsos (CMU) 40

41 T.3 : popularity over time # in links (log) -1.6 Post popularity drops-off exponentially? POWER LAW! Exponent? -1.6 close to -1.5: Barabasi s stack model and like the zero-crossings of a random walk days after post (log) C. Faloutsos (CMU) 41

42 -1.5 slope J. G. Oliveira & A.-L. Barabási Human Dynamics: The Correspondence Patterns of Darwin and Einstein. Nature 437, 1251 (2005). [PDF] Prob(RT > x) (log) Response time (log) C. Faloutsos (CMU) 42

43 Roadmap Introduction Motivation Patterns in graphs Diameter Connected components Tools: fraud detection on e-bay Conclusions C. Faloutsos (CMU) 43

44 HADI for diameter estimation Radius Plots for Mining Tera-byte Scale Graphs U Kang, Charalampos Tsourakakis, Ana Paula Appel, Christos Faloutsos, Jure Leskovec, SDM 10 Naively: diameter needs O(N**2) space and up to O(N**3) time prohibitive (N~1B) Our HADI: linear on E (~10B) Near-linear scalability wrt # machines Several optimizations -> 5x faster C. Faloutsos (CMU) 44

45 Count???? 19+ [Barabasi+] ~1999, ~1M nodes Radius C. Faloutsos (CMU) 45

46 Count?????? 19+ [Barabasi+] ~1999, ~1M nodes Radius YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) Largest publicly available graph ever studied. C. Faloutsos (CMU) 46

47 Count 14 (dir.)???? ~7 (undir.) 19+? [Barabasi+] Radius YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) Largest publicly available graph ever studied. C. Faloutsos (CMU) 47

48 Count 14 (dir.)???? ~7 (undir.) 19+? [Barabasi+] YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) 7 degrees of separation (!) Diameter: shrunk C. Faloutsos (CMU) Radius 48

49 Count???? ~7 (undir.) Radius YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) Q: Shape? C. Faloutsos (CMU) 49

50 YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges) effective diameter: surprisingly small. Multi-modality (?!) C. Faloutsos (CMU) 50

51 Roadmap Introduction Motivation Patterns in graphs Diameter Connected components Tools: fraud detection on e-bay Conclusions C. Faloutsos (CMU) 51

52 Generalized Iterated Matrix Vector Multiplication (GIMV) PEGASUS: A Peta-Scale Graph Mining System - Implementation and Observations. U Kang, Charalampos E. Tsourakakis, and Christos Faloutsos. (ICDM) 2009, Miami, Florida, USA. Best Application Paper (runner-up). C. Faloutsos (CMU) 52

53 Example: GIM-V At Work Connected Components 4 observations: Count Size C. Faloutsos (CMU) 53

54 Example: GIM-V At Work Connected Components Count 1) 10K x larger than next Size C. Faloutsos (CMU) 54

55 Example: GIM-V At Work Connected Components Count 2) ~0.7B singleton nodes Size C. Faloutsos (CMU) 55

56 Example: GIM-V At Work Connected Components Count 3) SLOPE! Size C. Faloutsos (CMU) 56

57 Example: GIM-V At Work Connected Components Count 300-size cmpt X size cmpt Why? X 65. Why? 4) Spikes! Size C. Faloutsos (CMU) 57

58 Example: GIM-V At Work Connected Components Count suspicious financial-advice sites (not existing now) Size C. Faloutsos (CMU) 58

59 Roadmap Introduction Motivation Patterns in graphs Tools: fraud detection on e-bay Conclusions C. Faloutsos (CMU) 59

60 E-bay Fraud detection w/ Polo Chau & Shashank Pandit, CMU [www 07] C. Faloutsos (CMU) 60

61 E-bay Fraud detection C. Faloutsos (CMU) 61

62 E-bay Fraud detection C. Faloutsos (CMU) 62

63 E-bay Fraud detection - NetProbe C. Faloutsos (CMU) 63

64 Popular press And less desirable attention: from Belgium police ( copy of your code? ) UMN CRAY, 2012 C. Faloutsos (CMU) 64

65 OVERALL CONCLUSIONS low level: Several new patterns (fortification, triangle-laws, conn. components, etc) New tools: belief propagation, gigatensor, etc Scalability: PEGASUS / hadoop C. Faloutsos (CMU) 65

66 OVERALL CONCLUSIONS high level BIG DATA: Large datasets reveal patterns/ outliers that are invisible otherwise C. Faloutsos (CMU) 66

67 Project info & thanks Thanks to: NSF IIS , IIS , CTA-INARC; Yahoo (M45), LLNL, IBM, SPRINT, C. Faloutsos (CMU) 67 Google, INTEL, HP, ilab

68 Cast Akoglu, Leman Beutel, Alex Chau, Polo Kang, U Koutra, Danai McGlohon, Mary Prakash, Aditya Papalexakis, Vagelis Tong, Hanghang C. Faloutsos (CMU) 68

69 Take-home message Tera/Peta-byte data Analytics Insights, outliers Big data reveal insights that would be invisible otherwise (even to experts) C. Faloutsos (CMU) 69

CMU SCS Mining Billion-Node Graphs - Patterns and Algorithms

CMU SCS Mining Billion-Node Graphs - Patterns and Algorithms Mining Billion-Node Graphs - Patterns and Algorithms Christos Faloutsos CMU Thank you! Panos Chrysanthis Ling Liu Vladimir Zadorozhny Prashant Krishnamurthy C. Faloutsos (CMU) 2 Resource Open source system

More information

CMU SCS Mining Large Graphs and Tensors - Patterns, Tools and Discoveries.

CMU SCS Mining Large Graphs and Tensors - Patterns, Tools and Discoveries. Mining Large Graphs and Tensors - Patterns, Tools and Discoveries. Christos Faloutsos CMU Thank you! Nikos Sidiropoulos Kuo-Chu Chang Zhi (Gerry) Tian C. Faloutsos (CMU) 2 Roadmap Introduction Motivation

More information

Graph Mining Techniques for Social Media Analysis

Graph Mining Techniques for Social Media Analysis Graph Mining Techniques for Social Media Analysis Mary McGlohon Christos Faloutsos 1 1-1 What is graph mining? Extracting useful knowledge (patterns, outliers, etc.) from structured data that can be represented

More information

School of Computer Science Carnegie Mellon Graph Mining, self-similarity and power laws

School of Computer Science Carnegie Mellon Graph Mining, self-similarity and power laws Graph Mining, self-similarity and power laws Christos Faloutsos University Overview Achievements global patterns and laws (static/dynamic) generators influence propagation communities; graph partitioning

More information

Graphs over Time Densification Laws, Shrinking Diameters and Possible Explanations

Graphs over Time Densification Laws, Shrinking Diameters and Possible Explanations Graphs over Time Densification Laws, Shrinking Diameters and Possible Explanations Jurij Leskovec, CMU Jon Kleinberg, Cornell Christos Faloutsos, CMU 1 Introduction What can we do with graphs? What patterns

More information

Scaling Up HBase, Hive, Pegasus

Scaling Up HBase, Hive, Pegasus CSE 6242 A / CS 4803 DVA Mar 7, 2013 Scaling Up HBase, Hive, Pegasus Duen Horng (Polo) Chau Georgia Tech Some lectures are partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John Stasko,

More information

The Shape of the Network. The Shape of the Internet. Why study topology? Internet topologies. Early work. More on topologies..

The Shape of the Network. The Shape of the Internet. Why study topology? Internet topologies. Early work. More on topologies.. The Shape of the Internet Slides assembled by Jeff Chase Duke University (thanks to and ) The Shape of the Network Characterizing shape : AS-level topology: who connects to whom Router-level topology:

More information

Introduction to Graph Mining

Introduction to Graph Mining Introduction to Graph Mining What is a graph? A graph G = (V,E) is a set of vertices V and a set (possibly empty) E of pairs of vertices e 1 = (v 1, v 2 ), where e 1 E and v 1, v 2 V. Edges may contain

More information

Big Graph Mining: Algorithms and Discoveries

Big Graph Mining: Algorithms and Discoveries Big Graph Mining: Algorithms and Discoveries U Kang and Christos Faloutsos Carnegie Mellon University {ukang, christos}@cs.cmu.edu ABSTRACT How do we find patterns and anomalies in very large graphs with

More information

Big Data Analytics Process & Building Blocks

Big Data Analytics Process & Building Blocks Big Data Analytics Process & Building Blocks Duen Horng (Polo) Chau Georgia Tech CSE 6242 A / CS 4803 DVA Jan 10, 2013 Partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John Stasko, Christos

More information

The ebay Graph: How Do Online Auction Users Interact?

The ebay Graph: How Do Online Auction Users Interact? The ebay Graph: How Do Online Auction Users Interact? Yordanos Beyene, Michalis Faloutsos University of California, Riverside {yordanos, michalis}@cs.ucr.edu Duen Horng (Polo) Chau, Christos Faloutsos

More information

Analyzing the Facebook graph?

Analyzing the Facebook graph? Logistics Big Data Algorithmic Introduction Prof. Yuval Shavitt Contact: shavitt@eng.tau.ac.il Final grade: 4 6 home assignments (will try to include programing assignments as well): 2% Exam 8% Big Data

More information

Graph Processing and Social Networks

Graph Processing and Social Networks Graph Processing and Social Networks Presented by Shu Jiayu, Yang Ji Department of Computer Science and Engineering The Hong Kong University of Science and Technology 2015/4/20 1 Outline Background Graph

More information

Expansion Properties of Large Social Graphs

Expansion Properties of Large Social Graphs Expansion Properties of Large Social Graphs Fragkiskos D. Malliaros 1 and Vasileios Megalooikonomou 1,2 1 Computer Engineering and Informatics Department University of Patras, 26500 Rio, Greece 2 Data

More information

Big Data Analytics Building Blocks. Simple Data Storage (SQLite)

Big Data Analytics Building Blocks. Simple Data Storage (SQLite) http://poloclub.gatech.edu/cse6242 CSE6242 / CX4242: Data & Visual Analytics Big Data Analytics Building Blocks. Simple Data Storage (SQLite) Duen Horng (Polo) Chau Georgia Tech Partly based on materials

More information

Graph models for the Web and the Internet. Elias Koutsoupias University of Athens and UCLA. Crete, July 2003

Graph models for the Web and the Internet. Elias Koutsoupias University of Athens and UCLA. Crete, July 2003 Graph models for the Web and the Internet Elias Koutsoupias University of Athens and UCLA Crete, July 2003 Outline of the lecture Small world phenomenon The shape of the Web graph Searching and navigation

More information

Big Data Analytics Building Blocks. Simple Data Storage (SQLite)

Big Data Analytics Building Blocks. Simple Data Storage (SQLite) http://poloclub.gatech.edu/cse6242 CSE6242 / CX4242: Data & Visual Analytics Big Data Analytics Building Blocks. Simple Data Storage (SQLite) Duen Horng (Polo) Chau Georgia Tech Partly based on materials

More information

Big Data Analytics Building Blocks; Simple Data Storage (SQLite)

Big Data Analytics Building Blocks; Simple Data Storage (SQLite) Big Data Analytics Building Blocks; Simple Data Storage (SQLite) Duen Horng (Polo) Chau Georgia Tech CSE6242 / CX4242 Jan 9, 2014 Partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John

More information

Big Data Analytics Building Blocks; Simple Data Storage (SQLite)

Big Data Analytics Building Blocks; Simple Data Storage (SQLite) Big Data Analytics Building Blocks; Simple Data Storage (SQLite) Duen Horng (Polo) Chau Georgia Tech CSE6242 / CX4242 Aug 21, 2014 Partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John

More information

Cost effective Outbreak Detection in Networks

Cost effective Outbreak Detection in Networks Cost effective Outbreak Detection in Networks Jure Leskovec Joint work with Andreas Krause, Carlos Guestrin, Christos Faloutsos, Jeanne VanBriesen, and Natalie Glance Diffusion in Social Networks One of

More information

Jure Leskovec (@jure) Stanford University

Jure Leskovec (@jure) Stanford University Jure Leskovec (@jure) Stanford University KDD Summer School, Beijing, August 2012 8/10/2012 Jure Leskovec (@jure), KDD Summer School 2012 2 Graph: Kronecker graphs Graph Node attributes: MAG model Graph

More information

The Internet Is Like A Jellyfish

The Internet Is Like A Jellyfish The Internet Is Like A Jellyfish Michalis Faloutsos UC Riverside Joint work with: Leslie Tauro, Georgos Siganos (UCR) Chris Palmer(CMU) Big Picture: Modeling the Internet Topology Traffic Protocols Routing,

More information

Social Network Mining

Social Network Mining Social Network Mining Data Mining November 11, 2013 Frank Takes (ftakes@liacs.nl) LIACS, Universiteit Leiden Overview Social Network Analysis Graph Mining Online Social Networks Friendship Graph Semantics

More information

MMap: Fast Billion-Scale Graph Computation on a PC via Memory Mapping

MMap: Fast Billion-Scale Graph Computation on a PC via Memory Mapping : Fast Billion-Scale Graph Computation on a PC via Memory Mapping Zhiyuan Lin, Minsuk Kahng, Kaeser Md. Sabrin, Duen Horng (Polo) Chau Georgia Tech Atlanta, Georgia {zlin48, kahng, kmsabrin, polo}@gatech.edu

More information

Search in BigData2 - When Big Text meets Big Graph 1. Introduction State of the Art on Big Data

Search in BigData2 - When Big Text meets Big Graph 1. Introduction State of the Art on Big Data Search in BigData 2 - When Big Text meets Big Graph Christos Giatsidis, Fragkiskos D. Malliaros, François Rousseau, Michalis Vazirgiannis Computer Science Laboratory, École Polytechnique, France {giatsidis,

More information

Hadoop Based Link Prediction Performance Analysis

Hadoop Based Link Prediction Performance Analysis Hadoop Based Link Prediction Performance Analysis Yuxiao Dong, Casey Robinson, Jian Xu Department of Computer Science and Engineering University of Notre Dame Notre Dame, IN 46556, USA Email: ydong1@nd.edu,

More information

Analyzing Big Data with AWS

Analyzing Big Data with AWS Analyzing Big Data with AWS Peter Sirota, General Manager, Amazon Elastic MapReduce @petersirota What is Big Data? Computer generated data Application server logs (web sites, games) Sensor data (weather,

More information

Exploring Big Data in Social Networks

Exploring Big Data in Social Networks Exploring Big Data in Social Networks virgilio@dcc.ufmg.br (meira@dcc.ufmg.br) INWEB National Science and Technology Institute for Web Federal University of Minas Gerais - UFMG May 2013 Some thoughts about

More information

Procedia Computer Science

Procedia Computer Science Procedia Computer Science 00 (2015) 1 8 Procedia Computer Science Scalable Tensor Mining Lee Sael a, Inah Jeon b, U Kang b, a Department of Computer Science, State University of New York Korea, Republic

More information

HADI: Mining Radii of Large Graphs

HADI: Mining Radii of Large Graphs HADI: Mining Radii of Large Graphs U KANG Carnegie Mellon University CHARALAMPOS E. TSOURAKAKIS Carnegie Mellon University ANA PAULA APPEL USP at São Carlos CHRISTOS FALOUTSOS Carnegie Mellon University

More information

(B ) Empirical observation: Shrinking diameters: The effective diameter is, in many cases, actually decreasing as the network grows.

(B ) Empirical observation: Shrinking diameters: The effective diameter is, in many cases, actually decreasing as the network grows. Graphs over Time: Densification Laws, Shrinking Diameters and Possible Explanations Jure Leskovec Carnegie Mellon University jure@cs.cmu.edu Jon Kleinberg Cornell University kleinber@cs.cornell.edu Christos

More information

How To Find Out How A Graph Densifies

How To Find Out How A Graph Densifies Graphs over Time: Densification Laws, Shrinking Diameters and Possible Explanations Jure Leskovec Carnegie Mellon University jure@cs.cmu.edu Jon Kleinberg Cornell University kleinber@cs.cornell.edu Christos

More information

Attend Part 1 (2-3pm) to get 1 point extra credit. Polo will announce on Piazza options for DL students.

Attend Part 1 (2-3pm) to get 1 point extra credit. Polo will announce on Piazza options for DL students. Attend Part 1 (2-3pm) to get 1 point extra credit. Polo will announce on Piazza options for DL students. Data Science/Data Analytics and Scaling to Big Data with MathWorks Using Data Analytics to turn

More information

Reconstruction and Analysis of Twitter Conversation Graphs

Reconstruction and Analysis of Twitter Conversation Graphs Reconstruction and Analysis of Twitter Conversation Graphs Peter Cogan peter.cogan@alcatellucent.com Gabriel Tucci gabriel.tucci@alcatellucent.com Matthew Andrews andrews@research.belllabs.com W. Sean

More information

NetProbe: A Fast and Scalable System for Fraud Detection in Online Auction Networks

NetProbe: A Fast and Scalable System for Fraud Detection in Online Auction Networks NetProbe: A Fast and Scalable System for Fraud Detection in Online Auction Networks Shashank Pandit, Duen Horng Chau, Samuel Wang, Christos Faloutsos Carnegie Mellon University Pittsburgh, PA 15213, USA

More information

Influence Propagation in Social Networks: a Data Mining Perspective

Influence Propagation in Social Networks: a Data Mining Perspective Influence Propagation in Social Networks: a Data Mining Perspective Francesco Bonchi Yahoo! Research Barcelona - Spain bonchi@yahoo-inc.com http://francescobonchi.com/ Acknowledgments Amit Goyal (University

More information

AN INTRODUCTION TO SOCIAL NETWORK DATA ANALYTICS

AN INTRODUCTION TO SOCIAL NETWORK DATA ANALYTICS Chapter 1 AN INTRODUCTION TO SOCIAL NETWORK DATA ANALYTICS Charu C. Aggarwal IBM T. J. Watson Research Center Hawthorne, NY 10532 charu@us.ibm.com Abstract The advent of online social networks has been

More information

W H I T E P A P E R. Deriving Intelligence from Large Data Using Hadoop and Applying Analytics. Abstract

W H I T E P A P E R. Deriving Intelligence from Large Data Using Hadoop and Applying Analytics. Abstract W H I T E P A P E R Deriving Intelligence from Large Data Using Hadoop and Applying Analytics Abstract This white paper is focused on discussing the challenges facing large scale data processing and the

More information

An Analysis of Verifications in Microblogging Social Networks - Sina Weibo

An Analysis of Verifications in Microblogging Social Networks - Sina Weibo An Analysis of Verifications in Microblogging Social Networks - Sina Weibo Junting Chen and James She HKUST-NIE Social Media Lab Dept. of Electronic and Computer Engineering The Hong Kong University of

More information

INFO 2950 Intro to Data Science. Lecture 17: Power Laws and Big Data

INFO 2950 Intro to Data Science. Lecture 17: Power Laws and Big Data INFO 2950 Intro to Data Science Lecture 17: Power Laws and Big Data Paul Ginsparg Cornell University, Ithaca, NY 29 Oct 2013 1/25 Power Laws in log-log space y = cx k (k=1/2,1,2) log 10 y = k log 10 x

More information

Social Networks and Social Media

Social Networks and Social Media Social Networks and Social Media Social Media: Many-to-Many Social Networking Content Sharing Social Media Blogs Microblogging Wiki Forum 2 Characteristics of Social Media Consumers become Producers Rich

More information

AGENDA. What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story. Our BIG DATA Roadmap. Hadoop PDW

AGENDA. What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story. Our BIG DATA Roadmap. Hadoop PDW AGENDA What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story Hadoop PDW Our BIG DATA Roadmap BIG DATA? Volume 59% growth in annual WW information 1.2M Zetabytes (10 21 bytes) this

More information

HADI: Fast Diameter Estimation and Mining in Massive Graphs with Hadoop

HADI: Fast Diameter Estimation and Mining in Massive Graphs with Hadoop HADI: Fast Diameter Estimation and Mining in Massive Graphs with Hadoop U Kang, Charalampos Tsourakakis, Ana Paula Appel, Christos Faloutsos, Jure Leskovec December 2008 CMU-ML-08-117 HADI: Fast Diameter

More information

Data Mining Meets HCI: Making Sense of Large Graphs Duen Horng (Polo) Chau

Data Mining Meets HCI: Making Sense of Large Graphs Duen Horng (Polo) Chau Data Mining Meets HCI: Making Sense of Large Graphs Duen Horng (Polo) Chau July 2012 CMU-ML-12-103 Data Mining Meets HCI: Making Sense of Large Graphs Duen Horng (Polo) Chau July 2012 CMU-ML-12-103 Machine

More information

Guilt-by-Constellation: Fraud Detection by Suspicious Clique Memberships

Guilt-by-Constellation: Fraud Detection by Suspicious Clique Memberships Guilt-by-Constellation: Fraud Detection by Suspicious Clique Memberships Véronique Van Vlasselaer KU Leuven Veronique.VanVlasselaer @kuleuven.be Leman Akoglu Stony Brook University leman@cs.stonybrook.edu

More information

Dynamics of information spread on networks. Kristina Lerman USC Information Sciences Institute

Dynamics of information spread on networks. Kristina Lerman USC Information Sciences Institute Dynamics of information spread on networks Kristina Lerman USC Information Sciences Institute Information spread in online social networks Diffusion of activation on a graph, where each infected (activated)

More information

Large-Scale Data Processing

Large-Scale Data Processing Large-Scale Data Processing Eiko Yoneki eiko.yoneki@cl.cam.ac.uk http://www.cl.cam.ac.uk/~ey204 Systems Research Group University of Cambridge Computer Laboratory 2010s: Big Data Why Big Data now? Increase

More information

Tutorial, IEEE SERVICE 2014 Anchorage, Alaska

Tutorial, IEEE SERVICE 2014 Anchorage, Alaska Tutorial, IEEE SERVICE 2014 Anchorage, Alaska Big Data Science: Fundamental, Techniques, and Challenges (Data Mining on Big Data) 2014. 6. 27. By Neil Y. Yen Presented by Incheon Paik University of Aizu

More information

Evaluating Online Payment Transaction Reliability using Rules Set Technique and Graph Model

Evaluating Online Payment Transaction Reliability using Rules Set Technique and Graph Model Evaluating Online Payment Transaction Reliability using Rules Set Technique and Graph Model Trung Le 1, Ba Quy Tran 2, Hanh Dang Thi My 3, Thanh Hung Ngo 4 1 GSR, Information System Lab., University of

More information

Bayesian networks - Time-series models - Apache Spark & Scala

Bayesian networks - Time-series models - Apache Spark & Scala Bayesian networks - Time-series models - Apache Spark & Scala Dr John Sandiford, CTO Bayes Server Data Science London Meetup - November 2014 1 Contents Introduction Bayesian networks Latent variables Anomaly

More information

Studying E-mail Graphs for Intelligence Monitoring and Analysis in the Absence of Semantic Information

Studying E-mail Graphs for Intelligence Monitoring and Analysis in the Absence of Semantic Information Studying E-mail Graphs for Intelligence Monitoring and Analysis in the Absence of Semantic Information Petros Drineas, Mukkai S. Krishnamoorthy, Michael D. Sofka Bülent Yener Department of Computer Science,

More information

Towards Modeling Legitimate and Unsolicited Email Traffic Using Social Network Properties

Towards Modeling Legitimate and Unsolicited Email Traffic Using Social Network Properties Towards Modeling Legitimate and Unsolicited Email Traffic Using Social Network Properties Farnaz Moradi Tomas Olovsson Philippas Tsigas Computer Science and Engineering Chalmers University of Technology,

More information

Marko Grobelnik marko.grobelnik@ijs.si Jozef Stefan Institute Ljubljana, Slovenia

Marko Grobelnik marko.grobelnik@ijs.si Jozef Stefan Institute Ljubljana, Slovenia Marko Grobelnik marko.grobelnik@ijs.si Jozef Stefan Institute Ljubljana, Slovenia Stavanger, May 8 th 2012 Introduction What is Big data? Why Big-Data? When Big-Data is really a problem? Techniques Tools

More information

Marko Grobelnik marko.grobelnik@ijs.si Jozef Stefan Institute

Marko Grobelnik marko.grobelnik@ijs.si Jozef Stefan Institute Marko Grobelnik marko.grobelnik@ijs.si Jozef Stefan Institute Kalamaki, May 25 th 2012 Introduction What is Big data? Why Big-Data? When Big-Data is really a problem? Techniques Tools Applications Literature

More information

On the Bursty Evolution of Online Social Networks

On the Bursty Evolution of Online Social Networks On the Bursty Evolution of Online Social Networks Sabrina Gaito, Matteo Zignani, Gian Paolo Rossi, Alessandra Sala Xiaohan Zhao, Haitao Zheng, Ben Y. Zhao Computer Science, Universitá degli Studi di Milano

More information

SIAM PP 2014! MapReduce in Scientific Computing! February 19, 2014

SIAM PP 2014! MapReduce in Scientific Computing! February 19, 2014 SIAM PP 2014! MapReduce in Scientific Computing! February 19, 2014 Paul G. Constantine! Applied Math & Stats! Colorado School of Mines David F. Gleich! Computer Science! Purdue University Hans De Sterck!

More information

The Big Data Paradigm Shift. Insight Through Automation

The Big Data Paradigm Shift. Insight Through Automation The Big Data Paradigm Shift Insight Through Automation Agenda The Problem Emcien s Solution: Algorithms solve data related business problems How Does the Technology Work? Case Studies 2013 Emcien, Inc.

More information

Big Graph Processing: Some Background

Big Graph Processing: Some Background Big Graph Processing: Some Background Bo Wu Colorado School of Mines Part of slides from: Paul Burkhardt (National Security Agency) and Carlos Guestrin (Washington University) Mines CSCI-580, Bo Wu Graphs

More information

Social Prediction in Mobile Networks: Can we infer users emotions and social ties?

Social Prediction in Mobile Networks: Can we infer users emotions and social ties? Social Prediction in Mobile Networks: Can we infer users emotions and social ties? Jie Tang Tsinghua University, China 1 Collaborate with John Hopcroft, Jon Kleinberg (Cornell) Jinghai Rao (Nokia), Jimeng

More information

Clustering Big Data. Anil K. Jain. (with Radha Chitta and Rong Jin) Department of Computer Science Michigan State University November 29, 2012

Clustering Big Data. Anil K. Jain. (with Radha Chitta and Rong Jin) Department of Computer Science Michigan State University November 29, 2012 Clustering Big Data Anil K. Jain (with Radha Chitta and Rong Jin) Department of Computer Science Michigan State University November 29, 2012 Outline Big Data How to extract information? Data clustering

More information

Outline. What is Big data and where they come from? How we deal with Big data?

Outline. What is Big data and where they come from? How we deal with Big data? What is Big Data Outline What is Big data and where they come from? How we deal with Big data? Big Data Everywhere! As a human, we generate a lot of data during our everyday activity. When you buy something,

More information

Mining Large Graphs. ECML/PKDD 2007 tutorial. Part 2: Diffusion and cascading behavior

Mining Large Graphs. ECML/PKDD 2007 tutorial. Part 2: Diffusion and cascading behavior Mining Large Graphs ECML/PKDD 2007 tutorial Part 2: Diffusion and cascading behavior Jure Leskovec and Christos Faloutsos Machine Learning Department Joint work with: Lada Adamic, Deepay Chakrabarti, Natalie

More information

Large-scale Data Mining: MapReduce and Beyond Part 2: Algorithms. Spiros Papadimitriou, IBM Research Jimeng Sun, IBM Research Rong Yan, Facebook

Large-scale Data Mining: MapReduce and Beyond Part 2: Algorithms. Spiros Papadimitriou, IBM Research Jimeng Sun, IBM Research Rong Yan, Facebook Large-scale Data Mining: MapReduce and Beyond Part 2: Algorithms Spiros Papadimitriou, IBM Research Jimeng Sun, IBM Research Rong Yan, Facebook Part 2:Mining using MapReduce Mining algorithms using MapReduce

More information

MLg. Big Data and Its Implication to Research Methodologies and Funding. Cornelia Caragea TARDIS 2014. November 7, 2014. Machine Learning Group

MLg. Big Data and Its Implication to Research Methodologies and Funding. Cornelia Caragea TARDIS 2014. November 7, 2014. Machine Learning Group Big Data and Its Implication to Research Methodologies and Funding Cornelia Caragea TARDIS 2014 November 7, 2014 UNT Computer Science and Engineering Data Everywhere Lots of data is being collected and

More information

Estimating PageRank Values of Wikipedia Articles using MapReduce

Estimating PageRank Values of Wikipedia Articles using MapReduce Estimating PageRank Values of Wikipedia Articles using MapReduce Due: Sept. 30 Wednesday 5:00PM Submission: via Canvas, individual submission Instructor: Sangmi Pallickara Web page: http://www.cs.colostate.edu/~cs535/assignments.html

More information

Introduction. A. Bellaachia Page: 1

Introduction. A. Bellaachia Page: 1 Introduction 1. Objectives... 3 2. What is Data Mining?... 4 3. Knowledge Discovery Process... 5 4. KD Process Example... 7 5. Typical Data Mining Architecture... 8 6. Database vs. Data Mining... 9 7.

More information

Big Data Analytics. Lucas Rego Drumond

Big Data Analytics. Lucas Rego Drumond Big Data Analytics Lucas Rego Drumond Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany Big Data Analytics Big Data Analytics 1 / 36 Outline

More information

Introduction to Data Mining

Introduction to Data Mining Introduction to Data Mining 1 Why Data Mining? Explosive Growth of Data Data collection and data availability Automated data collection tools, Internet, smartphones, Major sources of abundant data Business:

More information

BIG DATA TRENDS AND TECHNOLOGIES

BIG DATA TRENDS AND TECHNOLOGIES BIG DATA TRENDS AND TECHNOLOGIES THE WORLD OF DATA IS CHANGING Cloud WHAT IS BIG DATA? Big data are datasets that grow so large that they become awkward to work with using onhand database management tools.

More information

www.pwc.com/oracle Next presentation starting soon Business Analytics using Big Data to gain competitive advantage

www.pwc.com/oracle Next presentation starting soon Business Analytics using Big Data to gain competitive advantage www.pwc.com/oracle Next presentation starting soon Business Analytics using Big Data to gain competitive advantage If every image made and every word written from the earliest stirring of civilization

More information

A Tour of the Zoo the Hadoop Ecosystem Prafulla Wani

A Tour of the Zoo the Hadoop Ecosystem Prafulla Wani A Tour of the Zoo the Hadoop Ecosystem Prafulla Wani Technical Architect - Big Data Syntel Agenda Welcome to the Zoo! Evolution Timeline Traditional BI/DW Architecture Where Hadoop Fits In 2 Welcome to

More information

Using Data Mining to Detect Insurance Fraud

Using Data Mining to Detect Insurance Fraud IBM SPSS Modeler Using Data Mining to Detect Insurance Fraud Improve accuracy and minimize loss Highlights: combines powerful analytical techniques with existing fraud detection and prevention efforts

More information

Advanced Analytics & IoT Architectures

Advanced Analytics & IoT Architectures Advanced Analytics & IoT Architectures Presented by: Tom Marek and Orion Gebremedhin Use Case: ETL Offloading Have you outgrown your data delivery SLAs? Get the right data at the right time 2 ETL Processing

More information

Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware

Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware Created by Doug Cutting and Mike Carafella in 2005. Cutting named the program after

More information

Information Management course

Information Management course Università degli Studi di Milano Master Degree in Computer Science Information Management course Teacher: Alberto Ceselli Lecture 01 : 06/10/2015 Practical informations: Teacher: Alberto Ceselli (alberto.ceselli@unimi.it)

More information

Time Series Analysis and Forecasting Methods for Temporal Mining of Interlinked Documents

Time Series Analysis and Forecasting Methods for Temporal Mining of Interlinked Documents Time Series Analysis and Forecasting Methods for Temporal Mining of Interlinked Documents Prasanna Desikan and Jaideep Srivastava Department of Computer Science University of Minnesota. @cs.umn.edu

More information

Big Data Analytics of Multi-Relationship Online Social Network Based on Multi-Subnet Composited Complex Network

Big Data Analytics of Multi-Relationship Online Social Network Based on Multi-Subnet Composited Complex Network , pp.273-284 http://dx.doi.org/10.14257/ijdta.2015.8.5.24 Big Data Analytics of Multi-Relationship Online Social Network Based on Multi-Subnet Composited Complex Network Gengxin Sun 1, Sheng Bin 2 and

More information

Concept and Project Objectives

Concept and Project Objectives 3.1 Publishable summary Concept and Project Objectives Proactive and dynamic QoS management, network intrusion detection and early detection of network congestion problems among other applications in the

More information

Social Network Analysis

Social Network Analysis Social Network Analysis Challenges in Computer Science April 1, 2014 Frank Takes (ftakes@liacs.nl) LIACS, Leiden University Overview Context Social Network Analysis Online Social Networks Friendship Graph

More information

Hadoop for Enterprises:

Hadoop for Enterprises: Hadoop for Enterprises: Overcoming the Major Challenges Introduction to Big Data Big Data are information assets that are high volume, velocity, and variety. Big Data demands cost-effective, innovative

More information

Part 1: Link Analysis & Page Rank

Part 1: Link Analysis & Page Rank Chapter 8: Graph Data Part 1: Link Analysis & Page Rank Based on Leskovec, Rajaraman, Ullman 214: Mining of Massive Datasets 1 Exam on the 5th of February, 216, 14. to 16. If you wish to attend, please

More information

Using an In-Memory Data Grid for Near Real-Time Data Analysis

Using an In-Memory Data Grid for Near Real-Time Data Analysis SCALEOUT SOFTWARE Using an In-Memory Data Grid for Near Real-Time Data Analysis by Dr. William Bain, ScaleOut Software, Inc. 2012 ScaleOut Software, Inc. 12/27/2012 IN today s competitive world, businesses

More information

Sunnie Chung. Cleveland State University

Sunnie Chung. Cleveland State University Sunnie Chung Cleveland State University Data Scientist Big Data Processing Data Mining 2 INTERSECT of Computer Scientists and Statisticians with Knowledge of Data Mining AND Big data Processing Skills:

More information

Two Step Graph-based Semi-supervised Learning for Online Auction Fraud Detection

Two Step Graph-based Semi-supervised Learning for Online Auction Fraud Detection Two Step Graph-based Semi-supervised Learning for Online Auction Fraud Detection Phiradet Bangcharoensap 1, Hayato Kobayashi 2, Nobuyuki Shimizu 2, Satoshi Yamauchi 2, and Tsuyoshi Murata 1 1 Tokyo Institute

More information

SOCIAL MEDIA 80 78 76 74 72 70 68 66 64 Access to free content Series 1 To learn Advanced news of products Series 1 A Social Roadmap Understand how and why people use social media Map the social

More information

BIG DATA TOOLS. Top 10 open source technologies for Big Data

BIG DATA TOOLS. Top 10 open source technologies for Big Data BIG DATA TOOLS Top 10 open source technologies for Big Data We are in an ever expanding marketplace!!! With shorter product lifecycles, evolving customer behavior and an economy that travels at the speed

More information

Analytics in the Cloud. Peter Sirota, GM Elastic MapReduce

Analytics in the Cloud. Peter Sirota, GM Elastic MapReduce Analytics in the Cloud Peter Sirota, GM Elastic MapReduce Data-Driven Decision Making Data is the new raw material for any business on par with capital, people, and labor. What is Big Data? Terabytes of

More information

Big Data Analytics. with EMC Greenplum and Hadoop. Big Data Analytics. Ofir Manor Pre Sales Technical Architect EMC Greenplum

Big Data Analytics. with EMC Greenplum and Hadoop. Big Data Analytics. Ofir Manor Pre Sales Technical Architect EMC Greenplum Big Data Analytics with EMC Greenplum and Hadoop Big Data Analytics with EMC Greenplum and Hadoop Ofir Manor Pre Sales Technical Architect EMC Greenplum 1 Big Data and the Data Warehouse Potential All

More information

Hadoop Parallel Data Processing

Hadoop Parallel Data Processing MapReduce and Implementation Hadoop Parallel Data Processing Kai Shen A programming interface (two stage Map and Reduce) and system support such that: the interface is easy to program, and suitable for

More information

Large Scale Social Network Analysis

Large Scale Social Network Analysis Large Scale Social Network Analysis DATA ANALYTICS 2013 TUTORIAL Rui Sarmento email@ruisarmento.com João Gama jgama@fep.up.pt Outline PART I 1. Introduction & Motivation Overview & Contributions 2. Software

More information

Overview on Graph Datastores and Graph Computing Systems. -- Litao Deng (Cloud Computing Group) 06-08-2012

Overview on Graph Datastores and Graph Computing Systems. -- Litao Deng (Cloud Computing Group) 06-08-2012 Overview on Graph Datastores and Graph Computing Systems -- Litao Deng (Cloud Computing Group) 06-08-2012 Graph - Everywhere 1: Friendship Graph 2: Food Graph 3: Internet Graph Most of the relationships

More information

Community-Aware Prediction of Virality Timing Using Big Data of Social Cascades

Community-Aware Prediction of Virality Timing Using Big Data of Social Cascades 1 Community-Aware Prediction of Virality Timing Using Big Data of Social Cascades Alvin Junus, Ming Cheung, James She and Zhanming Jie HKUST-NIE Social Media Lab, Hong Kong University of Science and Technology

More information

Homophily in Online Social Networks

Homophily in Online Social Networks Homophily in Online Social Networks Bassel Tarbush and Alexander Teytelboym Department of Economics, University of Oxford bassel.tarbush@economics.ox.ac.uk Department of Economics, University of Oxford

More information

Unlocking the Intelligence in. Big Data. Ron Kasabian General Manager Big Data Solutions Intel Corporation

Unlocking the Intelligence in. Big Data. Ron Kasabian General Manager Big Data Solutions Intel Corporation Unlocking the Intelligence in Big Data Ron Kasabian General Manager Big Data Solutions Intel Corporation Volume & Type of Data What s Driving Big Data? 10X Data growth by 2016 90% unstructured 1 Lower

More information

SOCIAL NETWORK DATA ANALYTICS

SOCIAL NETWORK DATA ANALYTICS SOCIAL NETWORK DATA ANALYTICS SOCIAL NETWORK DATA ANALYTICS Edited by CHARU C. AGGARWAL IBM T. J. Watson Research Center, Yorktown Heights, NY 10598, USA Kluwer Academic Publishers Boston/Dordrecht/London

More information

Tree Ensembles: The Power of Post- Processing. December 2012 Dan Steinberg Mikhail Golovnya Salford Systems

Tree Ensembles: The Power of Post- Processing. December 2012 Dan Steinberg Mikhail Golovnya Salford Systems Tree Ensembles: The Power of Post- Processing December 2012 Dan Steinberg Mikhail Golovnya Salford Systems Course Outline Salford Systems quick overview Treenet an ensemble of boosted trees GPS modern

More information

Dynamics of Large Networks. Jurij Leskovec

Dynamics of Large Networks. Jurij Leskovec Dynamics of Large Networks Jurij Leskovec September 2008 CMU-ML-08-111 Machine Learning Department School of Computer Science Carnegie Mellon University Pittsburgh, PA Thesis Committee: Christos Faloutsos,

More information

Mining Large Datasets: Case of Mining Graph Data in the Cloud

Mining Large Datasets: Case of Mining Graph Data in the Cloud Mining Large Datasets: Case of Mining Graph Data in the Cloud Sabeur Aridhi PhD in Computer Science with Laurent d Orazio, Mondher Maddouri and Engelbert Mephu Nguifo 16/05/2014 Sabeur Aridhi Mining Large

More information

Oracle Advanced Analytics 12c & SQLDEV/Oracle Data Miner 4.0 New Features

Oracle Advanced Analytics 12c & SQLDEV/Oracle Data Miner 4.0 New Features Oracle Advanced Analytics 12c & SQLDEV/Oracle Data Miner 4.0 New Features Charlie Berger, MS Eng, MBA Sr. Director Product Management, Data Mining and Advanced Analytics charlie.berger@oracle.com www.twitter.com/charliedatamine

More information