CS 1944: Sophomore Seminar Big Data and Machine Learning

Transcription

1 CS 1944: Sophomore Seminar Big Data and Machine Learning B. Aditya Prakash Assistant Professor Nov 3, 2015

2 About me Assistant Professor, CS Member, Discovery Analy>cs Center Previously Ph.D. in Computer Science, Carnegie Mellon University B.Tech in Computer Science and Engg, Indian Ins>tute of Technology (IIT) Bombay Internships at Sprint, Yahoo, MicrosoN Research Prakash

3 Prakash

4 Data contains value and knowledge Prakash

5 Data and Business Data1and1business* 4* Recommended'linksA +79%''clicksA Personalized'' News'InterestsA +250%'clicksA Top'SearchesA +43%'clicksA vs.1randomly1selected* vs.1editorial1onedsizedfitsdall* vs.1editor1selected* Prakash 2015 Source: A. Machhanavajjhala 5

6 Data and Science Data1and1science* 5* Red:1official1numbers1from1Center1for1Disease1Control1and1Prevention;1weekly11 Black:1based1on1Google1search1logs;1daily1(potentially1instantaneously)* Detecting'influenza'epidemics'using'search' engine'query'data1 nature07634.html Prakash

7 Data and Government Data1and1government* 6* president/2013/06/14/1d71fe2e-d391-11e2- b05f-3ea3f0e7bb5a_story.html business/economy/democratspush-to-redeploy-obamas-voterdatabase/2012/11/20/ d14793a4-2e83-11e2-89d4-040c a_story.html Democratizing-Data edward-snowden-nsa-files-timeline Prakash 2015 Source: A. Machhanavajjhala 7

8 Data and Culture Data1and1culture* 7* Word1frequencies1in1 EnglishDlanguage1 books1in1google s1 database /03/20/what-are-you-inthe-mood-for-emotionaltrends-in-20th-century-books/ Prakash 2015 Source: A. Machhanavajjhala 8

9 8* Data1and1 1 your1favorite1subject* Sports* Journalism* Prakash

10 Good news: Demand for Data Mining Prakash

11 How to extract value from data? Manipulate Data CS, Domain exper>se Analyze Data Math, CS, Stat Communicate your results CS, Domain Exper>se Prakash

12 13* CommunicaEon is important! Communicating1results* The"British"government"spends"" 13"billion"a"year"on"universities. F So?* Try1instead1 bubbletree-map.html#/~/total/education/university On"average,"1"in"every"15"Europeans"" is"totally"illiterate. F True* But1about111in1every1141is1under171years1old!* Prakash

13 What is Data Mining? Given lots of data Discover paherns and models that are: Valid: hold on new data with some certainty Useful: should be possible to act on the item Unexpected: non-obvious to the system Understandable: humans should be able to interpret the pa]ern Prakash

14 Data Mining Tasks DescripEve methods Find human-interpretable pa]erns that describe the data Example: Clustering PredicEve methods Use some variables to predict unknown or future values of other variables Example: Recommender systems Prakash

15 Comp. Systems Theory & Algo. Biology Physics Social Science ML & Stats. Big data Econ. 15 Prakash 2015

16 Data at CS, VT Knowledge, Informa>on and Data h]p:// kid People: Fox, Harrison, Huang, Lu (in NVA), Ramakrishnan (in NVA), Rozovskaya, Prakash Prakash

17 Courses Background in some areas: CS3414 (Numerical Methods); also prob/stat 4000 level 4244 Internet SoNware Development 4604 Database Management Systems 4624 Capstone (Mul>media, Informa>on Access) 4634 Design of Informa>on (Capstone) 4804 AI 4984 Computa>onal Linguis>cs (Capstone) Prakash

18 Discovery AnalyEcs Center Prakash

19 MY RESEARCH Prakash

20 Networks are everywhere! Facebook Network [2010] Human Disease Network [Barabasi 2007] Gene Regulatory Network [Decourty 2008] The Internet [2005] Prakash

21 What else do they have in common? Prakash

22 High School DaEng Network Bearman et. al. Am. Jnl. of Sociology, Image: Mark Newman Blue: Male Pink: Female Interes>ng observa>ons? Prakash

23 The Internet Skewed Degrees Robustness Prakash

24 Karate Club Network Prakash

25 Dynamical Processes over networks are also everywhere! Prakash

26 Why do we care? Social collabora>on Informa>on Diffusion Viral Marke>ng Epidemiology and Public Health Cyber Security Human mobility Games and Virtual Worlds Ecology Prakash 2015

27 Why do we care? (1: Epidemiology) Dynamical Processes over networks [AJPH 2007] Diseases over contact networks Prakash 2015 CDC data: Visualiza>on of the first 35 tuberculosis (TB) pa>ents and their 1039 contacts 27

28 Why do we care? (1: Epidemiology) Dynamical Processes over networks Each circle is a hospital ~3000 hospitals More than 30,000 pa>ents transferred [US-MEDICARE NETWORK 2005] Prakash 2015 Problem: Given k units of disinfectant, whom to immunize? 28

29 Prakash Hospital-acquired inf. took 99K+ lives, cost $5B+ (all per year) Why do we care? (1: Epidemiology) ~6x fewer! [US-MEDICARE NETWORK 2005] CURRENT PRACTICE OUR METHOD

30 Why do we care? (2: Online Diffusion) > 800m users, ~$1B revenue [WSJ 2010] ~100m achve users > 50m users Prakash

31 Why do we care? (2: Online Diffusion) Dynamical Processes over networks Buy Versace! Followers Celebrity Prakash 2015 Social Media Marke>ng 31

32 Social ßà Biological Contagion AutomaEcally learn models Prakash

33 Why do we care? (3: To change the world?) Dynamical Processes over networks Social networks and CollaboraHve AcHon Prakash

34 High Impact MulEple Seangs epidemic out-breaks Q. How to squash rumors faster? products/viruses Q. How do opinions spread? transmit s/w patches Q. How to market beser? Prakash

35 Dynamical Processes = (a lot of) Networks + (some) Time-Series Prakash

36 Research Theme ANALYSIS Understanding DATA Large real-world POLICY/ ACTION Managing networks & processes Prakash

37 Research Theme Public Health ANALYSIS Will an epidemic happen? DATA Modeling # pa>ent transfers Prakash 2015 POLICY/ ACTION How to control out-breaks? 37

38 Research Theme Social Media ANALYSIS # cascades in future? DATA Modeling Tweets spreading Prakash 2015 POLICY/ ACTION How to market be]er? 38

39 A QuesEon How many of you think your friends have more friends than you? J A recent Facebook study Examined all of FB s users: 721 million people with 69 billion friendships. about 10 percent of the world s popula>on! Found that user s friend count was less than the average friend count of his or her friends, 93 percent of the >me. Users had an average of 190 friends, while their friends averaged 635 friends of their own. Prakash

40 Possible Reasons? You are a loner? Your friends are extroverts? There are more extroverts than introverts in the world? Prakash

41 Example Average number of friends? Source: S. Strogatz, NYT 2012 Prakash

42 Example Average number of friends = ( ) / 4 = 2 Source: S. Strogatz, NYT 2012 Prakash

43 Example Average number of friends = ( ) / 4 = 2 Average number of friends of friends Source: S. Strogatz, NYT 2012 Prakash

44 Example Average number of friends = ( ) / 4 = 2 Average number of friends of friends = ( )/8 = ((1x1) + (3x3) + (2x2) + (2x2))/8 Source: S. Strogatz, NYT 2012 Prakash

45 Example Average number of friends = ( ) / 4 = 2 Average number of friends of friends = ( )/8 = ((1x1) + (3x3) + (2x2) + (2x2))/8 = 2.25! Source: S. Strogatz, NYT 2012 Prakash

46 Actually it is (almost) always true! Proof? Prakash

47 Actually it is (almost) always true! Proof? E[X] = x i / N Prakash

48 Actually it is (almost) always true! Proof? E[X] = x i / N Var[X] = E[(X E[X]) 2 ] = E[X 2 ] E[X] 2 Prakash

49 Actually it is (almost) always true! Proof? E[X] = x i / N Var[X] = E[(X E[X]) 2 ] = E[X 2 ] E[X] 2 E[X 2 ] E[X] = E[X]+ Var[X] E[X] Prakash

50 Actually it is (almost) always true! Proof? EssenEally, it is true if there is any spread in # of friends (nonzero variance)! E[X] = x i / N Var[X] = E[(X E[X]) 2 ] = E[X 2 ] E[X] 2 E[X 2 ] E[X] = E[X]+ Var[X] E[X] Prakash

51 Immuniza>on ImplicaEons Figure 1. Network Illustrating Structural Parameters. This real network of 105 students shows variation in structural attributes and topological position. Each circle represents a person and each line represents a friendship tie. Nodes A and B have different degree, a measure that indicates the number of ties. Nodes with higher degree also tend to exhibit higher centrality (node A with six friends is more central than B and C who both only have four friends). If contagions infect people at random at the beginning of an epidemic, central individuals are likely to be infected sooner because they lie a shorter number of steps (on average) from all other individuals in the network. Finally, although nodes B and C have the same degree, they differ in transitivity (the probability that any two of one s friends are friends with each other). Node B exhibits high transitivity with many friends that know one another. In contrast, node C s friends are not connected to one another and therefore they offer more independent possibilities acquaintance immuniza>on for becoming infected earlier in the epidemic. doi: /journal.pone g001 Immunize friend-of-friend Early warning of outbreaks Again, monitor friends of friends the variance of the degree distribution divided by m. Hence, when there is variance in degree in a population, and especially when there is high variance, the mean number of contacts for the friends will be greater (and potentially much greater) than the mean for the random sample. This is sometimes known as the friendship paradox ( your friends have more friends than you do ) [15 19]. While the idea of immunizing such friends of randomly chosen people has previously been explored in a stimulating theoretical paper [12], to our knowledge, a method that uses nominated friends as sensors for early detection of an outbreak has not previously been proposed, nor has it been tested on any sort of real outbreak. To evaluate the effectiveness of nominated friends as social network sensors, we therefore monitored the spread of flu at Harvard College from September 1 to December 31, In the fall of 2009, both seasonal flu (which typically kills 41,000 Americans each year [20]) and the H1N1 strain were prevalent in the US, though the great majority of cases in 2009 have been attributed to the latter.[1] It is estimated that this H1N1 epidemic, which began roughly in April 2009, infected over 50 million Americans. Unlike seasonal flu, which typically affects individuals older than 65, H1N1 tends to affect young people. Nationally, according to the CDC, the epidemic peaked in late October 2009, and vaccination only became widely available in December Whether another outbreak of H1N1 will occur (for example, in areas and populations that have heretofore been spared) is a Figure 2. Theoretical expectations of differences in contagion between central individuals and the population as a whole. A contagious process passes through two phases, one in which the number of infected individuals exponentially increases as the contagion spreads, and one in which incidence exponentially decreases as susceptible individuals become increasingly scarce. These dynamics can be modeled by a logistic function. Central individuals lie on more paths in a network compared to the average person in a population and are therefore more likely to be infected early by a contagion that randomly infects some individuals and then spreads from person to person within the network. This shifts the S- shaped logistic cumulative incidence function forward in time for central individuals compared to the population as a whole (left panel). It also shifts the peak infection rate forward (right panel). doi: /journal.pone g002 Prakash

52 Thanks---QuesEons? B. Aditya Prakash 3160 F Torgersen Hall badityap@cs.vt.edu See my homepage for more details and papers: h]p:// Prakash