Almighty Google knows almost EVERYTHING! (Big-data & Complex Networks)

Size: px

Start display at page:

Download "Almighty Google knows almost EVERYTHING! (Big-data & Complex Networks)"

Hugo Osborne
8 years ago
Views:

1 Almighty Google knows almost EVERYTHING! (Big-data & Complex Networks) Hawoong Jeong (Dept. of Physics, KAIST, KOREA) D. Kim, J. Yun, H. Soh (KAIST), S.H. Lee (Oxford), P.J. Kim (APCTP), Y.Y. Ahn (Indiana U) Big data: The next Google What will happen in the next 10 years? Integration of the worlds of matter and information ELECTRONIC PAPER HAPTICS VIDEO VISORS PRODUCTS WITH MEMORIES AUTONOMOUS ROBOTS GENETIC INFORMATION OPEN CONTENT MANAGEMENT THREE-DIMENSIONAL ENVIRONMENTS THE SEMANTIC WEB BETTER BROWSERS (2008) 1

Integration of the worlds of matter and information ELECTRONIC PAPER HAPTICS VIDEO VISORS PRODUCTS WITH MEMORIES AUTONOMOUS

http://huzheng.cghub.com/ It s BIG ONLY BIG? Definition of BIG DATA by Gartner Inc. = 3V Volume : Really big in size Translation: Kor. Eng. << Kor. Jpn Eng.

2 It s BIG ONLY BIG? Definition of BIG DATA by Gartner Inc. = 3V Volume : Really big in size Translation: Kor. Eng. << Kor. Jpn Eng. Velocity : fast & real time analysis Twitter/facebook : create & diffuse fast Variety : various format, no-fixed-format User generated contents/unstructured data 2

3 Votes (millions) Importance of unstructured data Most of newly generated data is (>90%) video, music, message, social-media, geo-location etc, unstructured! Not a simple text or number anymore!!! Needs new analysis technique as well as new way of storage [e.g. CCTV recording] 2007 Presidential Election in Korea via Google Jeong M. Lee Pearson corr. R~ H. Lee Moon Dong-A Newspaper survey R~ Google hits (millions) 3

!! Needs new analysis technique as well as new way of storage

4 US presidential candidates (2008/4) Obama Hilary McCain Seoul Mayor (2011) Na vs Park 4

5 Ask Google for help?!? As of 2011/10/25 23:15 One day before the election 46.6Million 54.3Million Final result? Na : Park = 46.2 : 53.4 US 2012 Presidential Election As of 11/4 10:33PM 5

6 US 2012 Presidential Election As of 11/4 10:33PM US 2012 Presidential Election Google said 1350M vs 1290M = 51.1 vs 48.8 Result: 50 % vs 48% 6

7 This year in Korea? Park vs Moon People don t lie when they SEARCH! Search interest in weight loss peaks at every January! Google 7

8 Q: How many flu patients out there? Why interested? If it increases rapidly, outbreak! CDC (Center for Disease Control) collects % of patients with flu-like symptom from doctors of each state! Collect & statistics, it takes two weeks! Google claims that they can do better! In flu season, number of search for flu-related keywords is increasing! (Obviously! You don t??) Nature (2008) Find out best set of keywords for flu! Comparing 2003~2007 old (useless) CDC data with Google s search history, pick up 50 keywords real time prediction of # of flu patients in 2008 (2 weeks faster than CDC, with geo-location) 8

Google claims that they can do better! In flu season, number of search for flu-related keywords is increasing! (Obviously! You don t?

9 BUT, in 2013, Google Flu fails!! Why?? Media!! (vaccine shortage in 2013 Jan. flooded the news) Disruptions: Data Without Context Tells a Misleading Story Then what??? Don t forget the Power of Networks! Chanel Louis Vuitton Gucci Google 9

10 in Google full of data!!! Google N-gram Project! Civil right Civil War movements (1861-5) ( ) J.-B. Michael et al Science (2011) Grammar Correction via big-data 이상구 (SNU) 10

11 We have to combine Data together with networks! SNS = Data + Networks Twitter between politicians How do they communicate By H. Park(Sisa-In) Clustered 11

12 Social Network with Google??? S. H. Lee, P.-J. Kim, Y.-Y. Ahn, H. Jeong "Googling social interactions: Web search engine based social network construction", PLoS ONE e11233 (2010) Workshop speaker list 12

13 ESHIA Winter Workshop Speakers Basic idea: Using search engine for finding something Laszlo Barabasi s Google Hits (fame) = 175,000 13

14 To make a network, we need link information between 2 persons HOW? Ask! Laszlo Barabasi Hawoong Jeong Basic idea: Constructing weighted social networks by using web search engines Alessandro Vespignani w BV = 6130 w VJ = 2630 Laszlo Barabasi w BJ = 8530 (Google correlation) Hawoong Jeong and so on 14

networks by using web search engines Alessandro Vespignani w BV = 6130 w VJ

15 ESHIA Winter Workshop Speakers ESHIA Winter Workshop Speakers 15

16 ESHIA Winter Workshop Speakers ESHIA Winter Workshop Speakers 16

17 ESHIA Winter Workshop Speakers More familiar example: Transportation Networks with agents We are suffering everyday because of the traffic congestion Why? Can we solve the problem? Phys.Rev.Lett.(2008) 17

are suffering everyday because of the traffic

18 What is important? Dynamics of the networks : The topology of the network itself often evolves in time Dynamics on the networks : Agents are moving on the networks (E.g. Driver wants to find the shortest paths, Finding OPTIMAL PATH) What to optimize? Latency function (like time or cost) length 1 Latency # of travelers length width 1/width # of travelers CSSPL Global Optimum vs User Optimum (Please don t be too judgmental!) 18

networks : Agents are moving on the networks (E.g. Driver wants to find the shortest paths, Finding OPTIMAL PATH) What to optimize?

Network flow with congestion Cost function on path i Latency function length of path i # of agent on path i width of path i S T Given network with many agents going from S (source) to T (target),

19 Network flow with congestion Cost function on path i Latency function length of path i # of agent on path i width of path i S T Given network with many agents going from S (source) to T (target), what will be the optimized distribution of agents for best performance?? Based on the model of Roughgarden & Tardos, 2000 CSSP L Optimizations in physics Euler-Lagrange differential equation minimal free energy in thermodynamic physics Fitting experimental DATA with formula Low temperature behavior of disordered magnets There are two types of optimizations!!! Centralized control Minimizing Global Cost Global Optimization Decentralized control Each agent minimizes its own personal cost User Optimization (Nash equilibrium) CSSPL 19

? Based on the model of Roughgarden & Tardos, 2000 CSSP L Optimizations in physics Euler-Lagrange differential equation minimal free energy in thermodynamic physics Fitting

20 The Price of Anarchy Decentralized control Each agent minimizes its own personal cost Centralized control Minimizing Global Cost total cost of User Optimum total cost of Global Optimum Price of Anarchy Koutsoupias & Papadimitriou, Price of Anarchy (Roughgarden & Tardos, 2000) Price we have to pay not being coordinated by central agency Price of being selfish CSSP L Price of Anarchy: Contrived Example Pigous s example: Congestion sensitive network 10 agents want to Go from S to T. S T What will be the min total cost, i.e. Global Optimum =? If x a =x, then x b =10-x, total cost=10ᆞx + (10-x) ᆞ(10-x) = x 2-10x+100=(x-5) x a =x b =5 with total cost 75 CSSP L 20

agency Price of being selfish CSSP L Price of Anarchy: Contrived Example Pigous s example: Congestion sensitive network 10 agents want to Go from S to T.

21 Price of Anarchy: Contrived Example Envy The upper agents get envious of people with lower costs! x a = x b =5 S T BUT Global Optimum = 5x10 + 5x5 = 75 75/10 = 7.5$/person in average CSSP L Price of Anarchy: Contrived Example x a = 5 S x b = 5 T What will be the User Optimum? (Nash Equilibrium: everyone happy) CSSP L 21

22 Price of Anarchy: Contrived Example Move to Lower path x a = 5-1 x b = S T user cost = < 10 CSSP L Price of Anarchy: Contrived Example again +1 S x a = 4-1 x b = 6+1 T user cost = < 10 CSSP L 22

23 Price of Anarchy: Contrived Example again +1 S x a = 3-1 x b = 7+1 T user cost = < 10 CSSP L Price of Anarchy: Contrived Example again +1 S x a = 2-1 x b = 8+1 T user cost = < 10 CSSP L 23

24 Price of Anarchy: Contrived Example again +1 S x a = 1-1 x b = 9+1 T User Optimum = 10 x10 = 100 avg 10$/per travel cost > avg 7.5$/per travel cost CSSP L Price of Anarchy: Contrived Example There is a gap between global optimum & user optimum! x a = 5 vs 0 S x b = 5 vs 10 T User Optimum = 10 x10 = 100 Global Optimum = 5x10 + 5x5 = 75 4/3 Price of Anarchy! CSSP L 24

25 More realistic/complex example Assumption: traffic reaches at equilibrium Price of Anarchy on a real world the Boston Road Network (with real geometrical information like width, length, one-way etc) Traffic from central square (S) to copley square (T) CSSP L Boston Road Map CSSP L 25

26 Boston Road Network Start (nodes 59, edges 108, regular-like) Latency function = ax + b End Width length CSSP L More realistic/complex example Assumption: traffic reaches at equilibrium Price of Anarchy on a real world the Boston Road Network (with real geometrical information) Global optimum : mapping to Min-cost Max-flow problem User optimum ~ approximate optimum: Metropolis Algorithm and Annealing method to find out the optimum configurations CSSP L 26

27 User Optimum Global Optimum Number of traveler =1 CSSP L Congestion distribution on the edges User Optimum Global Optimum Number of Agents: 20 CSSP L 27

28 Variation of POA with Agent # CSSPL To write a paper Where to use?? Network design for better traffic control? 28

Making network more efficient without central government?? Lower PoA ~ better(?) system ( even w/o central control, user optimum is closer to global optimum, better!

29 Making network more efficient without central government?? Lower PoA ~ better(?) system ( even w/o central control, user optimum is closer to global optimum, better!) Let s make better network with lower PoA Simple thought (by stupid government): construct more roads with our tax money! But beware of Braess paradox!!! (counter-intuitive consequences) Braess s Paradox Again 10 travelers want to move from S to T. x 10 S 0: cost-free express road T 10 x increase User Optimum without middle arc = 150 = Global Optimum PoA=1 User Optimum with middle arc = 200 Price of Anarchy = 200/150 = 4/3 CSSPL 29

30 In real Boston Road Network? Start End CSSPL Better Boston Road Network! 30

31 London NEW YORK Quantifying Trading Behavior in Financial Markets Using Google Trends Tobias Preis, Helen Susannah Moat, H. Eugene Stanley Sci. Rep. (2013) 31

32 Data How to buy and sell? Dow Jones Industrial Average (DJIA) p(t) from 5 January 2004 until 22 February 2011 a set of 98 search terms related stock market Google trend from 2004 to 2011 (week) n(t): how many searches at week t Δn(t, Δt) = n(t) N(t 1, Δt) with N(t 1, Δt) = (n(t 1) + n(t 2) + + n(t Δt))/Δt If Δn(t-1, Δt)>0, sell p(t) at t and buy p(t+1) at t+1 <0, buy p(t) at t and sell p(t+1) at t+1 (i.e. search volume is ( ), stock price will be ( )) Using the word Debt Best performance 326% (Δt=3 weeks) If using p(t) instead of n(t), 33% only 32

33 if search volume increases(decreases) compared to last 3 weeks avg Sell(Buy) on Monday Buy(Sell) on next Monday i.e. stock price will go down(up) 33

34 When search volume decreases When search volume increases

35 In Korea??? (by MoneyWeek mag.) Google search volume too small, using Koran portal Naver Search vs KOSPI/KOSDAQ Stock (0.67) Fund (0.77) region (-0.71) Real estate (-0.64) 35

36 Conclusion Summary Google knows many things and can also predict many things We can also do many things using Google with big-data Especially, combining big-data & networks can do better Dynamics on complex network is fun! Which direction are we going??? ASK GOOGLE itself! Finally where are we??? 36

Quantifying human behaviour using online data. Suzy Moat Data Science Lab Behavioural Science, WBS Suzy.Moat@wbs.ac.uk

Quantifying human behaviour using online data Suzy Moat Data Science Lab Behavioural Science, WBS Suzy.Moat@wbs.ac.uk Data Science Lab Behavioural Science, WBS 1 The advantage of looking forward more Google