Rolling the Dice on Big Data. Ilse Ipsen Department of Mathematics

Transcription

1 Rolling the Dice on Big Data Ilse Ipsen Department of Mathematics

2 The Economist, 27 February 2010

3 Science, 11 February 2011

4 McKinsey Global Institute, May 2011

5

6

7 Rolling the Dice on Big Data What is Big?

8 Measuring Units 1 byte 1 character 10 bytes 1 word 100 bytes 1 sentence 1 kilobyte = 1,000 bytes 1 page

9 1 byte 1 character 10 bytes 1 word 100 bytes 1 sentence Measuring Units 1 kilobyte = 1,000 bytes 1 page 1 megabyte = 1,000 kilobytes complete works of Shakespeare 1 gigabyte = 1,000 megabytes a big shelf full of books

10 1 byte 1 character 10 bytes 1 word 100 bytes 1 sentence Measuring Units 1 kilobyte = 1,000 bytes 1 page 1 megabyte = 1,000 kilobytes complete works of Shakespeare 1 gigabyte = 1,000 megabytes a big shelf full of books 1 terabyte = 1,000 gigabytes all books in the Library of Congress

11 1 byte 1 character 10 bytes 1 word 100 bytes 1 sentence Measuring Units 1 kilobyte = 1,000 bytes 1 page 1 megabyte = 1,000 kilobytes complete works of Shakespeare 1 gigabyte = 1,000 megabytes a big shelf full of books 1 terabyte = 1,000 gigabytes all books in the Library of Congress 1 petabyte = 1,000 terabytes 20 million 4-door filing cabinets full of text

12 1 byte 1 grain of sand

13 1 byte 1 grain of sand 1 terabyte number of grains to fill a swimming pool

14 Rolling the Dice on Big Data Data

15 Rolling the Dice on Big Data

16 Rolling the Dice on Big Data Not quite

17 The Data in this Talk Given: Database: Collection of documents (data points) Query: Single document (data point) Want: Documents closest to query A tiny example to illustrate a big data problem

18 A Tiny Data Example Database: s from known authors 1: shipment of gold damaged in a fire 2: delivery of silver arrived in a silver truck 3: shipment of gold arrived in a truck Query: from unknown author gold silver truck Which s match the query best? These s may give clues about the author of query Simplest approach for matching: Word frequency

19 Tabulating s and Query Database (term document matrix) + Query Terms Query a arrived damaged delivery fire gold in of silver shipment truck

20 Basic Approach for Finding Matching s 1 Common words For each Count number of words common to and Query

21 Basic Approach for Finding Matching s 1 Common words For each Count number of words common to and Query 2 Length Count number of words in each , and in Query

22 Basic Approach for Finding Matching s 1 Common words For each Count number of words common to and Query 2 Length Count number of words in each , and in Query 3 Matching score for each Matching score = Number of common words (Length of ) (Length of Query) s with highest matching scores: May give clues about authors of Query

23 Count Common Words in Query and 1 Terms E 1 Q Multiply a arrived damaged delivery fire gold in of silver shipment truck Sum 1 # common words in 1 and Query: E 1 Q = 1

26 Basic Approach for Finding Matching s 1 Number of words common to s and Query E 1 Q = 1 E 2 Q = 3 E 3 Q = 2 2 Length Count number of words in each , and in query

27 Length of Query Terms Q Square a 0 0 arrived 0 0 damaged 0 0 delivery 0 0 fire 0 0 gold 1 1 in 0 0 of 0 0 silver 1 1 shipment 0 0 truck 1 1 Sum 3 Length of Query: Q = 3 1.7

28 Length of 2 Terms E 2 Square a 1 1 arrived 1 1 damaged 0 0 delivery 1 1 fire 0 0 gold 0 0 in 1 1 of 1 1 silver 2 4 shipment 0 0 truck 1 1 Sum 10 Length of 2: E 2 =

29 Basic Approach for Finding Matching s 1 Number of words common to s and Query E 1 Q = 1 E 2 Q = 3 E 3 Q = 2 2 Length of s and Query Q = E 1 = E 2 = E 3 = 7 2.6

30 Matching Score for each Matching score = Number of common words (Length of ) (Length of query) E 1 Q E 1 Q = E 2 Q E 2 Q = E 3 Q E 3 Q = is the best match for the query

31 Conclusion for Tiny Data Example Database: s from known authors 1: shipment of gold damaged in a fire 2: delivery of silver arrived in a silver truck 3: shipment of gold arrived in a truck Query: from unknown author gold silver truck Best matching 2: delivery of silver arrived in a silver truck

32 The Reason for the Weird Way of Counting Vector Space Model s, Query = vectors Matching score = cosine of angle between and Query E Q = cos (E, Q) E Q

33 What this means in practice Average number of s per day: 294 billion Number words in English language: at least 250,000 Matching one query with a single 250,000 operations (one for every possible word) Matching one query with all s: 250,000 * 294 billion = operations

34 What this means in practice Average number of s per day: 294 billion Number words in English language: at least 250,000 Matching one query with a single 250,000 operations (one for every possible word) Matching one query with all s: 250,000 * 294 billion = operations Fast PC (Intel Core i7 980 XE) 109 Gflops = floating point operations per second Matching one query with all s: about 8 days

35 What this means in practice Average number of s per day: 294 billion Number words in English language: at least 250,000 Matching one query with a single 250,000 operations (one for every possible word) Matching one query with all s: 250,000 * 294 billion = operations Fast PC (Intel Core i7 980 XE) 109 Gflops = floating point operations per second Matching one query with all s: about 8 days US supercomputer (Cray XT5, Opteron quad core 2.3GHz) Peak 1,381,400 Gflops Matching one query with all s: about 1 minute

36 Can the Matching be Performed Faster?

37 Can the Matching be Performed Faster? Yes! Ralph Abbey, Sarah Warkentin, Sylvester Eriksson-Bique, Mary Solbrig, Michael Stefanelli

38 Rolling the Dice on Big Data Rolling the Dice

39 Rolling the Dice on Big Data Rolling the Dice on which words to use for the matching

40 Idea Randomized Query Matching Algorithm Do not use every word in query and s Monte Carlo Sampling: Use only selected words {Downsize to smaller database with fewer words}

41 Idea Randomized Query Matching Algorithm Do not use every word in query and s Monte Carlo Sampling: Use only selected words {Downsize to smaller database with fewer words} Justification Don t need exact matching scores Identify only s with highest matching scores Database available for offline computation Derive statistics based on word frequencies Perform query matching online Use statistics to select words used for matching

42 Suggestions for Downsizing the Database Statistics n: number of words in database Q j : frequency of word j in query W j : frequency of word j in database Suggestion for selecting word j Probability of sampling word j W j Q j p j = W 1 Q W n Q n Frequently occurring words more likely to be sampled

43 Rolling the Dice = Downsizing the Database User input s: number of samples {number of words in downsized database} Monte Carlo Sampling {Roll the dice s times} For t = 1,..., s Sample index j t from {1,..., n} with probability p jt independently and with replacement Downsized database contains only s words: word j 1, word j 2,..., word j s

44 Matching with Downsized Database Downsized database: word j 1, word j 2,..., word j s Word frequency in Query: ˆQ = ( Q j1 Q j2... Q js ) For each E:

45 Matching with Downsized Database Downsized database: word j 1, word j 2,..., word j s Word frequency in Query: ˆQ = ( Q j1 Q j2... Q js ) For each E: Word frequency Ê = ( F j1 F j2... F js )

46 Matching with Downsized Database Downsized database: word j 1, word j 2,..., word j s Word frequency in Query: ˆQ = ( Q j1 Q j2... Q js ) For each E: Word frequency Ê = ( F j1 F j2... F js ) Approximate number of words common to and Query C = 1 ( Fj1 Q j1 + F j 2 Q j2 + + F ) j s Q js s p j1 p j2 p js {s, p j1, p j2,..., p js compensate for fewer words}

47 Matching with Downsized Database Downsized database: word j 1, word j 2,..., word j s Word frequency in Query: ˆQ = ( Q j1 Q j2... Q js ) For each E: Word frequency Ê = ( F j1 F j2... F js ) Approximate number of words common to and Query C = 1 ( Fj1 Q j1 + F j 2 Q j2 + + F ) j s Q js s p j1 p j2 p js {s, p j1, p j2,..., p js compensate for fewer words} Approximate matching score of C Ê ˆQ

48 Reuters Collection: Transcribed Subset 201 documents and 5601 words Number of sampled words s = 56 1 percent Ranking Deterministic Uniform Deterministic q Bucket of computed 25 best matches contains Correct 10 best matches in 99% of all cases

49 Wikipedia Dataset 200 documents and 198,853 words Average percent of correct 10 best matches as function of sample size % correct rankings Number of samples, c Sampling 1% of the words gives correct 9 best matches. More sampling does not help a lot.

50 Big data Summary Matching queries against document database Rolling the dice But... Randomized downsizing of database vocabulary Frequently occurring words more likely to be kept

51 Big data Summary Matching queries against document database Rolling the dice Randomized downsizing of database vocabulary Frequently occurring words more likely to be kept But... Why not use a predictable (deterministic) algorithm? Why use a randomized algorithm?

52 Big data Summary Matching queries against document database Rolling the dice Randomized downsizing of database vocabulary Frequently occurring words more likely to be kept But... Why not use a predictable (deterministic) algorithm? Why use a randomized algorithm? Advantages of randomized algorithm Easy to analyze Fast, and simple to implement As good in practice as deterministic algorithm (for this type of application)

53 The Bigger Picture Many different methods for fast query matching Algorithm in this talk: Randomized matrix vector multiplication Other randomized matrix algorithms: Matrix multiplication Subset selection Least squares problems (regression) Low rank approximation (PCA) Applications for randomized algorithms: Social network analysis, population genetics, circuit testing,...

54 NSF Web National Science Foundation, 29 March 2012 Press Release NSF Leads Federal Efforts In Big Data At White House event, NSF Director announces new Big Data solicitation, $10 million Expeditions in Computing award, and awards in cyberinfrastructure, geosciences, training UC Irvine's HIPerW advances earth sc and visualization Credit and Larger Hurricane Ike visualization created by Texas Advanced Computing Center (TACC) supercomputer Ranger. Credit and Larger Version March 29, 2012