Using CrowdSourcing for Data Analytics

Transcription

1 Using CrowdSouring for Dt Anlytis Hetor Gri-Molin (work with Steven Whng, Peter Lofgren, Adity Prmeswrn nd others) Stnford University 1 Big Dt Anlytis CrowdSouring 1

2 CrowdSouring 3 Rel World Exmples Ctegorizing Imges Serh Relevne Dt Gthering Imge Mthing Trnsltion 4 2

3 Mny Crowdsouring Mrketples! Mny Reserh Projets! 6 3

4 dt Exmple tsks: get missing dt verify results nlyze dt nlytis humns results 7 dt Exmple tsks: get missing dt verify results nlyze dt nlytis humns results Key Point: use humns judiiously 8 4

5 Tody will illustrte with Entity Resolution (my over nother topi riefly) 9 Trditionl Entity Resolution nlysis System 1 lensing... System n wht mthes wht?? 10 5

6 Why is ER Chllenging? Huge dt sets No unique identifiers Missing dt Lots of unertinty Mny wys to skin the t 11 Simple ER Exmple 12 6

7 Simple ER Exmple sim= sim=0.8 d 13 Simple ER Exmple sim= sim=0.8 d 14 7

8 ER: Ext vs Approximte mers ER resolved mers produts CDs ooks ER ER resolved CDs resolved ooks Simple ER Algorithm Compute pirwise similrities Apply threshold Perform trnsitive losure 16 8

9 Simple ER Algorithm Compute pirwise similrities Apply threshold Perform trnsitive losure Simple ER Algorithm Compute pirwise similrities Apply threshold Perform trnsitive losure threshold =

10 Simple ER Algorithm Compute pirwise similrities Apply threshold Perform trnsitive losure Crowd ER 20 10

11 Sme s this? 21 Crowd ER First Cut: For every pir of reords, sk workers if they mth (i.e., get similrity) 22 11

12 Crowd ER First Cut: For every pir of reords, sk workers if they mth (i.e., get similrity) Too expensive! Crowd ER Seond Cut: Compute similrities; workers verify "ritil" pirs 0.7 ritil??

13 Crowd ER Seond Cut: Compute similrities; workers verify "ritil" pirs 0.7 ritil?? Crowd ER Seond Cut: Compute similrities; workers verify "ritil" pirs 0.7 ritil??

14 reords pirwise nlysis generte questions new evidene rowd glol nlysis Key Point: use humns judiiously lusters 27 Key Issue: Semntis of Crowd Answer 28 14

15 Key Issue: Semntis of Crowd Answer C D E B A? 29 Also issue: Similrities s Proilities sim(,) pro(,) 30 15

16 Strtegy 0.2 urrent stte use ny given ER lgorithm 31 Strtegy 0.2 Q(,) urrent stte Q(,) onsider ALL possile questions (three in this exmple) Q(,) 32 16

17 Strtegy urrent stte 0.2 Q(,) Q(,) Y N Y new stte new stte new stte onsider possile outomes N new stte Q(,) Y new stte N new stte 33 Strtegy urrent stte Q(,) Y new stte exmple 34 17

18 Strtegy 0.2 Q(,) Y N new stte new stte sore? sore? urrent stte Q(,) Y new stte sore? N new stte sore? Q(,) Y new stte sore? N new stte sore? 35 Two Remining Issues How do we sore n? gold stndrd F sore Effiieny? 36 18

19 Gold Stndrd? sim to pro 37 Gold Stndrd? possile worlds sim to pro

20 Gold Stndrd? possile worlds possile lustering (vi ER lgorithm) 0.68 sim to pro Strtegy urrent stte 0.2 Q(,) Q(,) Y N Y N new stte new stte new stte new stte sore vs GS? sore vs GS? sore vs GS? sore vs GS? Q(,) Y new stte sore vs GS? N new stte sore vs GS? 40 20

21 Evluting Effiiently See: Steven E. Whng, Peter Lofgren, nd H. Gri-Molin. Question Seletion for Crowd Entity Resolution. To pper in Pro. 39th Int'l Conf. on Very Lrge Dt Bses (PVLDB), Trento, Itly, Smple Result 42 21

22 Summry dt Exmple tsks: get missing dt verify results nlyze dt nlytis humns results Key Point: use humns judiiously 43 Now for something ompletely different! nlytis DBMS ig dt 44 22

23 Now for something ompletely different! nlytis DBMS ig dt humns 45 DeCo: Delrtive CrowdSouring wht is est prie for Nikon DSLR mers? End user DBMS dt humns 46 23

24 DeCo: Delrtive CrowdSouring wht is est prie for Nikon DSLR mers? End user DBMS dt humns model type rnd D7100 DSLR Nikon 7D DSLR Cnon P5000 omp Nikon 47 DeCo: Delrtive CrowdSouring wht is est prie for Nikon DSLR mers? End user DBMS dt model type rnd D7100 DSLR Nikon 7D DSLR Cnon P5000 omp Nikon humns wht is est prie for Nikon D7100 mer? Crowd 48 24

25 Exmple with it more detil: User view resturnt rting uisine Chez Pnisse 4.9 Frenh Chez Pnisse 4.9 Cliforni Bytes 3.8 Cliforni Exmple with it more detil: User view resturnt rting uisine Chez Pnisse 4.9 Frenh Chez Pnisse 4.9 Cliforni Bytes 3.8 Cliforni resturnt Chez Pnisse Bytes Anhor o resturnt rting Chez Pnisse 4.8 Chez Pnisse 5.0 Chez Pnisse 4.9 Bytes 3.6 Bytes 4.0 Dependent resturnt uisine Chez Pnisse Frenh Chez Pnisse Cliforni Bytes Cliforni Bytes Cliforni Dependent 50 25

26 Exmple with it more detil: User view resturnt rting uisine Chez Pnisse 4.9 Frenh Chez Pnisse 4.9 Cliforni Bytes 3.8 Cliforni o resturnt resturnt rting resturnt uisine Chez Pnisse Chez Pnisse 4.8 Chez Pnisse Frenh Bytes Chez Pnisse 5.0 Chez Pnisse Cliforni Chez Pnisse 4.9 Bytes Cliforni Anhor Bytes 3.6 Bytes Cliforni feth rule feth rule Bytes 4.0 Bytes Dependent Chez Pnisse Dependent feth rule 51 Exmple with it more detil: User view resturnt rting uisine Chez Pnisse 4.9 Frenh Chez Pnisse 4.9 Cliforni Bytes 3.8 Cliforni resturnt Chez Pnisse Bytes Anhor feth rule feth rule o resturnt rting resturnt uisine Chez Pnisse 4.8 Chez Pnisse Frenh Chez Pnisse 5.0 Chez Pnisse Cliforni Chez Pnisse 4.9 Bytes Cliforni Bytes 3.6 Bytes Cliforni Bytes 4.0 Chez Pnisse Bytes Frenh Dependent feth rule Dependent feth rule 52 26

27 Exmple with it more detil: User view resturnt rting uisine Chez Pnisse 4.9 Frenh Chez Pnisse 4.9 Cliforni Bytes 3.8 Cliforni o resolution rule resolution rule resturnt resturnt rting resturnt uisine Chez Pnisse Chez Pnisse 4.8 Chez Pnisse Frenh Bytes Chez Pnisse 5.0 Chez Pnisse Cliforni Chez Pnisse 4.9 Bytes Cliforni Anhor Bytes 3.6 Bytes Cliforni Bytes 4.0 Chez Pnisse Bytes Dependent Dependent 53 Exmple with it more detil: User view resturnt rting uisine Chez Pnisse 4.9 Frenh Chez Pnisse 4.9 Cliforni Bytes 3.8 Cliforni 1. Feth 2. Resolve 3. Join o resturnt resturnt rting resturnt uisine Chez Pnisse Chez Pnisse 4.8 Chez Pnisse Frenh Bytes Chez Pnisse 5.0 Chez Pnisse Cliforni Chez Pnisse 4.9 Bytes Cliforni Anhor Bytes 3.6 Bytes Cliforni Bytes 4.0 Dependent Dependent 54 27

28 Mny Query Proessing Chllenges SELECT n,l, FROM ountry WHERE l = Spnish ATLEAST 8 AtLest [8] Join Filter [l= Spnish ] Resolve[m3] Resolve[d.e] Join Resolve[m3] Sn D2(n,) Feth [n l,] Sn A(n) Feth [l n,] [ n] [l n] Sn D1(n,l) Feth [n l,] [n l] 55 Deo Prototype V

29 Conlusion Crowdsouring is importnt for mnging dt! Still mny hllenges hed!