BIG DATA. Rasmus Pagh. OpenITU, IT University of Copenhagen January 23, 2015

Transcription

1 BIG DATA Rasmus Pagh OpenITU, IT University of Copenhagen January 23, 2015 Licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license. This images of book and magazine covers and the copyright for them are most likely owned either by the artist who created the covers or the publishers. It is believed that the use of low-resolution images qualifies as fair use. Image by DARPA

2 What is BIG DATA?

3 What is BIG DATA?

4 What is the fuss about BIG DATA?

5 Big Data-based businesses Data Business Google (search engine) Amazon Facebook The web, clickstreams, Products, reviews, sales User profiles, interaction, Advertising Retail Advertising

6 Big Data-based businesses Data Business Question Google (search engine) The web, clickstreams, Advertising What web pages should I visit? Amazon Products, reviews, sales Retail What products should I buy? Facebook User profiles, interaction, Advertising What things of interest are going on in my network?

7 Big Data-based businesses Data Business Question Google (search engine) Amazon Facebook The web, clickstreams, Products, reviews, sales User profiles, interaction, Advertising In common: Using data to Retail enable better decisions. Advertising What web pages should I visit? What products should I buy? What things of interest are going on in my network?

8 Big Data on a smaller scale Photo by Joe Mabel.

9 Rough definition Big Data systems: Rely a lot of data (that will often fit on a laptop). - Combines all relevant data from many sources, possibly real-time data. Attempt to answer questions where it would be hard (or labour-intensive) for humans to give the answer. - Settles for what? or how much?, rather than why? Image by DARPA

10 Rough definition Big Data systems: Rely a lot of data (that will often fit on a laptop). - Combines all relevant data from many sources, possibly real-time data. Not: Playback of videos on Youtube and Netflix. Attempt to answer questions where it would be hard (or labour-intensive) for humans to give the answer. - Settles for what? or how much?, rather than why? Image by DARPA

11 Rough definition Big Data systems: Rely a lot of data (that will often fit on a laptop). - Combines all relevant data from many sources, possibly real-time data. Not: Playback of videos on Youtube and Netflix. Attempt to answer questions where it would be hard (or labour-intensive) for humans to give the answer. - Settles for what? or how much?, rather than why? Not: Computing the strength of a bridge. Image by DARPA

12 Rough definition Big Data systems: Rely a lot of data (that will often fit on a laptop). - Combines all relevant data from many sources, possibly real-time data. Attempt to answer questions where it would be hard (or labour-intensive) for humans to give the answer. - Settles for what? or how much?, rather than why? Image by DARPA

13 Rough definition Big Data systems: Rely a lot of data (that will often fit on a laptop). - Often combine all relevant data from many sources, possibly real-time data. Attempt to answer questions where it would be hard (or labour-intensive) for humans to give the answer. - Settles for what? or how much?, rather than why? Image by DARPA

14 What Rough is definition the fuss? Big Data systems: Rely a lot of data (that will often fit on a laptop). - Often combine all relevant data from many sources, possibly real-time data. Attempt to answer questions where it would be hard (or labour-intensive) for humans to give the answer. - Settles for what? or how much?, rather than why? Image by DARPA

15 Software as rules Image by Ilkant

16 Software as rules? What is the rule that will allow us to recognize this animal?

17 Software as rules? What is the rule that will allow us to recognize this animal?

18 The three Vs of big data Volume Velocity Variety

19 The three Vs of big data My research area Volume Velocity Variety

20 The three Vs of big data My research area Volume Velocity SCALABLE SIMILARITY SEARCH Variety

21 The three Ps of big data Technology Data sources Use of analysis

22 The three Ps of big data Technology Possibilities Data sources Pitfalls Privacy Use of analysis concerns

23 The three Ps of big data Technology Possibilities Data sources Pitfalls Privacy Use of analysis concerns

24 Pitfall 1: Unfairness Issue: If data reflects historical biases, these biases may be preserved.

25 Pitfall 1: Unfairness Issue: If data reflects historical biases, these biases may be preserved. big data technologies can cause societal harms [ ] such as discrimination against individuals and groups. This discrimination can be the inadvertent outcome of the way big data technologies are [ ] used.

26 Pitfall 2: Opaqueness Issue: Many machine learning methods are essentially black boxes. A human cannot comprehend the meaning of the models they construct.

27 Pitfall 2: Opaqueness Issue: Many machine learning methods are essentially black boxes. A human cannot comprehend the meaning of the models they construct. some transparency is appropriate when a consumer s experience is being altered based on their personal information, particularly in situations where companies offer differential pricing

28 Pitfall 3: Uncertain accountability Issue: If it turns out that your Big Data-based decisions discriminate, can you claim that you were in good faith?

29 Pitfall 3: Uncertain accountability Issue: If it turns out that your Big Data-based decisions discriminate, can you claim that you were in good faith? big data will require monitoring and transparency [ ] Algorithmists would take a wow of impartiality and confidentiality, much as accountants [ ] do now.

30 Pitfall 4: Spurious discovery From

31 Pitfall 4: Spurious discovery From

32 Pitfall 4: Spurious discovery From

33 The last P: Privacy concerns Issue: What does it even mean that certain data about me cannot be used?

34 The last P: Privacy concerns Issue: What does it even mean that certain data about me cannot be used?

35 The last P: Privacy concerns Issue: What does it even mean that certain data about me cannot be used? Many technologists are of the view that de-identification of data as a means of protecting individual privacy is, at best, a limited proposition.

36 Can we do it?

37 Can we do it?

38 Take-home message Big Data presents many Possibilities, but attention should be paid to Pitfalls, e.g.: Unfairness Opaqueness Uncertain accountability Spurious discovery Privacy regulations need to be phrased in a balanced and technically meaningful way. Image by DARPA

39 Thank you! Slides available at

40 Some pointers Cathy O Neil: mathbabe.org/2014/08/25/gilian-tett-gets-itvery-wrong-on-racial-profiling/ Moritz Hardt: medium.com/@mrtz/how-big-data-isunfair-9aa544d739de Workshops on Differential Privacy; Fairness, Accountability, and Transparency in Machine Learning: simons.berkeley.edu/ workshops/bigdata2013-4; Spurious discovery: scientists-think-science-self-correcting-alarmingdegree-it-not-trouble, xkcd.com/882/