D- t- Science, Big D- t- and Sta/s/cs: Can we all live together? Chalmers Ini-a-ve Seminar on Big Data Tuesday 25th March 2014

Transcription

1 D- t- Science, Big D- t- and Sta/s/cs: Can we all live together? Chalmers Ini-a-ve Seminar on Big Data Tuesday 25th March

2 Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it... Dan Ariely, 2013 In God we trust; all others must bring data. W.E. Deming 2

3 Overview of this talk 1. What is (are) Big Data? 2. Sta-s-cs and Big Data 3. Removing Unwanted Varia-on 3

4 1. What is (are) Big Data? 4

5 Interest in Big Data is growing Google Trends: Big Data 5

6 for a while, at least Google Trends: Bioinforma-cs Google Trends : Sta-s-cs 6

7 for a while, at least Google Trends: Bioinforma-cs Google Trends : Data Mining 7

8 The Gartner Hype Cycle 8

9 Some specifics 9

10 Big Data isn t new to sta/s/cians

11 Big Data isn t new to sta/s/cians Part I Treatment of "Huge'' Data Sets

12 But perhaps the name is new Part I Treatment of "Huge'' Data Sets KDD

13 For some, size maners big data refers to things one can do at a large scale, that cannot be done at a smaller one, to extract new insights, or create new forms and value, in ways that change markets, organisaaons, the relaaonship between governments, ciazens and more. p.6 Big Data: A Revolu-on that Will Transform how We Live, Work, and Think by Viktor Mayer- Schönberger, and Kenneth Cukier

14 Managing the environmental impact on rivers by streaming informa-on!"#$"%&$'()($ *+,,-..$*)/0%-. Applies emerging technologies to deliver instantaneous people searches Analyzing huge volumes of customer comments in real -me delivers a compe--ve edge Analyzes real- -me data streams to iden-fy traffic pa_erns Puang real- -me data to work and providing a plaborm for technology development Helping companies deliver the web experience their customers want. Streaming data technology supports covert intelligence and surveillance sensor systems Leveraging key data to provide proac-ve pa-ent care Streaming real- -me data supports large scale study of space weather Turning climate into capital with big data 14

15 For others, it s not just size ICSA Bulle-n, Jan

16 For some, it s about novel forms of data Data from sensors, posts to social networking sites, digital images, videos posted online, transac-on records of online purchases, data from mobile phone GPS signals, bus records, traffic, BIG players: Google, Microsog, Amazon, Verizon, Walmart,,, NSA, GCHQ, 16

17 Big Data, or linle science and technology? Soil sensors can help dairy farmers op-mise fer-liser use. UTAS 17

18 For yet others, old- fashioned data is o.k. Titles from a forthcoming Big Data conference Big insights into pa-ent flow Iden-fying and predic-ng community care and health needs Enhancing diagnos-cs for invasive Aspergillosis using machine learning Big data from small sites: Crea-ng a primary care data warehouse Spa-o- temporal visualisa-on of disease incidence and respec-ve interven-on strategies 18

19 Whatever it is: Big Data is here! 19

20 What is Big Data for the US NIH? The term 'Big Data' is meant to capture the opportuni-es and challenges facing all biomedical researchers in accessing, managing, analyzing, and integra-ng datasets of diverse data types [e.g., imaging, phenotypic, molecular (including various ' omics'), exposure, health, behavioral, and the many other types of biological and biomedical and behavioral data] that are increasingly larger, more diverse, and more complex, and that exceed the abili-es of currently used approaches to manage and analyze This effec-vely. is exactly what I have been trying to do for the last 15 years 20

21 From NIH s BD2K: Major challenges in using biomedical Big Data include: Loca-ng data and sogware tools. Geang access to the data and sogware tools. Standardizing data and metadata. Extending policies and prac-ces for data and sogware sharing. Organizing, managing, and processing biomedical Big Data. Developing new methods for analyzing & integra-ng biomedical data. Training researchers who can use biomedical Big Data effec-vely. 21

22 A new ini/a/ve in Australia Centre of Excellence for FronAers in MathemaAcs and StaAsAcs: Big Data, Big Models, New Insights Centre Director: Professor Peter Hall From the 350 character descrip-on This Centre will create innova-ve mathema-cal and sta-s-cal models for analysing big data sets. The insight gained from analysing these models 22

23 Conclusion: Big Data is about data! 23

24 2. Big Data and Sta/s/cians 24

25 Buja A, Keller- McNulty S. Introduc-on to the special sec-on on massive datasets. J Comp Graph StaAst 1999 The area of massive datasets, though currently of great interest to Computa-onal sta-s-cians and to many data analysts, has not yet become part of mainstream sta-s-cal science. However, we see it as an area of ferment that is likely to generate problems of interest to the en-re field and to contribute to the changes in the teaching of sta-s-cs. 25

26 It didn t happen Huber (1994) 33 cites Huber (1998) 21 cites Massive Data sets Proceedings of a Conference (1998) 0 cites Buja & Keller- McNulty (1999) 3 cites Lambert (2003) What use is sta-s-cs for massive data? 8 cites Ke_enring (2009) Massive datasets 12 cites Maybe this is not an issue, as we ve already seen that Big Data is not just about size. But 26

27 What s worse: We seem to have gone backwards, at least in the eyes of some California Magazine, Winter 2013 (of alumni of UC Berkeley) 27

28 28

29 29

30 30

31 The absence of sta/s/cians in Big Data ac/vi/es is striking (to a sta/s/cian) The US NSF convened a working group on Big Data in 2012 of 100 experts: guess how many sta-s-cians? The US NIH BD2K Execu-ve Commi_ee of 17 has how many sta-s-cians? Conferences and commi_ees are popping up around the world with no sta-s-cians involved (thanks for the invita-on to come here!) $Ms (and other currencies) are being thrown at Big Data experts (no, this is not envy, it s puzzlement) And so on 31

32 Summing up If I am typical of sta-s-cians (and I d like to think I was a bit ahead of most in this respect), then I didn t see this coming (what is this?) I s-ll don t fully understand it I am s-ll unsure how best to react I have been told many sta-s-cians saw it coming, (Leo Breiman certainly did), so the above may well be my personal shortcoming. However, if many is true, why are we s-ll excluded so fully? 32

33 How my sta/s/cal career seems right now Genomics and bioinforma-cs. Not Big, but much bigger than before. My early years Inevitable decline TIME Re-re? Big Data 33

34 Why? Some first reac/ons Many problems in Big Data are poorly defined, and perhaps we tend to shy away from these, rather than help sharpen the problem Perhaps we prefer to be precisely wrong than approximately right Perhaps we are misunderstood, or, worse, understood We lack the IT (tempted to add: and the marke-ng ) skills Perhaps subject ma_er considera-ons put an unacceptably high cogni-ve burden on us Many Big Data projects involve large teams Your views please, now or later. 34

35 My personal sta/s/cal paradigm I use sta-s-cal models, which are sets of equa-ons involving random variables, with associated distribu-onal assump-ons, devised in the context of a ques-on and a body of data concerning some phenomenon, with which tenta-ve answers can be derived, along with measures of uncertainty concerning these answers. quesaons + data (real world) model (equaaons, distribuaons) answers + measures of uncertainty How relevant is this to Big Data? It s part of the story. 35

36 Is this a general view? Mike Flowers, a lawyer, and NYC s first Director of Analy-cs "I had no interest in very experienced sta-s-cians," he says. "I was a li_le concerned that they would be reluctant to take this novel approach to problem solving." Earlier, when he had interviewed tradi-onal stats guys for the financial fraud project, they had tended to raise arcane concerns about mathema-cal methods. "I wasn't even thinking about what model I was going to use. I wanted ac-onable insight, and that was all I cared about," he says. Chapter 10, Big Data: A Revolu-on That Will Transform How We Live, Work and Think. 36

37 There is a lot of work to be done for Big Data Algorithm complexity has to be linear or sub- linear to run on big data Data take up memory Calcula-ons on data also take up memory because mul-ple copies of data are typically made during calcula-ons CPU and memory efficient algorithms are needed, rela-ve to available CPUs and RAM. 37

38 All this leads to a need for new algorithms. There is lots of good work going on here. Embarrassingly parallel problems (e.g. the Lasso) Parallel Computa-on with communica-on (e.g. Massive ParallelizaAon of Serial Inference Algorithms with GPGPU, Suchard et al 2012) Reducing data size via randomized algorithms and/or sampling (many authors: Drineas, Mahoney, Candes, etc) Use of MapReduce or Hadoop (see Wikipedia) 38

39 Should sta/s/cians have a place at the Big Data table? I think the answer is unequivocally yes! (but then I m a sta-s-cian, and so I would say that) I do think I could argue cogently for that, and will do so a li_le (and look forward to tomorrow s panel discn) However, we sta-s-cians need to change! See next slide for the 1999 perspec-ve. 39

40 Buja A, Keller- McNulty S. Introduc-on to the special sec-on on massive datasets. J Comp Graph StaAst 1999 In addiaon, we see two paracular messages emerging from the following aracles: (1) the interdisciplinary nature of problem soluaons that integrate staasacs, databases and subject maver issues; and (2) the increased computaaonal literacy required of future staasacians. Both points should be noted by young researchers who may do well to educate themselves in new computaaonal tools, as well as in sciences and insatuaons that generate these massive datasets. 40

41 McKinsey Report (2011) Big data: The next fron1er for innova1on, compe11on, and produc1vity. 6. There will be a shortage of talent necessary for organiza-ons to take advantage of big data. By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analy-cal skills as well as 1.5 million managers and analysts with the know- how to use the analysis of big data to make effec-ve decisions. What will they learn? Who will teach them? 7. Several issues will have to be addressed to capture the full poten-al of big data. Policies related to privacy, security, intellectual property, and even liability will need to be addressed in a big data world. 41

42 Who are the data scien/sts? Sta/s/cians are, and Many others as well, from machine learning, computer science, signal processing, physics, applied mathema-cs, domain sciences and social sciences, ci-zen science, 42

43 What skills are needed for Big Data? Interpersonal and leadership : access to important problems (engineers are more used to team work) Computa-onal (gateway to accessing data, and analysis tools) Theore-cal (analyses under idealized models give insight) Making sense of data: cri-cal thinking and common sense (sta-s-cians advantage?) And recall, much of Big Data isn t about big data at all 43

44 Three dimensions of the big data enterprise (a_er Bin Yu) Human Data Computa/on Context 44

45 3. Removing Unwanted Varia/on Discuss RUV as -me permits (-me did not permit) 45

46 Two final thoughts This is a Golden age of sta-s-cs but not necessarily for sta-s-cians Gerry Hahn Those who ignore sta-s-cs are condemned to re- invent it. Brad Efron 46

47 References/Acknowledgements J.M. Jordan & D.K.J. Lin (2014) Sta-s-cs for Big Data: Are Sta-s-cians ready for Big Data? ICSA BulleAn, January B. Yu (2007) Embracing Sta-s-cal Challenges in the Informa-on Technology Age. Technometrics v.49. B. Yu (2012) President s Welcome. Bull IMS B. Yu (2013) The relaave size of big data: experiences from an interdisciplinary staasacian. Tutorial talk at JSM Many thanks to Bin Yu, UC Berkeley Sta-s-cs & EECS 47