Big Data and Big Analy-cs Trends: The Promise and the Hype. Gregory Piatetsky KDnuggets

Transcription

1 Big Data and Big Analy-cs Trends: The Promise and the Hype Gregory Piatetsky KDnuggets KDnuggets

2 My Data PhD in applying Machine Learning to databases Researcher at GTE Labs started the first project on Knowledge Discovery in Databases in 1989 Organized first 3 Knowledge Discovery and Data Mining (KDD) workshops ( ), cofounded Knowledge Discovery and Data Mining (KDD) conferences (1995) Chief ScienSst at 2 analyscs startups Co- founder SIGKDD (1998), Chair, AnalyScs/Data Mining Consultant, KDnuggets

3 KDnuggets Stands for Knowledge Discovery Nuggets started KDnuggets News newsleyer (~ 12,000 subscribers now) early website in 1994, in 1997, blog- style in best year: 50,000 unique visitors/month twiyer.com/kdnuggets ~6,000 followers facebook.com/kdnuggets page group: KDnuggets AnalyScs & Data Mining KDnuggets

4 What do we call what we do? StaSsScs Data mining Knowledge Discovery in Data (KDD) PredicSve AnalyScs Data Science Big AnalyScs? Core Idea: Finding Useful PaBerns in Data KDnuggets

5 Pre- history ( ): StaSsScs sta-s-cs is the biggest term in 20 th century, Analy-cs is used increasingly thru 20 th century data mining appears in late 1990s From Google Ngram viewer English language books Search case sensisve used most popular version. Other languages, especially Chinese, need to be considered for full picture KDnuggets

6 20 th Century AnalyScs vs Data Mining data mining AnalyScs Data Mining analy-cs?? Google N- grams search is case sensisve; Note: data mining > Data Mining usage While analyscs < AnalyScs KDnuggets

7 Data Mining Surges in 1996 data mining AnalyScs analy-cs Data Mining KDD- 95, 1 st Conference on Knowledge Discovery and Data Mining, Montreal Advances in Knowledge Discovery and Data Mining, AAAI/MIT Press, 1996, Eds: U. Fayyad, G. Piatetsky- Shapiro, P. Smyth, and R. Uthurusamy KDnuggets

8 Recent History: data mining AnalyScs analy-cs Knowledge Discovery analy-cs has been used since 1980, but started to rise in 2005 data mining surges around 1996 (soon amer first KDD conference) but slowly declines amer 2003 (TIA controversy, associated with Govt invasion of privacy). Knowledge Discovery appears in 1989, jumps in 1996, and plateaus amer 2000 (Google N- grams, smoothing =1) KDnuggets

9 Earliest use of data mining 1962? Amer eliminasng many following data. Mining cost is examples which refer to Mining of minerals, and books from 1958 that have a CD ayached (errors in book year) The earliest data mining reference I found is Source: Google Books (c) KDnuggets

10 Google Trends: Amer 2006, AnalyScs > Data Mining Global all regions (c) KDnuggets

11 >50% of AnalyScs searches are for Google AnalyScs Google AnalyScs introduced, Dec 2005 (c) KDnuggets

12 Google Trends observasons (as of Sep 2012) Decline in analyscs in 2012? Compe9ng on Analy9cs book, Apr 2007 December vacason drops (c) KDnuggets 2012

13 Global View: searches for data mining, analyscs - google Google Insights (c) KDnuggets

14 AnalyScs: Business > Data> PredicSve > Text Google Insights, Jan Sep 2012, Global (c) KDnuggets

15 Data Mining >> Business/Data/PredicSve AnalyScs Google Insights, Jan Sep 2012, Global (c) KDnuggets

16 Data Mining > Big Data >> PredicSve AnalyScs > Data Science Big Data Surge Google Insights, Jan Sep 2012, Global (c) KDnuggets

17 What will replace Big Data buzzword? Poll: will- replace- big- data.html KDnuggets

18 History StaSsScs 1960s Data Mining = bad ac9vity, data dredging Data Mining is good, surges in Data Mining plateaus (bad, invasion of privacy?) Google AnalyScs Business/Data/PredicSve AnalyScs Big Data ?? KDnuggets

19 AnalyScs, Big Data, Data Mining Today KDnuggets Polls Findings (c) KDnuggets

20 KDnuggets

21 Where did you apply Analy-cs/Data Mining? Avg. Number of Industries 2.8 Most Popular: - CRM - Banking - Health Care - EducaSon - Fraud DetecSon Highest growth in: Travel / Hospitality Social Networks EducaSon Biotech/Genomics Credit Scoring applied- anayscs- data- mining.html KDnuggets

22 Data Types Analyzed/Mined Most popular: - Table data - Time series - Text - - itemsets/transacsons Most growing: - XML data - text (free- form) - social network data - JSON types- analyzed- data- mined.html KDnuggets

23 Largest Dataset Analyzed? Big Data Miners elite group 2012 median dataset size ~20-40 GB, vs GB in dataset- analyzed- data- mined.html KDnuggets

24 Largest Dataset Analyzed by Region Big Data Miners: TeraBytes and Petabytes 18-24% KDnuggets

25 Which methods/algorithms did you use for data analysis Most popular: - Decision Trees - Regression - Clustering - StaSsScs - VisualizaSon analyscs- data- mining.html (c) KDnuggets

26 Algorithms with highest Industry Affinity Industry Affinity = How much this algorithm is more used among industry data miners = analyscs- data- mining.html (c) KDnuggets

27 Academic algorithms lowest Industry affinity analyscs- data- mining.html (c) KDnuggets

28 Cloud Analy-cs is not common (yet) Big data tools use grew 5- fold, from about 3% in 2011 to about 15% of respondents in 2012 analyscs somware poll (c) KDnuggets

29 JOBS AND SKILLS (c) KDnuggets

30 Shortage of Skills McKinsey: shortage by 2018 in the US of ,000 people with deep analyscal skills 1.5 M managers/analysts with the know- how to use the analysis of big data to make effecsve decisions. Source: big_data/ (c) KDnuggets

31 Indeed.com fastest growing jobs Top 10 skills: HTML5 MongoDB ios Android Mobile app Puppet Hadoop jquery PaaS Social Media Hadoop MongoDB KDnuggets

32 Big Data grows faster than MongoDB Hadoop Big Data MongoDB KDnuggets

33 Data Mining >> Hadoop (c) KDnuggets

34 Demand for Data Scien-sts surging Data ScienSst Fastest growing term on 1% of jobs in % of jobs in % of jobs 2012, Jan- Sep Data ScienSst sexiest job of the 21 st Century (???) say Thomas H. Davenport and D.J. PaSl, (HBR, Oct 2010) KDnuggets

35 Rebranding from Data Mining to Big Data Data Mining Big Data Data Scien-st Data mining jobs are much more common, but Big Data jobs are surging much faster than Data ScienSst (c) KDnuggets

36 LinkedIn Skills: Data Mining ~ 105,000 members with Data Mining skill (Sep 2012) Dwarfs related skills - 1% growth (according to LinkedIn) But in Oct 2011 there were 75K members with Data Mining Skill, which gives 40% annual growth (c) KDnuggets

37 Cloud (Big Data) AnalyScs Skills (c) KDnuggets

38 LinkedIn AnalyScs/Data Mining Skills Ground analyscs skills most common Cloud analyscs skills grow fastest Text AnalyScs skills less common SenSment Analysis fastest growing (c) KDnuggets

39 LinkedIn Analy-c Tools Skills SAS- cersfied KDnuggets

40 Big Data 2 nd Industrial RevoluSon Do old acsvises beyer Create new acsvises/businesses (c) KDnuggets

41 ApplicaSon areas Doing old things beyer Churn predicson Direct markesng/customer modeling RecommendaSons Fraud detecson Security/Intelligence CompeSSon will level companies (c) KDnuggets

42 Limit to PredicSng Human Behavior? There is randomness in human behavior and once we find 1- level effects, more data or beyer algorithms will give diminishing returns in most cases Example: Neylix Prize: the most advanced algorithms were only a few percentages beyer than basic algorithms (c) KDnuggets

43 Big Data Enables New Things! Google first big success of big data Social networks (facebook, TwiYer, LinkedIn, ) success depends on network size, i.e. big data LocaSon analyscs Health- care Personalized medicine SemanScs and AI? Imagine IBM Watson, Siri in 2020? (c) KDnuggets

44 Big Data Bubble? Big Data Gartner Hype Cycle 2012 KDnuggets 44

45 Gartner Hype Cycle for Big Data, 2012 Social Network Analysis, 5-10 Data ScienSst, 2-5 yrs Social AnalyScs, 2-5 PredicSve AnalyScs, <2 MapReduce & AlternaSve - Disillusionment KDnuggets

46 QuesSons? KDnuggets: Analy;cs, Big Data, Data Mining News, Jobs, Sodware, Data, ConsulSng, Courses, MeeSngs, PublicaSons, Webcasts, Subscribe to KDnuggets News at to editor1@kdnuggets.com KDnuggets

47 Data Mining in 1902?? KDnuggets

48 Research and Industry Disconnect? Uplim modeling needs more research AssociaSon rules need less papers Data Mining with Privacy research industry use? Conferences should bring researchers and industry people together (c) KDnuggets

49 Direct MarkeSng Lim: Random and Model- sorted Lists CPH: CumulaSve Pct Hits Random Model Pct list 5% of random list have 5% of hits 5% of model- score ranked list have 21% of hits. Lid(5%) = 21%/5% = 4.2

50 Most lim curves are surprising similar Study of lift curves in banking, telecom Best lift curves are similar Special point T=Target percentage Lift Actual lift(t) Est. lift(t) Lift(T) ~ sqrt (1/T) 4 2 G. Piatetsky- Shapiro, B. Masand, Es-ma-ng Campaign Benefits and Modeling Lid, in Proceedings of KDD- 99 Conference, ACM Press, *T% (c) KDnuggets