The Bloor Group. The Pillars of Data Science

Transcription

1 The Pillars of Data Science

2 The Three Pillars DOMAIN KNOWLEDGE The DS/DA needs to know the business STATISTICAL SKILLS Knowing how to use software to analyze data TECHNOLOGY KNOWLEDGE Knowing how to leverage technology

3 This is Essentially a New Role Ø NEW STATISTICAL SOFTWARE TOOLS (R, Python, Mahout, KNIME, etc.) Ø New Techniques to some degree Ø NEW TECHNOLOGY (Hadoop++,data access capabilities, analytical databases, etc.) Ø New Data (external data, mobile data, sensor data, log data)

4 The Big Data Trend Ø Corporate data volumes grow at about 55% per annum Ø This is exponential Ø Data has been growing at this rate for at least 20 years, maybe even 40 years Ø As such there is nothing new about big data, except The Bloor Group

5 The Forces of Disruption The Bloor Group

6 Moore s Law Cubed Ø The biggest databases are NEW databases Ø They seem to grow by volume at the CUBE OF MOORE S LAW Ø Moore s Law = 10X EVERY 6 YEARS Ø VLDB: 1000x every 6 years 1991/2 megabytes 1997/8 gigabytes 2003/4 terabytes 2009/10 petabytes 2015/16 exabytes The Bloor Group

7 The SQL Inconvenience SQL has: DDL (for data definition) DML (for Select, Project and Join) But it has no MML (Math) or TML (Time) Ø Usually result sets are brought to the client for further analytical manipulation, but this creates problems Ø Alternatively, doing all analytical manipulation in the database creates problems The Bloor Group

8 The Avalanche of New Technology Hadoop extremely inexpensive scale-out data store with parallelism (MR) Scale-out database engines The Cloud really fast deployment of hardware resources Parallelism is becoming a way of life the acceleration has increased (I think)

9 This is the Field of BI HINDSIGHT Regular reporting Operational BI OVERSIGHT Dashboards OLAP BPM, etc. INSIGHT Data mining Statistical analysis FORESIGHT Predictive analytics

10 There s a Schism in BI BI is FRAGMENTING between traditional BI AND ANALYTICS HINDSIGHT & OVERSIGHT Insight & Foresight HINDSIGHT & OVERSIGHT Relatively easy to accommodate technically INSIGHT & FORESIGHT Modeling, interactive & conversational, highly variable

11 The DS/DA Provides the Insight The Bloor Group

12 The Cast of Characters Data Analyst IT Developers IT Operations BI User

13 The Whole Enchilada The DS/DA never had complete control of this For data access, IT s help was needed For deployment, IT s help was needed Complete self-service is desirable

14 The Data Analysis Process in Detail The Bloor Group

15 Please Note Ø This is a non-standard and unpredictable workload Ø It is iterative but the workload varies considerably Ø Speed is an advantage The Bloor Group

16 Here s Your Problem The IDEAL would be to provide the DS/ DA with end-to-end self-service But that means that the DS/DA needs to understand workloads and software architecture It also means that the DS/DA needs to be capable of selecting software tools

17 The Secret Ingredient DATA FLOW

18 Here s a Question Should you take the or the The Bloor Group

19 The Data Flow Issue The Bloor Group

20 Here are Two Scenarios one Data is in Hadoop, so move the processing to the data? two The first call for data is a big query, all other calls are small, so move the data to an analytic DBMS? HOW DO YOU EVEN PREDICT SCENARIOS?

21 The Four Key Latencies Time taken to ANALYZE (and produce an actionable result) Time taken to IMPLEMENT the result Time taken to PROVIDE ACTIONABLE INFORMATION to the user Time taken to REFRESH the model

22 The Goal TEAR DOWN THESE LATENCIES This cannot be done reliably without really well thought-out: Data Flow Architecture

23

24 THANK YOU BloorGroup.com InsideAnalysis.com The image on Slide 8 provided by Chagai at en.wikipedia (Transferred from en.wikipedia) [Public domain], from Wikimedia Commons Avalanche_on_Everest.JPG