Big Data Terminology - Key to Predictive Analytics Success. Mark E. Johnson Department of Statistics University of Central Florida F2: Statistics

Transcription

1 Big Data Terminology - Key to Predictive Analytics Success Mark E. Johnson Department of Statistics University of Central Florida F2: Statistics

2 Outline Big Data Phenomena Terminology Role Background on ISO TC69 NIST Initiative Joint ISO TC69 and NIST Effort Current and Future Developments Oct Joint Technical Communities Conference 2

3 Big Data Phenomena See: Olap.com Oct Joint Technical Communities Conference 3

4 Wiki Land Related Topics Apache Accumulo Hortonworks Apache Hadoop NoSQL Big Memory Programming in R Data mining Sqrrl Cask (company) Talend Cloudera Tranreality gaming Data Analytics Accel. Lib Tuple space HPCC Systems Unstructured data Internet of Things Spark Map Reduce Oct Joint Technical Communities Conference 4

5 Big Data Phenomena Average salaries (Bill Snyder, InfoWorld): MapReduce: $127,315 Cloudera: $126,816 HBase: $126,369 Pig: $124,563 Flume: $123,186 Hadoop: $121,313 Hive: $120,873 Zookeeper: $118,567 Data Architect: $118, Oct Joint Technical Communities Conference 5

6 Google search counts 9/20/2015 Data: 5.59 billion Big: 4.07 billion Big data: 753 million Business analytics: Business intelligence: Predictive analytics: Statistics lies: Machine learning: Kardashian: 127 million 108 million 12.4 million 217 million 60 million 221 million Oct Joint Technical Communities Conference 6

7 Opportunities/Challenges Managers/Deans/Provosts throwing $$$ at Big Data Discipline Discipline = Statistics / Computer Science / Info Sciences / MIS / Industrial Engineering / Business College Skill Set: IT + Statistics One Thing Abundantly, Massively, Redundantly Clear: Big Data means different things to different people Overwhelming Statistical types need partners to get the data; IT folks need help with the (simple) statistical analyses Oct Joint Technical Communities Conference 7

8 Even our Sponsored Research folks got involved Oct Joint Technical Communities Conference 8

9 Is Terminology Necessary? Google Search (electronic dictionary)? Familiar with definitions ability to write them Lack of standardization does it matter? (think C p, C pk ) 沟 通 コミュニケーション viestintä επικοινωνία 의사소통 общение iletişim спілкування การต ดต อส อสาร sự truyền đạt ا ت ص ال Oct Joint Technical Communities Conference 9

10 Big Data Definition DRAFT NIST Big Data Interoperability Framework: Volume 1, Definitions Abstract sets the playing field: Big Data is a term used to describe the new deluge of data in our networked, digitized, sensorladen, information-driven world. While great opportunities exist with Big Data, it can overwhelm traditional technical approaches and its growth is outpacing scientific and technological advances in data analytics Oct Joint Technical Communities Conference 10

11 Brief History Moore s Law (1965): transistors on boards doubling every 2 years (or, computing capability) Growth of Data Volumes: doubling every 1.5 yrs Development of relational data bases Increasing amounts of unstructured data Hardware pushed Software pushed via massive parallel computing Oct Joint Technical Communities Conference 11

12 Drilling down into Big Data Four key aspects of the Big Data Revolution (1) characteristics of the datasets, (2) analysis of the datasets, (3) performance of the systems that handle the data, (4) the business considerations of cost effectiveness Oct Joint Technical Communities Conference 12

13 Big Data V s NIST working definition: Big Data consists of extensive datasets primarily in the characteristics of volume, variety, velocity, and/or variability that require a scalable architecture for efficient storage, manipulation, and analysis. Volume (data set size) Variety (multiple data types, various domains and repositories) Velocity (rate of flow or creation of data) Variability (non-constancy of other V s) Oct Joint Technical Communities Conference 13

14 The Big Data paradigm consists of the distribution of data systems across horizontally coupled, independent resources to achieve the scalability needed for the efficient processing of extensive datasets. Horizontal scaling: clustering multiple resources to act as one system Vertical scaling: limited by physical constraints (e.g., Moore s law) Oct Joint Technical Communities Conference 14

15 Definitions Big Data engineering includes advanced techniques that harness independent resources for building scalable data systems when the characteristics of the datasets require new architectures for efficient storage, manipulation, and analysis. Non-relational models, frequently referred to as NoSQL, refer to logical data models that do not follow relational algebra for the storage and manipulation of data. A federated database system is a type of meta-database management system, which transparently maps multiple autonomous database systems into a single federated database Oct Joint Technical Communities Conference 15

16 More NIST Definitions Schema-on-read is the application of a data schema through preparation steps such as transformations, cleansing, and integration at the time the data is read from the database. Computational portability is the movement of the computation to the location of the data. The data science paradigm is extraction of actionable knowledge directly from data through a process of discovery, hypothesis, and hypothesis testing. The data lifecycle is the set of processes that transform raw data into actionable knowledge. Analytics is the synthesis of knowledge from information Oct Joint Technical Communities Conference 16

17 Data science is the empirical synthesis of actionable knowledge from raw data through the complete data lifecycle process A data scientist is a practitioner who has sufficient knowledge in the overlapping regimes of business needs, domain knowledge, analytical skills, and software and systems engineering to manage the end-to-end data processes through each stage in the data lifecycle Oct Joint Technical Communities Conference 17

18 Skills Needed in Data Science Oct Joint Technical Communities Conference 18

19 Even More V s Veracity: accuracy of the data Value: value of the analytics to the organization Volatility: tendency for data structures to change over time Validity: appropriateness of the data for its intended use Some non-v s: quality control, metadata, and data provenance Oct Joint Technical Communities Conference 19

20 Oxford English Dictionary: big data n. Computing (also with capital initials) data of a very large size, typically to the extent that its manipulation and management present significant logistical challenges; (also) the branch of computing involving such data. Cathy O Neill: Big data is more than one thing, but an important aspect is its use as a rhetorical device, something that can be used to deceive or mislead or overhype Oct Joint Technical Communities Conference 20

21 ISO TC69 Applications of Statistical Methods Standardization in the application of statistical methods, including generation, collection (planning and design), analysis, presentation and interpretation of data. ( No Specific Big Data Role TC69 Chair (Dr. M. Boulanger) launched ad hoc committee to work with ISO/IEC JTCG1/WG9 (i.e., NIST effort) Oct Joint Technical Communities Conference 21

22 ISO TC69 Structure SC1 Terminology and symbols SC4 Applications of statistical methods in process management SC5 Acceptance sampling SC6 Measurement methods and results SC7 Applications of statistical and related techniques for the implementation of Six Sigma SC8 Application of statistical and related methodology for new technology TC 69/WG 3 Statistical interpretation of data Oct Joint Technical Communities Conference 22

23 NIST Initiative, Draft Documents Volume 1, Definitions Volume 2, Taxonomies Volume 3, Use Cases and General Requirements Volume 4, Security and Privacy Volume 5, Architectures White Paper Survey Volume 6, Reference Architecture Volume 7, Standards Roadmap Oct Joint Technical Communities Conference 23

24 Joint Effort ISO TC69: NIST Group Standardization and Statistics Expertise IT expertise including comprehensive proposed architecture/ecosystem for Big Data Oct Joint Technical Communities Conference 24

25 NIST Initiative via ISO/IEC JTC 1/WG9 Big Data Public Working Group (NBD-PWG) has developed a Big Data Interoperability Framework DRAFT NIST Big Data Interoperability Framework: Volume 2, Big Data Taxonomies Particular interest is Figure 1 from this Document (one of 7 volumes)

26 Figure 1

27 Simplified Diagram Data Provider Big Data Application Provider Collection Preparation/Curation Analytics Visualization Access Data Consumer Information Technology Realm

28 1. Collection 2. Preparation/Curation 3. Analytics 4. Visualization 5. Access Components of Big Data Application Provider

29 Collection Moving the data from its repository to an accessible location (large volumes and confidentiality) What/which data (merge); granularity; Sampling (if any) Prior to data preparation and cleansing Interactive/Iterative process likely Statistician connects Data Provider and Data Consumer

30 Preparation/Curation Validation (e.g., check sums, format checks) Cleansing (eliminate clear mistakes, duplicate records; not Outlier removal however ) Conversion (hiding sensitive data; summarization?) Partition implementation (data >> PC storage; schema on read)

31 Analytics NPD-PWD thinks of analytics as the steps of discovery that equate to rapid hypothesis-test cycle for finding value in large data sets (i.e., agile analytics such as correlations and trends) Structured versus unstructured data (containing value) Possible parallel environment for analysis May not have the raw data in a table (summary tables or relational data bases)

32 Analytics (continued) Complexity issues (execution time of method) Latency (real-time or streaming, near real-time or interactive, batch or offline) Human-in-the-loop analytics lifecycle (e.g., discovery, hypothesis, hypothesis testing)

33 Visualization 1. Exploratory data visualization for data understanding (e.g., browsing, outlier detection, boundary conditions) 2. Explicatory visualization for analytical results (e.g., confirmation, near real-time presentation of analytics, interpreting analytic results) 3. Explanatory visualization to tell the story (e.g., reports, business intelligence, summarization)

34 Visualization (continued) Note similarity to Exploratory Data Analysis Confirmatory Data Analysis Implementation Feasible with very large data sets Data rich allows training, validation, verification Not necessary to squeeze all information out

35 Access Could be an explicit implementation of the actionable items as a culmination of the previous four categories. The access activity of the Big Data Application Provider should mirror all actions of the Data Provider, since the Data Consumer may view this system as the Data Provider for their follow-on tasks. = score future observations based on an analytical model (including cleaning/aggregation). Interpretation of 5 categories is a Work in Progress

36 Summary Big Data Public Working Group (NBD-PWG) has developed a Big Data Interoperability Framework useful to ISO/TC69 for developing future statistical standards of use to both us (developers and users) and them (IT people who rely on us plus the End Users) Some standards in TC69 apply now (but gap analysis in order) We need them and they need us! Hardware/Software/Info Tech material handled by NBD- PWG (we can focus on statistics) Invitation to participate with JTC/WG9 is a great opportunity. Take advantage now or some other entity will.