CS 493N: Big Data Engineering - Overview

Transcription

1 CS 493N: Big Data Engineering - Overview Introduction Data Structures for Big Data Data structures Basic Search Algorithms for Big Data Machine Learning and Predictive Analytics Intro to machine learning Regression and prediction Clustering & classification High Performance Computing for Big Data Hadoop & MapReduce HPC, Massively parallel computing, CUDA, etc. Social Computing and Crowdsourcing Social computing, social network analysis Crowdsourcing Visualization in Big Data Social Media Analytics NLP, social media analytics 1 1

2 CS493N Overview: Assessment Quizzes and Tests (50%) Assessed for each section by individual professors. Projects (50%) Group projects (Groups of 1 to 3) Project List and Selection (released by Sep. 21 ) Project Proposal presentation (Week of Oct. 5) (20%) Project progress report (Week of Nov. 2 ) (10%) Final Project Presentation (Week of Nov. 30) (50%) Final Project Report (Week of Dec. 7) (20%) Class Participation (5% on Quizzes and Tests) No Finals Grade Assignment A >= 85; B: 75-84; C: 65-74; D:50-64; F<50 2 2

3 Why Border? 3 3

4 Defining Big Data? Relates to data volume, variety, complexity, See: The Big Data Conundrum: How to Define It?, MIT Technology Review, Oct. 3, 2013, and the comments that followed ( One example definition from this site: Big data is a term describing the storage and analysis of large and or complex data sets using a series of techniques including, but not limited to: NoSQL, MapReduce and machine learning. -- JS Ward and A Barker, University of St Andrews, Scotland. But should the definition be technology driven? Another example definition from the site: Big data is a term describing the analysis of complex, highly variable and often large-scale data sets in order to convert raw data into oct. 4,

5 Big Data Drivers --- Technology View Technological Capabilities Improvements in various computing technologies. Big Data Sources & Generators Sensors of various forms images, audio, video, Molecular/nanoscale (genomics, next generation sequencing) Macro-scale sensing and imaging (think astronomy) The web and social media sources Internet of Things Capture, Collection, and Storage New Analytics Techniques Machine Learning, Data Mining, Data representations; NLP Big Data Tools & Platforms Hadoop, MapReduce, NoSQL, Hive, Spark, 5 5

6 Growth of Computational Capability From Moore s Law entry in Wikipedia 6 6

7 Growth of Computational Capability From Supercomputer entry in Wikipedia 7 7

8 Growth of Computational Capability Top Supercomputer speed (log scale) over the past 60 years, From Supercomputer entry in Wikipedia 8 8

9 Growth of Storage Capacity From Moore s Law entry in Wikipedia 9 9

10 From Moore s Law entry in Wikipedia 10

11 From Fernando Sancho Caparrini: 11

12 Big Data Drivers Users & Opportunities Businesses unprecedented business insights improved decision-making untapped novel sources of revenue and profit Big Government Threat prediction & prevention; Crime prediction & prevention Social program fraud, waste and errors Tax compliance, esp. fraud and abuse Healthcare; Emergency response and coordination Market Projections Projected to grow at 45% annually Optimism, Expectations, & Caution 12 12

13 Big Data Drivers Users & Opportunities Businesses business insight; improved decision-making; untapped revenue From: A Passion for Research: softwarestrategiesblog.com 13 13

14 Big Data Drivers Users & Opportunities Government Threat prediction & prevention; Crime prediction & prevention Social program fraud, waste and errors Tax compliance - fraud and abuse Healthcare; Emergency response and coordination Image from: Hadoop Use Cases: Big Data for the Government,

15 Big Data Drivers Users & Opportunities Market Projections From: Big Data Vendor Revenue and Market Forecast , Feb, 2014: 15 org/wiki/v/big Data Vendor Revenue and Market Forecast

16 Big Data Drivers Technology view Tools & Techniques business insight; improved decision-making; untapped revenue From:

17 Big Data Drivers Users & Opportunities Optimism, Great Expectations, & Caution The optimism is palpable, The applications are endless, The promise is unprecedented, Yet, we must still exercise caution with respect to what can realistically be expected from this emerging and exciting field. -axiom-of-reed%e2%80%99s-lawthe-financial-science-behind-big-data 17 17

18 So, What s Next in CS 493N? In this course, we will take a general look at key concepts that underlie the field of big data, from the systems platforms needed to data representations, to the analytics techniques. Along the way, we will get to use one or more of the key tools in big data. The project will be the primary driver in using hands-on tools to solving big data problems. No specific programming language is required. You will be free to work in any environment or language as dictated by your project, or by personal preference

19 Nature of Big Data Big Data from Multiple Views From: 1

20 Nature of Big Data The 3 V s of Big Data 2

21 From Fernando Sancho Caparrini:

22 Nature of Big Data The 5 V s of Big Data From: 4

23 Nature of Big Data The 10 V s of Big Data 5Vs+5 more Viscosity Visualization Variability (NOT variety) Virality Visibility.. still more From: 5

24 Nature of Big Data Connection to Data Science 6

25 Big Data Applications Various Government Threat prediction & prevention; Crime prediction & prevention Social program fraud, waste and errors Tax compliance - fraud and abuse Healthcare; Emergency response and coordination Image from: Hadoop Use Cases: Big Data for the Government, 7

26 Big Data Challenges Business View From: Careers/News-And-Events/News/2012/Challenges-Big-Data2.png 8

27 Big Data Challenges Business View m: com/read/10 greatest challenges preventing businesses from /372 9

28 Big Data Challenges Business View m: com/read/10 greatest challenges preventing businesses from /372 10

29 Big Data -- Nature, Challenges & Usage From: 11