What is Big Data? The three(or four) Vs in Big Data In 2013 the total amount of stored information is estimated to be Volume.

Transcription

1 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall CS535/CS581A BIG DATA What is Big Data? PART 0. INTRODUCTION 1. INTRODUCTION TO BIG DATA 2. COURSE INTRODUCTION PART 0. INTRODUCTION 1. INTRODUCTION TO BIG DATA Sangmi Lee Pallickara Computer Science, Colorado State University 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Big Data Things one can do at a large scale that cannot be done at a smaller one To extract new insights Create new forms of values Big Data is about analytics of huge quantities of data in order to infer probabilities Big Data is NOT about trying to teach a computer to think like humans Providing a quantitative dimension it never had before The three(or four) Vs in Big Data In 2013 the total amount of stored information is estimated to be Volume around 1,200 exabytes Voluminous Only 2 percent is non-digital It does not have to be certain number of petabytes or quantity. Velocity How fast the data is coming in? How fast you need to be able to analyze and utilize it Variety Number of sources or incoming vectors Real-time data analysis Veracity Can you trust the data itself, source of the data, or the process? User entry errors, redundancy, corruption of the values Data cleaning 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Use all of the data, or just a little? 1/3 John Graunt s population example Instead of counting every person in London, he inferred the population size using sampling Conducting censuses Costly and time-consuming The US Constitution mandated one every decade As the growing country measured itself in millions Use all of the data, or just a little? 2/3 By the late nineteenth century, even that was proving problematic Data outstripped the Census Bureau s ability to keep up. The 1880 census took 8 years to complete The information was obsolete even before it became available! The 1890 census was estimated to require a full 13 years 1

2 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Use all of the data, or just a little? 3/3 In 2009 a new flu virus was discovered. The Census Bureau contracted with Herman Hollerith An American inventor used his idea of punch cards and tabulation machines for the 1890 census Reduced duration to less than one year Methods of acquiring and analyzing big data is very expensive Image of the newly identified H1N1 influenza virus, taken in the CDC Influenza Laboratory. Source: 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall H1N1 virus Combines elements of the viruses that causes bird flu and swine flu Some projected outbreak to be on the scale of the 1918 Spanish flu that had: Infected half a billion people Killed tens of millions In the United States, The Centers for Disease Control and Prevention (CDC) Requested doctors inform patients of new flu cases Tabulated the numbers once a week What will be the problems with this system? No vaccine was ready Public health authorities needed to know where it already was. 8/26/2014 CS581 Big Data - Fall Detecting influenza epidemics using search engine query data -1/2 J. Ginsberg, et al., Detecting influenza epidemics using search engine query data Nature 457 pp ~ 1014, February nature07634.html 8/26/2014 CS581 Big Data - Fall Detecting influenza epidemics using search engine query data -2/2 Predict the spread of the winter flu in the United States Nationally Specific regions and states Looking at what people were searching for on the Internet Google More than three billion search queries every day Saves all the queries 50 million most common search terms Americans type and compared the list with CDC data On the spread of seasonal flu between 2003 and 2008 Identify areas infected by the flu virus by what people searched for on the Internet 2

3 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall google.org: Flu Trends Example More information How did they build a prediction model? (1/3) Aggregating historical logs of online web search queries Between 2003 and 2008 Computed a time series of weekly counts for 50 million of the most common search queries in the US Separate aggregate weekly counts were kept for every query in each state Simple model that estimates the probability that a random physician visit in a particular region is related to an influenza-like illness(ili) 8/26/2014 CS581 Big Data - Fall How did they build a prediction model? (2/3) The probability that a random search query submitted from the same region is ILI-related 8/26/2014 CS581 Big Data - Fall How did they build a prediction model? (3/3) Each of the 50 million candidate queries was separately tested To identify the search queries that could most accurately model the CDC ILI visit percentage in each region logit(i(t)) = αlogit(q(t))+ε where: I(t) is the percentage of ILI physician visits at time t Q(t) is the ILI-related query fraction at time t α is the multiplicative coefficient ε is the error term logit(p) = ln(p/(1 p)) Generated the highest scoring search queries 8/26/2014 CS581 Big Data - Fall Evaluation of combining top-scoring queries Maximal performance at estimating out-of sample points during cross-validation was obtained by summing top 45 search queries J Ginsberg et al. Nature 000, 1-3 (2008) doi: /nature07634 After adding query 81 oscar nominations 8/26/2014 CS581 Big Data - Fall 2014 In the top 100, 18 topics like high school basket ball was not Included. It coincided with influenza season in the US Most co-related queries Search query topic Top 45 queries Next 55 queries n Weighted n Weighted Influenza complication Cold/flu remedy General influenza symptoms Term for influenza Specific influenza symptom Symptoms of an influenza complication Antibiotic medication remedies General influenza Symptoms of a related disease Antiviral medication Related disease Unrelated to influenza Total The top 45 queries were used in the final model; the next 55 queries are presented for comparison purposes. The number of queries in each topic is indicated, as well as queryvolume-weighted counts, reflecting the relative frequency of queries in each topic. 3

4 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Data sources Public dataset from the CDC s Influenza Sentinel Provider Surveillance Network Average percentage of all outpatient visits to sentinel providers that were ILI-related Query submitted to the Google search engine 2003 ~ 2008 What makes this work innovative? There have been other approaches: CDC s weekly report Call volume to telephone triage advice lines Tracking over-the-counter drug sales Online activity for influenza surveillance Search queries submitted to a Swedish medical website Tracking search keys ( flu, or influenza ) in Yahoo search queries What is the innovation in Google s approach? Google processes more than 24 Petabytes of data per day 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Related research areas Storage systems How can we efficiently resolve queries on massive amounts of input data? The input dataset may be presented in the form of a distributed data stream Machine learning How can we efficiently solve large-scale machine learning problems? The input data may be massive; stored in a distributed cluster of machines Distributed computing How can we efficiently solve large-scale optimization problems in distributed computing environments? For example, how can we efficiently solve large-scale combinatorial problems, e.g. processing of large scale graphs? PART 0. INTRODUCTION 1. INTRODUCTION TO BIG DATA 2. COURSE INTRODUCTION PART 0. INTRODUCTION 2. COURSE INTRODUCTION Sangmi Lee Pallickara Computer Science, Colorado State University CS581A BIG DATA 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Goal of this course Understanding fundamental concepts in Big Data Analytics Learn about existing technologies and how to apply them Related courses Machine Learning/Statistics Distributed Systems Database Systems Computer Communications Computer Security 4

5 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall When is the best time to take this course? About me Research area Big Data Storage, retrievals, analytics and visualization Projects Galileo Distributed data storage system for large scale geospatial time-series datasets Mendel Genomic data storage system GeoLens Visual Analytics 8/26/2014 CS581 Big Data - Fall Galileo: Big Picture 8/26/2014 CS581 Big Data - Fall Galileo: key features Data is voluminous Outpaces what is available on a single hard drive Storage must be over a collection of machines Avoid central coordinators Cope with failures Preserve data locality without introducing storage imbalances And the accompanying query hotspots Support range queries and fast ingest of new data Support for large numbers (10 9 ) of small files High throughput storage and retrieval Data is multidimensional with multiple types Time-series data Support for exact match and range queries (with wildcards) along multiple dimensions Support for multiple data formats netcdf, BUFR, HDF 4/5, and data from the Defense Meteorological Satellite Program 8/26/2014 CS581 Big Data - Fall Galileo Design Considerations 8/26/2014 CS581 Big Data - Fall Distributed Hash Tables (DHTs) Symmetric storage nodes No special-function or controller nodes Storage and retrievals may go to any node, and will be forwarded to the targeted node(s) Incremental scale-up Failure-resiliency These are decentralized systems Each node maintains information about a subset of nodes Each node uses its local information to reach any other node in the system Typically used as <key, value> stores 5

6 8/26/2014 CS581 Big Data - Fall Storage Throughput Block is about 8 KB of data 56,000 blocks per second in a system with 48-nodes 8/26/2014 CS581 Big Data - Fall Communications Course Website Announcements: Check the course website at least twice a week. Schedule (course materials, readings, assignments) Policies RamCT Assignment submission Discussion board Grades Contact Instructor sangmi@cs Office hour: T, TH 11:00AM ~ noon and by appointment Office: CSB456 URL: 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Course Structure Programming Assignments Assignment 2 Part 0 Part 1 Part 2 Part 3 Part 0 Part 1 Part 2 Part 3 Introduction to Big Data Data Storage models Scalable algorithms and Big Data Analytics Frameworks of Big Data Analytics Introduction to Big Data Data Storage models Scalable algorithms and Big Data Analytics Frameworks of Big Data Analytics Assignment 1 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Term Project-1/2 Phase 1. Motivating articles Read 3 motivating articles and summarize them Phase 2. Term project proposal Choose your topic Challenging Big Data topics e.g. Simple distributed storage systems, N-gram analysis engine Big Data challenges in your research domain e.g. Simple analytical tool for large scale data analysis in bioinformatics Plan your project Software design Functional requirements Timeline Evaluation method Team/Individual interview Term Project-2/2 Phase 3. Final submission/demo Description and code Phase 4. Presentation You may work on the term project (Phase 2 ~ 4) individually or as a team (only two member teams will be allowed) You should submit phase 1 individually. Group efforts must be correspondingly bigger If you are working with a teammate, please your team by September 8. 6

7 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Term Project Grading Phase 1 (Motivating articles): 10% Phase 2 (Proposal): 25% Phase 3 (Final submission/demo): 40% Phase 4 (Presentation): 25% Course Timeline Phase 1. September 3 Phase 2. PA1 PA2 Phase 4. September 17 October 6 November 5 Phase 3. Week 16 December 2 August September October November December Midterm I October 2 Midterm 2 December 4 Phase 1. description has been posted. 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Quizzes and Participation There will be 10+ pop quizzes in class 1-2 simple questions Open-book, open-material, not open-internet Minute survey Grading Policy Category Points Percentage of Final Grade Programming Assignments (15% x 2) % Term Project % Quizzes and participation % Midterm Exams (15% x 2) % 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Final Letter Grade Course Policy Letter Grade A Total Percentage % and higher No make-up for missed exams Except in extraordinary circumstances (e.g., major illness, family emergency) B ~ % C ~ % D ~ % No make-up for missed quizzes Except for the case where student provided an advance notice to the instructor based on an emergency F Below % 7

8 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall Course Policy No Cell-phones in the class. No Laptops during the exam, and quiz. If you need to use a laptop, please sit in the back row. Course Policy: Assignments Assignment submission RamCT Late policy for the assignments Up to a maximum of 2 day past the deadline. 10% penalty per day will be applied I will ask you to turn off your laptop if needed. Grading will be done by interview. 8/26/2014 CS581 Big Data - Fall /26/2014 CS581 Big Data - Fall More importantly, Questions? Attend the class, ask questions, and discuss Check the course web page and RamCT regularly Try new technologies and apply them Share your experiences with other students in class 8