Mining Big Data. Pang-Ning Tan. Associate Professor Dept of Computer Science & Engineering Michigan State University

Transcription

1 Mining Big Data Pang-Ning Tan Associate Professor Dept of Computer Science & Engineering Michigan State University Website:

2 Google Trends Big Data Smart Cities

3 Big Data and Smart Cities

4 Outline Smart Cities Big Data and Its Challenges Mining Big Data

5 Smart Cities Cities are growing steadily, and the process of urbanization is a common trend in the world. Although cities are getting bigger, they are not necessarily getting better smart cities, founded on the use of information and communication technologies, aim at tackling many local problems, from local economy and transportation to quality of life and e-governance. [Martínez-Ballesté et al. IEEE Communications 2013]

6 Examples of Smart Cities Smart Buildings E-Governance Transportation Healthcare DATA $$$$ Education Energy Water Waste management Public safety What are the key resources needed to realize this?

7 Types of Data from Smart Cities Sensor time series Surveillance video streams GPS trajectories from mobile devices Smart card Social media Structured data

8 Why Mine/Analyze the Data? The data contains useful information that can be harnessed for various purposes: Monitoring/surveillance Event detection Adaptation Decision making Planning Forecasting Etc..

9 Outline Smart Cities Big Data and Its Challenges Mining Big Data

10 Big Data: How Much Data is Out There? Source:

11 How much is a Zettabyte? 1 ZettaByte= 1000 ExaBytes= 10 6 PetaBytes = 10 9 TeraBytes= GigaBytes A DVD stores about 5 GB data and its case is ~1cm thick 1 ZettaByte ~ / = 200 billion DVDs to store them Distance from Earth to moon = 384,000 km = cm ** If you stack all the DVDs that contain 1 ZB of data, it is about 3 times the distance to the moon and back

12 Challenges of Big Data Volume: large amount of data that is continuously growing Velocity: rapid streams of data collected Variety: structured and unstructured data obtained from (potentially) multiple data sources Veracity: messiness or trustworthiness of the data Value: usefulness of the data; needs a careful cost/benefit analysis before embarking on big data project

13 Outline Smart Cities Big Data and Its Challenges Mining Big Data

14 What is Data Mining? A collection of computer algorithms and techniques to automatically extract useful information from large data repositories Big Data Analytics Pipeline

15 Garbage In, Garbage Out Quality of output information depends on quality of input data

16 Data Preprocessing Helps to alleviate many of the data quality issues Noise Outliers Missing values (incomplete data) Duplicate data Data with irrelevant attributes Data with redundant attributes Data of varying format, scales, etc

17 Types of Data Analysis Simple, descriptive statistics Mean/Median/Mode Standard deviation/mean absolute deviation Quartiles, percentiles, top-k Example: Heavy-hitter problem Find the hot topics (e.g., trending hashtags) used over the past 24 hours

18 TrendMap

19 Finding Hot Topics (Unbounded storage) Data Stream 2013 discount holiday 2013 MSU Associative array, f Memory discount 1 holiday 1 MSU 1 Naïve algorithm; Assume storage space is unbounded

20 Finding Hot Topics (Limited Storage) Data Stream 2013 discount holiday Associative array, f Memory discount 1? holiday 1 Which one to replace? Any theoretical guarantees that solution will always be in the array?

21 Misra-Gries Algorithm Data Stream 2013 discount holiday 2013 MSU Associative array, f Memory discount MSU 1 0 holiday 1 Algorithm guarantees that all hot items that appear at least m/k+1 times will be in the buffer (where m is length of data stream and k is number of buffers)

22 Summary Even simple analysis becomes harder to compute when you have big data Need for fast and scalable algorithms that can produce good, approximate solutions

23 10 Advanced Data Mining Analysis Data Tid Refund Marital Status Taxable Income Cheat 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes 11 No Married 60K No 12 Yes Divorced 220K No 13 No Single 85K Yes 14 No Married 75K No 15 No Single 90K Yes Ranking/ Recommendation

24 Predictive Modeling: Classification To infer the value of a nominal attribute based on the values of other observed attributes Examples: Autonomous driving Traffic sign recognition Open lane detection Smart Home/Building: Appliance identification based on electricity utilization

25 Predictive Modeling: Regression To infer the value of a continuous attribute based on the values of other observed attributes Examples: mhealth Monitoring heart rate and body temperature using wearable devices Intelligent Transportation System Traffic volume prediction Smart Building Electricity/Water demand prediction

26 Framework for Predictive Modeling Labeled examples Unlabeled examples congestion No congestion Test Set Training Set Train Model Model

27 Cluster Analysis Find groups of observations such that the observations in the same group are more similar to each other than to those in other groups Intra-cluster distances are minimized Inter-cluster distances are maximized

28 Applications of Cluster Analysis Crime hotspot detection GPS trajectory segmentation

29 Association Analysis Extract patterns of frequently co-occurring events Time Sensor ID State 3/1/ :48:05 BR1 OFF 3/1/ :48:07 LR1 ON 3/1/ :48:10 LR6 ON 3/1/ :48:20 BT1 ON 3/1/ :48:40 LR6 OFF 3/1/ :49:30 BT3 ON Weekday, 7-8am, BR2 = OFF, BR1 = OFF, LR6 = ON LR1=ON Weekday, 10-11pm, BR1 = ON, BR2 = ON, LR6 = OFF LR1 = OFF

30 Applications of Association Analysis Traffic Accident Analysis Smart Health Adverse drug interactions

31 Anomaly Detection Detect significant deviations from normal observations

32 Applications of Anomaly Detection Smart Transportation Congestion detection Sensor fault detection Smart Home/Building Water theft detection Pipe burst detection

33 Ranking (Recommendation) Given a query q, recommend items in specific rank order based on their relevance to q Examples: Location-aware services Smart home assistant

34 Other Challenges: Privacy

35 Other Challenges: Security

36 Summary Mining big data is both a challenge and an opportunity

37 CSE Courses on Data Mining CSE 491/891: Computational Techniques for Large-Scale Data Analysis CSE 881: Data Mining

38 References Pang-Ning Tan, Knowledge Discovery from Sensor Data, Feature Article in Sensors Magazine, March Pang-Ning Tan, Michael Steinbach, and VipinKumar, Introduction to Data Mining, Addison Wesley, 2006