Mining Big Data Pang-Ning Tan Associate Professor Dept of Computer Science & Engineering Michigan State University Website: http://www.cse.msu.edu/~ptan
Google Trends Big Data Smart Cities
Big Data and Smart Cities
Outline Smart Cities Big Data and Its Challenges Mining Big Data
Smart Cities Cities are growing steadily, and the process of urbanization is a common trend in the world. Although cities are getting bigger, they are not necessarily getting better smart cities, founded on the use of information and communication technologies, aim at tackling many local problems, from local economy and transportation to quality of life and e-governance. [Martínez-Ballesté et al. IEEE Communications 2013]
Examples of Smart Cities Smart Buildings E-Governance Transportation Healthcare DATA $$$$ Education Energy Water Waste management Public safety What are the key resources needed to realize this?
Types of Data from Smart Cities Sensor time series Surveillance video streams GPS trajectories from mobile devices Smart card Social media Structured data
Why Mine/Analyze the Data? The data contains useful information that can be harnessed for various purposes: Monitoring/surveillance Event detection Adaptation Decision making Planning Forecasting Etc..
Outline Smart Cities Big Data and Its Challenges Mining Big Data
Big Data: How Much Data is Out There? Source: http://www.emc.com/leadership/digital-universe/index.htm
How much is a Zettabyte? 1 ZettaByte= 1000 ExaBytes= 10 6 PetaBytes = 10 9 TeraBytes= 10 12 GigaBytes A DVD stores about 5 GB data and its case is ~1cm thick 1 ZettaByte ~ 10 21 / 5 10 9 = 200 billion DVDs to store them Distance from Earth to moon = 384,000 km = 3.84 10 10 cm ** If you stack all the DVDs that contain 1 ZB of data, it is about 3 times the distance to the moon and back
Challenges of Big Data Volume: large amount of data that is continuously growing Velocity: rapid streams of data collected Variety: structured and unstructured data obtained from (potentially) multiple data sources Veracity: messiness or trustworthiness of the data Value: usefulness of the data; needs a careful cost/benefit analysis before embarking on big data project
Outline Smart Cities Big Data and Its Challenges Mining Big Data
What is Data Mining? A collection of computer algorithms and techniques to automatically extract useful information from large data repositories Big Data Analytics Pipeline
Garbage In, Garbage Out Quality of output information depends on quality of input data
Data Preprocessing Helps to alleviate many of the data quality issues Noise Outliers Missing values (incomplete data) Duplicate data Data with irrelevant attributes Data with redundant attributes Data of varying format, scales, etc
Types of Data Analysis Simple, descriptive statistics Mean/Median/Mode Standard deviation/mean absolute deviation Quartiles, percentiles, top-k Example: Heavy-hitter problem Find the hot topics (e.g., trending hashtags) used over the past 24 hours
TrendMap
Finding Hot Topics (Unbounded storage) Data Stream 2013 discount holiday 2013 MSU Associative array, f Memory 2013 12 discount 1 holiday 1 MSU 1 Naïve algorithm; Assume storage space is unbounded
Finding Hot Topics (Limited Storage) Data Stream 2013 discount holiday Associative array, f Memory 2013 1 discount 1? holiday 1 Which one to replace? Any theoretical guarantees that solution will always be in the array?
Misra-Gries Algorithm Data Stream 2013 discount holiday 2013 MSU Associative array, f Memory 2013 1 0 discount MSU 1 0 holiday 1 Algorithm guarantees that all hot items that appear at least m/k+1 times will be in the buffer (where m is length of data stream and k is number of buffers)
Summary Even simple analysis becomes harder to compute when you have big data Need for fast and scalable algorithms that can produce good, approximate solutions
10 Advanced Data Mining Analysis Data Tid Refund Marital Status Taxable Income Cheat 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes 11 No Married 60K No 12 Yes Divorced 220K No 13 No Single 85K Yes 14 No Married 75K No 15 No Single 90K Yes Ranking/ Recommendation
Predictive Modeling: Classification To infer the value of a nominal attribute based on the values of other observed attributes Examples: Autonomous driving Traffic sign recognition Open lane detection Smart Home/Building: Appliance identification based on electricity utilization
Predictive Modeling: Regression To infer the value of a continuous attribute based on the values of other observed attributes Examples: mhealth Monitoring heart rate and body temperature using wearable devices Intelligent Transportation System Traffic volume prediction Smart Building Electricity/Water demand prediction
Framework for Predictive Modeling Labeled examples Unlabeled examples congestion No congestion Test Set Training Set Train Model Model
Cluster Analysis Find groups of observations such that the observations in the same group are more similar to each other than to those in other groups Intra-cluster distances are minimized Inter-cluster distances are maximized
Applications of Cluster Analysis Crime hotspot detection GPS trajectory segmentation
Association Analysis Extract patterns of frequently co-occurring events Time Sensor ID State 3/1/2015 07:48:05 BR1 OFF 3/1/2015 07:48:07 LR1 ON 3/1/2015 07:48:10 LR6 ON 3/1/2015 07:48:20 BT1 ON 3/1/2015 07:48:40 LR6 OFF 3/1/2015 07:49:30 BT3 ON Weekday, 7-8am, BR2 = OFF, BR1 = OFF, LR6 = ON LR1=ON Weekday, 10-11pm, BR1 = ON, BR2 = ON, LR6 = OFF LR1 = OFF
Applications of Association Analysis Traffic Accident Analysis Smart Health Adverse drug interactions
Anomaly Detection Detect significant deviations from normal observations
Applications of Anomaly Detection Smart Transportation Congestion detection Sensor fault detection Smart Home/Building Water theft detection Pipe burst detection
Ranking (Recommendation) Given a query q, recommend items in specific rank order based on their relevance to q Examples: Location-aware services Smart home assistant
Other Challenges: Privacy http://techland.time.com/2012/02/17/how-target-knew-a-high-school-girl-was-pregnant-before-her-parents/
Other Challenges: Security http://arstechnica.com/security/2012/12/how-an-internet-connected-samsung-tv-can-spill-your-deepest-secrets/
Summary Mining big data is both a challenge and an opportunity
CSE Courses on Data Mining CSE 491/891: Computational Techniques for Large-Scale Data Analysis CSE 881: Data Mining
References Pang-Ning Tan, Knowledge Discovery from Sensor Data, Feature Article in Sensors Magazine, March 1 2006 Pang-Ning Tan, Michael Steinbach, and VipinKumar, Introduction to Data Mining, Addison Wesley, 2006