Why big data? Lessons from a Decade+ Experiment in Big Data

Transcription

1 Why big data? Lessons from a Decade+ Experiment in Big Data David Belanger PhD Senior Research Fellow Stevens Institute of Technology dbelange@stevens.edu 1

2 What Does Big Look Like? 7 Image Source Page: 1,000 Image Source Page: ~C(10^5) DGB 5/2013 2

3 Challenge: Getting More Value from Data 9,570,000,000,000,000,000,000 Bytes** Managing Risk: Security Fraud Compliance Customer* Experience Data Analysis: Industry leading Information Mining Technology including RT Decision Support Data In Flight Efficient, Reliable, Secure Data Transportb Hundreds of Data Sources Volume Velocity Variability Information Visualization: The most effective ways to deliver Information & Alerts to decision makers? Operations: Market Network Services Process Control: Provisioning Forecasting & Planning ** * Customer 3 Specific Data is anonymized or aggregated. CS & Data Management: Storage and processing architectures That operate at scale, and in real time Vertical Services: Medical Smart Grid Hospitality Managed Services

4 Where Does It Come From? Transactional Data Internet + WWW + BB Access Mobility, Anywhere, Anytime Sensors, Ad Hoc Networks, IoT Integration Communications Entertainment 4

5 Applications of Big Data Application Type Service Oriented Retrieval, Individual Precision, Sparse Data Paths, Graphs, Relationships Diversity of Sources Real Time, Predictive, Data in Flight Driving Technology

6 Sparse Targets & Individual Precision Then Precision for many measurements, and most targets, are aggregate. Sampling is often used. Accessing a dataset which is a small subset of a huge dataset is difficult, especially for unstructured data. Surveys used for customer experience. Now Analysis is of individual events, and measured against individual metrics. Map/Reduce useful for finding relatively small subsets of very large datasets. Customer behavior/results can be used for customer experience. Search Misuse Fraud Customer Experience photoblog.nbcnews.com At&t Labs - Research DGB 5/2013 6

7 Paths, Graphs, & Relationships Then Most analysis and visualization done on graphs is relatively small scale, and seldom interactive. Now Networks, including explicit, implicit, and inferred, are analyzed and visualized at very large scale. DGB 5/2013 7

8 Diversity of Sources - Crowd Then Sources of large databases are largely from transactional systems such as credit card swipe, ATM, Financial Markets, or engineered systems such as communication networks, as well as science. Now Sources range from individual contributors, to sensors, to networks. The WWW and Mobile Smartphones have turned consumption and generation of data into an always on, anywhere activity. Including video. Health Vital Signs Personal Environment Sensors Ad Hoc Sensor Networks Image Source Page: /09/07/follow-us-on-twitter-and-like-us-on- Image Source Page:

9 Consumer Items & Sensors The next crowd Devices That Can Be Networked & IP Addressable How can we best exploit the billions of devices, many mobile, as computing, sensing, and communications platforms? Pallets and Cases Machinery Home Appliances Vehicles & Handheld Devices Crowd Computers b Invisible Computing 10 Consumer Items Pallets and Cases Home Appliances Machinery Vehicles and Handheld Devices Sensors Machine to Machine Internet Will Far outnumber current IT Devices Page 9 DGB 5/2013 9

10 Convergence of Communication and Entertainment Can I find what I want and watch, when I want? Base Issue How do we differentiate our media services in a world of commodity media services? Vision Access to all media, from all time, from anywhere Challenges User interface, especially search Indexing Storage Distribution 10 Source: AT&T Labs

11 Diversity of Data Types - Variety Then Most large data sets are either combinations of alphanumeric fields, or text. Now Data types range from the traditional, structured alpha-numic fields, to semi-structured (e.g. Web), to unstructured (text, speech, video, image). All of these, and mixtures of them, are analyzed at scale. Vodeo/Image Mining Personal Environment Sensors Speech Mining Customer Experience Customer Interaction Record

12 Then Real time systems are custom engineered and controlled, typically with relatively small data in flight. Data communications expensive at scale. Data in Flight Now Analysis of individual events, and measured against individual metrics, and at very large scale is becoming relatively common. Internet of Things starting to drive another spike in growth rate. Health - Smart Slippers Safety, Gaming Location Based Services DGB 5/

13 Data Analysis Lifecycle: Process Control Monitor Analyze Instrument Decide Control 13

14 Then Data Analysis For large datasets, it is usually the case that relatively small samples must be used. Customer studies are often based on surveys. Study results are frequently on aggregate data. Data numeric or text. Now Characterized by analytics on the population of data, though some datasets are still so big that sampling must be used. Customer studies based on behavior, and at extreme detail. Wide scale use of relationships e.g. social networks. Data numeric, text, speech, image. Graphs, networks, and paths Relationships Visualized Recommender Systems Visual Pattern Recognition Machine Learning openclassroom.stanford.edu 14

15 Information Visualization Human in the loop Then Largely descriptive and embedded in reports or dashboards. Aggregate measures most common, and created from a fairly restricted set of models characterized by statistical system. Now Characterized by scale, interactivity, and integration. Usually real time with immediate drill down facilities. Often with powerful new models to express detail Sometimes derived from gaming systems. Graphs, networks, and paths Word Clouds Relationships Visualized Recommender Systems VizGems Transparency, Integration, Control Through Visualization DGB 5/

16 Then Driving Technology Innovation within Data Management over a period of 30 years largely controlled by Relational Model, with some exceptions such as OODB Data Manaaement Systems Interactive Visualization & Data Integration Now Dynamic research in new tools embodying technologies such as NoSQL, Map/Reduce, DSMS, DSW, R, et al.. Much driven by large web companies. Some basic technologies, e.g. compression, become much moer important. DSWS Data Stream Warehouse Systems DS1 Graph Analysis at Large Scale DS2 DS3 DSMS DSW Image Source Page: DGB 5/

17 Then Meta Challenges Significant systems containing sensitive data are not easily accessed. Complex semantics and poor integrity often exist, but impact is hidden because data is relatively closed. Integration, outside of joins, uncommon at scale. Now Protection of SPI data a constant problem. Transparency of use, integrity of data a concern. Open data provides much more opportunity for interesting new apps from integration, and semantic confusion. Integration complex. Data Governance Security Privacy Integrity Semantics Integration DGB 5/

18 Organizing for Innovation Then Classical research or exploratory development teams create new products, often in large teams with significant timelines. Careful attention is paid to decision gates to prevent runaway costs. Due to costs, decisions often top down. Now Small, elite teams create prototypes of potential products quickly, trial the prototypes, and, when successful, present for funding to go to market. Go to trial, very quick. Classical research provides technology base to prototyping teams, and partners with them. Crowd Sourced Funnel Innovation Laboratory InfoLab DGB 5/

19 It s About the Data: Opportunity An Environment for Discovery InfoLab Sandbox Data Production Research InfoLab Clients Transfer Customer Applications Platform Incubation Informa1on & Data Product/Marketing IT InfoLab Cycle Engineering Operations Source: 19 AT&T Labs

20 Some Lessons Learned Strategy A defendable niche provides time to mature scale It s all about the data The Goal is to Enable Fundamental Process and Product Changes Ask Big Questions: e.g. Can an IP Network Run Itself? DGB 5/2013 Page 20

21 Some Lessons Learned Technology Multidimensional technical expertise is essential: Network Computing Data Analysis Visualization - Domains The Nature of analytics has changed: parallel, streams, predictive, geospatial Data Feed management can scale linearly. That is really bad Tradeoffs: Optimize Speed vs. Accuracy Depth & Volatility: Rules vs. Differences DGB 5/

22 Some Lessons Learned Ecosystem Few corporations can ignore the broader technology world, and none should. Sometimes the most effective way to impress management is to go outside e.g. Netflix, Idol Customer Focus - Choose your partners well, and make them heroes Components of an ecosystemthe above image of an ecosystem includes the DGB 5/2013 following components some of... Page 22

23 A More Complete Picture Data Analysis Data Management Visualization Applications Sandbox Privacy Data Governance Policy, Process Data Security OA&M Software Sustainability Integrity Semantics Framing Questions Distribution & Ownership of Results DGB 5/2013 Page 23

24 Where Information Services Are Going Pervasive Monitoring/Control Internet of Things OPEN DATA COSM Xively 1 Traditional Services TP, DW, Analytic Reports 3 Next- Generation Value from Data Immersive, Augmented Reality Interfaces s2 s1 s3 Multiple RT Streams 2 Info in Flight Real Time Stream Mining Mining Unstructured Data Mobility Next Gen Analytics, Prediction Data Stream Mining Speech/Text Mining DGB 5/2013 Anywhere, AnyDevice 24