Big Data and Analytics: Challenges and Opportunities Dr. Amin Beheshti Lecturer and Senior Research Associate University of New South Wales, Australia (Service Oriented Computing Group, CSE) Talk: Sharif University of Technology 4 July 2015
We are Generating Vast Amounts of Data!! 2 Remote patient monitoring Product sensors Healthcare Social media Manufacturing books, music, videos, etc. Retail Real time location data Digitalization of Artefacts Location-Based Services
We are Generating Vast Amounts of Data!! Air Bus A380: generate 10 TB every 30 min Twitter: Generate approximately 12 TB of data per day. Facebook: Facebook data grows by over 500 TB daily. New York Stock: Exchange 1TB of data everyday. 3
We are Generating Vast Amounts of Meta-data!! 4 Provenance Data Versioning Privacy Security We are Tracing everything: Who did What? When? Where?
We are Generating Vast Amounts of Meta-data!! 5 Provenance Data Versioning Privacy Security We are Tracing everything: Who did What? When? Where?
We are Generating Vast Amounts of Meta-data!! Reading a book, e.g. Kindle tracks: what you are reading, when you are reading it, how often you read it, etc. Listening to music, e.g. mp3 player tracks: what you are listening to, when and how often, in what order, etc. Smart phones, e.g. iphone tracks: our location, our speed, what apps we are using, who we are ringing, etc. 6
We are Generating Vast Amounts of Meta-data!! Reading a book, e.g. Kindle tracks: what you are reading, when you are reading it, how often you read it, etc. Listening to music, e.g. mp3 player tracks: what you are listening to, when and how often, in what order, etc. Smart phones, e.g. iphone tracks: our location, our speed, what apps we are using, who we are ringing, etc. 7
Big Data and Big Meta-Data 8 Big share, comment, review, crowdsource, etc.
So, What is Big Data? Big data refers to our ability to collect and analyse the ever expanding amounts of data and meta-data that we are generating every second! Challenges: Capture, Storage, Search, Sharing, Transfer, Analysis, Visualization, etc. 9
So, What is Big Data? Big data refers to our ability to collect and analyse the ever expanding amounts of data and meta-data that we are generating every second! Challenges: Capture, Storage, Search, Sharing, Transfer, Analysis, Visualization, etc. 10
Volume What Makes it Big Data? the vast amounts of data generated every second. Velocity the speed at which new data is generated and moves around. Variety the increasingly different types of data. Veracity the quality of data, e.g. the messiness of the data. Needs detecting and correcting noisy and inconsistent data Value Statistical, Events, Correlation, Hypothetical 11
Challenges: How to Store and Process? 12 Big data is high volume, high velocity, and/or high variety information assets. Require new forms of storage and processing. On-hand database management tools? Traditional data processing applications?
Challenges: Big Data Storage NoSQL databases: 13 Employs less constrained consistency models. Simple retrieval and appending operations. Significant performance benefits. Examples: Key value Store Document Store Graph Database
(Graphs are Everywhere) Challenges: Big Data Storage 14 Social Network User Collaborative Filtering Netflix Movie Probabilistic Analysis Text Analysis Docs Wiki Words
Challenges: Big Data Analytics (Graphs are Everywhere) 15 Social Network User Collaborative Filtering Netflix Movie Probabilistic Analysis Text Analysis Docs Wiki Words
Challenges: Big Data Analytics (Graphs are Everywhere) 16 Social Network User Collaborative Filtering Netflix Movie Probabilistic Analysis Text Analysis Docs Wiki Words
Challenges: Big Data Processing Apache Hadoop: 17 Hadoop is an open source framework that uses a simple programming model to enable distributed processing of large data sets on clusters of computers. Apache Hadoop solution: Distributed File System (HDFS) MapReduce Pig HCatalog Who Use Hadoop? Amazon Facebook Google IBM New York Times Yahoo!
Challenges: Big Data Processing Apache Spark: Fast and Expressive Cluster Computing Engine Compatible with Apache Hadoop 18 Efficient In-memory storage Usable Rich APIs in Java, Scala, Python
Challenges: Big Data Processing Apache Spark: Fast and Expressive Cluster Computing Engine Compatible with Apache Hadoop 19 Efficient In-memory storage Usable Rich APIs in Java, Scala, Python Resilient Distributed Dataset (RDD), Spark's data storage model
Challenges: Big Data Integration 20 Workflows IT Systems Web Services People Example Scenario: Business Processes (BPs) BPs Execution Log..
Challenges: Big Data Integration 21 Workflows IT Systems Web Services People Example Scenario: Business Processes (BPs) BPs Execution Log..
Challenges: Big Data Integration Messy, schema-less and complex Big Data world. Less than 10% of Big Data world are genuinely relational. 22 e.g. Linked Data
Challenges: Big Data Integration Big Data-as-a-Service: Effective processing of big data within acceptable processing time Easy access of the big data and the big data analysis results API Engineering 23 ProgrammableWeb - APIs, Mashups and the Web as Platform; www.programmableweb.com/ DataSift, CSDL
Challenges: Big data requires a broad set of skills 24 Math and Operations Research Expertise Data Experts Data architecture, management, governance, policy Develop analytic algorithms Decision Making Executive and Management Apply information to solve business issues Tool Developers Mask complexity and analytics to lower skills boundaries Visualization Expertise Interpret data sets, determine correlations and present in meaningful ways Industry Vertical Domain Expertise Develop hypothesis, identify relevant business issues, ask the right questions
Challenges: Big Data Analytics Analytics can be defined in many ways, but what matters is the purpose of analytics. Most definitions agree on the following: 25 Analytics is used to gain insights from data in order to make better decisions, using mathematical or scientific methods. Data Insight Action Analyse Decide Manage the Data Understand the Data Act on the Data
Challenges: Big Data Analytics Analytics can be defined in many ways, but what matters is the purpose of analytics. Most definitions agree on the following: 26 Analytics is used to gain insights from data in order to make better decisions, using mathematical or scientific methods. Data Insight Action Analyse Decide Manage the Data Understand the Data Act on the Data
Challenges: Big Data Analytics 27
Challenges: Big Data Analytics 28
Challenges: Big Data Analytics Example: Beheshti et al., Scalable Graph-based OLAP Analytics over Process Execution Data, DAPD Journal (2015). Beheshti et al., A Framework and a Language for On-Line Analytical Processing on Graphs, WISE Conference (2012). 29 OLAP, is an approach to answering multi-dimensional analytical queries swiftly. Problem: extension of existing OLAP techniques to analysis of graphs is not straightforward. key business insights remain hidden in the interactions among objects. Solution: On-Line Analytical Processing on Graphs
Challenges: Big Data Analytics 30
(Graph Data Model) Challenges: Big Data Analytics Nodes (Entities, Folders, and Paths) Entities: 31 Structured/Unstructured typed/un-typed data objects. Paper Author Venue
(Graph Data Model) Challenges: Big Data Analytics Nodes (Entities, Folders, and Paths) Folder Nodes: Contains a set of inter-related entities (e.g. events). Can be a placeholder for the result of a given query. 32 Set of related authors. Set of related papers.
(Graph Data Model) Challenges: Big Data Analytics Nodes (Entities, Folders, and Paths) Path Nodes: Contains a set of paths (i.e. a path is a transitive relationship between two entities) 33 Alex author-of published-in VLDB Paper: Big Data
(Graph Data Model) Challenges: Big Data Analytics Nodes (Entities, Folders, and Paths) Relationships: Is a directed link between a pair of entities. Can be explicit or implicit. 34 Folder Implicit (e.g. member-of) Explicit (e.g. triggered-by)
(Graph OLAP) Challenges: Big Data Analytics 35
(Graph OLAP) Challenges: Big Data Analytics 36
(Graph OLAP) Challenges: Big Data Analytics 37
(Graph OLAP) Challenges: Big Data Analytics 38
Challenges: Big Data Analytics Big Data Analytic benefits from: NLP Machine Learning pattern recognition, learning, KG NLP Example: Beheshti, et al,, A Systematic Review and Comparative Analysis of Cross-Document Coreference Resolution Methods and Tools, Computing Journal (2015), submitted. 39
(Graph Data Model) Challenges: Big Data Analytics 40 NLP Example: Beheshti S.M.R., et al,, A Systematic Review and Comparative Analysis of Cross-Document Coreference Resolution Methods and Tools, Computing Journal (2015), submitted.
(Graph Data Model) Challenges: Big Data Analytics NLP Example: Beheshti S.M.R., et al,, A Systematic Review and Comparative Analysis of Cross- Document Coreference Resolution Methods and Tools, Computing Journal (2015), submitted. 41
(Graph Data Model) Challenges: Big Data Analytics 42 Big Data Analytics benefits from: NLP Machine Learning Pattern recognition, Learning, Extraction, Classification, Enrichment, Linking, etc. Knowledge Graph (KG) KG Construction Open Source Data
(Graph Data Model) Challenges: Big Data Analytics 43 Big Data Analytics benefits from: NLP Machine Learning Pattern recognition, Learning, Extraction, Classification, Enrichment, Linking, etc. Knowledge Graph (KG) KG Construction Open Source Data
Big Data Leadership!! Industry has been in the lead Google, Amazon, Yahoo!, etc. University researchers have been left behind!! due to lack of access to large-scale cluster computing facilities Government agencies are making heavy investments Investments in big-data computing will have extraordinary near-term and long-term benefits. Cloud computing must be considered a strategic resource 44
Big Data: Opportunities 45 Varieties of Data Text Social Media Networks Multimedia Machine Data Sensors Analytics Organizing Big Data Navigating through data Summarizing Big Data Process Analytics Support decision-making Integration Integrating enterprise and public data Linking data/context Entity Extraction and Integration Knowledge Graph Big Data Performance In memory New Benchmarks and Architecture User Experience automation and intelligent guidance Visualizing with Analytics Interacting with Analytics Storytelling
Big Data: Opportunities 46 Varieties of Data Text Social Media Networks Multimedia Machine Data Sensors Analytics Organizing Big Data Navigating through data Summarizing Big Data Process Analytics Support decision-making Integration Integrating enterprise and public data Linking data/context Entity Extraction and Integration Knowledge Graph Big Data Performance In memory New Benchmarks and Architecture User Experience automation and intelligent guidance Visualizing with Analytics Interacting with Analytics Storytelling
Conclusion Why Big Data is different from past Very Large Datasets? Meta-Data!! 47 Having the ability to analyse Big Data is of limited value if users cannot understand the analysis. How can the industry and academia collaborate towards solving Big Data challenges!! What is big today maybe not be big tomorrow!
Questions / Suggestions 48