Introduction to Big Data and the Lambda Architecture

Transcription

1 Introduction to Big Data and the Lambda Architecture Marc Schöni Meinrad Weiss April 2014 BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA 1

2 What is Big Data, why do we care? 2

3 The world of data has changed 10x increase every five years 4.3 connected devices per adult 85% from new data types Data explosion Consumerization of IT 27% using social media input By 2015, organizations integrating high-value, diverse, new information types and sources into a coherent information management infrastructure will outperform their industry peers financially by more than 20%. Gartner, Regina Casonato et al., Information Management in the 21st Century 3

4 Big Data 4

5 Personal Corporate BI Team BI Big Data PDW Polybase SQL Hive HDInsight Fasttrack Appliances ODBC OLE DB C# Web Services Azure Cloud xvelocity BISM SSAS SQL SERVER 2012 Analysis Services DAX SSRS Reporting Services Tabular Column Store Index Sharepoint Partitioning VB Skript MDX SSIS SQL Server Data Tools Integration Services TSQL UDM PowerPivot Maps PowerView Office Excel Access SelfService BI PerformancePoint 5

6 Big Data Big data solutions deal with complexities of: VOLUME (Size) VELOCITY (Speed) VARIETY (Structure) 6 VALUE Hadoop/HDInsight

7 Big Data Patterns Petabytes Terabytes Gigabytes Megabytes Data Complexity: Variety and Velocity 7 Hadoop/HDInsight

8 The modern data warehouse Data sources Non-Relational Data 8

9 Traditional DW/BI Environment ETL Data Warehouse

10 Tomorrows DW/BI Environment ETL Data Warehouse Social Networks Sensor Data Log Data Business Critical HDInsight 10 RFID Data Automated Data

11 Zookeeper Ambari What is Hadoop? HCatalog Oozie HBase/Cassandra/Couch/ MongoDB Hive Mahout R Cascading Pig Flume Sqoop HBase (column DB) Avro Hadoop = MapReduce + HDFS Distributed, scalable system on commodity hardware composed of: HDFS distributed file system MapReduce programming model Others: HBase, R, Pig, Hive, Flume, Mahout, Avro, Zookeeper 11

12 Hadoop capabilities Extract Load Transform Distributed Compute Predictive Analysis Machine Learning Graph Processing 12

13 Hadoop is not C# A replacement for Data Warehouse A place to learn how to code A place for low latency data 13

14 Limitations: Analysis with Big Data today Steep learning curve, slow and inefficient Hadoop ecosystem Move HDFS into the warehouse before analysis Learn new skills SQL Build Integrate Manage Maintain Support ETL 14

15 Microsoft Hadoop Vision Better on Windows and Azure Active Directory System Center.Net Programmability Microsoft Data Connectivity SQL Server / SQL Parallel Data Warehouse Azure Storage / Azure Data Market Microsoft Business Intelligence (BI) Hive ODBC Connectivity BI Tools for Big Data Collaborate with and Contribute to OSS Collaborate with HortonWorks Provide improvements and Windows support back to OSS "Microsoft is far ahead of everyone else in terms of what they're contributing back to the community" Hortonworks Founder and Architect Arun Murthy 15

16 Big Data Lambda Architecture 16

17 Big Data Lambda Architecture Batch layer Stores master dataset Compute arbitrary views Speed layer Fast, incremental algorithms Batch layer eventually overrides speed layer Serving layer Random access to batch views Updated by batch layer 17

18 The Batch Layer Stores master dataset (in append mode) Unrestrained computation Horizontally scalable High latency 18

19 The Speed Layer Stream processing of data Stores a limited window of data Dynamic computation 19

20 The Serving Layer Queries the batch and real-time views Merges the results 20

21 Microsoft Lambda Architecture Support 21

22 E.g. STRUCTURED & UNSTRUCTURED DATA Extremely large volume of unstructured web logs Ad hoc analysis of logs to prototype patterns Hadoop data cluster feeds large 24TB cube Business users analyze cube data 22

23 Apache Hadoop Staging Database SQL Server Analysis Service (SSAS) Microsoft Excel and PowerPivot Other BI Tools and Custom Applications SQL Server Connector (Hadoop Hive ODBC) Hadoop Data Third Party Database SQL Server Analysis Services (SSAS Cube) 23 + Custom Applications

24 Questions? 25