Decoding the Big Data Deluge a Virtual Approach. Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco

Transcription

1 Decoding the Big Data Deluge a Virtual Approach Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco

2 High-volume, velocity and variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.

3 What Changed? 2011 Internet-connected things: 15+ billion permanent 50+ billion intermittent 2020 Internet-connected things: 30+ billion permanent >200 billion intermittent Content & Services via Connected Products Biometrics Sensors & Devices Building & Infrastructure Management 3

4 What Are Your Big Data Plans? Have Invested Investing within next year Investing within two years No Plans Don t Know 4

5 Big Data Integration Challenge

6 Traditional Data Warehouse One Place to Go for Data Business View of Data Business Intelligence Data Warehouses &Marts Operational Stores ETL Transactional Applications

7 Challenges Accelerating Business Demand Proliferation of Distributed Silos Self-Service Analytics Business Intelligence Dashboard, Portals Many Different Access Protocols Data Warehouses &Marts Operational Stores Big Data & NoSQL Analytic Stores ETL Transactional Applications SaaS Applications Web Services

8 Analytic Stores Big Data & NoSQL Self-Service Analytics Transactional Applications Data Warehouses Business Intelligence SaaS Applications Operational Stores Executive Dashboard Data Marts Data Marts Data Marts Web Services

9 Logical Data Warehouse One Logical Place to Go for Data Business View of Data Self-Service Analytics Business Intelligence Cisco Data Virtualization Dashboard, Portals Abstract Federate Cache Big Data & NoSQL Analytic Stores Operational Stores Data Warehouses &Marts Transactional Applications SaaS Applications Web Services

10 Cisco Data Virtualization Business Intelligence Analytics Cisco Data Virtualization Structured Data Big Data Cloud Data

11 What is Cisco DV, Why is it Unique, Why is it Better? Cisco DV is agile data integration software that makes it easy to access data, no matter where it resides, and query it across the network as if it is in a single place. The ability to derive real value from a data virtualization platform is dependent upon Federated query optimization Breadth of data source systems Breath of consuming applications Enterprise security, scalability & manageability Network level query optimization

12 Cisco Data Virtualization Better Business Outcomes, Faster, for Less Business Intelligence Immediate Access Business Intelligence/Analytics Cisco Data Virtualization 5-10x Faster Analytics Up to 75% Cost Savings Higher Impact More Agile Less Expensive

13 Benefits of Data Virtualization: Business Agility Reduced development effort makes it easier to adapt to changes Get to solving the business problem sooner in the project timeline Same data does not have to be re-modeled for representation in a different DBMS Application programmers don t have to design data movement code; custom data movement code or ETL code doesn t have to be tested and supported Application designers can focus on solving the business problem Let data virtualization worry about locality, format, etc.

14 Benefits of Data Virtualization: Cost Reduction Duplicate storage eliminated Duplicate storage management also eliminated Data can be moved just when needed; dropped when no longer needed automatically Development effort for application-level data copying eliminated Data access achieved by configuration, not programming Also, Faster Time to Data Reduces Op Ex

15 Data Warehouse Optimization (DWO) As data volume increases, cost of warehousing grows substantially Combine Hadoop data with DW data for a more comprehensive view of history Data Virtualization Platform 3 4 Add operational data for greater insight and agility in analytics and BI 1 2 Offload data to less expensive Hadoop cluster to save on data management costs HDFS HDFS HDFS

16 Cisco Data Virtualization Data Abstraction Reference Architecture Layered Architecture View BI Server Customer Customer Purchase Orders ESB/BPM Orders Order Orders Details App Server Customer Customer Order Buyer Order Products XML Data Consumers Application Layer Business Layer Physical Layer Data Sources Customers DB Purchasing DB Orders DB Product Catalog

17 Data Virtualization Runtime Production Steps 1. Application invokes request 2. Optimized query (single statement) executes Cisco Information Server 3. Deliver data in proper form Benefits Low latency data Optimized performance Less replication required

18 Query Optimization Optimal Query Plan (pre-fetch) Auto SQL code generation Cost-based optimizer using statistics Rule-based optimizer using user knowledge Minimum Network Impact (fetch) On-the-fly plan regeneration Push down joins leverage source optimizers Minimal data retrieval necessary to complete request Streaming results Streaming-XML technology Maximum Join Efficiency (post-fetch) Auto-selection of fastest join operators Cisco Information Server

19 Data Virtualization Architectural Overview Cisco Data Virtualization Platform Development Environment Business Directory SQL (ODBC, JDBC, ADO.NET) Front-end Applications Cisco Information Server Web Services (HTTP, REST, SOAP, JSON, OData) Messaging (JMS) Hadoop (Input Format) Management Environment Manager Discovery Views, SQLScript (Database Centric) Security Metadata Deployment Manager Studio Caching Federated Query Optimization Data Source Capabilities Monitor Security Adapters SQL (ODBC, JDBC) Web Services (REST, SOAP) Messaging (JMS) URI Hadoop Java MF Adapter Application APIs Active Cluster XML Applications Big Data Stores Excel Files Flat Files Mainframes Cloud Messages XML Docs RDBMS Web Services

20 Key Takeaways 1. Big is here or its coming 2. Big Data requires new skills 3. Big Data is not a replacement for the traditional DWH 4. Big Data often results in one (or more) new data silos 5. Often times the users wants to augment Big data with enterprise data or vice versa 6. Data Virtualization provides a way to quickly integrate Big Data with enterprise data in a manner that is consistent with your current EIM practices 7. Everyone wins