News and trends in Data Warehouse Automation, Big Data and BI. Johan Hendrickx & Dirk Vermeiren

Transcription

1 News and trends in Data Warehouse Automation, Big Data and BI Johan Hendrickx & Dirk Vermeiren

2 Extreme Agility from Source to Analysis DWH Appliances & DWH Automation

3 Typical Architecture 3

4 What Business wants : A DWH that delivers business value very fast. Agility A DWH that can change with the business. Extensibility A DWH that can scale with the business. Scalability 4

5 What Business gets : It takes a year to deliver the first business value. When I want new data it takes months to years to get it. Every query and/or Dashboard needs specific tuning to get good performance. 5

6 Appliance & DWH Automation Agility TWS DWH Accelerator by generating the Data Structure and ETL (DWH Automation). Appliance performance to allow querying on the foundation (integration) layer Extensibility Data Vault modeling delivers the ability to extend the data model. DWH Accelerator will generate the code for the new data entities Scalability Appliances to deliver agile performance and scalabillity (No complex tuning required) 6

7 Agility 7

8 Traditional approach 8

9 DWH Accelerator & Exadata/Exalitics Generated data models/code/mappings 9

10 DWH Accelerator Based on Accelerator Metadata (Generated and manual input) Generates Data Vault model (90% generated) Based on source model Generates Inflow Transformations (80% to 90% generated) for Incremental loads of : Hub objects Satellite objects Link objects Generate Inflow for incremental extraction of the Data Integration area towards the data exploitation area (100% generated) 10

11 DWH Accelerator Metadata driven Accelerator used in other environments Generator has proven support of other models, for example Marketing Data Platform project based on Banking Data Warehouse data model at ING, 82% of Data Integration mappings are generated Quote ING Belgium Tripwire has been a real asset for ING, through their expert knowledge, experience and passion for data warehousing. They participated in a struggling project and caused a turning point by introducing a generic approach including automated code generators. 11

12 Flexibility 12

13 Data Vault Model, Components Customer Relationship history Product Order 13

14 Support Incremental Approach S Fase 2 S H Fase 1 H L S L H S H S L H S No Changes to existing model 14

15 A comparison Data Vault modeling Pro Extensible model because of externalization of keys & relations. Standardized approach supports generation of data model & ETL code Fully solved model delivers good extract performance Con Multiplies the number of entities by a factor 3 minimum compared to 3NF. 15

16 Combined offer Oracle Appliances & DWH accelerator Agility Extensibility Scalability Simplicity Speed 16

17 17

18 Value of Engineered Systems 00:00 00:01 00:02 Enterprise-Scale Performance Faster Time to Production Reduce Costs 18 Copyright 2012, Oracle and/or its affiliates. All rights reserved. 18

19 Oracle Engineered Systems Hardware and Software, Engineered to Work Together 19

20 introducing EXADATA X3 DATABASE IN-MEMORY MACHINE 20

21 Oracle Exadata X X Warehouse OLTP & VLDB Scale-Up Massive Flash All I/Os to Flash Disk-Centric Flash Acceleration Flash-Centric DRAM Acceleration 21

22 Oracle Exadata X3 Database In-Memory Machine Same price as Oracle Exadata X2 4X Larger Flash Memory 20X More Write Performance 33% More Data Throughput 10% to 30% Lower Power 33% Faster Database CPUs 75% More Memory All X3 Database Machines X3-2 Database Servers 22 TB of Flash Memory per Rack Exadata Smart Flash Write Caching 100 GB/sec running SQL Up to 3 Kilowatt Reduction per Rack 8-Core Xeon SandyBridge E TB to 2 TB per Rack Full 10Gb 10% Ethernet to 20% to Lower Data Center Power 40X 10Gb ports per Rack 22

23 Oracle Exadata X3 Database In-Memory Machine Mass Memory for Extreme Performance 2 or 4 TB DRAM 22 TB PCI FLASH 500 TB DISK Hottest Data Active Data Cold Data X3 mass memory hierarchy delivers extreme performance Automatically moves all active data from disk to memory DRAM memory expanded to 2 or 4 TB for hottest data 4 to 40 TB of compressed user data Flash memory expanded 4X to 22 TB per rack 40 to 200 TB of compressed user data ALL active data 1.5 Million SQL random read I/Os per second for OLTP Comparable to 15,000 disk drives in 150 array frames 100 GB/sec SQL data scan rate for reporting and warehouses Comparable to 1,000 disk drives in 10 array frames 23

24 Oracle Exadata X3 Database In-Memory Machine Oracle Exadata Smart Flash Cache Write-Back New software caches write I/Os in flash memory 1 Million 8K write IOPS from SQL Now all reads and writes of active data go to flash 1 Million 8K random write I/Os per second on X3 Comparable to 1000 disks on 100 array frames Half-million random writes on existing V2 and X2 systems Cache is persistent and highly available Faster restarts 24

25 Hardware + Software Scalability on Demand Architecture Start Small, expand your Database Machine as you grow Growth HW upgrade 1 HW Start config HW upgrade 2 HW = EIGHTH Rack QUARTER Rack 2 DB Nodes (SW ½ CPU Cores) 3 Storage Server (SW ½ Disks) 2.4 TB Flash 4,5 TB (*1) 23 TB (*2) 2 DB Nodes 3 Storage Server 4.8 TB Flash 9,5 TB (*1) 48 TB (*2) HALF Rack 4 DB Nodes 7 Storage Server 11.2 TB Flash 22,5 TB (*1) 112 TB (*2) FULL Rack 8 DB Nodes 14 Storage Server 22.4 TB Flash 45 TB (*1) 224 TB (*2) (*) Available Usable Capacity for DB with NO Compression usage with (*1)High Performance disks/ (*2)High Capacity disks : 25

26 26 Copyright 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 13 26

27 Driving Better Business Outcomes Exalytics Big Data Applianc e Exadata 27

28 Oracle Big Data Appliance Big Data for Big Data processing Servers, RAM, Storage/ Rack; easy rack expansion NoSQL, Cloudera Hadoop, Oracle R Oracle Loader, Oracle Data Integrator, HDFS Connector for integration Integrates into your architecture Streams data into TB/hour 28

29 Oracle Exadata Your Data Database server pool Storage server pool InfiniBand network faster load and query times storage savings, less power, and lot less space Lower data center costs through consolidation 29

30 Why & How Visualize the Best Course of Action? FASTER ACTION Update Forecasts Daily or Hourly Analyze & Act in Real-Time What-if Analysis 30

31 Oracle Exalytics In-Memory Machine Predict Discover Plan Analyze In-Memory Processing Responsiveness And Throughput Columnar Storage & Compression Frequently Used Data is Sub-Second Interactivity Desktop And Mobile User Experience Tightly integrated with Exadata and Big Data Appliance 31

32 Oracle Endeca Information Discovery Understand the complete picture with context from any source. BI Star Schemas! Enterprise Content Management! Oracle Endeca Websites Enterprise Applications! How do we avoid costly product recalls? Social Media! 32

33 Oracle Exalytics with Information Discovery Fastest, most powerful hardware platform for Unstructured Analytics In-Memory BI, EPM, and Unstructured Analytics! Designed for Performance Endeca Software Breakthroughs Multi-Core Parallel Query Evaluation In-Memory Columnar Storage and Dynamic Cache Adaptive In-Memory Data Mapping Embedded Search Indices Free Form Search, Exploration, and Analysis Optimized Hardware Processor: 4 Intel Xeon E7-4870, 40 cores total Memory: 1 Terabyte DRAM Hard Disk: 3.6 Terabytes 33

34 Oracle Business Analytics Drive Better Business Outcomes Exalytics Big Data Appliance Exadata 34