BIG DATA & the Data Warehouse

Transcription

1 25568 Genesee Trail Rd Golden, Colorado (303) Data Vault Modeling and Approach DW2.0 and Unstructured Data Master Data Management and Metadata BIG DATA & the Data Warehouse 2012 Genesee Academy, LLC Genesee Trail Rd Golden, Colorado Hans Hultgren 2012 Genesee Academy, LLC

2 BIG DATA and the Data Warehouse WHAT TO DO WHEN THE DATA WAREHOUSE MEETS HUGE VOLUMES OF RAPIDLY ARRIVING & SHAPE- SHIFTING DATA Asser-on What it Means

3 About BIG DATA Typical Data Big Data Typical Data Huge Data Volumes v v A v v v v B n- Structured & Very Complex v v C Streaming & Shape- ShiBing

4 Big Data and architecture Big Data solu-ons are separate from the EDW solu-on

5 IBM View IBM note on Architecture:

6 Oracle View Oracle note on Architecture:

7 Teradata View Teradata note on Architecture:

8 MicrosoB View MicrosoB note on Architecture:

9 Big Data and the EDW today Big Data solu-ons are separate from the EDW solu-on Architectures see Big Data components as Separate layers for other forms of analy-cs Ini-al landing areas (persisted and shared) Pre- processing layers becoming sources to EDW Data pools for integrated or hybrid downstream Marts (repor-ng) The main factors defining the differences for the two layers include Schema- on- Write versus Schema- on- Read Model- driven versus Data- driven analy-cs Model- based seman-cs versus Metadata- based seman-cs All- Data versus Selected Data- on- Demand

10 BIG DATA and the Data Warehouse Asser-on All EDW Data is n- structured What it Means Dealing with n- structured data is not op-onal for enterprise data warehouse programs.

11 BIG DATA and the Data Warehouse Asser-on All EDW Data is n- structured Full EDW data integra-on is impossible What it Means Dealing with n- structured data is not op-onal for enterprise data warehouse programs. Seman-c Integra-on is the only meaningful integra-on. The EDW can address integra-on to a point, then alignment and reconcilia-on.

12 BIG DATA and the Data Warehouse Asser-on All EDW Data is n- structured Full EDW data integra-on is impossible All EDW BI is Fuzzy BI What it Means Dealing with n- structured data is not op-onal for enterprise data warehouse programs. Seman-c Integra-on is the only meaningful integra-on. The EDW can address integra-on to a point, then alignment and reconcilia-on. With parsing, business rules- based logic, and interpre-ve (subjec-ve) transforms, all downstream EDW BI is fuzzy BI.

13 BIG DATA and the Data Warehouse Asser-on All EDW Data is n- structured Full EDW data integra-on is impossible All EDW BI is Fuzzy BI For the EDW, Big Data equals Data What it Means Dealing with n- structured data is not op-onal for enterprise data warehouse programs. Seman-c Integra-on is the only meaningful integra-on. The EDW can address integra-on to a point, then alignment and reconcilia-on. With parsing, business rules- based logic, and interpre-ve (subjec-ve) transforms, all downstream EDW BI is fuzzy BI. The true EDW architecture sees Big Data in much the same way as all Data. So Big Data tools and techniques are applicable to the en-re EDW.

14 BIG DATA and the Data Warehouse Asser-on All EDW Data is n- structured Full EDW data integra-on is impossible All EDW BI is Fuzzy BI For the EDW, Big Data equals Data The EDW and Big Data can live together What it Means Dealing with n- structured data is not op-onal for enterprise data warehouse programs. Seman-c Integra-on is the only meaningful integra-on. The EDW can address integra-on to a point, then alignment and reconcilia-on. With parsing, business rules- based logic, and interpre-ve (subjec-ve) transforms, all downstream EDW BI is fuzzy BI. The true EDW architecture sees Big Data in much the same way as all Data. So Big Data tools and techniques are applicable to the en-re EDW. Future Big Data solu-ons and EDW programs can be deployed on a common architecture. Historized metadata layers will enable solu-ons.

15 EDW & Big Data: Integrated Architecture Historized Semantic Integration Metadata Source Stage Integrated Architecture Marts Pool EDW FAS BB BNYM Manual Kurre TCM Other

16 EDW & Big Data: Integrated Architecture Historized Semantic Integration Metadata Source Stage Integrated Architecture Marts Pool EDW FAS BB BNYM Manual Kurre TCM Other

17 EDW & Big Data: Integrated Architecture Historized Semantic Integration Metadata Source Stage Integrated Architecture Marts Pool EDW FAS BB BNYM Manual Kurre TCM Other

18 Integrated Architecture Modeling PaYern Ensemble Modeling Unified Decomposi-on Data Vault Modeling

19 Data Vault Model

20 Links and Informa-on Data Vault Cer-fica-on Course December Sydney Register Today Book Launch Modeling the Agile Data Warehouse with Data Vault Hans Hultgren Twitter: gohansgo Hanshultgren.wordpress.com YouTube: DataVaultAcademy Online, on-demand training DataVaultAcademy.com