M6P. TDWI Data Warehouse Automation: Better, Faster, Cheaper You Can Have It All. Mark Peco

Transcription

1 M6P European TDWI Conference with June 22 24, 2015 MOC Munich / Germany TDWI Data Warehouse Automation: Better, Faster, Cheaper You Can Have It All Mark Peco TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

2 Better, Faster, Cheaper... You Can Have It All TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

3 COURSE OBJECTIVES You will learn about: Concepts, principles, and practices of Data Warehouse Automation The current state of Data Warehouse Automation technology Automation opportunities and benefits when building or managing a data warehouse How to get started with Data Warehouse Automation Best practices and mistakes to avoid with Data Warehouse Automation TDWI takes pride in the educational soundness and technical accuracy of all of our courses. Please send us your comments we d like to hear from you. Address your feedback to: [email protected] Publication Date: February 2015 Copyright 2015 by TDWI. All rights reserved. No part of this document may be reproduced in any form, or by any means, without written permission from The Data Warehousing Institute. ii TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

4 TABLE OF CONTENTS Module 1 Data Warehouse Automation Concepts and Principles Module 2 Building and Managing the Data Warehouse Module 3 Using Data Warehouse Automation Module 4 Data Warehouse Automation in Action Module 5 Getting Started with Data Warehouse Automation Appx A Bibliography and References... A-1 Appx B Data Warehouse Automation Platforms... B-1 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY iii

5 iv TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

6 Data Warehouse Automation Concepts and Principles Module 1 Data Warehouse Automation Concepts and Principles Topic Page Data Warehouse Automation Basics 1-2 Why Data Warehouse Automation? 1-10 The Foundation 1-16 Activities and Deliverables 1-22 The Technology Landscape 1-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-1

7 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Data Warehouse Automation Basics Data Warehouse Automation Defined 1-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

8 Data Warehouse Automation Concepts and Principles Data Warehouse Automation Basics Data Warehouse Automation Defined COVERING THE WAREHOUSING LIFECYCLE CHANGING BEST PRACTICES Data warehouse automation is more than simply automation of ETL development, or even the entire development process. It encompasses the entire data warehousing lifecycle from planning, analysis, and design through development and extending into operations, maintenance, and change management. Adoption of data warehouse automation changes the way that we think about building data warehouses. The widely accepted best practice of extensive up-front analysis, design, and modeling can be left behind as the mindset changes from get it right the first time to develop fast and develop frequently an approach that is aligned with today s agile development practices. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-3

9 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Data Warehouse Automation Basics Automated vs. Hand-Crafted 1-4 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

10 Data Warehouse Automation Concepts and Principles Data Warehouse Automation Basics Automated vs. Hand-Crafted BUILD IT BETTER, FASTER, AND AT LOWER COST CHANGE WHEN AND AS NEEDED WHY HANDCRAFT? Automation in data warehousing has many of the same benefits as in manufacturing: Increased productivity and speed of production Reduction of manual effort Improved quality and consistency Better controls and process optimization opportunities The manufacturing parallel hold when building a data warehouse; we can think of it as an information factory. But data warehousing is more complex than product manufacturing. Manufactured products are typically delivered to a consumer and the job is done. Data warehouses must be sustained through a long life cycle where changes in source data, business requirements, and underlying technologies are ongoing considerations. Automation helps to implement the right changes in the right ways and as quickly as they are needed. There are, of course, popular arguments in favor of handcrafting. In consumer goods, for example, advocates of hand made use terms such as unique, personal, and human touch. These are all valid reasons to buy a hand made scarf, but are they qualities that are needed in a data warehouse? TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-5

11 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Data Warehouse Automation Basics Automation vs. Customization 1-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

12 Data Warehouse Automation Concepts and Principles Data Warehouse Automation Basics Automation vs. Customization NOT MUTUALLY EXCLUSIVE Automation does not preclude customization, nor does it imply mass production. Neither is practical in data warehousing as most data warehouses need to have some level of customization. The purpose of customizing is to meet unique and specific needs that are likely to exist for every data warehouse. Strategic and business-critical uses typically have greater need for customizing than routine and repetitive uses. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-7

13 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Data Warehouse Automation Basics Data Warehouse Automation vs. CASE 1-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

14 Data Warehouse Automation Concepts and Principles Data Warehouse Automation Basics Data Warehouse Automation vs. CASE SMART AUTOMATION Some may recall the popularity of Computer Aided Systems Engineering (CASE) in the 1980 s. The purpose of CASE was to automatically generate program code and database schema from models and specifications collected and stored in a repository. The objective was to fully automate the process of systems development. The labor-intensive activities shifted from programming to analysis/design/specification with the goal of populating the repository. Methodology was a predominant factor and systems quality almost a singular goal. Data warehouse automation is distinctly different from CASE. The goal is to automate that which is smart to automate not necessarily to automate everything. Analysis can reasonably be accelerated with requirements discovery and design optimized by use of design patterns. Methodology is applied to support the processes, not to define and dominate them. And the goals include speed, quality, efficiency, effectiveness, and adaptability. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-9

15 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Why Data Warehouse Automation? Business Benefits 1-10 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

16 Data Warehouse Automation Concepts and Principles Why Data Warehouse Automation? Business Benefits QUALITY AND EFFECTIVENESS AGILE BUSINESS SPEED COST SAVINGS Data warehouse automation delivers quality and effectiveness through ability to build better solutions. Better solutions are those that best meet real business requirements, and it is especially difficult to get complete and correct requirements when limited to an early phase of a linear development process. With data warehouse automation the business can make changes much later in the development process and change can occur more frequently with less disruption, waste, and rework. Ability to change fast and frequently extends beyond the warehouse development process. Changes that occur in business requirements can be met with quick response. Responding to change in real time and without the delay of lengthy projects is the essence of business agility. Speed is the critical factor that enables agility both for agile business and for agile development. Ability to generate quickly and to regenerate equally fast when change occurs are fundamental automation capabilities. Ultimately building better, building faster, and changing quickly when needed bring substantial cost savings to data warehouse development, operation, maintenance, and evolution. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-11

17 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Why Data Warehouse Automation? Technical Benefits 1-12 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

18 Data Warehouse Automation Concepts and Principles Why Data Warehouse Automation? Technical Benefits THE VALUE FOR IT ORGANIZATIONS The benefits of automation aren t solely for business. IT organizations also derive real value from data warehouse automation in all of the ways that are illustrated on the facing page. The first four items agility, time savings, adaptability and cost savings are important and valuable for developers. The remaining part of the list documentation, impact analysis, testing, and maintainability arguably have greater impact because they apply beyond the development project and throughout the life of the data warehouse. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-13

19 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Why Data Warehouse Automation? Barriers and Resistance 1-14 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

20 Data Warehouse Automation Concepts and Principles Why Data Warehouse Automation? Barriers and Resistance REASONS FOR RESISTANCE PROCESS BARRIERS AND CONSTRAINTS HUMAN BARRIERS AND CONSTRAINTS Despite the obvious benefits, adoption of data warehouse automation is not a given. You may encounter resistance from two perspectives processes that aren t automation ready and people who are reluctant to change. Be prepared to adjust and adapt your data warehousing processes development, operation, and maintenance when implementing data warehouse automation. Processes that are designed to minimize risk in non-automated environments create conflict and limit the benefits of automation. We ll discuss the process considerations of change management in Module 5: Getting Started with Data Warehouse Automation. People will resist automation for a variety of reasons that are generally associated with fear that they will lose something loss of job, loss of status, loss of control, etc. We ll discuss the human aspects of change management in Module 5: Getting Started with Data Warehouse Automation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-15

21 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation The Foundation Components of Data Warehousing 1-16 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

22 Data Warehouse Automation Concepts and Principles The Foundation Components of Data Warehousing DATA FLOW FROM DISPARATE DATA TO INTEGRATED INFORMATION The facing page illustrates the core elements of data warehousing beginning with disparate data sources at the bottom of the diagram and leading to integrated information resources at the top. The information resources aren t necessarily the end of the line they are simply the end of the data integration processes. Business value is created when they are used for reporting, business intelligence, decision-making, analytics, etc. The center of the diagram shows the processing steps to get from data sources to integrated information using two methods data consolidation through extract-transform-load processing and data virtualization through a series of abstract data views. Consolidation and virtualization are often combined to optimize the flow of data. Data warehouse automation can implement both approaches independently or in a mix-and-match form. ALL OF THE PARTS Data warehousing encompasses many different techniques and produces many components to enable the data-to-information flow. Among these are architectural standards, data models, mappings, data transformations, database load procedures, tests and controls, events and errors, and metadata. Data warehouse automation includes capabilities to create, connect, manage, and apply all of these components. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-17

23 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation The Foundation Data Warehousing Projects 1-18 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

24 Data Warehouse Automation Concepts and Principles The Foundation Data Warehousing Projects PROGRAM VS. PROJECT Data warehousing is more than just a project or a series of projects. It involves many projects within a program context that also includes evolving architecture and ongoing operations. Projects are iterative, timebounded activities to produce specific data warehousing capabilities. The program is a continuous investment of resources that encompasses projects, operations, and value creation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-19

25 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation The Foundation Design Patterns 1-20 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

26 Data Warehouse Automation Concepts and Principles The Foundation Design Patterns ENCAPSULATING STANDARDS AND CONVENTIONS As described earlier, one of the benefits of automation is accelerated through use of design patterns. Design patterns can encapsulate architectural standards as well as best practices for data design, data management, data integration, and data usage. The facing page identifies common design patterns in all of these categories. Creating and applying patterns in a data warehouse automation tool supports the goal of accelerated design, but more importantly it drives compliance with standards and consistency of data warehousing results. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-21

27 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Activities and Deliverables Data Warehousing Processes 1-22 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

28 Data Warehouse Automation Concepts and Principles Activities and Deliverables Data Warehousing Processes LABOR INTENSITY, COMPLEXITY, AND VOLATILITY A typical data warehouse development process is burdened by laborintensive activities, a high degree of complexity and dependency among deliverables, and volatility of data and requirements. Some of the data warehousing pain that can be relieved by automation includes: Slow, laborious, and difficult to get it right requirement gathering Interdependency between source data analysis and warehouse data modeling Manually mapping sources to targets Detailed specification of data transformation logic Translating data models into schema and building databases Hand coding and manual testing of ETL processing Preparing and loading initial start-up data Acceptance testing and deployment to production These activities would be difficult enough in a linear process, but in reality data warehousing projects are never linear. We re always going back because: Data modeling may change requirements Source-target mapping may affect data modeling Data transformation design may change mappings Integration process design may affect database design Incremental testing may cause changes throughout Acceptance testing may discover incorrect requirements and so on... When we survive all of these things and deploy the data warehouse it doesn t remain stable. Business needs, source data, and technology will all change (at different times and speeds) driving continuous change in the warehouse. And through it all we need to manage projects, track changes, manage code and system versions, isolate production from development and testing environments, sustain daily operations, and try to keep metadata and documentation synchronized with what is actually implemented. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-23

29 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Activities and Deliverables What Can You Automate? 1-24 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

30 Data Warehouse Automation Concepts and Principles Activities and Deliverables What Can You Automate? USING DATA WAREHOUSE AUTOMATION Data warehouse automation is capable of relieving much of the pain discussed on the previous pages. Nearly everything in the diagram on the facing page can be automated. All automation tools support core activities such as data modeling and data integration. The most robust automation products support all of the activities. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-25

31 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation The Technology Landscape Automation Tools and Vendors 1-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

32 Data Warehouse Automation Concepts and Principles The Technology Landscape Automation Tools and Vendors COMMON TOOLS The facing page illustrates the four most common tools for data warehouse automation Attunity Compose, Magnitude Software Kalido, timextender, and WhereScape. These are the pure DWA product vendors. Others, such as Birst, include some automation capabilities in a broader product suite. Later in the course you ll see more about the approaches and capabilities of Attunity Compose, Magnitude Software Kalido, and WhereScape each with different perspectives and capabilities. The intent is not to compare the tools or to perform product evaluation, but to show a representative sample of mainstream DWA technology. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-27

33 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation 1-28 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

34 Building and Managing the Data Warehouse Module 2 Building and Managing the Data Warehouse Topic Page The Data Warehousing Lifecycle 2-2 Building the Data Warehouse 2-22 Managing the Data Warehouse 2-30 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-1

35 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle A Closed Loop 2-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

36 Building and Managing the Data Warehouse The Data Warehousing Lifecycle A Closed Loop CONTINUOUS WITHOUT AN EXIT POINT The diagram on the facing page revisits the data warehousing lifecycle with emphasis on the blending of program and project work. The important introductory concept is that the lifecycle is a closed-loop system. There is no exit point in the diagram and no end to continuous evolution, change, and redevelopment. This is the fundamental reason that conventional design-and-build cycles don t work well for data warehousing. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-3

37 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Architecture and Planning 2-4 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

38 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Architecture and Planning ARCHITECTURE DELIVERABLES The facing page expands the architecture phase of the data warehousing lifecycle to illustrate common deliverables of architecture and planning including: Program Charter describing goals, purpose, resources, governance, sponsorship, etc. Readiness Assessment identifying strengths, weaknesses, risks for the program Business Architecture identifying stakeholders and their interests, business applications, and business processes to be affected Organization Architecture describing the people, purpose, processes, and structure of warehouse development, deployment, operation, and application Data Architecture with standards for structure, storage, distribution, access, etc. Integration Architecture describing the standards and patterns for flow of data from disparate sources, to integrated information, and access by data consumers Process Architecture with standards, conventions, and patterns for data sources, mappings, and integration procedures TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-5

39 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Architecture and Planning - Preparation 2-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

40 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Architecture and Planning - Preparation AUTOMATION BENEFITS Preparation activities of charter and readiness assessment can get value from automation in several ways. Automation helps to: Maximize finances Achieve goals Meet expectations Optimize people Automate processes Increase capabilities TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-7

41 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Architecture and Planning - Architecture 2-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

42 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Architecture and Planning - Architecture AUTOMATION BENEFITS Architectural activities can use automation to: Engage stakeholders Capture business rules Focus on purpose Encapsulate data standards Encapsulate integration standards Automate DW methodology TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-9

43 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Implementation 2-10 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

44 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Implementation IMPLEMENTATION DELIVERABLES The facing page illustrates high-level categories of deliverables from data warehouse implementation projects including: Source Data Understanding describing content, meaning, and quality of data sources Target Data Models describing the organization and structure of warehouse data Source-Target Mappings associating warehouse data fields with the sources from which they are populated Data Acquisition Processes procedures to obtain data from sources Data Transform Processes procedures to change the data to achieve data warehousing characteristics (integrated, subject oriented, time variant, non-volatile, and sometimes dimensional) Database Load Processes procedures to load data into warehouse databases Warehousing Databases the physical databases in which warehouse data is stored TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-11

45 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Implementation Analysis and Design 2-12 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

46 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Implementation Analysis and Design AUTOMATION BENEFITS The analysis and design activities of implementation projects source data understanding, target data modeling, and source-target mapping use automation for: Profiling and exploration of data sources Stakeholder driven requirements Applied data standards & templates Data modeling & schema generation Encapsulation of integration standards Metadata driven source-target mapping TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-13

47 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Implementation Design and Build 2-14 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

48 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Implementation Design and Build AUTOMATION BENEFITS The design and build activities of implementation projects building processes and databases use automation to: Generate processes & procedures Sequence & link processes & procedures Test processes & procedures Generate database schema TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-15

49 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Deployment and Operations 2-16 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

50 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Deployment and Operations DEPLOYMENT AND OPERATIONS DELIVERABLES The facing page illustrates deliverables and activities of data warehouse deployment and operations including: Managed Environments separating development, testing, and production environments Acceptance Testing confirming that the implementation meets business requirements and formalizing acceptance as a production system Transfer to Operations transfer of knowledge, responsibility, and accountability from development to operations team User Training and Support enabling effective use with the right knowledge, skills, and support structure Scheduling and Execution ensuring that data warehousing processes run when and as they should with attention to dependencies on source systems and among warehousing processes Maintenance and Versioning making small changes and repairs with complete change history and version management Change and Evolution identifying and initiating architectural changes and new projects due to changed data sources, business requirements, or technology capabilities TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-17

51 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Deployment and Operations Production Implementation 2-18 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

52 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Deployment and Operations Production Implementation AUTOMATION BENEFITS The production implementation activities use automation for: Incremental testing Data warehouse versioning Environment management and migration Documentation and metadata especially as applied for initial training of operations staff and users TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-19

53 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Deployment and Operations Execution and Maintenance 2-20 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

54 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Deployment and Operations Execution and Maintenance AUTOMATION BENEFITS The execution and maintenance activities use automation for: Incremental testing Source & version management Scheduling Validation testing Documentation and metadata especially as applied for troubleshooting, investigation, and ongoing training Impact analysis TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-21

55 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Building the Data Warehouse Requirements Driven Development 2-22 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

56 Building and Managing the Data Warehouse Building the Data Warehouse Requirements Driven Development A TRADITIONAL APPROACH Requirements driven development is the traditional process that begins with requirements gathering and proceeds in a linear fashion through analysis, modeling and design, specification and coding, and deployment. This approach is a legacy of conventional software development processes that doesn t work well for data warehousing. It is characterized by: Intensive and detailed planning Long development timelines Limited business participation early in the process Substantial staffing and skills demand Difficulty in requirements gathering and analysis Much cycling back to previous steps to correct errors and oversights Much waste and rework Post-deployment discovery of incorrect and missed requirements TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-23

57 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Building the Data Warehouse Modeling and Metadata Driven Development 2-24 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

58 Building and Managing the Data Warehouse Building the Data Warehouse Modeling and Metadata Driven Development A DESIGN APPROACH Model and metadata driven development focuses on capturing business requirements and translating them to a design that is recorded as models and metadata. The models and metadata are then used in conjunction with design patterns to generate data warehouse deliverables. This approach is an incremental improvement over requirements-driven development in ability to build models and collect metadata that are useful for many related projects. Model and metadata driven development naturally implies some degree of automation. It is characterized by: Need for stated business requirements and well defined scope Development timeline that correlates with project scope Business participation early and late in the process Specialized staffing and skills to perform critical modeling tasks Evolving granularity of requirements through model development Some cycling back to previous steps as a result of model refinement Iterative development with many related and dependent projects TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-25

59 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Building the Data Warehouse Discovery Driven Development 2-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

60 Building and Managing the Data Warehouse Building the Data Warehouse Discovery Driven Development A LEARNING APPROACH Discovery driven development acknowledges the problems of traditional requirements approaches in data warehousing, replacing them with the concept of requirements through discovery. Fuzzy requirements become clear through cycles of prototyping and refinement of requirements. This approach is a substantial shift away from requirements first approaches. It is highly dependent on automation to enable rapid cycles of prototyping, and is characterized by: Projects of small to modest scope with fuzzy requirements Accelerated development timelines achieved through automation Extensive business participation throughout the prototyping activities and again at acceptance and deployment Demand for data warehouse automation skills and business subject expertise Discovery of requirements through prototyping and evaluation Some rework when inevitable source data surprises occur Multiple projects and iterative development TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-27

61 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Building the Data Warehouse Data Driven Development 2-28 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

62 Building and Managing the Data Warehouse Building the Data Warehouse Data Driven Development AN AGILE APPROACH Data driven development adapts the discovery approach to recognize a reality of data warehousing that source data is as challenging as requirements gathering and is often filled with surprises. Combining data exploration with prototyping and requirements discovery creates a truly agile process for data warehousing. Rework caused by unexpected conditions in source data is eliminated by early data exploration. Automation is a must for this approach which is characterized by: Many projects of small scope appropriate for agile development Accelerated development achieved through automation Extensive business participation throughout the entire development process Demand for data warehouse automation skills, agile development mindset, and business subject expertise Discovery of business requirements through prototyping Discovery of data requirements through data exploration Elimination of rework with agile development practices Many small projects and iterative development cycles TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-29

63 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Managing the Data Warehouse Operations 2-30 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

64 Building and Managing the Data Warehouse Managing the Data Warehouse Operation AUTOMATION FOR OPERATIONS Operations comprises a core set of activities in managing a data warehouse, with attention to: Sequencing Dependencies Scheduling Execution Verification Validation Error Handling Automation aids data warehouse operations with features and functions for: Scheduling Documentation & Metadata Managed Environments Validation Testing TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-31

65 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Managing the Data Warehouse Monitoring and Tuning 2-32 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

66 Building and Managing the Data Warehouse Managing the Data Warehouse Monitoring and Tuning AUTOMATION FOR MONITORING AND TUNING Monitoring and tuning are essential activities of managing a data warehouse, with attention to: Usage Performance Availability Access Security Growth Capacity Automation aids data warehouse monitoring and tuning with features and functions for: Applied Standards Documentation & Metadata Managed Environments Iterative Testing TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-33

67 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Managing the Data Warehouse Maintenance and Change 2-34 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

68 Building and Managing the Data Warehouse Managing the Data Warehouse Maintenance and Change AUTOMATION FOR WAREHOUSE MAINTENANCE AND CHANGE The need for maintenance and change is inevitable when managing a data warehouse. Specific activities include: Tracing And Troubleshooting Bug Fixes Data Correction Error Recovery Upgrades Enhancements Automation aids data warehouse maintenance and change management with features and functions for: Applied Standards Impact Analysis Documentation & Metadata Managed Environments Iterative Testing Versioning & Source Control TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-35

69 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Managing the Data Warehouse Evolution and Change 2-36 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

70 Building and Managing the Data Warehouse Managing the Data Warehouse Evolution and Change AUTOMATION FOR WAREHOUSE EVOLUTION Evolution is a natural part of the data warehouse lifecycle driven by ongoing change and needs for: Extension Expansion Migration Modernization Re-Architecting Re-Sourcing Refactoring Automation offers many features and functions to support data warehouse evolution including: Applied Standards Impact Analysis Documentation & Metadata Data Source Profiling Managed Environments Data Warehouse Versioning Encapsulated Standards Schema Generation Iterative Testing Versioning & Source Control TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-37

71 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Managing the Data Warehouse Support 2-38 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

72 Building and Managing the Data Warehouse Managing the Data Warehouse Support AUTOMATION FOR WAREHOUSE SUPPORG Support is critical to success and sustainability of every data warehouse. Automation aids data warehouse support with features and functions for: Applied Standards Documentation & Metadata Managed Environments Versioning & Change Control TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-39

73 Building and Managing the Data Warehouse TDWI Data Warehouse Automation 2-40 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

74 Using Data Warehouse Automation Module 3 Using Data Warehouse Automation Topic Page Automation Use Cases 3-2 Case Studies 3-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-1

75 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Building a New Data Warehouse 3-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

76 Using Data Warehouse Automation Automation Use Cases Building a New Data Warehouse SCENARIO CHALLENGES OPPORTUNITIES You need to build a new data warehouse either where none exists or as a complete replacement of an existing and dysfunctional warehouse. Without automation, all of the normal data warehousing challenges exist. Source data is messy, warehouses are hard to build, they take too long to build, and they re obsolete before they are deployed. Automation opportunities are abundant in this scenario. You can automate everything from planning to deployment and operation using any of model-driven, discovery-driven, or data-driven approaches. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-3

77 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Building Dependent Data Marts 3-4 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

78 Using Data Warehouse Automation Automation Use Cases Building Dependent Data Marts SCENARIO CHALLENGES OPPORTUNITIES You have a data warehouse but you need to build dependent data marts sourced from that warehouse. Sometimes the data marts are star-schema or cubes and sometimes non-dimensional reporting databases. Data marts are user facing so requirements are critical, but requirements are as difficult for data marts as for data warehouses. Handcrafting of schema and ETL is time-consuming, labor-intensive, and likely to require rework due to unclear requirements. Automation is a natural fit for this scenario. Overcome the requirements challenges with discovery-driven or data-driven development and accelerate the process with automated schema and process generation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-5

79 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Building Dimensional Data Marts 3-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

80 Using Data Warehouse Automation Automation Use Cases Building Dimensional Data Marts SCENARIO CHALLENGES OPPORTUNITIES You need to build dimensional data marts that are sourced directly from operational systems without using a hub data warehouse. This scenario often involves high demand with every business function and many work groups wanting data marts quickly. Matching marts to business needs and source data with mart requirements is difficult. Handcrafted star-schema and ETL processing is too slow to meet demand when the business wants more and faster. This is another natural fit for automation. Any of model-driven, discovery-driven, or data-driven approaches step up to the challenges of business alignment, and automated schema and ETL generation help to meet the demand to deliver more and faster. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-7

81 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Conforming Independent Data Marts 3-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

82 Using Data Warehouse Automation Automation Use Cases Conforming Independent Data Marts SCENARIO CHALLENGES OPPORTUNITIES You have many independent data marts that were built without considering the need to conform the dimensions. Now the business wants the ability to drill across and perform analysis that involves more than one data mart. This scenario is challenged to analyze and understand the differences among variations of each dimension, to establish conformity standards for each dimension, to analyze the impact of dimension conformity on each existing data mart, and to rebuild the data marts with dimension standards applied. Automatically documenting the data marts is a good first step to analyze and understand the differences. Standards for conformed dimensions can be encapsulated as design patterns in an automation tool. When the tool is populated with metadata about the existing data marts (a necessary step in automating documentation) then analyzing the impact of change can also be aided by automation. Ultimately data-driven or model-driven techniques can be used to rebuild the data marts with conformed dimensions. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-9

83 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Data Warehouse Prototyping 3-10 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

84 Using Data Warehouse Automation Automation Use Cases Data Warehouse Prototyping SCENARIO CHALLENGES OPPORTUNITIES You need to engage the business and understand requirements before committing time and resources to data warehouse development. And you re working with business users whose perspective on requirements is I ll know what I need when I see it! Prototyping doesn t work without business participation, and it doesn t work with long lag time between prototype building, prototype evaluation, and the next cycle. Automation and prototyping fit together quite naturally. Both discoverydriven and data-driven approaches make it possible to produce visible and tangible results quickly, and to apply the findings and feedback of evaluation in the spirit of rapid prototyping. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-11

85 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Agile Data Warehousing 3-12 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

86 Using Data Warehouse Automation Automation Use Cases Agile Data Warehousing SCENARIO CHALLENGES OPPORTUNITIES You need all of the benefits of agile development fast, focused, and business driven when building the data warehouse. But agile data seems to be much harder to achieve than agile processing. Not only is agile data harder than agile process, but agile techniques are challenged by constraints of the existing data warehouse and of the source data. You need engaged business subject experts, very fast development cycles, and the ability to make the data tangible as requirements come to the surface. Handcrafted schema and ETL is too slow for agile. The agile value of working software over comprehensive documentation is a concern. You need to be agile without sacrificing the documentation and metadata that are essential to warehouse operations, maintenance, usage, and evolution. Automation enables agile data warehousing by supporting very fast development and redevelopment cycles while capturing essential metadata as part of the process. Existing warehouse constraints are easily incorporated, and with a data-driven approach source data constraints are also easily addressed. Generated schema and ETL, quick changes, and versioning all help when applying agile methods to data warehouse development. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-13

87 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Data Warehouse Migration 3-14 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

88 Using Data Warehouse Automation Automation Use Cases Data Warehouse Migration SCENARIO CHALLENGES OPPORTUNITIES You have an existing data warehouse that is functionally sound but needs to be moved to a new technology platform. Business and logical views of the data warehouse are sound, but the physical differences between databases and data integration platforms are a consideration. Only the physical metadata is reliable because conceptual and logical models and designs haven t kept pace with changes to the existing warehouse. Having the warehouse offline for more than a very brief period is not acceptable. Manually rebuilding schema and data integration processes will take too long and is likely to be error-prone. In addition to moving the schema and processes to a new platform, you also need to move the data. Overcome the inadequacies of documentation by populating the automation tool with physical metadata from the old warehouse and then extracting the design. Rebuild schema and processes for the new platform by generating them with the automation tool. Map old schema to new schema, then build and execute data migration processes to move the data. A quick turnaround from old platform to new, with automated testing and validation, minimizes downtime that the warehouse is not available for business use. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-15

89 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Re-Architecting the Data Warehouse 3-16 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

90 Using Data Warehouse Automation Automation Use Cases Re-Architecting the Data Warehouse SCENARIO Your data warehouse needs a comprehensive architectural update perhaps to overcome long-standing architectural deficiencies, to align with today s best practices, to meet real time data demands, to incorporate unstructured data, or for a variety of other reasons. CHALLENGES OPPORTUNITIES The challenges are similar to those for data warehouse migration. The significant difference is that the architecture not the platform is changing. Obsolete documentation, downtime constraints, and the time and cost to rebuild manually are real challenges. But these challenges are compounded by the probability that re-architecting will bring structural and standards changes that have greater impact than technology platform changes. Migration opportunities to automate apply here too. Extracting the design, generating schema and process, automating data migration, and automating testing and validation are all valuable. New opportunities also exist to embed architectural structures as design patterns and to encapsulate architectural standards as part of automation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-17

91 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Warehouse Management Use Cases 3-18 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

92 Using Data Warehouse Automation Automation Use Cases Warehouse Management Use Cases SCENARIO CHALLENGES OPPORTUNITIES Data warehouse operation is challenged by poorly understood data structures, data mappings, and data transformation logic. Documentation is incomplete and inaccurate. Complex schedules are a challenge and managing dependencies with source systems and among warehousing processes is difficult. It is difficult to make a business case and get a sponsor to commit time, funding, and resources for retroactive documentation of an existing data warehouse. Manually documenting is labor intensive, error prone, and a lengthy project that is working against a moving target. Use automation to quickly extract the data structures, data mappings, and transformation rules, then apply that metadata to retroactively document the existing data warehouse. Expand and enrich the metadata by adding schedule and dependency information as a one-time documenting effort that will pay dividends in automating data warehouse operation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-19

93 Using Data Warehouse Automation TDWI Data Warehouse Automation Case Studies Arch Insurance Group 3-20 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

94 Using Data Warehouse Automation Case Studies Arch Insurance Group INSURANCE COMPLEXITIES Arch Insurance Group is a specialty insurer providing property, casualty, and specialty insurance for corporations, professional firms and financial institutions across the United States and Canada. Their complex business includes large volumes of data, regulatory pressures, and retroactive events and data related to claims. The data warehouse is central to both management decisions and regulatory compliance. Keeping pace with volatility and managing time variance impacts of retroactive data became data warehouse quality concerns. Attunity Compose automates the data warehouse process allowing business and regulatory changes to be applied to a business model, and then propagated to changes in the data warehouse. Attunity Compose s history of history method resolves the difficulties of retroactive data in the warehouse. Automation enables Arch to meet the challenges of regulatory changes, eliminates quality issues from retroactive data, and helps them adapt to continuing business changes. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-21

95 Using Data Warehouse Automation TDWI Data Warehouse Automation Case Studies British American Tobacco 3-22 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

96 Using Data Warehouse Automation Case Studies British America Tobacco COMPETITIVE BUSINESS AND POST-MERGER INTEGRATION British American Tobacco is the second largest listed tobacco company in the world with brands sold in more than 180 markets worldwide. As with any consumer goods company, the global market for British American Tobacco s products are fiercely competitive, hugely volatile and require a localized approach to each market. In this competitive environment they simultaneously faced a merger and a company-wide SAP implementation. High quality information to retain their competitive position in the face of extreme volatility was a real concern. British American Tobacco worked with Magnitude Software Kalido, using automation to provide critical, actionable market share information and to standardize management practices for business-critical information. Automation made it possible to do this quickly and costeffectively while adapting to change and sustaining a high level of quality. The solution enables them to sustain competitive position through brand management, operational planning, marketing efficiencies, and postmerger reporting that are information dependent. Additional benefits of automation include agility of data warehousing processes and substantial reduction of maintenance costs. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-23

97 Using Data Warehouse Automation TDWI Data Warehouse Automation Case Studies F5 Networks 3-24 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

98 Using Data Warehouse Automation Case Studies F5 Networks SALES-DRIVEN COMPANY NEEDS GOOD DATA This company is the global leader in application delivery networking. They make network appliances that work as the backbone of IT infrastructure in large companies. They are a heavily sales driven organization and consequently need data to help manage sales effectiveness. This was traditionally done via spreadsheets where data from Salesforce.com, the billing systems and the installation/service return systems were manually consolidated each week. This was a very manually intensive process and prone to time delays and data errors. There was no ability to perform any serious analytics such as trending without a mammoth effort by analysts. The WhereScape Red solution pulled data from Salesforce.com, the billing systems and the installation/service return systems and consolidated this into fact tables around sales opportunities, sales/billing, installed products and services and servicing returns. This for the first time enabled the company to gain a full picture of each customer and potential customer and allowed the sales management hierarchy to have visibility of the sales pipeline. Within the first month after go-live, a sales rep made a $1.5 million sale using the new analytics to uncover an opportunity that would have otherwise not been found. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-25

99 Using Data Warehouse Automation TDWI Data Warehouse Automation Case Studies Lamar 3-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

100 Using Data Warehouse Automation Case Studies Lamar LOTS OF DATA BUT NOT ENOUGH INFORMATION This outdoor advertising company owns (among other things) electronic billboards. Each billboard has a computer in the back that sends a file each day to head office with the playlog what advert played when, in eight-second increments. This produces a huge volume of data, and the existing system was old, slow and had data integrity issues. It was also not scalable so as more billboards were added the system was unable to cope. There was also a project to upgrade the software on the billboards, which would result in a richer file being sent back. This new file was unable to be loaded by the old system. There are around 3000 billboards sending data back each day. The WhereScape Red solution created a process to load all the files (of both new and old formats) rapidly into the new datamart (SQL Server). A detail fact table was created to hold 90 days of playlogs and an aggregate fact table was built to provide all-history analytics. 3 months from start to finish. The company is now able to provide much better analytics to their clients about when their ads played compared to contracted plays. It also allows them to identify new opportunities to sell advertising spots. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-27

101 Using Data Warehouse Automation TDWI Data Warehouse Automation Case Studies ProShop 3-28 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

102 Using Data Warehouse Automation Case Studies ProShop RETAIL PRICING AND STOCKING INFORMATION ProShop is an online retailer of consumer electronics located in Denmark. For many years they built up a SQL Server database with steadily increasing amounts of useful information about sales, customers and much more. But it was expensive a lot of costly consultant hours were used to retrieve this information from the system so it could be used to set the right prices and have the right products on the shelves. Costly consultants were replaced with a timextender and Excel solution, giving a fast overview of the data and an up-to-date basis for decision making. The solution uses timextender to move data from SQL Server to a data cube from which Excel retrieves all the data. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-29

103 Using Data Warehouse Automation TDWI Data Warehouse Automation Case Studies Shell Oil 3-30 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

104 Using Data Warehouse Automation Case Studies Shell Oil GLOBAL SEGMENTATION WITHOUT COMPROMISING LOCALIZATION Shell Oil Products, the worldwide fuel and lubricants businesses within the Royal Dutch/Shell Group, have segmented their global markets bringing vital information on customers and marketing initiatives. The solution brings clarity and speed to performance measurement, and produces harmonized, segmented management information, helping Shell market their products more effectively. Shell needed to segment their global business to gain understanding of products, customers and channel profitability. They also needed to monitor the performance of global marketing initiatives without affecting the local organization of operating units. The semi-autonomous nature of constituent local companies meant that there was no common data standard for management information, making global customer and product management time-consuming, inaccurate and inefficient. Business analysts were spending much of their time gathering and classifying data, and very little time actually analyzing it. Using Magnitude Software Kalido, Shell has linked management information systems across 120 separate operational units, creating a truly global view of customer and product data for the first time. The solution brings together management information to support standardization and segmentation, as part of the broader Oil Products Management Information System. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-31

105 Using Data Warehouse Automation TDWI Data Warehouse Automation Case Studies Union Bank 3-32 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

106 Using Data Warehouse Automation Case Studies Union Bank INTEGRATING DATA AND ENABLING ANALYSIS Union Bank was formed as a consolidation of several California and Japanese banks. As a result of acquisition of subsidiaries and of new federal banking laws, the Consumer Lending department needed a consolidated, clean source of data about consumer lending. Previously this data was stored piecemeal around the bank in different databases. Ability to perform timely, high quality analyses was non-existent. The WhereScape Red solution provided a centralized, consolidated, cleansed set of data with a comprehensive set of lending KPIs. This comprised fact tables and cubes for daily loan balances, loan origination (and life of loan history. For the first time, executives can perform a variety of analytics around consumer Lending in a very timely manner, with no delay while analysts prepare data and reports to answer ad hoc requests all these are selfserviced via the cubes. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-33

107 Using Data Warehouse Automation TDWI Data Warehouse Automation 3-34 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

108 Data Warehouse Automation in Action Module 4 Data Warehouse Automation in Action Topic Page Attunity Compose in Action 4-2 Magnitude Software Kalido in Action 4-4 TimeXtender in Action 4-6 WhereScape in Action 4-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 4-1

109 Data Warehouse Automation in Action TDWI Data Warehouse Automation Attunity Compose in Action Product Overview 4-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

110 Data Warehouse Automation in Action Attunity Compose in Action Product Overview A BLACK BOX APPROACH The facing page illustrates a comparison of typical data warehouse architecture and the Attunity Compose architecture. The core of Attunity Compose s automation approach is a business model that drives data staging and business views into the data. Everything between source data and business views is treated as a black box where data transformations and integration processes are determined by the business model perspective. BIReady will generate either CIF (Inmon-sytle) or Data Vault data warehouses and then build star-schema data marts as views of the warehouse. The data marts may be either virtual or physical. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 4-3

111 Data Warehouse Automation in Action TDWI Data Warehouse Automation Magnitude Software Kalido in Action Product Overview 4-4 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

112 Data Warehouse Automation in Action Magnitude Software Kalido in Action Product Overview THE SCOPE OF AUTOMATION This is a high level list of all the processes / components that Magnitude Software Kalido automates. The Business Information Model drives all of this automation. The core is to look at common repetitive technical tasks, componentize them and then automate them. This allows the project team to focus on solving the business problem rather than focusing on the mundane technical tasks that need to be accomplished to produce a scalable, consistent, efficient system. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 4-5

113 Data Warehouse Automation in Action TDWI Data Warehouse Automation TimeXtender in Action Product Overview 4-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

114 Data Warehouse Automation in Action TimeXtender in Action Product Overview PRODUCT FEATURES The facing page illustrates the core features of TimeXtender selecting data from multiple sources, automating warehouse generation processes, and automating deployment and execution processes. Note that TimeXtender supports a variety of data sources but limits cube generation to Microsoft SQL Server. TimeXtender also highlights features for rule-based processing, flexible data loads, reverse engineering, maintaining history and versions, data quality checking, a multi-user environment, manual controls, and OLAP write-back capability. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 4-7

115 Data Warehouse Automation in Action TDWI Data Warehouse Automation WhereScape in Action Products Overview 4-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

116 Data Warehouse Automation in Action WhereScape in Action Products Overview PLAN WITH 3D BUILD WITH RED 3D is the planning tool of the WhereScape toolset. It supports ability to: Discover, profile, explore and document any potential source system for a data warehousing project, including detailed examination of source data at the table and column levels Design, populate and test any target data warehouse schema, whether normal form, dimensional, data vault or hybrid Implement, test and populate any purchased enterprise data model in the planned target environment Perform a complete source-to-target mapping between profiled and documented source systems, and designed target schema Test planned schema, populated with real data, for functionality and coverage, with end users, before building activities commence View, manipulate and associate conceptual and logical views of the proposed data warehouse or data mart Capture user stories, interviews and all other requirements artefacts as an integral part of the planning process, in a managed repository Capture the design rationale for the new environment, while the project team is in planning Generate complete project documentation, for governance, funding or internal communications 3D is the building tool of the WhereScape toolset. It supports ability to: Automate the entire data warehousing life cycle, from design, through implementation, operation and renovation. We provide ELT facilities, but we re the market s first and best IDE for the data warehouse build process, end to end. Utilize the integrated team-aware metadata repository for all code, audit and workflow components, with full versioning, code promotion, impact analysis tools (trackback and track-forward features), metadata search and automated documentation production. Support agile as well as traditional project approaches, including Live Prototyping - rapid prototyped and fully functional data warehouses in hours, for user reviews and feasibility testing. Provide industry best practices out-of-the-box, including generation of surrogate keys, time dimensions, and target indices. Deliver full extraction, load and transformation (ELT) facilities, with integrated dependency management and scheduling. Generate code automatically, with complete editing freedom post TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 4-9

117 Data Warehouse Automation in Action TDWI Data Warehouse Automation WhereScape in Action WhereScape RED 4-10 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

118 Data Warehouse Automation in Action WhereScape in Action WhereScape RED COMPONENTS OF RED The facing page illustrates the technical architecture of RED. At the center is the data warehouse comprising databases, metadata, scripts, procedures, and tasks. Developers, administrators, and operations work with RED through the corresponding clients desktop, setup administrator, and scheduler management. Runtime operations are driven by the scheduler, which encompasses timing, sequencing, and dependencies of tasks. The scheduler may optionally be connected with an enterprise scheduling system. Source data, of course, is fundamental to data warehousing as is the ability to produce warehouse dependent objects such as cubes and data exports. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 4-11

119 Data Warehouse Automation in Action TDWI Data Warehouse Automation 4-12 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

120 Getting Started with Data Warehouse Automation Module 5 Getting Started with Data Warehouse Automation Topic Page Step by Step 5-2 Human Factors 5-4 New Horizons 5-6 Next Steps 5-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 5-1

121 Getting Started with Data Warehouse Automation TDWI Data Warehouse Automation Step by Step A Process for Automation 5-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

122 Getting Started with Data Warehouse Automation Step by Step A Process for Automation BUSINESS CASE SPONSORSHIP AND BUY-IN SHORT LIST PROOF OF CONCEPT TECHNOLOGY SELECTION ORGANIZATIONAL CHANGE TRAINING AND IMPLEMENTATION Build the business case for data warehouse automation based on business benefits, not technical benefits. Focus on value creating benefits such as speed, agility, solution quality, and cost savings. Seek a sponsor who can secure the funding, resources, and political will to drive data warehouse automation. Then identify other key stakeholders and work to secure their buy-in. Identify the candidate list of vendors and products that fit your needs, constraints, and culture. When developing the short list, also identify the criteria that you ll use to make a final selection. Identify one or two proof-of-concept projects and ask each vendor on the short list to illustrate how they will do the work. Before executing the projects establish a baseline for comparison either by: (1) automating something that you ve already done and have known time and cost to build it manually, or (2) by building something new for which you ve estimated time and cost to build manually. Based on proof of concept results and previously defined selection criteria, make your technology decision and install the automation tools. Recognize the need for and undertake the necessary organizational changes. The processes, roles, responsibilities, and team configurations for manual data warehousing are not what you ll want for automation. Train people to use the automation tools, and train them to understand their new roles and responsibilities with data warehouse automation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 5-3

123 Getting Started with Data Warehouse Automation TDWI Data Warehouse Automation Human Factors People and Automation 5-4 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

124 Getting Started with Data Warehouse Automation Human Factors People and Automation GETTING PEOPLE ON BOARD Earlier in the course we discussed the reality of and reasons for resistance to automation. Most resistance is based in fear and uncertainty. Getting from resistance to adoption and impact demands that you acknowledge the resistance. Don t dismiss it, but recognize and acknowledge that automation brings change: Change in individual roles and responsibilities Change in team configuration and competencies Change in individual skills and competencies Be aware that desired change only occurs when it is managed, and that it doesn t happen all at once. Change is a process, not an event. Apply all of the proven organizational change management techniques to ensure that your people are ready for data warehouse automation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 5-5

125 Getting Started with Data Warehouse Automation TDWI Data Warehouse Automation New Horizons Automation and Data Warehousing Trends 5-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

126 Getting Started with Data Warehouse Automation New Horizons Automation and Data Warehousing Trends AUTOMATION AND AGILITY AUTOMATION AND BIG DATA Agile data warehousing is a hot topic today. It can be a motivator for the change manager who needs to overcome resistance. It can be a selling point when building your business case for automation. And most importantly it can become a reality for your data warehousing program when enabled with data warehouse automation. The buzz around big data is enormous perhaps louder than the agile data warehousing buzz. Data warehouse automation enables the changes and refactoring that may be necessary to bring big data into your warehouse. Big data is certainly a motivator for technical people and can be counted among the technical benefits. It may also be a business motivator if you ve established a business case to integrate big data into the warehouse. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 5-7

127 Getting Started with Data Warehouse Automation TDWI Data Warehouse Automation Next Steps Making Your Plans 5-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

128 Getting Started with Data Warehouse Automation Next Steps Making Your Plans DISCUSSION QUESTIONS So you ve learned a lot about data warehouse automation today. Is it for you and your organization? What will you do next? Some points for discussion as we wrap up the class, and when you get back to your office include: When to start? Who are the stakeholders Who should sponsor? Who are good change agents? What is your business case? What are the barriers? What are the risks? What are the opportunities? What are some potential POC projects? What impact on existing data warehouse? What impact on existing organizations? TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 5-9

129

130 Bibliography and References Appendix A Bibliography and References TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY A-1

131 Bibliography and References TDWI Data Warehouse Automation A-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

132 Bibliography and References Bibliography and References. Agile Data Warehousing, Hughes, iuniverse, Bloomington IN, 2008 Agile Data Warehouse Design, Corr and Stagnitto, DecisionOne Press, Leeds UK, 2011 Building the Data Warehouse (Second Edition), Inmon, John Wiley & Sons, New York NY, 1996 Data Model Patterns: Conventions of Thought, Hay, Dorset House Publishing, New York NY, 1996 Data Warehouse: From Architecture to Implementation, Devlin, Addison-Wesley Longman, Reading MA, 1997 The Data Warehouse Challenge: Taming Data Chaos, Brackett, John Wiley & Sons, New York NY, 1996 The Data Warehouse Lifecycle Toolkit, Kimball, John Wiley & Sons, New York NY, 1998 The websites, case studies, and white papers of Attunity Compose, Magnitude Software Kalido, timextender, and WhereScape. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY A-3

133

134 Data Warehouse Automation Platforms Appendix B Data Warehouse Automation Platforms TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-1

135 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Attunity Compose in Action Product Overview B-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

136 Data Warehouse Automation Platforms Attunity Compose in Action Product Overview A BLACK BOX APPROACH The facing page illustrates a comparison of typical data warehouse architecture and the Attunity Compose architecture. The core of Attunity Compose s automation approach is a business model that drives data staging and business views into the data. Everything between source data and business views is treated as a black box where data transformations and integration processes are determined by the business model perspective. Attunity Compose will generate either CIF (Inmon-style) or Data Vault data warehouses and then build star-schema data marts as views of the warehouse. The data marts may be either virtual or physical. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-3

137 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Attunity Compose in Action Business Model B-4 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

138 Data Warehouse Automation Platforms Attunity Compose in Action Business Model BUSINESS VIEW OF DATA This screen shot illustrates a part of the business model (in this case the business model was obtained by reverse engineering the Microsoft Northwind demo database). You are looking at the entity type Orders together with relationships to other entity types and attributes. For each attribute and/or relationship one can specify the type of history that has to be kept in the Data Warehouse. This is the only metadata that is required for the generation of the enterprise (central) data warehouse. In a more advanced mode some more design decisions can be indicated. For instance if one knows that some attributes are frequently changing, while others are only slowly changing. In that case it makes sense to store them physically in different tables. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-5

139 Data Warehouse Automation Platforms TDWI Data Warehouse Automation BI Ready in Action Data Mart Wizard B-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

140 Data Warehouse Automation Platforms BI Ready in Action Data Mart Wizard FROM WAREHOUSE TO DATA MARTS For the design of the Data Marts some additional design decisions have to be made. A wizard can support this process. First the fact tables have to be discovered (step 1). Second we establish the dimensions. The first proposal of the tool in this example is that there should be an Orders dimension and a Products dimension (and the rest should be denormalized into these two). With the latter (the Products dimension) we agree, but the Orders dimension seems to be too big a dimension. So we uncheck the Orders in this screen. Then the tool guesses that we probably want to have Customers, Employees and Shippers as dimensions. We judge this as an excellent idea, so we click Next. The remaining steps are less illustrative. Note that, leaving them unchecked, means that the entity types Suppliers and Categories will be denormalized into the Products dimension. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-7

141 Data Warehouse Automation Platforms TDWI Data Warehouse Automation BI Ready in Action Dimension and Facts Edit B-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

142 Data Warehouse Automation Platforms BI Ready in Action Dimension and Facts Edit REFINING DIMENSIONS After using the Data Mart wizard you can edit the metadata of the Data Marts even further. You can add remove and edit dimensions and/or fact tables. The transaction date for the facts can be altered and so on. You can see that there is apparently no need for contact title, Address or Country By right clicking on an attribute one can define a filter condition. In the example we are only interested in sales of product supplied by suppliers from the UK. This filter condition can be specified with the attribute Country. Attributes with a filter can be identified by the funnel symbol. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-9

143 Data Warehouse Automation Platforms TDWI Data Warehouse Automation BI Ready in Action Generated Star Schema B-10 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

144 Data Warehouse Automation Platforms BI Ready in Action Generated Star Schema THE DATA MART STRUCTURE The structure of the Data Mart that is generated for the Northwind case is shown here. It is generated in Microsoft SQL Server and shown via the diagram wizard of Microsoft SQL Server. Note that each dimension has a VERSION_ID as primary key. This is Data Warehouse Key (meaningless number) identifying one version of an object. This key (pointer!) is also the foreign key in the fact table (e.g. S_CUSTOMERS_VID). But both in the fact table and in the dimension tables we also have the OBJECT_ID. This is the data warehouse key identifying the object itself. Normally we will join the fact table and a dimension table by the version id, because most of the time we want the transactions together with the dimension object info as it was at the time of the transaction. But the object id s (which are already present in the Data Warehouse) allow ad hoc querying. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-11

145 Data Warehouse Automation Platforms TDWI Data Warehouse Automation BI Ready in Action Mapping B-12 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

146 Data Warehouse Automation Platforms BI Ready in Action Mapping CONNECTING SOURCE TO TARGET To support the mapping from real operational sources to the staging area, simple mappings can be defined and maintained. So ETL is supported and often this will suffice. Some situations the ETL functionality that comes with the database products (for example OWB for Oracle or MSIS for SQL Server) to populate the business staging tables. The example shows the case where we need one separate mapping only for the attribute ContactName. This is so, because in the source data an explicit date field ContactSince exists, which tells us as from which date the person is in charge as our contact within the customer. We want to use this date as effective date and we can accomplish this by drawing a line from ContactSince on the left hand side to the staging field From date TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-13

147 Data Warehouse Automation Platforms TDWI Data Warehouse Automation BI Ready in Action Process Instructions B-14 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

148 Data Warehouse Automation Platforms BI Ready in Action Process Instructions SPECIFYING THE PROCESSING The start of the generation of the process instructions is accompanied by a window as shown here. The process instructions are stored in tables. These tables are read and executed by the Process Engine. In the same screen you can indicate if you want to (re-) generate staging and/or logging tables. The barrel signs indicate that there are already tables present. That is why by default the (re-) generate checkboxes are unchecked. If they are white the tables are empty. If the barrel seems to be filled, then there are rows in the tables. If the color is green, it is no problem to drop the tables, because they contain derived data (like process instructions) which can be easily computed again. If the color is red, then you might lose information dropping the tables. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-15

149 Data Warehouse Automation Platforms TDWI Data Warehouse Automation BI Ready in Action Starting the Process B-16 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

150 Data Warehouse Automation Platforms BI Ready in Action Starting the Process PROCESS EXECUTION You can start a process online or just generate a command file, such that the processes can be scheduled overnight. Either way you can specify some parameters. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-17

151 Data Warehouse Automation Platforms TDWI Data Warehouse Automation BI Ready in Action Run Log B-18 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

152 Data Warehouse Automation Platforms BI Ready in Action Run Log AN AUDIT TRAIL Each run produces a report of what has happened in that run. You can drill drown in this screen to see more details, for instance the individual inserts or some examples of them. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-19

153 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Magnitude Software Kalido in Action Product Overview B-20 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

154 Data Warehouse Automation Platforms Magnitude Software Kalido in Action Product Overview THE SCOPE OF AUTOMATION This is a high level list of all the processes / components that Magnitude Software Kalido automates. All of this automation is driven by the Business Information Model. The core is to look at common repetitive technical tasks, componentize them and then automate them. This allows the project team to focus on solving the business problem rather than focusing on the mundane technical tasks that need to be accomplished to produce a scalable, consistent, efficient system. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-21

155 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Magnitude Software Kalido in Action Modeling the Solution B-22 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

156 Data Warehouse Automation Platforms Magnitude Software Kalido in Action Modeling the Solution MODELING OBJECTIVES THE BUSINESS INFORMATION MODEL Model real world Automatic support for ragged / unbalanced hierarchies Supertyping and subtyping Multi grain transactions Business entity definition management Data validation rules Magnitude Software Kalido uses the concept of a Business Information Model to capture the requirements for a solution. The Business Information Model represents how a business user (someone who makes business decisions based on data) describes their environment. The Business Information Model is not as detailed as a typical Logical ER model but not as high level as a typical Conceptual Model. It uses business terms so that business users can relate to the model resulting in a common language that both the business team as well as the technical implementation team use to solve the problem jointly. There are 2 major concepts captured by the Business Information Model. Measures that are contained within business events. These business events are given context by reference or master data. This modeling paradigm has numerous real world constructs to cater for common scenarios that are found in most organizations e.g. structures that automatically cater for ragged or unbalanced hierarchies, supertyping and subtyping, dealing with similar data that comes in at different granularity etc. The model also drives data integration. It explicitly caters for the fact that a single real world object can be identified in multiple different ways, treats associations as objects and expects granularity to change so the integration rules can be catered for directly in the model. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-23

157 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Magnitude Software Kalido in Action Model Validation and Impact Analysis B-24 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

158 Data Warehouse Automation Platforms Magnitude Software Kalido in Action Model Validation and Impact Analysis OBJECTIVES Built in intelligence to ensure model integrity System driven impact analysis to existing data & processes A DATA AWARE MODEL Since the model drives all operations it is uniquely aware of the data that flows through those operations. This means that the model is aware of the data that is currently loaded into the repository. This enables a wide range of validation to occur which would not be possible in a traditional modeling tool. The simplest example would be turning an optional association to mandatory. One can always accomplish this task in a modeling tool but it is only on deploying the model that the developer finds that there are existing records that do not conform to this stricter rule. Since the Business Information Model is aware of the data that resides in the repository the user is immediately made aware of the impact of the change. The system will not allow changes to be applied that will break the integrity of any existing rules. The user will be made aware of each of these cases and only once all cases have been dealt with will the change be allowed. In the example above a default would need to be supplied for the existing records that currently do not contain the association. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-25

159 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Magnitude Software Kalido in Action System Deploy B-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

160 Data Warehouse Automation Platforms Magnitude Software Kalido in Action System Deploy THE TECHNICAL VIEW The business user view of the Business Information Model intentionally hides the technical deployment details of the model. A more technical resource can then control the algorithms that are used to turn the Business Information Model into a deployed physical model. The software uses standard best practices to produce a range of physical models from normalized to denormalized. The user can guide this process which is typically driven by the combination of the database that is used for persistence and the reporting tools that is used for visualization. The unique power of having the physical model be generated and the loading into this model be automated is that the physical model can be changed post deploy for performance optimization. The user could start with a more normalized physical model and once they get into testing realize that they would get better performance by denormalizing a few objects. This process is done automatically not just for the physical schema but for the data that currently resides in that schema. This means that making the choice of what paradigm to use for the physical model shifts from being a pure design time decision to being closer to a runtime decision. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-27

161 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Magnitude Software Kalido in Action Loading Data B-28 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

162 Data Warehouse Automation Platforms Magnitude Software Kalido in Action Loading Data OBJECTIVES Delta Detection Key Management Time Variance Catering for Dependencies ELT architecture Exploits the native capabilities of the underlying platform THE LOAD PROCESS The loading process has a significant amount of automation. The largest change is that when a load is defined a physical source of data is mapped to the Business Information Model. It is not mapped to the physical schema. This has several benefits. Loads definitions stay consistent as the physical model is changed (normalization / denormalization) as the software will automatically reconfigure the load to fit the new physical structure Impact analysis is automatic when models change as the model is aware of all loads into the model The actual loading is abstracted. E.g. the source could be a single denormalized object that holds data for multiple Business Model objects. The user needs to perform just a single load mapping. The software however will decompose that and automatically configure individual loads for each of the multiple physical objects in the warehouse catering for any dependencies within that process Since the loads are driven by software automation if there is a new feature on a particular DB platform that could speed up the loading process that will automatically be applied to all existing load processes if applicable The load process automatically does key management, delta detection, insert vs. update and stores all data with attribute level time variance. None of that needs to be configured by the user and is a consistent automated process that will happen on all loads. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-29

163 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Magnitude Software Kalido in Action Data Validation and Suspense Processing B-30 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

164 Data Warehouse Automation Platforms Magnitude Software Kalido in Action Data Validation and Suspense Processing OBJECTIVES Automatic rules checking Automatic Suspense Enables Stewardship processes Ensures integrity of data DATA QUALITY The Business Information Model describes rules about the quality, completeness and consistency of data expected. These rules are automatically applied during the load process. Any data that does not conform to these rules are automatically moved into a suspense area. This allows the appropriate person to interact with the data to assess if the data was suspended due to the model being too strict and not catering for real world scenarios or due to the data really being poor enough that it would corrupt the repository if it were allowed in. There are automatic capabilities to rollback loads or to reprocess suspense e.g. due to late arriving reference data. This process ensures that there are no implicit nulls in the data loaded enabling all SQL joins to be inner. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-31

165 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Magnitude Software Kalido in Action Model-Driven Result Set Generation B-32 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

166 Data Warehouse Automation Platforms Magnitude Software Kalido in Action Model-Driven Result Set Generation FEATURES Currency Conversion Time Variance Resolve rollup paths Ensure integrity of data TOPIC The rich metadata in the model can be used to drive extracting data from the repository. A user can use the model to extract data avoiding having to write manual SQL that could end up being complex. E.g. The measure that is loaded into the repository holds currency values that are in the local currency of the transaction. The report needs a consistent reporting currency. The model understands this & will be able to extract data applying the appropriate currency conversion in the process Since all data stored in the repository is automatically time variant the user can select the time basis that they would like when extracting data (Current Hierarchy, Time of Transaction, Point in Time) Calculated measures can be defined just once & consistently used for multiple outputs Since the software is generating the SQL for the extract this means that the extract is abstracted from the physical schema. The software will therefore automatically update the extract if the underlying physical scheme changes. Since the model is aware of the extracts if model change is proposed it s impact on interfaces out can be immediately assessed. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-33

167 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Magnitude Software Kalido in Action Reporting Tool Semantic Synchronization B-34 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

168 Data Warehouse Automation Platforms Magnitude Software Kalido in Action Reporting Tool Semantic Synchronization THE BI LAYER The Business Information Model holds the businesses view of data as well as how that data is physically persisted. This is the core of most reporting tool semantic layers. The process of generating the semantic layer & keeping it up to date can therefore be automated. The richness of the model can be used to resolve loops and ensure that the integrity of the data is maintained in the semantic layer e.g. trying to ensure that Balances are not aggregated across time. Hierarchies can be defined and the point in time that data needs to be reported off configured. The team that manages the BI layer can therefore focus on the user experience rather than the plumbing of how to access data. The business user also sees consistency as the business terms that are defined in the Business Information Model are the same terms that are presented in the BI interface. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-35

169 Data Warehouse Automation Platforms TDWI Data Warehouse Automation Magnitude Software Kalido in Action Operations B-36 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

170 Data Warehouse Automation Platforms Magnitude Software Kalido in Action Operations FEATURES Consistent Logging Consistent Audit Consistent Return Codes System Driven Impact Analysis RUNNING THE DATA WAREHOUSE All warehouse operations from loading data to enabling the migration process from development to production are all driven by the Business Information Model. This means that users have full visibility into all operations and all operations occur consistently. All processes are logged in the same manner. All processes are audited in the same manner. There is operational consistency through the system. The developer does not need to focus on low value operational consistency but can shift into focusing on high value design consistency. Any improvements to processes are done at the application layer and catered for through the upgrade process. This frees up operational resources to focus on data issues rather than mundane updates to existing processes. The rich metadata of the model and process also means that issues are much easier to troubleshoot. The software can capture what portion of the model each process affects and what physical objects from landing through staging to the final interface out was used in the process. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-37

171 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Products Overview B-38 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

172 Data Warehouse Automation Platforms WhereScape in Action Products Overview PLAN WITH 3D 3D is the planning tool of the WhereScape toolset. It supports ability to: Discover, profile, explore and document any potential source system for a data warehousing project, including detailed examination of source data at the table and column levels Design, populate and test any target data warehouse schema, whether normal form, dimensional, data vault or hybrid Implement, test and populate any purchased enterprise data model in the planned target environment Perform a complete source-to-target mapping between profiled and documented source systems, and designed target schema Test planned schema, populated with real data, for functionality and coverage, with end users, before building activities commence View, manipulate and associate conceptual and logical views of the proposed data warehouse or data mart Capture user stories, interviews and all other requirements artefacts as an integral part of the planning process, in a managed repository Capture the design rationale for the new environment, while the project team is in planning Generate complete project documentation, for governance, funding or internal communications TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-39

173 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Products Overview B-40 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

174 Data Warehouse Automation Platforms WhereScape in Action Products Overview BUILD WITH RED 3D is the building tool of the WhereScape toolset. It supports ability to: Automate the entire data warehousing life cycle, from design, through implementation, operation and renovation. We provide ELT facilities, but we re the market s first and best IDE for the data warehouse build process, end to end. Utilize the integrated team-aware metadata repository for all code, audit and workflow components, with full versioning, code promotion, impact analysis tools (trackback and track-forward features), metadata search and automated documentation production. Support agile as well as traditional project approaches, including Live Prototyping - rapid prototyped and fully functional data warehouses in hours, for user reviews and feasibility testing. Provide industry best practices out-of-the-box, including generation of surrogate keys, time dimensions, and target indices. Deliver full extraction, load and transformation (ELT) facilities, with integrated dependency management and scheduling. Generate code automatically, with complete editing freedom post TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-41

175 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action WhereScape RED B-42 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

176 Data Warehouse Automation Platforms WhereScape in Action WhereScape RED COMPONENTS OF RED The facing page illustrates the technical architecture of RED. At the center is the data warehouse comprising databases, metadata, scripts, procedures, and tasks. Developers, administrators, and operations work with RED through the corresponding clients desktop, setup administrator, and scheduler management. Runtime operations are driven by the scheduler, which encompasses timing, sequencing, and dependencies of tasks. The scheduler may optionally be connected with an enterprise scheduling system. Source data, of course, is fundamental to data warehousing as is the ability to produce warehouse dependent objects such as cubes and data exports. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-43

177 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Building a Dimension with RED B-44 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

178 Data Warehouse Automation Platforms WhereScape in Action Building a Dimension with RED AN AGILE EXAMPLE TABLE NAMING While WhereScape RED is able to support many development approaches, it is particularly strong for agile development. For the purpose of this in action illustration we ll use an agile example. We begin by building a dimension. Using RED s data-driven approach, simply drag and drop the source table into the dimension panel of the desktop client. This triggers the process of creating a production quality dimension object. Defined naming standards are applied to the object. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-45

179 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Building a Dimension with RED B-46 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

180 Data Warehouse Automation Platforms WhereScape in Action Building a Dimension with RED COLUMN DEFINITION SYSTEM MAINTAINED COLUMNS INDEXING AND CONSTRAINTS The columns for the new dimension table are automatically defined based on metadata from the source table. Initial column definitions can be modified as needed. Additional columns not derived from the source table are added: dim_product_key is the surrogate key for the dimension dss_start_date and dss_end_date are date/time stamps required for type 2 slowly changing dimensions dss_current_flag is an indicator that is set to y in the row that is the current version of each dimension member dss_version shows the sequence of versions dss_update_time contains the last update date and time for the row Red automatically generates indexes and key constraints for primary and foreign keys. Indexing can be modified and additional indexes created as needed. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-47

181 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action ETL/ELT with RED B-48 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

182 Data Warehouse Automation Platforms WhereScape in Action ETL/ELT with RED MOVING THE DATA LOADING THE DIMENSION TRANSFORMING THE DATA Moving data from source to warehouse requires processing to load, transform, de-duplicate, cleanse, etc. Whether using ETL or ELT approach (both supported by RED) you need to generate the procedures to perform the processing. The pane at the top of the facing page shows the steps to create a table load function as a T*SQL stored procedure. The stored procedure will handle requirements of a type 2 dimension and will handle error conditions and messages. Data transformation is typically the largest part of the work in moving data from source to target. Transformations are easily defined for any column at any point in the load process. RED supports all native SQL functions. New columns can be defined to support derived data. Transformations become part of the update logic and are also included in the documentation components. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-49

183 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action ETL/ELT with RED B-50 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

184 Data Warehouse Automation Platforms WhereScape in Action ETL/ELT with RED BUSINESS KEYS SCD COLUMNS DENORMALIZING Business keys need to be identified so RED can create the logic to map them to surrogate keys in the warehouse. This is illustrated at the top of the facing page. Each column for which history is desired in a slowly changing dimension needs to be identified. The pane at the center of the facing page illustrates this capability in RED. It is common to join multiple tables when loading a warehouse table. RED detects when multiple source tables are required and provides the means to describe the join logic. This is illustrated at the bottom of the facing page. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-51

185 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Loading Warehouse Tables with RED B-52 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

186 Data Warehouse Automation Platforms WhereScape in Action Loading Warehouse Tables with RED INITIAL VS.INCREMENTAL LOADS SURROGATE KEY LOOKUPS THE CODE The logic that we ve built so far is sufficient to initially load the table, but ongoing loads require more. We need logic to detect changes and to incrementally and correctly update records in the dimension tables. We also need a fast and efficient way to handle surrogate keys. RED creates a stored procedure to meet that need. At the bottom of the facing page you can see a sample of the high-quality, standard code that is generated by RED. The code is open so it may be viewed and modified after it is generated. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-53

187 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action User-Driven Development B-54 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

188 Data Warehouse Automation Platforms WhereScape in Action User-Driven Development CYCLES OF CHANGE The process just illustrated is fast. It is equally quick to modify and regenerate the data warehouse. RED is particularly well suited to the many variations of user-driven development including rapid prototyping, requirements discovery, test driven development, and agile methodologies. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-55

189 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Database Refactoring B-56 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

190 Data Warehouse Automation Platforms WhereScape in Action Database Refactoring CHANGING THE DATABASE The quick changes of business logic, data transformations, etc. are readily handled by RED functions that you ve already seen. But some changes may require modifying the database as well as the code. RED s database refactoring capabilities support this requirement with the steps shown on the facing page: add and change columns as needed apply the changes to the intermediate staging and work tables validate metadata changes against existing tables to ensure consistency propagate the changes to the warehouse tables TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-57

191 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Automated Documentation B-58 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

192 Data Warehouse Automation Platforms WhereScape in Action Automated Documentation DOCUMENTATION MADE EASY SCHEMA DOCUMENTATION DATA SOURCE TRACKING Documentation is often the most overlooked activity of development, and it quickly gets out-of-sync with implementation when changes occur. RED overcomes these difficulties with automated documentation. As you develop the data warehouse RED tracks object properties and dependencies, join conditions, table relationships and other information needed to support and operate the warehouse. Using this metadata, creating documentation is a menu function in RED. Schema diagrams are created from metadata as shown at the center of the facing page. Similarly, data lineage diagrams are created from metadata as illustrated at the bottom of the page. This is especially valuable as the data warehouse grows in scope, functionality, and complexity. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-59

193 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Version Control B-60 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

194 Data Warehouse Automation Platforms WhereScape in Action Version Control KEEPING TRACK OF CHANGES Change is continuous throughout data warehousing, so change management is essential. Version control is a fundamental capability of change management. RED supports version control at four levels: Procedure versioning tracks changes in logic and stored procedures. Object versioning tracks changes for all data warehouse objects tables, columns, etc. Project versioning provides the ability to track changes for a group of objects and procedures at a project milestone. Data warehouse versioning provides ability to track changes at each deployment of the data warehouse. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-61

195 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Testing B-62 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

196 Data Warehouse Automation Platforms WhereScape in Action Testing INCREMENTAL TESTING VALIDATION TESTING Each database object can be linked to an external stored procedure or script that will be called after the load procedure has executed. The test procedure or script tests the output of the load procedure against criteria that is defined before development and writes error conditions to the scheduler log. Ensuring continued quality of the data warehouse requires ongoing testing after deployment and while the data warehouse is in operation. The production framework in RED supports validation scripts executed for any object at load time. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-63

197 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Schedules and Dependencies B-64 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

198 Data Warehouse Automation Platforms WhereScape in Action Schedules and Dependencies RUNNING THE WAREHOUSE The RED scheduler supports many scheduling options for warehouse jobs including daily weekly, monthly, annually and extensively customizable. The scheduler pane is shown at the top of the facing page. Schedules alone, however, don t capture all that is needed to run the warehouse processes. The scheduler schedules jobs. But jobs must account for dependencies among procedures. WhereScape s job builder provides default dependencies based on object type. The dependencies are fully configurable and customizable including ability to specify parallel processing. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-65

199 Data Warehouse Automation Platforms TDWI Data Warehouse Automation WhereScape in Action Errors and Recovery B-66 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

200 Data Warehouse Automation Platforms WhereScape in Action Errors and Recovery HANDLING ERROR CONDITIONS Load errors are a given in data warehousing systems where source data and systems often change without awareness of the downstream effects on the data warehouse. WhereScape RED is designed to detect, handle, and recover from errors. Error notification is shown at the top of the facing page. The center of the page illustrates ability to drill to detail and investigate the causes of errors. The bottom of the page shows how easy it is to restart processing when the error condition is handled. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY B-67

201 Data Warehouse Automation Platforms TDWI Data Warehouse Automation B-68 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY