M6P. TDWI Data Warehouse Automation: Better, Faster, Cheaper You Can Have It All. Mark Peco

Transcription

1 M6P European TDWI Conference with June 22 24, 2015 MOC Munich / Germany TDWI Data Warehouse Automation: Better, Faster, Cheaper You Can Have It All Mark Peco TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

2 Better, Faster, Cheaper... You Can Have It All TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

3 COURSE OBJECTIVES You will learn about: Concepts, principles, and practices of Data Warehouse Automation The current state of Data Warehouse Automation technology Automation opportunities and benefits when building or managing a data warehouse How to get started with Data Warehouse Automation Best practices and mistakes to avoid with Data Warehouse Automation TDWI takes pride in the educational soundness and technical accuracy of all of our courses. Please send us your comments we d like to hear from you. Address your feedback to: info@tdwi.org Publication Date: February 2015 Copyright 2015 by TDWI. All rights reserved. No part of this document may be reproduced in any form, or by any means, without written permission from The Data Warehousing Institute. ii TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

4 TABLE OF CONTENTS Module 1 Data Warehouse Automation Concepts and Principles Module 2 Building and Managing the Data Warehouse Module 3 Using Data Warehouse Automation Module 4 Data Warehouse Automation in Action Module 5 Getting Started with Data Warehouse Automation Appx A Bibliography and References... A-1 Appx B Data Warehouse Automation Platforms... B-1 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY iii

5 iv TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

6 Data Warehouse Automation Concepts and Principles Module 1 Data Warehouse Automation Concepts and Principles Topic Page Data Warehouse Automation Basics 1-2 Why Data Warehouse Automation? 1-10 The Foundation 1-16 Activities and Deliverables 1-22 The Technology Landscape 1-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-1

7 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Data Warehouse Automation Basics Data Warehouse Automation Defined 1-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

8 Data Warehouse Automation Concepts and Principles Data Warehouse Automation Basics Data Warehouse Automation Defined COVERING THE WAREHOUSING LIFECYCLE CHANGING BEST PRACTICES Data warehouse automation is more than simply automation of ETL development, or even the entire development process. It encompasses the entire data warehousing lifecycle from planning, analysis, and design through development and extending into operations, maintenance, and change management. Adoption of data warehouse automation changes the way that we think about building data warehouses. The widely accepted best practice of extensive up-front analysis, design, and modeling can be left behind as the mindset changes from get it right the first time to develop fast and develop frequently an approach that is aligned with today s agile development practices. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-3

9 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Data Warehouse Automation Basics Automated vs. Hand-Crafted 1-4 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

10 Data Warehouse Automation Concepts and Principles Data Warehouse Automation Basics Automated vs. Hand-Crafted BUILD IT BETTER, FASTER, AND AT LOWER COST CHANGE WHEN AND AS NEEDED WHY HANDCRAFT? Automation in data warehousing has many of the same benefits as in manufacturing: Increased productivity and speed of production Reduction of manual effort Improved quality and consistency Better controls and process optimization opportunities The manufacturing parallel hold when building a data warehouse; we can think of it as an information factory. But data warehousing is more complex than product manufacturing. Manufactured products are typically delivered to a consumer and the job is done. Data warehouses must be sustained through a long life cycle where changes in source data, business requirements, and underlying technologies are ongoing considerations. Automation helps to implement the right changes in the right ways and as quickly as they are needed. There are, of course, popular arguments in favor of handcrafting. In consumer goods, for example, advocates of hand made use terms such as unique, personal, and human touch. These are all valid reasons to buy a hand made scarf, but are they qualities that are needed in a data warehouse? TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-5

11 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Data Warehouse Automation Basics Automation vs. Customization 1-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

12 Data Warehouse Automation Concepts and Principles Data Warehouse Automation Basics Automation vs. Customization NOT MUTUALLY EXCLUSIVE Automation does not preclude customization, nor does it imply mass production. Neither is practical in data warehousing as most data warehouses need to have some level of customization. The purpose of customizing is to meet unique and specific needs that are likely to exist for every data warehouse. Strategic and business-critical uses typically have greater need for customizing than routine and repetitive uses. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-7

13 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Data Warehouse Automation Basics Data Warehouse Automation vs. CASE 1-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

14 Data Warehouse Automation Concepts and Principles Data Warehouse Automation Basics Data Warehouse Automation vs. CASE SMART AUTOMATION Some may recall the popularity of Computer Aided Systems Engineering (CASE) in the 1980 s. The purpose of CASE was to automatically generate program code and database schema from models and specifications collected and stored in a repository. The objective was to fully automate the process of systems development. The labor-intensive activities shifted from programming to analysis/design/specification with the goal of populating the repository. Methodology was a predominant factor and systems quality almost a singular goal. Data warehouse automation is distinctly different from CASE. The goal is to automate that which is smart to automate not necessarily to automate everything. Analysis can reasonably be accelerated with requirements discovery and design optimized by use of design patterns. Methodology is applied to support the processes, not to define and dominate them. And the goals include speed, quality, efficiency, effectiveness, and adaptability. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-9

15 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Why Data Warehouse Automation? Business Benefits 1-10 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

16 Data Warehouse Automation Concepts and Principles Why Data Warehouse Automation? Business Benefits QUALITY AND EFFECTIVENESS AGILE BUSINESS SPEED COST SAVINGS Data warehouse automation delivers quality and effectiveness through ability to build better solutions. Better solutions are those that best meet real business requirements, and it is especially difficult to get complete and correct requirements when limited to an early phase of a linear development process. With data warehouse automation the business can make changes much later in the development process and change can occur more frequently with less disruption, waste, and rework. Ability to change fast and frequently extends beyond the warehouse development process. Changes that occur in business requirements can be met with quick response. Responding to change in real time and without the delay of lengthy projects is the essence of business agility. Speed is the critical factor that enables agility both for agile business and for agile development. Ability to generate quickly and to regenerate equally fast when change occurs are fundamental automation capabilities. Ultimately building better, building faster, and changing quickly when needed bring substantial cost savings to data warehouse development, operation, maintenance, and evolution. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-11

17 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Why Data Warehouse Automation? Technical Benefits 1-12 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

18 Data Warehouse Automation Concepts and Principles Why Data Warehouse Automation? Technical Benefits THE VALUE FOR IT ORGANIZATIONS The benefits of automation aren t solely for business. IT organizations also derive real value from data warehouse automation in all of the ways that are illustrated on the facing page. The first four items agility, time savings, adaptability and cost savings are important and valuable for developers. The remaining part of the list documentation, impact analysis, testing, and maintainability arguably have greater impact because they apply beyond the development project and throughout the life of the data warehouse. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-13

19 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Why Data Warehouse Automation? Barriers and Resistance 1-14 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

20 Data Warehouse Automation Concepts and Principles Why Data Warehouse Automation? Barriers and Resistance REASONS FOR RESISTANCE PROCESS BARRIERS AND CONSTRAINTS HUMAN BARRIERS AND CONSTRAINTS Despite the obvious benefits, adoption of data warehouse automation is not a given. You may encounter resistance from two perspectives processes that aren t automation ready and people who are reluctant to change. Be prepared to adjust and adapt your data warehousing processes development, operation, and maintenance when implementing data warehouse automation. Processes that are designed to minimize risk in non-automated environments create conflict and limit the benefits of automation. We ll discuss the process considerations of change management in Module 5: Getting Started with Data Warehouse Automation. People will resist automation for a variety of reasons that are generally associated with fear that they will lose something loss of job, loss of status, loss of control, etc. We ll discuss the human aspects of change management in Module 5: Getting Started with Data Warehouse Automation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-15

21 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation The Foundation Components of Data Warehousing 1-16 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

22 Data Warehouse Automation Concepts and Principles The Foundation Components of Data Warehousing DATA FLOW FROM DISPARATE DATA TO INTEGRATED INFORMATION The facing page illustrates the core elements of data warehousing beginning with disparate data sources at the bottom of the diagram and leading to integrated information resources at the top. The information resources aren t necessarily the end of the line they are simply the end of the data integration processes. Business value is created when they are used for reporting, business intelligence, decision-making, analytics, etc. The center of the diagram shows the processing steps to get from data sources to integrated information using two methods data consolidation through extract-transform-load processing and data virtualization through a series of abstract data views. Consolidation and virtualization are often combined to optimize the flow of data. Data warehouse automation can implement both approaches independently or in a mix-and-match form. ALL OF THE PARTS Data warehousing encompasses many different techniques and produces many components to enable the data-to-information flow. Among these are architectural standards, data models, mappings, data transformations, database load procedures, tests and controls, events and errors, and metadata. Data warehouse automation includes capabilities to create, connect, manage, and apply all of these components. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-17

23 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation The Foundation Data Warehousing Projects 1-18 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

24 Data Warehouse Automation Concepts and Principles The Foundation Data Warehousing Projects PROGRAM VS. PROJECT Data warehousing is more than just a project or a series of projects. It involves many projects within a program context that also includes evolving architecture and ongoing operations. Projects are iterative, timebounded activities to produce specific data warehousing capabilities. The program is a continuous investment of resources that encompasses projects, operations, and value creation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-19

25 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation The Foundation Design Patterns 1-20 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

26 Data Warehouse Automation Concepts and Principles The Foundation Design Patterns ENCAPSULATING STANDARDS AND CONVENTIONS As described earlier, one of the benefits of automation is accelerated through use of design patterns. Design patterns can encapsulate architectural standards as well as best practices for data design, data management, data integration, and data usage. The facing page identifies common design patterns in all of these categories. Creating and applying patterns in a data warehouse automation tool supports the goal of accelerated design, but more importantly it drives compliance with standards and consistency of data warehousing results. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-21

27 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Activities and Deliverables Data Warehousing Processes 1-22 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

28 Data Warehouse Automation Concepts and Principles Activities and Deliverables Data Warehousing Processes LABOR INTENSITY, COMPLEXITY, AND VOLATILITY A typical data warehouse development process is burdened by laborintensive activities, a high degree of complexity and dependency among deliverables, and volatility of data and requirements. Some of the data warehousing pain that can be relieved by automation includes: Slow, laborious, and difficult to get it right requirement gathering Interdependency between source data analysis and warehouse data modeling Manually mapping sources to targets Detailed specification of data transformation logic Translating data models into schema and building databases Hand coding and manual testing of ETL processing Preparing and loading initial start-up data Acceptance testing and deployment to production These activities would be difficult enough in a linear process, but in reality data warehousing projects are never linear. We re always going back because: Data modeling may change requirements Source-target mapping may affect data modeling Data transformation design may change mappings Integration process design may affect database design Incremental testing may cause changes throughout Acceptance testing may discover incorrect requirements and so on... When we survive all of these things and deploy the data warehouse it doesn t remain stable. Business needs, source data, and technology will all change (at different times and speeds) driving continuous change in the warehouse. And through it all we need to manage projects, track changes, manage code and system versions, isolate production from development and testing environments, sustain daily operations, and try to keep metadata and documentation synchronized with what is actually implemented. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-23

29 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation Activities and Deliverables What Can You Automate? 1-24 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

30 Data Warehouse Automation Concepts and Principles Activities and Deliverables What Can You Automate? USING DATA WAREHOUSE AUTOMATION Data warehouse automation is capable of relieving much of the pain discussed on the previous pages. Nearly everything in the diagram on the facing page can be automated. All automation tools support core activities such as data modeling and data integration. The most robust automation products support all of the activities. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-25

31 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation The Technology Landscape Automation Tools and Vendors 1-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

32 Data Warehouse Automation Concepts and Principles The Technology Landscape Automation Tools and Vendors COMMON TOOLS The facing page illustrates the four most common tools for data warehouse automation Attunity Compose, Magnitude Software Kalido, timextender, and WhereScape. These are the pure DWA product vendors. Others, such as Birst, include some automation capabilities in a broader product suite. Later in the course you ll see more about the approaches and capabilities of Attunity Compose, Magnitude Software Kalido, and WhereScape each with different perspectives and capabilities. The intent is not to compare the tools or to perform product evaluation, but to show a representative sample of mainstream DWA technology. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 1-27

33 Data Warehouse Automation Concepts and Principles TDWI Data Warehouse Automation 1-28 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

34 Building and Managing the Data Warehouse Module 2 Building and Managing the Data Warehouse Topic Page The Data Warehousing Lifecycle 2-2 Building the Data Warehouse 2-22 Managing the Data Warehouse 2-30 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-1

35 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle A Closed Loop 2-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

36 Building and Managing the Data Warehouse The Data Warehousing Lifecycle A Closed Loop CONTINUOUS WITHOUT AN EXIT POINT The diagram on the facing page revisits the data warehousing lifecycle with emphasis on the blending of program and project work. The important introductory concept is that the lifecycle is a closed-loop system. There is no exit point in the diagram and no end to continuous evolution, change, and redevelopment. This is the fundamental reason that conventional design-and-build cycles don t work well for data warehousing. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-3

37 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Architecture and Planning 2-4 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

38 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Architecture and Planning ARCHITECTURE DELIVERABLES The facing page expands the architecture phase of the data warehousing lifecycle to illustrate common deliverables of architecture and planning including: Program Charter describing goals, purpose, resources, governance, sponsorship, etc. Readiness Assessment identifying strengths, weaknesses, risks for the program Business Architecture identifying stakeholders and their interests, business applications, and business processes to be affected Organization Architecture describing the people, purpose, processes, and structure of warehouse development, deployment, operation, and application Data Architecture with standards for structure, storage, distribution, access, etc. Integration Architecture describing the standards and patterns for flow of data from disparate sources, to integrated information, and access by data consumers Process Architecture with standards, conventions, and patterns for data sources, mappings, and integration procedures TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-5

39 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Architecture and Planning - Preparation 2-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

40 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Architecture and Planning - Preparation AUTOMATION BENEFITS Preparation activities of charter and readiness assessment can get value from automation in several ways. Automation helps to: Maximize finances Achieve goals Meet expectations Optimize people Automate processes Increase capabilities TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-7

41 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Architecture and Planning - Architecture 2-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

42 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Architecture and Planning - Architecture AUTOMATION BENEFITS Architectural activities can use automation to: Engage stakeholders Capture business rules Focus on purpose Encapsulate data standards Encapsulate integration standards Automate DW methodology TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-9

43 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Implementation 2-10 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

44 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Implementation IMPLEMENTATION DELIVERABLES The facing page illustrates high-level categories of deliverables from data warehouse implementation projects including: Source Data Understanding describing content, meaning, and quality of data sources Target Data Models describing the organization and structure of warehouse data Source-Target Mappings associating warehouse data fields with the sources from which they are populated Data Acquisition Processes procedures to obtain data from sources Data Transform Processes procedures to change the data to achieve data warehousing characteristics (integrated, subject oriented, time variant, non-volatile, and sometimes dimensional) Database Load Processes procedures to load data into warehouse databases Warehousing Databases the physical databases in which warehouse data is stored TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-11

45 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Implementation Analysis and Design 2-12 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

46 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Implementation Analysis and Design AUTOMATION BENEFITS The analysis and design activities of implementation projects source data understanding, target data modeling, and source-target mapping use automation for: Profiling and exploration of data sources Stakeholder driven requirements Applied data standards & templates Data modeling & schema generation Encapsulation of integration standards Metadata driven source-target mapping TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-13

47 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Implementation Design and Build 2-14 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

48 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Implementation Design and Build AUTOMATION BENEFITS The design and build activities of implementation projects building processes and databases use automation to: Generate processes & procedures Sequence & link processes & procedures Test processes & procedures Generate database schema TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-15

49 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Deployment and Operations 2-16 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

50 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Deployment and Operations DEPLOYMENT AND OPERATIONS DELIVERABLES The facing page illustrates deliverables and activities of data warehouse deployment and operations including: Managed Environments separating development, testing, and production environments Acceptance Testing confirming that the implementation meets business requirements and formalizing acceptance as a production system Transfer to Operations transfer of knowledge, responsibility, and accountability from development to operations team User Training and Support enabling effective use with the right knowledge, skills, and support structure Scheduling and Execution ensuring that data warehousing processes run when and as they should with attention to dependencies on source systems and among warehousing processes Maintenance and Versioning making small changes and repairs with complete change history and version management Change and Evolution identifying and initiating architectural changes and new projects due to changed data sources, business requirements, or technology capabilities TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-17

51 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Deployment and Operations Production Implementation 2-18 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

52 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Deployment and Operations Production Implementation AUTOMATION BENEFITS The production implementation activities use automation for: Incremental testing Data warehouse versioning Environment management and migration Documentation and metadata especially as applied for initial training of operations staff and users TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-19

53 Building and Managing the Data Warehouse TDWI Data Warehouse Automation The Data Warehousing Lifecycle Deployment and Operations Execution and Maintenance 2-20 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

54 Building and Managing the Data Warehouse The Data Warehousing Lifecycle Deployment and Operations Execution and Maintenance AUTOMATION BENEFITS The execution and maintenance activities use automation for: Incremental testing Source & version management Scheduling Validation testing Documentation and metadata especially as applied for troubleshooting, investigation, and ongoing training Impact analysis TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-21

55 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Building the Data Warehouse Requirements Driven Development 2-22 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

56 Building and Managing the Data Warehouse Building the Data Warehouse Requirements Driven Development A TRADITIONAL APPROACH Requirements driven development is the traditional process that begins with requirements gathering and proceeds in a linear fashion through analysis, modeling and design, specification and coding, and deployment. This approach is a legacy of conventional software development processes that doesn t work well for data warehousing. It is characterized by: Intensive and detailed planning Long development timelines Limited business participation early in the process Substantial staffing and skills demand Difficulty in requirements gathering and analysis Much cycling back to previous steps to correct errors and oversights Much waste and rework Post-deployment discovery of incorrect and missed requirements TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-23

57 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Building the Data Warehouse Modeling and Metadata Driven Development 2-24 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

58 Building and Managing the Data Warehouse Building the Data Warehouse Modeling and Metadata Driven Development A DESIGN APPROACH Model and metadata driven development focuses on capturing business requirements and translating them to a design that is recorded as models and metadata. The models and metadata are then used in conjunction with design patterns to generate data warehouse deliverables. This approach is an incremental improvement over requirements-driven development in ability to build models and collect metadata that are useful for many related projects. Model and metadata driven development naturally implies some degree of automation. It is characterized by: Need for stated business requirements and well defined scope Development timeline that correlates with project scope Business participation early and late in the process Specialized staffing and skills to perform critical modeling tasks Evolving granularity of requirements through model development Some cycling back to previous steps as a result of model refinement Iterative development with many related and dependent projects TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-25

59 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Building the Data Warehouse Discovery Driven Development 2-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

60 Building and Managing the Data Warehouse Building the Data Warehouse Discovery Driven Development A LEARNING APPROACH Discovery driven development acknowledges the problems of traditional requirements approaches in data warehousing, replacing them with the concept of requirements through discovery. Fuzzy requirements become clear through cycles of prototyping and refinement of requirements. This approach is a substantial shift away from requirements first approaches. It is highly dependent on automation to enable rapid cycles of prototyping, and is characterized by: Projects of small to modest scope with fuzzy requirements Accelerated development timelines achieved through automation Extensive business participation throughout the prototyping activities and again at acceptance and deployment Demand for data warehouse automation skills and business subject expertise Discovery of requirements through prototyping and evaluation Some rework when inevitable source data surprises occur Multiple projects and iterative development TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-27

61 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Building the Data Warehouse Data Driven Development 2-28 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

62 Building and Managing the Data Warehouse Building the Data Warehouse Data Driven Development AN AGILE APPROACH Data driven development adapts the discovery approach to recognize a reality of data warehousing that source data is as challenging as requirements gathering and is often filled with surprises. Combining data exploration with prototyping and requirements discovery creates a truly agile process for data warehousing. Rework caused by unexpected conditions in source data is eliminated by early data exploration. Automation is a must for this approach which is characterized by: Many projects of small scope appropriate for agile development Accelerated development achieved through automation Extensive business participation throughout the entire development process Demand for data warehouse automation skills, agile development mindset, and business subject expertise Discovery of business requirements through prototyping Discovery of data requirements through data exploration Elimination of rework with agile development practices Many small projects and iterative development cycles TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-29

63 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Managing the Data Warehouse Operations 2-30 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

64 Building and Managing the Data Warehouse Managing the Data Warehouse Operation AUTOMATION FOR OPERATIONS Operations comprises a core set of activities in managing a data warehouse, with attention to: Sequencing Dependencies Scheduling Execution Verification Validation Error Handling Automation aids data warehouse operations with features and functions for: Scheduling Documentation & Metadata Managed Environments Validation Testing TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-31

65 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Managing the Data Warehouse Monitoring and Tuning 2-32 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

66 Building and Managing the Data Warehouse Managing the Data Warehouse Monitoring and Tuning AUTOMATION FOR MONITORING AND TUNING Monitoring and tuning are essential activities of managing a data warehouse, with attention to: Usage Performance Availability Access Security Growth Capacity Automation aids data warehouse monitoring and tuning with features and functions for: Applied Standards Documentation & Metadata Managed Environments Iterative Testing TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-33

67 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Managing the Data Warehouse Maintenance and Change 2-34 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

68 Building and Managing the Data Warehouse Managing the Data Warehouse Maintenance and Change AUTOMATION FOR WAREHOUSE MAINTENANCE AND CHANGE The need for maintenance and change is inevitable when managing a data warehouse. Specific activities include: Tracing And Troubleshooting Bug Fixes Data Correction Error Recovery Upgrades Enhancements Automation aids data warehouse maintenance and change management with features and functions for: Applied Standards Impact Analysis Documentation & Metadata Managed Environments Iterative Testing Versioning & Source Control TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-35

69 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Managing the Data Warehouse Evolution and Change 2-36 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

70 Building and Managing the Data Warehouse Managing the Data Warehouse Evolution and Change AUTOMATION FOR WAREHOUSE EVOLUTION Evolution is a natural part of the data warehouse lifecycle driven by ongoing change and needs for: Extension Expansion Migration Modernization Re-Architecting Re-Sourcing Refactoring Automation offers many features and functions to support data warehouse evolution including: Applied Standards Impact Analysis Documentation & Metadata Data Source Profiling Managed Environments Data Warehouse Versioning Encapsulated Standards Schema Generation Iterative Testing Versioning & Source Control TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-37

71 Building and Managing the Data Warehouse TDWI Data Warehouse Automation Managing the Data Warehouse Support 2-38 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

72 Building and Managing the Data Warehouse Managing the Data Warehouse Support AUTOMATION FOR WAREHOUSE SUPPORG Support is critical to success and sustainability of every data warehouse. Automation aids data warehouse support with features and functions for: Applied Standards Documentation & Metadata Managed Environments Versioning & Change Control TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 2-39

73 Building and Managing the Data Warehouse TDWI Data Warehouse Automation 2-40 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

74 Using Data Warehouse Automation Module 3 Using Data Warehouse Automation Topic Page Automation Use Cases 3-2 Case Studies 3-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-1

75 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Building a New Data Warehouse 3-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

76 Using Data Warehouse Automation Automation Use Cases Building a New Data Warehouse SCENARIO CHALLENGES OPPORTUNITIES You need to build a new data warehouse either where none exists or as a complete replacement of an existing and dysfunctional warehouse. Without automation, all of the normal data warehousing challenges exist. Source data is messy, warehouses are hard to build, they take too long to build, and they re obsolete before they are deployed. Automation opportunities are abundant in this scenario. You can automate everything from planning to deployment and operation using any of model-driven, discovery-driven, or data-driven approaches. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-3

77 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Building Dependent Data Marts 3-4 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

78 Using Data Warehouse Automation Automation Use Cases Building Dependent Data Marts SCENARIO CHALLENGES OPPORTUNITIES You have a data warehouse but you need to build dependent data marts sourced from that warehouse. Sometimes the data marts are star-schema or cubes and sometimes non-dimensional reporting databases. Data marts are user facing so requirements are critical, but requirements are as difficult for data marts as for data warehouses. Handcrafting of schema and ETL is time-consuming, labor-intensive, and likely to require rework due to unclear requirements. Automation is a natural fit for this scenario. Overcome the requirements challenges with discovery-driven or data-driven development and accelerate the process with automated schema and process generation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-5

79 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Building Dimensional Data Marts 3-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

80 Using Data Warehouse Automation Automation Use Cases Building Dimensional Data Marts SCENARIO CHALLENGES OPPORTUNITIES You need to build dimensional data marts that are sourced directly from operational systems without using a hub data warehouse. This scenario often involves high demand with every business function and many work groups wanting data marts quickly. Matching marts to business needs and source data with mart requirements is difficult. Handcrafted star-schema and ETL processing is too slow to meet demand when the business wants more and faster. This is another natural fit for automation. Any of model-driven, discovery-driven, or data-driven approaches step up to the challenges of business alignment, and automated schema and ETL generation help to meet the demand to deliver more and faster. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-7

81 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Conforming Independent Data Marts 3-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

82 Using Data Warehouse Automation Automation Use Cases Conforming Independent Data Marts SCENARIO CHALLENGES OPPORTUNITIES You have many independent data marts that were built without considering the need to conform the dimensions. Now the business wants the ability to drill across and perform analysis that involves more than one data mart. This scenario is challenged to analyze and understand the differences among variations of each dimension, to establish conformity standards for each dimension, to analyze the impact of dimension conformity on each existing data mart, and to rebuild the data marts with dimension standards applied. Automatically documenting the data marts is a good first step to analyze and understand the differences. Standards for conformed dimensions can be encapsulated as design patterns in an automation tool. When the tool is populated with metadata about the existing data marts (a necessary step in automating documentation) then analyzing the impact of change can also be aided by automation. Ultimately data-driven or model-driven techniques can be used to rebuild the data marts with conformed dimensions. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-9

83 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Data Warehouse Prototyping 3-10 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

84 Using Data Warehouse Automation Automation Use Cases Data Warehouse Prototyping SCENARIO CHALLENGES OPPORTUNITIES You need to engage the business and understand requirements before committing time and resources to data warehouse development. And you re working with business users whose perspective on requirements is I ll know what I need when I see it! Prototyping doesn t work without business participation, and it doesn t work with long lag time between prototype building, prototype evaluation, and the next cycle. Automation and prototyping fit together quite naturally. Both discoverydriven and data-driven approaches make it possible to produce visible and tangible results quickly, and to apply the findings and feedback of evaluation in the spirit of rapid prototyping. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-11

85 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Agile Data Warehousing 3-12 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

86 Using Data Warehouse Automation Automation Use Cases Agile Data Warehousing SCENARIO CHALLENGES OPPORTUNITIES You need all of the benefits of agile development fast, focused, and business driven when building the data warehouse. But agile data seems to be much harder to achieve than agile processing. Not only is agile data harder than agile process, but agile techniques are challenged by constraints of the existing data warehouse and of the source data. You need engaged business subject experts, very fast development cycles, and the ability to make the data tangible as requirements come to the surface. Handcrafted schema and ETL is too slow for agile. The agile value of working software over comprehensive documentation is a concern. You need to be agile without sacrificing the documentation and metadata that are essential to warehouse operations, maintenance, usage, and evolution. Automation enables agile data warehousing by supporting very fast development and redevelopment cycles while capturing essential metadata as part of the process. Existing warehouse constraints are easily incorporated, and with a data-driven approach source data constraints are also easily addressed. Generated schema and ETL, quick changes, and versioning all help when applying agile methods to data warehouse development. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-13

87 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Data Warehouse Migration 3-14 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

88 Using Data Warehouse Automation Automation Use Cases Data Warehouse Migration SCENARIO CHALLENGES OPPORTUNITIES You have an existing data warehouse that is functionally sound but needs to be moved to a new technology platform. Business and logical views of the data warehouse are sound, but the physical differences between databases and data integration platforms are a consideration. Only the physical metadata is reliable because conceptual and logical models and designs haven t kept pace with changes to the existing warehouse. Having the warehouse offline for more than a very brief period is not acceptable. Manually rebuilding schema and data integration processes will take too long and is likely to be error-prone. In addition to moving the schema and processes to a new platform, you also need to move the data. Overcome the inadequacies of documentation by populating the automation tool with physical metadata from the old warehouse and then extracting the design. Rebuild schema and processes for the new platform by generating them with the automation tool. Map old schema to new schema, then build and execute data migration processes to move the data. A quick turnaround from old platform to new, with automated testing and validation, minimizes downtime that the warehouse is not available for business use. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-15

89 Using Data Warehouse Automation TDWI Data Warehouse Automation Automation Use Cases Re-Architecting the Data Warehouse 3-16 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY

90 Using Data Warehouse Automation Automation Use Cases Re-Architecting the Data Warehouse SCENARIO Your data warehouse needs a comprehensive architectural update perhaps to overcome long-standing architectural deficiencies, to align with today s best practices, to meet real time data demands, to incorporate unstructured data, or for a variety of other reasons. CHALLENGES OPPORTUNITIES The challenges are similar to those for data warehouse migration. The significant difference is that the architecture not the platform is changing. Obsolete documentation, downtime constraints, and the time and cost to rebuild manually are real challenges. But these challenges are compounded by the probability that re-architecting will bring structural and standards changes that have greater impact than technology platform changes. Migration opportunities to automate apply here too. Extracting the design, generating schema and process, automating data migration, and automating testing and validation are all valuable. New opportunities also exist to embed architectural structures as design patterns and to encapsulate architectural standards as part of automation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY 3-17