Agile Data Warehouse Automation

Transcription

1 Agile Data Warehouse Automation Executive Summary Today s data architects and business intelligence (BI) teams want to rapidly integrate and transform data to meet fast changing, real-time business analytics requirements. But, they are struggling with brittle, hand-coded, complex data warehousing processes that cause project and decision delays. These challenges can create management complexity, developer resource constraints and cost overruns. This white paper explains how Attunity Compose offers a fresh approach to data warehouse automation for data architects that meets these challenges. It will show how Attunity Compose automates the design, implementation and change process for data warehouses and data marts. It replaces manual, error-prone coding with a metadata-based solution that automates data modeling, ETL commands and complex transformations and accelerates change propagation.

2 Table of Contents Executive Summary...1 Overview...3 Business Challenges...3 Introducing Attunity Compose...3 Key Features...4 Agile Data Warehouse Automation...4 Real-time Data Integration and Automated ETL...6 Physical Data Warehouse Management...8 Generation of Data Marts...10 Operational Features and Life Cycle Management...12 Conclusion

3 Overview Business Challenges Traditional methods of designing, developing, and implementing data warehouses consume prohibitively large amounts of time and resources. The multi-month, error-prone ETL development effort to set up a data warehouse typically taking 60-80% of preparation time and requiring specialized developer expertise often means that the data model is out of date before the BI project even starts. In addition, traditional design, development, and implementation processes often result in data warehouses that cannot be adapted to continually changing business requirements. Modifying data warehouses diverts skilled resources from more innovative activities. In short, data warehouses become a bottleneck as much as an enabler of analytics. To speed time to analytics, aspects of the data warehouse creation and management lifecycle must be streamlined wherever possible. The goal of the BI team is to address strategic business questions with minimum tactical and software development work. The processes of extracting data from disparate source systems and wrangling it into the data warehouse, then updating the data warehouse to meet changing requirements, are ripe for automation. Testimonial We had initially planned on 45 days of ETL coding. With Attunity Compose we had it completed in two days. Senior Information Management Analyst at a global insurance company Introducing Attunity Compose Businesses demand agility and responsiveness from their BI environments. To support these requirements, Attunity Compose provides a metadata-based platform built for data architects rather than developers, that automates the manual, repetitive aspects of data warehouse design, development, testing, deployment, operations, impact analysis, and change management. Attunity Compose does much more than simple ETL generation. Its capabilities span development processes, requirements gathering, deployment and maintenance. Attunity Compose automatically generates the ETL commands, data warehouse structures and documentation needed to efficiently execute projects while tracking data lineage and ensuring integrity. Attunity Compose automates routine IT tasks associated with designing, creating, populating, and managing data warehouses and data marts (DMs). It also accelerates tasks that cannot be completely automated. With Attunity Compose, IT teams can respond in days to business requests with accurate time, cost, and resource estimates. Once projects are approved, it s possible to deliver completed data warehouses, data marts, and BI environments in far less time. Enterprises gain efficiencies, reduce risk and enable more effective analytics. Agile Data Warehouse Automation with Attunity Compose 3

4 Key Features Attunity Compose provides the following comprehensive set of automation features to eliminate the cumbersome and error-prone manual coding associated with the many repetitive steps of data warehouse design and implementation. In addition, Attunity Compose provides the operational features needed for ongoing maintenance of the data warehouse and data marts. Automation Features Optimized for either model-driven or data-driven approaches to data warehousing Real-time source data integration Automated ETL generation Physical data warehouse management Generation of data marts Operational Features Monitoring Workflow designer and scheduler Notifications Data profiling and quality enforcement Lineage and impact analysis Project documentation generation Migration between environments (DTAP) Agile Data Warehouse Automation Create, Import and Reverse-Engineer Data Models The conceptual data modeling process is complicated by source-to-target data mappings. Attunity Compose eliminates this complexity by defining the physical model in several ways, and then automatically mapping the source databases to the model according to best practices for data warehouse generation. Attunity Compose Data Warehouse model designer enables users to generate models in three ways: Use Attunity Compose as a modeling tool and define the entity relationship diagram Import third party models like CA Erwin or other industry-standard models Discover the model by reverse engineering the source databases 4

5 The Attunity Compose model designer offers the following key features: An Attribute Domain glossary for all the attributes in the data warehouse Definition of entities with attributes and relationships between the entities Definition of history type for each attribute (type 1 and type 2) Definition of one or many satellites per entity Built-in date model that can be linked to any date attribute These capabilities deliver a compelling set of benefits. With Attunity Compose, it s possible to create a model based on business requirements, then automatically generate and implement that logical model without additional manual work. Attunity Compose automatically generates the mappings using data warehouse best practices for higher consistency and quality. Enable agile development (Implement-Change-Adjust) Starting a new Data Warehouse/Business Intelligence (DW/BI) initiative can be lengthy, expensive and risky. Attunity Compose reduces the risk of such projects and helps deliver a DW/BI solution that satisfies end users. Due to a variety of factors, data warehouse requirements often change throughout the project lifecycle. Only an incremental and agile development approach will succeed. Attunity Compose is purpose built with this in mind. Any changes to the data sources, models and business rules can be introduced and deployed rapidly, speeding project execution and the learning process while reducing risk. Modify and enhance the data model with derived attributes and lookup tables Derived attributes in the data warehouse are used to enhance the data model, and are usually calculated from other attributes, such as multiplying an employee s monthly salary by twelve to calculate an annual salary or deriving a person s full name from first name and last name attributes. To create those derived attributes, typically the data warehouse administrator and ETL developers must manually modify the data warehouse tables and write ETL code that calculates their values. With Attunity Compose, the derived attributes are defined as part of the model definition. These derived attributes will be evaluated and updated as part of ongoing, automated data warehouse management. Derived attributes are defined as part of the model definition and are evaluated and updated. In addition, Attunity Compose helps to automate the process of replacing the values in the source data with the actual data that you want to store in the data warehouse. This is done by implementing lookup tables functionality. For example, a lookup table could be used to replace a zip code with a full address or, conversely, to replace a full address with a zip code. 5

6 Support date/time based modeling for flexible BI and analytics Incorporating time into data warehouse dimensions is one of the most common requirements for BI and analytics projects. Time dimension attributes represent periods such as years, semesters, quarters, months or days that provide varying levels of granularity for analysis and reporting. The attributes are organized in hierarchies, and the granularity of the time dimension is determined largely by the business and reporting requirements for historical data. For example, most financial and sales data in business intelligence applications use months or quarters. Attunity Compose completely automates the construction management and ongoing updates of time dimensions. Real-time Data Integration and Automated ETL Easy-to-use GUI to load data from source systems Using a clearly defined user interface, Attunity Compose is designed to hide the complexity of data integration and transformation from users while still offering a rich feature set. Through its innovative and easy to use interface, it s possible to easily design data models and build data warehouses and data marts with just a few clicks. Once users have finished designing and building the environment, they can switch to a monitoring console to track progress. Real-time loading of source data with Change Data Capture (CDC) Attunity Compose is integrated with Attunity Replicate - high-performance data replication and loading software application. Together these products support the entire lifecycle of data warehouses and data marts, including source mapping, model definition, data warehouse and DM creation, data warehouse/dm population and ongoing updates. Users can load data from many heterogeneous data sources simultaneously with real-time change data capture (CDC) technology. Zero footprint technology with no agents on source systems Attunity s zero footprint technology means that no agents need be placed on the source databases, eliminating performance overhead and administrative burden for mission-critical systems. Support heterogeneous source systems RDBMS/Mainframe/Hadoop Through integration with Attunity Replicate, Attunity Compose offers support for the industry s widest ecosystem of heterogeneous database sources, with real-time and query-based change data capture (CDC.) This platform supports all major databases, from legacy to modern data stores. Whether sourcing from or to RDBMS, Hadoop, EDW, cloud and on-premises, the Attunity Replicate management environment provides the ability to work across these endpoints in a similar form. It is purpose built for heterogeneous data replication. 6

7 ETL Automation Attunity Compose provides automated ETL generation and execution for loading data to the staging 1 area from the landing 2 area in the data warehouse: Every combination of source table to staging table can include a mapping definition. Mapping definitions connect source columns to attributes in a data warehouse entity. Attunity Compose uses the same mapping and transformations for the initial load of the data warehouse from replica tables, and for updating the data warehouse from change tables. Supported transformation functions include: Mapping an attribute in the staging area to an expression based on the source Filtering records from the source according to a logical expression Providing mappings based on a landing table, view, or SQL query Users may also include custom SQL-based ETL steps in the processing flow to leverage pre-defined procedures and enhance supported transformation features. Apply advanced transformations with GUI-based expression builder The Attunity Compose graphical Expression Builder provides an easy way to build an expression that calculates new derived attributes or transforms existing fields. It provides easy access to the required elements for an expression without typing any information manually. After building an expression, users can list the expression and test drive it, without leaving the Expression Builder editor. This provides an easy way to create and apply transformations to data elements, and ensures that users will receive the desired results. 1 The staging is an area in the data warehouse that includes tables in the structure of the data warehouse model that are the source for the data to load or update the data warehouse. 2 The landing area (per operational data source) in the data warehouse includes tables replicated from the source, as well as change tables. 7

8 Physical Data Warehouse Management Automatically generate code to create data staging, a data warehouse and data marts Staging Staging tables are the mirror images of the source table inside the data warehouse. They are used to collate and prepare data for loading into the data warehouse tables. Staging tables are automatically generated by Attunity Compose when discovering the source databases. Data warehouse Once source metadata is discovered and mapped, and the model is generated, Attunity Compose can build the data warehouse with a click of a button. The DDL statements are generated based on normalized enterprise data warehouse principles. But unlike the traditional approach to normalized data warehouse implementation, Attunity Compose does not suffer from the same downsides: long development time, hard maintenance and change propagation. Once the DDL is generated, the statements are executed. Over the life of the data warehouse, the existing model is constantly compared to the sources. Physical adjustments to the data warehouse structure are automatically introduced as the data warehouse model evolves without generating ETL code. This makes Attunity Compose-generated data warehouses compliant with data vault modeling techniques, meaning: Completely auditable architecture Lends itself for real-time processing data warehouse model is aligned with the business Easy to load to a dimensional model model Layered isolation from change Extremely adaptable to (business) changes Can be incrementally built, easily extended Designed and optimized for the data warehouse Attunity Compose s data warehouse automation features include: Transparently managing surrogate keys and naturalto-surrogate key 3 mappings Handling of history types and slowly changing dimensions Supporting and integrating history information coming from operational sources with history Eliminating the need to deal with the order in Data Mart Attunity Compose automates dimensional modeling (star schema) of the data marts. The generated data marts are suitable for multi-dimensional analysis, and organized per subject area; easy to understand for business users. To build a data mart, Attunity Compose provides a wizard that walks through all the steps required to build and populate the data mart: 1. Selecting fact tables 2. Selecting dimension tables 3. Defining transaction date(s) 4. Creating deformalized tables based on facts and dimensions defined above 5. Generate ETL code to incrementally populate the data mart which the data warehouse is loaded via handling of premature facts 4 Handling updates to the data warehouse based on partial records (updating only changed fields) Asynchronously loading entity from several sources. This means sources loading a single record in data warehouse doesn t have to use same loading schedules as standard ETL processes. 3 A key is one or more data attributes that uniquely identify an entity. A natural key is formed of attributes that already exist in the real world. For example, U.S. citizens are issued a SSN. A surrogate key has no business meaning and is internally generated by the system. Surrogate keys are used to simplify the management of complex natural keys (more than one attribute in the key) or when integrated data sources have a different representation of the natural key for the same values. 4 Premature facts are facts that are loaded to the data warehouse before the dimensions that describe them are loaded. For example, an order record that is loaded before the customer record that made that order. 8

9 Rapidly propagate data model changes to physical structures without manual ETL coding With Attunity Compose, introducing a new data source, modifying an existing source, or simply adding new attributes to an existing model does not require any ETL code modifications. Users simply follow the same steps and wizards that they used during the original design stage. Table/ETL code changes are automatically generated and reflected in the new version of the data warehouse. In this way, users can implement business changes with a few clicks. Additionally, all ETL code to implement a change is automatically generated, eliminating the need for manual coding. Preserve history for slow changing dimensions (Type 2) without any manual coding This method tracks historical data by creating multiple records for a given entity in the dimensional/hub tables. For example, a data warehouse application might seek to track and store customer addresses every time a new customer location is registered. Attunity Compose automatically identifies Type 2 attributes and recommends a table structure accordingly. Before deploying the model, the Attunity Compose user can manually overwrite the recommendation and change the attribute to Type 1 if history tracking is not required. Attunity Compose automatically tracks and handles changes in the history which greatly reduces the complexity of implementation and eliminates the need for manual coding. Generates optimized data warehouse architecture using hubs and satellite table layout Attunity Compose uses a data warehousing design pattern that makes designing a robust, agile, and historical data warehouse an easy process. Attunity Compose uses Hub and Satellite tables to model an optimized data warehouse. These are the names given to a type of table, in the same way a star schema is made up of dimensions and facts in the data mart. As the first step in modeling, Attunity Compose breaks the model into core entities, such as Customer, Sale, Employee, etc. Each core subject area is then turned into a Hub table. The Hub table contains just Type1 attributes, like the business key from the source system(s), such as Customer_Id. Satellite tables contain all the descriptive information about a Hub. Each Hub can have one to many Satellite tables that contain information about that business key (Type2 attributes). In the context of the Customer_Hub, there could be a Customer_Name_Satellite and a Customer_Address_Satellite. The Hub and Satellite-based design makes it easy to add sources to the data warehouse in the future. New Hubs and Satellites can be created from new source systems and joined to already existing Hubs. In the same manner, new Satellites can also be created for pre-existing Hubs. This prevents modifications to already existing tables. Organizations no longer feel pressured to create a perfect data warehouse design at the outset. With Attunity Compose, they can continually edit and add to the data warehouse as business needs change. This agile approach makes it possible to build segments of a data warehouse quickly, which pleases business owners who want immediate results more quickly. Traditionally, developers have spent months or even years building and testing the logic of data warehouses, while the business users wait to see any benefit. 9

10 Generation of Data Marts Automates process of data mart definition generation Since a data mart is essentially a subset of the data warehouse, Attunity Compose users can create any number of data marts according to their business needs. Users can also create multiple star schemas for a single data mart using pre-coded design patterns. Star schemas allow you to reuse existing dimension tables within the same data mart, thereby saving space in the data warehouse platform while at the same time improving query performance. For example, you could create one star schema with an Order Details fact table and Customers and Products dimensions and another star schema with the same dimensions but a different fact type (or the same fact type, but different dimensions). This also allows to generate BI reports using different facts that share the same dimensions. Additionally, in a star schema, dimensions are linked with each other through one join path intersecting the fact table, facilitating accurate and consistent query results. Attunity Compose supports three types of data marts out of the box: Transactional A star schema with a transactional fact table allows you to retrieve the desired data, even if a dimension table contains multiple versions of the same record. To use an example from the automotive industry, selecting OrderDate as the Transaction Date would allow you to generate a report for the number of customers who bought cars in New York between 2013 and 2016, even if a customer moved to a different city (which would also result in a new record being added to the Customers dimension). Aggregated A star schema with an aggregated fact table allows you to make aggregate calculations based on the fact table attributes. For instance, you could create an aggregated fact that shows the total freight costs per shipping region and product category. Additionally, the presence of a transaction date in the fact table makes it possible to retrieve the desired data, even if a dimension contains multiple versions of the same record. To use an example from the shipping industry, a shipper could use an aggregated fact to generate a report for the total cost of shipping rice to Australia from State Oriented A star schema with a state oriented fact supports Type 2 columns in the fact table. This is useful in cases where the fact is not a singular event in time, but rather, consists of multiple states or events that occur over time. Typical example of facts with multiple states are insurance claims or flight reservations. There are also cases when the same entity is treated as both a fact and a dimension - for example, Customers. In such cases, a report could be generated that relates to the state of the fact, such as the time a claim was submitted to the time it was approved. 10

11 Physically creates and manages data marts Once defined, Attunity Compose automates the management of the data mart, and: Automates the building of the physical data mart tables Generates the ETL set to load the data mart from the data warehouse Makes incremental updates to the data mart as the data warehouse is updated Associates facts with the matching version of their dimensions according to a user-selected date attribute Out-of-the-box enablement for BI visualization tools Attunity Compose designs data marts to work with BI visualization tools out of the box. This is accomplished by exposing the applications to all the primary and foreign keys, and by using an application-friendly attribute name that matches with the tables in the data marts. This enables visualization applications like Microsoft PowerPivot, Tableau and Qlik to work on top of Attunity Compose as it generates data marts out of the box. 11

12 Operational Features and Life Cycle Management Workflow Designer, Monitor and Scheduler Keeping data warehouses and data marts up to date is a complex process comprised of multiple tasks. The different tasks included collecting data from multiple sources, staging it in the landing area, transforming and integrating the data before loading it into the data warehouse, and then loading the relevant information into a data mart. Attunity Compose provides a built-in workflow designer that enables you to run all of your data warehouse and data mart ETL tasks as a single, end-to-end process. In a project with a single data mart, all tasks run sequentially, starting with the data warehouse ETL tasks and ending with the data mart ETL task. However, in projects with multiple data marts, the data mart ETL tasks can be designed to run in parallel, following the completion of the data warehouse ETL tasks. The Attunity Compose monitor shows the current status of the workflows and independent tasks and enables users to drill-down for additional information. Workflows can be enabled to run immediately or scheduled to run in the future (either once or at set intervals). Notifications Attunity Compose can be configured to send notification messages about events that occur when Attunity Compose tasks are running. Notifications are set via the Attunity Compose monitor, where users select events, on the occurrence of which, a notification will be sent to the specified recipients. Users can manage notifications that they created from the Notifications list. This list provides information about each notification defined and lets users activate/ deactivate a notification. In addition, users can make changes to the definitions of existing notifications or delete them. Lineage and Impact Analysis Attunity Compose automatically collects and stores metadata during design and implementation stages. Attunity Compose connects directly to different data sources and collects information about the source schemas, tables, attributes, primary and secondary keys. The collected information is stored in the Attunity Compose central repository. This information provides a visual map of data flow from source data to data warehouse and data marts, then regenerates data lineage when changes are implemented. Before editing an entity or attribute, users should run the Impact Analysis report to check which other entities/ attributes in the entity s/attribute s lineage will be impacted by the change. For example, removing the Discount attribute from a table will affect the Total Price. Data Profiling and Quality Management: Attunity Compose can be used to profiles data and improve its quality before loading it into the data warehouse for accurate reporting. Data profiling features enable BI teams to validate data in the landing area tables to eliminate surprises as they execute on their data model. They can identify format discrepancies or issues such as null or blank fields, then repair them, for example, adjusting source tables or creating new rules, before loading the data. Migration between environments BI teams also can improve data quality by configuring and enforcing rules before loading data to automatically discover issues with values, formats, data ranges, duplicates, etc., and define exception policies (i.e., diverting exceptions to an error mart) and remediation tasks. Attunity Compose provides an Export/Import Project Configuration facility that lets users capture the configuration settings of an existing data warehouse project including databases, sources, data warehouse, data marts, transformations, filters, scheduling, and notifications. This is helpful, for example, when users need to move configuration settings from a test environment to a production environment. 12

13 Conclusion Attunity Compose enables businesses to advance their BI and analytics goals by providing a comprehensive, agile data warehouse automation platform for data architects. Attunity Compose uses key technologies and innovations to enable efficient, effective data warehouse lifecycle management. BI personnel are able to eliminate manual coding when designing, creating, populating and managing data warehouses and data marts. They can implement either a model driven approach, based on business process, or data driven approach based on source structures. Analytics productivity, time to value, quality and consistency all improve as a result. With the controls and process optimization of Attunity Compose, businesses can realize better insights with fewer resources to accelerate BI project ROI. Testimonial Adding Attunity Compose to our EDW environment versus our prior efforts is like comparing apples to oranges. We went from taking days to build a data mart down to just a single day. Data Architect at a U.S. furniture manufacturer 13

14 About Attunity Attunity is a leading provider of information availability solutions that enable access, sharing and distribution of data across heterogeneous enterprise platforms, organizations, and the cloud. Our software solutions include data replication, change data capture (CDC), data connectivity, enterprise file replication (EFR) and managed-file-transfer (MFT). Using Attunity s software solutions, our customers enjoy significant business benefits by enabling real-time access and availability of data and files where and when needed, across the maze of heterogeneous systems making up today s IT environment. Attunity has supplied innovative software solutions to its enterprise-class customers for nearly 20 years and has successful deployments at thousands of organizations worldwide. Attunity provides software directly and indirectly through a number of partners such as Microsoft, Oracle, IBM and HPE. Headquartered in Boston, Attunity serves its customers via offices in North America, Europe, and Asia Pacific and through a network of local partners. For more information, visit Over 2,000 companies, including half of the Fortune 500, trust Attunity solutions. Attunity exclusively focuses on getting the right data to the right place at the right time. The company engineers high-performance data management solutions that are fast to deploy and easy to operate, empowering enterprises to simply and cost-effectively ensure business-critical information is accessible and manageable when, where and how it s needed to become a more agile, intelligent enterprise. Americas sales@attunity.com Europe / Middle East / Africa 44 (0) info-uk@attunity.com Asia Pacific (852) info-hk@attunity.com 2016 Attunity LTD. All rights reserved.