Breadboard BI. Advanced ETL with Pentaho Data Integration By Christopher Lavigne

Transcription

1 Breadboard BI Advanced ETL with Pentaho Data Integration By Christopher Lavigne Introduction My recent quote in the industry press about the great potential of Pentaho Data Integration (formerly named Kettle) drew a lot of questions (and a few critical comments) from many of my peers. In general, they wanted to know why they should include Pentaho Data Integration in the list of tools they evaluate at the start of consulting engagements. While detailing two simple dimension & fact transformations, this white paper attempts to answer my colleagues en masse and introduce the community to the functionality native to Breadboard BI s ETL Solution Slices. Although many may benefit, the intended audience is proficient with at least one data integration tool and familiar with dimensional modeling concepts. Terms & Transformation Overview A brief attempt to compare terminology to other data integration tools may be helpful. I will use Informatica PowerCenter and Oracle Warehouse Builder (OWB) for reference. A Pentaho Transformation is equivalent to a PowerCenter Map or OWB Mapping. Pentaho Transformations are composed of multiple Steps, like PowerCenter Transformations or OWB Operators. Finally, a Pentaho Mapping corresponds to a PowerCenter Mapplet or an OWB Pluggable Map (in the OWB Beta). To see Pentaho Data Integration in a data warehouse context, refer to Figures 1 & 2. Figure 1 represents the Breadboard BI Order Type Dimension Transformation; Figure 2 represents the Breadboard BI Market Basket Fact Transformation.

2 Figure 1 Order Type Dimension Transformation Figure 2 Market Basket Fact Transformation Deceptively Simple, Fully Featured While the dimension and fact tables loaded by these Transformations are simple, the features offered by the Steps are quite robust. The dimension Transformation example includes the following functionality: error processing, slowly changing dimension type 1 (SCD1) or type 2 (SCD2) updates, surrogate keys, staging table maintenance, and audit table support. The fact Transformation example includes many of the same features, but also adds reusable object and late-arriving fact functionality. Each function is detailed in the following sections. Error Processing In many cases, it makes great sense to exclude data meaningless to your business users from your fact and dimension tables. After all, how useful is a sales representative without a name or an order without a product? Using Pentaho Data Integration, flexible error processing logic can be implemented with the following Step types: Filter, Add Constants, Update, and Dimension Lookup/Update. Examples of these Step types from the Order Type Dimension Transformation are displayed in Figure 3.

3 Figure 3 Error Processing Steps Subset This subset begins with the ROUTE_ROWS_WITH_ERRORS Filter step. It evaluates one or more data quality conditions defined by the business. If the row passes the quality checks, it continues to the DIMENSION_ORDER_TYPE Dimension Lookup/Update Step. However if the row fails the checks, it proceeds to the DW_ERROR_IND Add Constants Step to pick up a Y constant, then terminates at the STAGE_ORDER_TYPE_errors Update Step where the row is flagged (using the Y constant) as a failure in the staging table from which it originated. In our example, a single condition is tested, i.e., is the ORDER_TYPE_NAME value NULL. See Figure 4 for details.

4 Figure 4 Expanded ROUTE_ROWS_WITH_ERRORS Filter Step Slowly Changing Dimension Updates & Surrogate Keys Capturing changes to dimensional data across time has traditionally been a challenging aspect of designing a data warehouse. However, Pentaho Data Integration has made this process a whole lot easier. Slowly changing dimension and surrogate key functionality are supported by the Dimension Lookup/Update Step. This Step type, like a few other Step types, makes you realize the developer of this tool was a business intelligence architect that just happened to also know Java. (This very likely also applies to the other contributors to this project.) I have not encountered anything like this Step in any of the proprietary data integration tools on the market today. The best way to describe some of the advanced functionality offered by the Dimension Lookup/Update Step type is to walk through a few different data scenarios. In the first scenario, a row that is new to the DIMENSION_ORDER_TYPE table arrives at the DIMENSION_ORDER_TYPE Dimension

5 Lookup/Update Step. Based on the target table key you ve defined, the Step realizes it is new to the target table. Since the row is new, the Step increments the database sequence used to value the surrogate key, and then inserts the new row with the surrogate key value and other SCD support values into the DIMENSION_ORDER_TYPE_TABLE. Figure 4 displays the Table tab of the DIMENSION_ORDER_TYPE step. You can see that the dimension s surrogate key is named in the Technical key field, and the database sequence is defined in Optional Sequence field. (If your flavor of DBMS supports identity columns instead of sequences, you can check the use auto increment field? checkbox and leave the Optional Sequence field blank.) Other SCD support fields shown on the Table Tab include the Version field, Date range start field, Table date range end. Figure 5 Table Tab of the DIMENSION_ORDER_TYPE Step This functionality alone makes it a great Step type, but as they say on TV, wait, there s more! In a second data scenario, a particular key combination already exists in the target dimension, but the order type name has changed in the new row. (Based on a previously defined business rule, it was decided a change to this column will trigger a SCD2 response.) As such, the changed order type triggers a new insert into the order type dimension table with a new surrogate key, a new version number, and a new start date. Further, the row with the old order type name value will be assigned a stop date. In a third data scenario, a particular row already exists in the order type dimension,

6 but the order type description has changed. (Based on a previously defined business rule, it was decided this change will trigger a SCD1 response.) Accordingly, a changed order type description does NOT trigger a new insert, but only a SCD1 update to the existing row. Controlling the SCD behavior is quite simple using this Step type. If you look at Figure 6, you can see both the ORDER_TYPE_NAME and ORDER_TYPE_DESC entries. The only difference is the value in the Type of dimension update column. This amazing Step type gives you SCD control at the column level! Figure 6 Fields Tab of the DIMENSION_ORDER_TYPE Step Staging Table Maintenance If you use staging tables, then you may want to either hard or soft delete the rows that you have successfully loaded into your target dimensions or facts. The Order Type Dimension Transformation example utilizes the STAGE_ORDER_TYPE_delete Delete Step type to hard delete rows from the staging table. (Alternatively, the Update Step type could be used to flag stage tables as softdeleted). The implementation of hard deletes is quite easy - simply define the source system key in the Delete Step type, then place the Delete Step type after the step that loads your dimension or fact table. In Figure 7, you can see that the STAGE_ORDER_TYPE_delete step follows the DIMENSION_ORDER_TYPE step. Figure 7 Staging Table Maintenance Steps Subset

7 Audit Table Support Information about a particular data load can be used in a variety of ways (e.g., incremental load logic, system maintenance, etc.). The simple example depicted in Figure 8 uses the Get System Data and Insert/Update Step types to insert or update a single row into the ADMIN_LOAD_CONTROL. Specifically, the SET_ADMIN_LOAD_CONTROL Step adds the transformation name and the system date to the data flow that terminates at the ADMIN_LOAD_CONTROL table. Aside from these two elements, there are 33 additional pieces of information that could also be added to the data flow by this Step type. A few of these include: transformation file name, host name, ip address, user that last modified the transformation, last modified date, etc. Figure 8 Audit Table Support Subset Reusable Logic An extremely important element to any well-designed data integration system is the inclusion of reusable objects. From dimension surrogate key lookups to standard methods for computing gross margin, reusable objects speed the development and ease the maintenance of your system. Pentaho Data Integration uses the Mapping Step type to meet this need. As you can see from Figure 9, a particular Transformation can contain one or more Mapping Step types.

8 Figure 9 - Reusable Logic Subset Each Mapping Step type hides the underlying complexity of one or more reusable Steps. Figure 10 shows the simple MAP_LOOKUP_PRODUCT_1 Mapping Step expanded to reveal the underlying Steps. Figure 10 Expanded MAP_LOOKUP_PRODUCT_1 Mapping Type Step Late-Arriving Facts The final topic of this white paper is the implementation of late-arriving facts using Pentaho Data Integration. To demonstrate this functionality, we return to the Dimension Lookup/Update Step type. This Step type was discussed in the dimension context to demonstrate SCD and surrogate key functionality. In the context of a fact, it provides latearriving fact functionality. To take full advantage of this function, one only needs to add a date value to the Stream datefield field on the table tab of the Step. The example in Figure 11 lists the STREAM_DATE value in this field. In the fact table Transformation, this STREAM_DATE value could represent a transaction date, order date, accounting date, or any other date relevant to the facts you are loading. Once connected, the Dimension Lookup/Update Step type uses the value in the Stream datefield field to compare to the Daterange start field (DW_START_DATE) and the table daterange end (DW_STOP_DATE) to return the appropriate surrogate key from the DIMENSION_PRODUCT table for use in the FACT_MARKET_BASKET table.

9 Figure 11 Table Tab of the LOOKUP_DIMENSION_PRODUCT Step Summary Hopefully the error processing, SCD update, surrogate key, staging table maintenance, audit table support, reusable object, and late arriving functionality built into this tool are enough to explain my previous quote and convince you to include Pentaho Data Integration in your next tool evaluation task. If that is not enough, then you should also consider the fact that, out of the box, Pentaho Data Integration connects to just about every imaginable data store. This includes proprietary and open source databases, flat files, XML, AS400, etc. Finally, you might also consider the fact that the box is FREE. About Breadboard BI Breadboard BI is the leading provider of packaged analytic content in the open source business intelligence space. Breadboard s Solution Area content spans the customer, finance, supply chain, and workforce areas. Breadboard s Solution Slices - data models, ETL, OLAP cubes, Dashboards, and Reports - provide their clients and consulting partners with a fast track to project success. Visit us on the web at About Pentaho Pentaho provides a full spectrum of open source Business Intelligence (BI) capabilities including reporting, analysis, dashboards, data mining, data integration, and a BI platform. Formed by a highly experienced team of industry veterans, Pentaho s mission is to bring

10 innovative, high quality technology and professional support to the BI market. Pentaho uses a revolutionary approach to development, distribution and support made possible by an open source business model. Pentaho s technologies support a wide range of business initiatives from sales and profitability analysis, customer analysis, HR reporting, Financial reporting, KPI dashboards, Supply Chain analytics, and operational reporting. For more information: Pentaho s web site is located at