Oracle Data Integrator Best Practices for a Data Warehouse

Transcription

1 An Oracle White Paper August 2010 Oracle Data Integratr

2 Preface... 4 Purpse... 4 Audience... 4 Additinal Infrmatin... 4 Intrductin t Oracle Data Integratr (ODI)... 5 Objectives... 5 Business-Rules Driven Apprach... 5 Traditinal ETL versus E-LT Apprach... 6 Understanding Oracle Data Integratr Interfaces... 7 A Business Prblem Case Study... 8 Implementatin using Manual Cding Implementatin using Traditinal ETL tls Implementatin using ODI s E-LT and the Business-rule Driven Apprach 14 Benefits f E-LT Cmbined with a Business-rule Driven Apprach Using ODI in yur Data Warehuse Prject ODI and the Data Warehuse Prject Organizing the Teams Reverse-engineering, Auditing and Prfiling Surce Applicatins Designing and Implementing the Data Warehuse s Schema Specifying and Designing Business Rules Building a Data Quality Framewrk Develping Additinal Cmpnents Packaging and Releasing Develpment Versining Develpment Scheduling and Operating Scenaris Mnitring the Data Quality f the Data Warehuse Publishing Metadata t Business Users Planning fr Next Releases Oracle Data Integratr fr Oracle Best Practices Architecture f ODI Repsitries Reverse-engineering an Oracle Schema Oracle Lading Strategies Using Changed Data Capture Oracle Integratin Strategies Defining a Data Quality Strategy Setting up Agents in an Oracle envirnment Architecture Case Studies

3 Setting up Repsitries Using ODI Versin Management Ging t Prductin Setting up Agents Backing up Repsitries Appendices Appendix I. Oracle Data Integratr fr Teradata Best Practices Architecture f ODI Repsitries Reverse-engineering a Teradata Schema Teradata Lading Strategies Teradata Integratin Strategies Setting up Agents in a Teradata envirnment Appendix II: Additinal Infrmatin Acrnyms used in this dcument

4 Preface Purpse This dcument describes the best practices fr implementing Oracle Data Integratr (ODI) fr a data warehuse slutin. It is designed t help setup a successful envirnment fr data integratin with Enterprise Data Warehuse prjects and Active Data Warehuse prjects. This dcument applies t Oracle Data Integratr 11g. Audience This dcument is intended fr Data Integratin Prfessinal Services, System Integratrs and IT teams that plan t use Oracle Data Integratr (ODI) as the Extract, Lad and Transfrm tl in their Enterprise r Active Data Warehuse prjects. Additinal Infrmatin The fllwing resurces cntain additinal infrmatin: Oracle website: Oracle Data Integratr 11g n-line dcumentatin: Java reference : Jythn reference 4

5 Intrductin t Oracle Data Integratr (ODI) Objectives The bjective f this chapter is t Intrduce the key cncepts f a business-rule driven architecture Intrduce the key cncepts f E-LT Understand what an Oracle Data Integratr (ODI) interface is Thrugh a business prblem case study, understand and evaluate sme different develpment appraches including: Manual cding Traditinal ETL ODI s business-rule driven apprach cmbined with E-LT Business-Rules Driven Apprach Intrductin t Business rules Business rules specify mappings, filters, jins and cnstraints. They ften apply t metadata t transfrm data and are usually described in natural language by business users. In a typical data integratin prject (such as a Data Warehuse prject), these rules are defined during the specificatin phase in dcuments written by business analysts in cnjunctin with prject managers. Business Rules usually define What t d rather than Hw t d it. They can very ften be implemented using SQL expressins, prvided that the metadata they refer t is knwn and qualified in a metadata repsitry. Examples f business rules are given in the table belw: BUSINESS RULE TYPE SQL EXPRESSION Sum f all amunts f items sld during May 2010 multiplied by the item price Prducts that start with CPU and that belng t the hardware categry Custmers with their rders Mapping Filter Jin SUM( CASE WHEN SALES.YEARMONTH= THEN SALES.AMOUNT * PRODUCT.ITEM_PRICE ELSE 0 END ) Upper(PRODUCT.PRODUCT_NAME) like CPU% And PRODUCT.CATEGORY = HARDWARE CUSTOMER.CUSTOMER_ID = ORDER.ORDER_ID And ORDER.ORDER_ID = ORDER_LINE.ORDER_ID 5

6 BUSINESS RULE TYPE SQL EXPRESSION and rder lines Reject duplicate custmer names Reject rders with a link t an nn-existent custmer Unique Key Cnstraint Reference Cnstraint CONSTRAINT CUST_NAME_PK PRIMARY KEY (CUSTOMER_NAME) CONSTRAINT CUSTOMER_FK FOREIGN KEY (CUSTOMER_ID) REFERENCES CUSTOMER(CUSTOMER_ID) Mappings A mapping is a business rule implemented as a SQL expressin. It is a transfrmatin rule that maps surce clumns (r fields) nt ne f the target clumns. It can be executed by a relatinal database server at run-time. This server can be the surce server (when pssible), a middle tier server r the target server. Jins A jin peratin links recrds in several data sets, such as tables r files. Jins are used t link multiple surces. A jin is implemented as a SQL expressin linking the clumns (fields) f tw r mre data sets. Jins can be defined regardless f the physical lcatin f the surce data sets invlved. Fr example, a JMS queue can be jined t a relatinal table. Depending n the technlgy perfrming the jin, it can be expressed as an inner jin, right uter jin, left uter jin and full uter jin. Filters A filter is an expressin applied t surce data sets clumns. Only the recrds matching this filter are prcessed by the data flw. Cnstraints A cnstraint is an bject that defines the rules enfrced n data sets data. A cnstraint ensures the validity f the data in a given data set and the integrity f the data f a mdel. Cnstraints n the target are used t check the validity f the data befre integratin in the target. Traditinal ETL versus E-LT Apprach Traditinal ETL tls perate by first Extracting the data frm varius surces, Transfrming the data n a prprietary, middle-tier ETL engine, and then Lading the transfrmed data nt the target data warehuse r integratin server. Hence the term ETL represents bth the names and the rder f the peratins perfrmed, as shwn in Figure 1 belw. 6

7 Figure 1: Traditinal ETL apprach cmpared t E-LT apprach In respnse t the issues raised by ETL architectures, a new architecture has emerged, which in many ways incrprates the best aspects f manual cding and autmated cde-generatin appraches. Knwn as E-LT, this new apprach changes where and hw data transfrmatin takes place, and leverages existing develper skills, RDBMS engines and server hardware t the greatest extent pssible. In essence, E-LT mves the data transfrmatin step t the target RDBMS, changing the rder f peratins t: Extract the data frm the surce tables, Lad the tables int the destinatin server, and then Transfrm the data n the target RDBMS using native SQL peratrs. Nte, with E-LT there is n need fr a middle-tier engine r server as shwn in Figure 1 abve. Understanding Oracle Data Integratr Interfaces An interface is an ODI bject stred in the ODI Repsitry that enables the lading f ne target datastre with data transfrmed frm ne r mre surce datastres, based n business rules implemented as mappings, jins, filters and cnstraints. A datastre can be: a table stred in a relatinal database an ASCII r EBCDIC file (delimited, r fixed length) a nde frm a XML file a JMS tpic r queue frm a Message Oriented a nde frm a LDAP directry an API that returns data in the frm f an array f recrds Figure 2 shws a screensht f an ODI interface that lads data int the FACT_SALES target table. Surce Data is defined as a hetergeneus query n the CORRECTIONS file, the ORDERS and LINES tables. 7

8 Mappings, jins, filters and cnstraints are defined within this windw. Figure 2: Example f an ODI Interface Wherever pssible, ODI interfaces generate E-LT peratins that relegate transfrmatins t the target RDBMS servers. A Business Prblem Case Study Figure 3 describes an example f a business prblem t extract, transfrm and lad data frm a Micrsft SQL Server database and a file int a target Oracle table. Data is cming frm tw Micrsft SQL Server tables (ORDERS jined t LINES) and is cmbined with data frm the CORRECTIONS file. The target SALES Oracle table must match sme cnstraints such as the uniqueness f the ID clumn and valid reference t the SALES_REP table. Data must be transfrmed and aggregated accrding t sme mappings as shwn in Figure 3. 8

9 Figure 3: Example f a business prblem Translating these business rules frm natural language t SQL expressins is usually straightfrward. In ur example, the rules that appear in the figure culd be translated as fllws: TYPE RULE SQL EXPRESSION / CONSTRAINT Filter Only ORDERS marked as clsed ORDERS.STATUS = CLOSED Jin Mapping A rw frm LINES has a matching ORDER_ID in ORDERS Target s SALE is the sum f the rder lines AMOUNT gruped by sales rep., with the crrectins applied. ORDERS.ORDER_ID = LINES.ORDER_ID SUM(LINES.AMOUNT + CORRECTIONS.VALUE) Mapping Sales Rep = Sales Rep ID frm ORDERS ORDERS.SALES_REP_ID Cnstraint ID must nt be null ID is set t nt null in the data mdel Cnstraint ID must be unique A Primary Key is added t the data mdel with (ID) as set f clumns Cnstraint The Sales Rep. ID shuld exist in the Target sales Rep table A Reference (Freign Key) is added in the data mdel n SALES.SALES_REP = SALES_REP.SALES_REP_ID 9

10 Implementatin using Manual Cding When implementing such a data flw using, manual cding, ne wuld prbably use several steps, several languages, and several scripting tls r utilities. Figure 4 gives an verview f the different steps needed t achieve such an extract, transfrm and lad prcess. Figure 4: Sequence f Operatins fr the Prcess There are, f curse, several technical slutins fr implementing such a prcess. One f them (prbably the mst efficient, as it uses an Oracle data warehuse as a transfrmatin engine) is detailed in the fllwing table: STEP DESCRIPTION EXAMPLE OF CODE 1 Execute the jin between ORDERS and LINES as well as the filters n the surce Micrsft SQL Server database using a database view. Create view C$_SALES As select... frm ORDERS, LINES where ORDERS.STATUS = CLOSED and ORDERS.ORDER_ID = LINES.ORDER_ID Extract the cntent f the view int a flat file using the BCP utility. bcp C$_SALES ut c$_sales_extract.bcp c -S... -U... -P... t\b Use the SQL*Lader utility t lad the temprary BCP file int the TEMP_1 Oracle table. 2 Use the SQL*Lader utility t lad the CORRECTIONS ASCII file int the TEMP_2 Oracle table. 3 Jin, transfrm and aggregate the 2 temprary tables TEMP_1 and TEMP_2 and lad the results int a 3 rd table (TEMP_SALES) using SQL sqlldr cntrl=temp_1.ctl lg=lgfile.lg userid=.../... sqlldr cntrl=temp_2.ctl lg=lgfile.lg userid=.../... insert int TEMP_SALES (...) select SUM(TEMP_1.AMOUNT+TEMP_2.VALUE), TEMP1.SALES_REP_ID,... frm TEMP_1, TEMP_2 where TEMP_1.LINEID = TEMP_2.CORR_ID) 10

11 STEP DESCRIPTION EXAMPLE OF CODE... 4 Check Unique cnstraints using SQL and insert the errrs int the Errrs table insert int Errrs(...) select... frm TEMP_SALES where ID in (select ID frm TEMP_SALES grup by ID having cunt(*) > 1) Check Reference cnstraints using SQL and insert the errrs int the Errr table insert int Errrs(...) select... frm TEMP_SALES where SALES_REP nt in (select SALES_REP_ID frm SALES_REP) 5 Finally, use SQL lgic t insert / update int the target SALES table using a query n TEMP_SALES update SALES set... frm... where ID in (select ID frm TEMP_SALES where IND_UPDATE= U ) insert int SALES (...) select... frm TEMP_SALES where IND_UPDATE= I... The benefits f this apprach are: High perfrmance: Cde flexibility: Uses pure set-riented SQL t avid rw-by-rw peratins Uses Oracle as a transfrmatin engine t leverage the pwer f the RDBMS Uses in-place utilities such as External Tables Leverages the latest features f Oracle such as the built-in transfrmatin functins 11

12 Hwever, this apprach raises several issues that becme painful as the Enterprise Data Warehuse prjects grw, and mre develpers get invlved. These issues are: Pr prductivity Every lad prcess needs t be develped as a set f scripts and prgrams, within different envirnments and with several languages. Business rules ( what happens t the data SUM(AMOUNT + VALUE)) are mixed with technical lgic ( hw t mve / lad the data SQL*Lader, External Table, insert etc.) Mving t prductin is ften difficult when develpers haven t designed envirnment variables, r variable qualified names fr their bjects. High cst f data quality implementatin Hard maintenance N prject flexibility Data cleansing / data quality accrding t predefined cnstraints is usually avided due t the cst f its implementatin Every prject has its wn definitin f the data quality withut any centralized framewrk (default structure f the errr tables, errr recycling etc.) Even when prject managers have set up a framewrk, every script may reinvent the wheel and cntain specific rutines that make it hard t understand Develpments are spread in several machines and flders, withut a central repsitry Impact analysis is impssible as there is n metadata management and n crss references mechanism The cst f a change t the data mdels r t the business rules becmes such a cnstraint that IT teams refuse it, leading t frustratin amngst the business users. Implementatin using Traditinal ETL tls Traditinal ETL tls perfrm all the transfrmatins in a prprietary engine. They ften require additinal hardware t stage the data fr the transfrmatins. Nne f them really leverages the pwer f the RDBMS. A typical ETL architecture is shwn in Figure 5. 12

13 Figure 5: Implementatin Using an ETL Every transfrmatin step requires a specific cnnectr r transfrmer. ETL tls are ften knwn fr the fllwing advantages: Centralized develpment and administratin Easier maintenance Single graphical user interface Centralized repsitry Impact analysis (fr certain tls) Unfrtunately this ETL apprach presents several drawbacks: Pr perfrmance Bad prductivity As the data needs t be prcessed in the engine, it is ften prcessed rw by rw When data frm the target database is referenced - table lkups fr example - it needs t be extracted frm the database, int the engine and then mved back again t the target database. Very few mappings, jins, aggregatins and filters are given t the pwerful engine f the RDBMS Every lad prcess needs t be develped as a set f steps that mix business rules ( what happens t the data SUM(AMOUNT + VALUE)) with technical lgic ( hw t lad the data cnnectr 1, cnnectr 2 etc.) Mving t prductin is ften difficult when develpers haven t designed envirnment variables, r variable qualified names within their queries Sme f them still require the use f heavy manual cding t achieve certain particular tasks, r t leverage the RDBMS pwerful transfrmatin functins 13

14 High Cst ETL tls require additinal hardware ETL tls require specific skills Implementatin using ODI s E-LT and the Business-rule Driven Apprach Implementing a business prblem using ODI is a very easy and straightfrward exercise. It is dne by simply translating the business rules int an interface. Every business rule remains accessible frm the Diagram panel f the interface s windw. Specifying the Business Rules in the Interface Figure 6 gives an verview f hw the business prblem is translated int an ODI interface: The ORDERS, LINES and CORRECTION datastres are dragged and drpped int the Surce panel f the interface The Target SALES datastre is drpped in the Target Datastre panel Jins and filters are defined by dragging and drpping clumns in the Surce panel Mappings are defined by selecting every target clumn and by dragging and drpping clumns r by using the advanced expressin editr. Cnstraints are defined in the Cntrl tab f the interface. They define hw flw data is ging t be checked and rejected int the Errrs table. Figure 6: Implementatin using Oracle Data Integratr 14

15 Business Rules are Cnverted int a Prcess Business rules defined in the interface need t be split int a prcess that will carry ut the jins, filters, mappings and cnstraints frm surce data t target tables. Figure 7 defines the prblem t be slved. Figure 7: Hw t cnvert business rules int a prcess? By default, Oracle Data Integratr will use the RDBMS as a staging area fr lading surce data int temprary tables and applying all the required mappings, staging filters, jins and cnstraints. The staging area is a separate area in the RDBMS (a user/database) where ODI creates its temprary bjects and executes sme f the rules (mapping, jins, final filters, aggregatins etc). When perfrming the peratins this way, ODI leverages the E-LT architecture as it first extracts and lads the temprary tables and then finishes the transfrmatins in the target RDBMS. In sme particular cases, when surce vlumes are small (less than 500,000 recrds), this staging area can be lcated in memry in ODI S in-memry relatinal database ODI Memry Engine. ODI wuld then behave like a traditinal ETL tl. Figure 8 shws the prcess autmatically generated by Oracle Data Integratr t lad the final SALES table. The business rules, as defined in Figure 7 will be transfrmed int cde by the Knwledge Mdules (KM). The cde prduced will generate several steps. Sme f these steps will extract and lad the data frm the surces t the staging area (Lading Knwledge Mdules - LKM). Others will transfrm and integrate the data frm the staging area t the target table (Integratin Knwledge Mdule - IKM). T ensure data quality, the Check Knwledge Mdule (CKM) will apply the user defined cnstraints t the staging data t islate errneus recrds in the Errrs table. 15

16 Figure 8: ODI Knwledge Mdules in actin ODI Knwledge Mdules cntain the actual cde that will be executed by the varius servers f the infrastructure. Sme f the cde cntained in the Knwledge Mdules is generic. It makes calls t the ODI Substitutin API that will be bund at run-time t the business-rules and generates the final cde that will be executed. Figure 9 illustrates this mechanism. During design time, rules are defined in the interfaces and Knwledge Mdules are selected. During run-time, cde is generated and every API call in the Knwledge Mdules (enclsed by <% and %>) is replaced with its crrespnding bject name r expressin, with respect t the metadata prvided in the Repsitry. Fr example, a call t <%=diref.gettable( TARG_NAME )%> will return the name f the target table f the interface with the apprpriate qualifier accrding t cntext infrmatin, tplgy setup etc. A typical SQL INSERT statement wuld be cded in a Knwledge Mdule as fllws: INSERT INTO <%=diref.gettable( TARG_NAME )%>... This template f cde will f curse generate different SQL statements depending n the target table ( INSERT INTO MyDB1.SALES... when the target is the SALES table, INSERT INTO DWH_DB.PRODUCT when the target is the PRODUCT table etc.). This is als particularly useful when migrating ODI prcesses frm an envirnment t anther (prmting prcesses frm Develpment t QA fr example) as ODI will autmatically substitute the crrect 16

17 schema infrmatin based n the specified envirnment withut requiring any cde mdificatins. Figure 9: Hw Knwledge Mdules generate native cde Once the cde is generated, it is submitted t an ODI Agent, which will either redirect it t the apprpriate database engines and perating systems, r will execute it when needed (memry engine transfrmatin, java r jythn cde etc.) In mst cases, the agent is simply a cnductr that des nt tuch the data. Benefits f E-LT Cmbined with a Business-rule Driven Apprach Cmpared t ther architectures (manual cding and traditinal ETL), ODI mixes the best f bth wrlds: Prductivity / Maintenance The business-rules driven apprach delivers greater prductivity as develpers simply need t cncentrate n the What withut caring abut the Hw. They define SQL expressins fr the business rules, and ODI Knwledge Mdules generate the entire set f SQL peratins needed t achieve these rules. When a change needs t be made in peratinal lgic (such as creating a backup cpy f every target table befre lading the new recrds ), it is 17

18 High Perfrmance: Lw Cst: simply applied in the apprpriate Knwledge Mdule and it autmatically impacts the hundreds f interfaces already develped. With a traditinal ETL apprach, such a change wuld have necessitated pening every jb and manually adding the new steps, increasing the risk f mistakes and incnsistency. Flexibility and a shallw learning curve are ensured by leveraging the RDBMS latest features. With a centralized repsitry that describes all the metadata f the surces and targets and a single unified and cmprehensive graphical interface, maintenance is greatly ptimized as crss-references between bjects can be queried at any time. This gives the develpers and the business users a single entry pint fr impact analysis and data lineage ( What is used where?, Which surces ppulate which targets? etc.) In the ODI repsitry, the tplgy f the infrastructure is defined in detail, and mving bjects between different executin cntexts (Develpment, Testing, QA, Prductin, etc.) is straightfrward. With a pwerful versin cntrl repsitry, several teams can wrk n the same prject within different release stages, with guaranteed cnsistency f deliverables. With a centralized framewrk fr Data Quality, develpers spend less time n defining technical steps, and mre time n the specificatin f data quality rules. This helps t build a cnsistent and standardized Data Warehuse. The E-LT architecture leverages the pwer f all the features f in-place databases engines. ODI generates pure set-riented SQL ptimized fr each RDBMS which can take advantage f advanced features such as parallel prcessing r ther advanced features. Native database utilities can be invked by the ODI Knwledge Mdules prvided. When data frm the target database is referenced - table lkups fr example, it desn t need t be extracted frm the database, int an engine. It remains where it is, and it is prcessed by database engine. Oracle Data Integratr desn t require a dedicated server. The lads and transfrmatins are carried ut by the RDBMS. In cnclusin, with its business-rule driven E-LT architecture, Oracle Data Integratr is the best slutin fr taking advantage f bth manual cding and traditinal ETL wrlds. 18

19 Using ODI in yur Data Warehuse Prject ODI and the Data Warehuse Prject The main gal f a Data Warehuse is t cnslidate and deliver accurate indicatrs t business users t help them make decisins regarding their everyday business. A typical prject is cmpsed f several steps and milestnes. Sme f these are: Defining business needs (Key Indicatrs) Identifying surce data that cncerns key indicatrs; specifying business rules t transfrm surce infrmatin int key indicatrs Mdeling the data structure f the target warehuse t stre the key indicatrs Ppulating the indicatrs by implementing business rules Measuring the verall accuracy f the data by setting up data quality rules Develping reprts n key indicatrs Making key indicatrs and metadata available t business users thrugh ad-hc query tls r predefined reprts Measuring business users satisfactin and adding/mdifying key indicatrs Oracle Data Integratr will help yu cver mst f these steps, frm surce data investigatin t metadata lineage and thrugh lading and data quality audit. With its repsitry, ODI will centralize the specificatin and develpment effrts and prvide a unique architecture n which the prject can rely t succeed. Organizing the Teams As Oracle Data Integratr relies n a centralized repsitry, different types f users may need t access it. The list belw describes hw ODI may be used by yur teams. PROFILE DESCRIPTION ODI MODULES USED Business User Business Analyst Develper Business users have access t the final calculated key indicatrs thrugh reprts r ad-hc queries. In sme cases, they need t understand what the definitin f the indicatrs is, hw they are calculated and when they were updated. Alternatively, they need t be aware f any data quality issue regarding the accuracy f their indicatrs. Business Analysts define key indicatrs. They knw the surce applicatins and specify business rules t transfrm surce data int meaningful target indicatrs. They are in charge f maintaining translatin data frm peratinal semantics t the unified data warehuse semantic. Develpers are in charge f implementing the business rules in respect t the specificatins described by the Business Analysts. They release their wrk by prviding executable scenaris t the prductin team. Develpers must have ODI Cnsle Designer Navigatr (limited access) ODI Cnsle Tplgy Navigatr (read nly access) Designer Navigatr: 19

20 PROFILE DESCRIPTION ODI MODULES USED bth technical skills regarding the infrastructure and business knwledge f the surce applicatins. Limited access t Mdels Metadata Administratr Metadata Administratrs are in charge f reverse-engineering surce and target applicatins. They guarantee the verall cnsistency f Metadata in the ODI Repsitry. They have an excellent knwledge f the structure f the surces and targets and they have participated in the data mdeling f key indicatrs. In cnjunctin with Business Analysts, they enrich the metadata by adding cmments, descriptins and even integrity rules (such as cnstraints). Metadata Administratrs are respnsible fr versin management. Full access t Prjects Operatr Navigatr ODI Cnsle Tplgy Navigatr (limited access) Designer Navigatr: Full access t Mdels Restre access t Prjects ODI Cnsle Database Administratr System Administratr Security Administratr Operatr Database Administratrs are in charge f defining the technical database infrastructure that supprts ODI. They create the database prfiles t let ODI access the data. They create separate schemas and databases t stre the Staging Areas. They make the envirnments accessible by describing them in the Tplgy System Administratrs are in charge f maintaining technical resurces and infrastructure fr the prject. Fr example, they may install and mnitr Scheduler Agents backup / restre Repsitries install and mnitr ODI Cnsle Setup envirnments (develpment, test, maintenance etc.) The Security Administratr is in charge f defining the security plicy fr the ODI Repsitry. He r she creates ODI users and grants them rights n mdels, prjects and cntexts. Operatrs are in charge f imprting released and tested scenaris int the prductin envirnment. They schedule the executin f these scenaris. They mnitr executin lgs and restart failed sessins when needed. Tplgy Navigatr (full access) Designer Navigatr (full access) Operatr Navigatr (full access) ODI Cnsle Agents Tplgy Navigatr (limited access) ODI Cnsle Security Navigatr (full access) Designer Navigatr (read access) Tplgy Navigatr (read access) ODI Cnsle Operatr Navigatr ODI Cnsle Oracle Enterprise Manager Plug-in Fr ODI 20

21 The ODI Master Repsitry cntains built-in default prfiles that can be assigned t users. The fllwing table suggests hw t use these built-in prfiles: PROFILE Business User Business Analyst Develper Metadata Administratr Database Administratr System Administratr Security Administratr Operatr BUILT-IN PROFILE IN ORACLE DATA INTEGRATOR CONNECT, NG REPOSITORY EXPLORER CONNECT, NG REPOSITORY EXPLORER, NG DESIGNER CONNECT, DESIGNER CONNECT, METDATA ADMIN, VERSION ADMIN CONNECT, DESIGNER, METADATA ADMIN, TOPOLOGY ADMIN CONNECT, OPERATOR CONNECT, SECURITY ADMIN CONNECT, OPERATOR Reverse-engineering, Auditing and Prfiling Surce Applicatins A gd starting pint fr prjects is t understand the cntents and the structure f surce applicatins. Rather than having paper-based dcumentatin n surce applicatins, a gd practice is t cnnect t these surce applicatins using ODI and t capture their metadata. Once this is achieved, it is usually helpful t define sme data quality business rules in ODI Repsitry. This helps check the data cnsistency in the surce and validate yur understanding f the data mdels. At this phase f the prject it is imprtant t start having an answer t the fllwing questins: Hw many different surces d we have t take in accunt t calculate ur indicatrs? Is the data needed fr ur indicatrs present in the surce systems? What data quality challenges will we have t address t ensure the accuracy f the target warehuse? What surce system(s) will prvide the master data (dimensins) that will be referenced by ur indicatrs? What data vlumes are we ging t manipulate? And s frth. In sme cases, surce applicatins are nt accessible directly. Only ASCII r binary files extracts frm these applicatins are prvided. Starting t wrk with these surce files befre implementing the data warehuse mdel is recmmended, as they represent the visin f the truth that the prductin surce systems want t give yu. The metadata f these files shuld typically be described in the repsitry. Samples f the files can als be initially laded int temprary target tables t validate their structure and cntent. 21

22 All this can be implemented in Oracle Data Integratr as fllws: Cnnect surce applicatins r files in Tplgy Navigatr Define a lgical architecture in Tplgy Navigatr Create ne mdel per lgical schema in Designer Navigatr Reverse Engineer mdels when pssible r manually describe datastres Use standard JDBC Reverse-engineering t get database metadata Use Custmized Reverse-engineering strategies (Reverse Knwledge Mdules) when standard JDBC reverse is nt applicable (r nt accurate) Use Cbl Cpy Bk imprt fr ASCII r binary files if available Use Delimited File Reverse fr ASCII delimited files Enrich the metadata by adding infrmatin when it is nt available: Datastres and Clumns descriptins and titles Unique keys infrmatin (Primary Keys, Alternate Keys) Referential Integrity assumptins between datastres (Freign Keys) Check cnstraints (fr example t check the dmain f values fr a clumn) When the surce data is stred in files, develp simple interfaces t lad these files int a temprary area in the target database t better evaluate their level f data quality. Lcate the imprtant entities in the surce data and use the fllwing features t prfile the cntents: View Data n a datastre View distributin f values fr a clumn f a datastre Cunt the number f recrds Run ad-hc SQL queries frm Designer Navigatr Validate the cnstraints that yu have defined in the mdels by perfrming data quality cntrl: Schedule r interactively run data quality static cntrl n the datastres and mdels Access and understand the cntents f the errr tables Prpse alternatives fr handling data discrepancies in the future Define the acceptable level f data quality fr the target warehuse Of curse, these actin items shuld be dne in cnjunctin with business analysts. Nrmally, in parallel t this phase, yu will be designing the data mdel f the target Data Warehuse. 22

23 Data quality audit errrs and data prfiling will help yu refine yur understanding f the truth, and cnsequently, will lead yu t a better mdeling f the target warehuse. Designing and Implementing the Data Warehuse s Schema This sectin culd be a bk n its wn. The gal here is nt t prvide yu with guidelines in Data Warehuse mdeling, but rather t give sme general hints and tips that may affect yur ODI develpment. It is always a gd practice t have an Operatinal Data Stre (ODS) t stre raw peratinal data. Usually, the data mdel f the ODS is very clse t the OLTP surce applicatins data mdel. Any surce data shuld be accepted in the ODS and almst n data quality rule shuld be implemented. This ensures t have a stre representing all the data-f-the-day frm the peratinal systems. When designing the Data Warehuse schema, the fllwing general tips may be helpful: Where pssible, describe clumns inside the data dictinary f the database (fr example use COMMENT ON TABLE and COMMENT ON COLUMN SQL statements). By ding s, yu allw ODI t retrieve these cmments and descriptins int its metadata repsitry. Design a strage space fr the Staging Area where ODI will create its necessary temprary tables and data quality errr tables. D nt use primary keys f the surce systems as primary key fr yur target tables. Use cunters r identity clumns whenever it is pssible. This makes a flexible data mdel that will easily evlve ver time. Design referential integrity (RI) and reverse engineer freign keys in ODI mdels. D nt implement these freign keys in the target database as they may lead t perfrmance issues. With the autmatic data quality checks, ODI will guarantee the cnsistency f data accrding t these RI rules. Standardize yur bject naming cnventins. Fr example, use three letters t prefix table names by subject area. Avid using very lng names as ODI may add a prefix t yur table names when creating temprary bjects (fr example E$_Custmer will be the errr table fr table Custmer). Use f either 3 rd Nrmal Frm mdeling (3NF) r dimensinal mdeling ( Snw Flakes r Star Schemas ) desn t have any impact n ODI, but rather n the way yu will design yur future business rules develpment and business users reprts. Once the ODS and DW are designed, yu will have t create the apprpriate mdels in Designer and perfrm reverse-engineering. ODI cmes with mdeling capabilities in the frm f a feature called Cmmn Frmat Designer. It can help yu design the structure f the ODS and the DW frm the existing structures f the surce systems by simply drag and drpping tables clumns and cnstraints. 23

24 The generated Data Descriptin Language statements (DDL) autmatically include specific features f the target database. It als keeps track f the rigin f surce clumns t autmatically generate the interfaces fr lading the target, thus resulting in significant time savings. Best practices n the usage f Cmmn Frmat Designer are ut f the scpe f this dcument. Specifying and Designing Business Rules Usually this step starts almst at the same time as the design f the target schema. Understanding the business needs is clearly a key factr in the success f the verall prject. This influences the key indicatrs selected fr the warehuse as well as the rules used t transfrm surce t target data. The mre accurate the specificatin f a business rule, the easier the design f this rule in ODI. Several successful prjects have used ODI Designer t specify the business rules, thus aviding having discrepancies between a Wrd dcument and the final develped rules. The fllwing table summarizes what needs t be specified fr every target table f the ODS r the DW befre designing business rules. TARGET DATASTORE: DESCRIPTION OF THE TRANSFORMATION: INTEGRATION STRATEGY: Give the name f the target datastre, pssibly preceded by a qualificatin prefix (such as the database r schema name) Give a shrt descriptin f the purpse f the transfrmatin Define hw data shuld be written t the target. Examples: Replace cntent with new data frm surces Append incming data nt the target Update existing recrds and insert new nes accrding t an update key (specify the key here) Use a Slwly Changing Dimensin strategy. Specify the surrgate key, the slwly changing attributes, the updatable nes, the clumns fr start and end date etc. A hme-made strategy (fr example: BASEL II cmpliance, SOX histrical audit trail etc.) Every strategy specified here will crrespnd t an ODI Integratin Knwledge Mdule. REFRESH FREQUENCY: DEPENDENCIES: SOURCE DATASTORES: Specify when this datastre shuld be laded (every night, every mnth, every 2 hurs etc) Give all dependencies in the lad prcess (e.g. what datastres have t be laded prir t the current ne; what specific jbs need t be executed prir t this ne etc.) Give the list f surce datastres invlved fr lading the target table. This list shuld include all the lkup tables as well. Fr each surce datastre, include the fllwing infrmatin. SOURCE ODI MODEL Give the name f the surce mdel as it appears in Designer DATASTORE NAME Give the name f the datastre 24

25 PURPOSE/DESCRIPTION FIELD MAPPINGS AND TRANSFORMATIONS: TARGET COLUMN Indicate whether it is the main surce set r a lkup ne. If Change Data Capture is t be used, it shuld be indicated here. Fr every target field (r clumn), indicate what transfrmatins need t be applied t the surce fields. These transfrmatins shuld be written as expressins r frmulas as ften as pssible. Give the target clumn name MAPPING DESCRIPTION Describe the purpse f the mapping MAPPING EXPRESSION Give the expressin using the surce clumn names. Try t express it in pseud cde. LINKS OR JOIN CRITERIA DATASTORE 1 Fr every pair f surce datastres, specify the criteria t lkup matching recrds. This is ften knwn as an SQL jin. Name f the first datastre DATASTORE 2 Name f the secnd datastre LINK EXPRESSION Give the expressin used t link the 2 datastres in pseud cde DESCRIPTION Describe this link by specifying if it is a left, right r full uter jin. FILTERS: FILTER DESCRIPTION Give the list f filters that apply t surce data. Filters shuld be expressed in natural language and in pseud cde whenever pssible. Describe this Filter. FILTER DESCRIPTION Give the expressin used t implement the filter in pseud cde. DATA QUALITY REQUIREMENTS: CONSTRAINT NAME DESCRIPTION CONSTRAINT EXPRESSION List here all the data quality requirements including errr recycling if apprpriate. The requirements shuld be expressed as cnstraints whenever pssible. Name r shrt descriptin f the cnstraint t validate. Purpse f the data quality check Expressin in pseud cde required t cntrl the data T illustrate the use f such a frm, the fllwing table gives an example fr the business prblem case study defined in the chapter Intrductin t Oracle Data Integratr (ODI). TARGET DATASTORE: DESCRIPTION OF THE TRANSFORMATION: INTEGRATION STRATEGY: REFRESH FREQUENCY: DEPENDENCIES: Oracle Warehuse.SALES Aggregate rders and rder lines frm the Micrsft SQL Server surce prductin system. Append new recrds f the day Every night The fllwing tables need t be laded prir t this ne fr referential integrity: PRODUCT SALES_REP SOURCE DATASTORES: 25

26 SOURCE ODI MODEL DATASTORE NAME PURPOSE/DESCRIPTION Micrsft SQL Server Surce ORDERS Orders frm the prductin system Micrsft SQL Server Surce LINES Order lines. Main surce fr aggregatin User Data Files CORRECTIONS File cntaining the value t add t the amunt sld if any manual crrectin has ccurred fr a particular rder line. FIELD MAPPINGS AND TRANSFORMATIONS: TARGET COLUMN MAPPING DESCRIPTION MAPPING EXPRESSION ID Order Id ORDERS.ORDER_ID PRODUCT_PK Prduct Id as it appears in the rder lines LINES.PRODUCT_ID SALES_REP Sales Rep Id as it appears in the rder ORDERS.SALES_REP_ID SALE Amunt sld. If a crrectin fr this amunt exists in the crrectins file, the value shuld be added. The ttal value needs t be summed. SUM ( LINES.AMOUNT + (CORRECTIONS.VALUE when it exists) ) QUANTITY Ttal quantity f prduct sld. SUM ( LINES.QUANTITY ) LINKS OR JOIN CRITERIA DATASTORE 1 DATASTORE 2 LINK EXPRESSION DESCRIPTION ORDERS LINES Link rders and rder lines n the rder ID. Every rder line must match an existing rder LINES CORRECTIONS Lkup a crrectin value frm the crrectin file if it exists given the line item ID (left jin) ORDERS.ORDER_ID = LINES.ORDER_ID LINES.LINE_ITEM_ID = CORRECTIONS.CORR_ID Filters: FILTER DESCRIPTION Orders f the day Orders validated FILTER EXPRESSION ORDER.ORDER_DATE between yesterday and nw ORDER.STATUS = Clsed DATA QUALITY REQUIREMENTS: CONSTRAINT DESCRIPTION CONSTRAINT EXPRESSION Reference t SALES_REP Reference t PRODUCT Ensure that every SALES_REP ID exists in the SALES_REP reference table Ensure that every PRODUCT_PK exists in the PRODUCT reference table SALES_REP references SALES_REP(ID) PRODUCT_PK references PRODUCT(ID) ID Nt null ID Clumn is mandatry Check ID nt null Quantity greater than 0 The quantity sld must always be psitive QUANTITY > 0 Uniqueness f sale A sales rep can t sell the same prduct twice inside the same rder (PRODUCT_PK, SALES_REP) is unique 26

27 The next step is t design these business rules in the Designer Navigatr. The translatin frm the specificatin t the design is straightfrward. Every specificatin f the lading f a target table will be cnverted int an interface. This prcess is even faster if the specificatin is made directly using the Designer Navigatr. The steps t implement the interface are typically as fllws. Drag and drp the target datastre Drag and drp surce datastres Fr every target field, translate the pseud cde f the transfrmatin int SQL expressins. When pssible, decide where t execute the expressin (surce, staging area, target) Fr every jin, define the jin in the surce panel f the interface. Depending n the specificatin, define the jin as inner, left, right, natural r full uter jin. When pssible, try t execute jins n the surces t minimize netwrk traffic. Fr every filter, translate the pseud cde int an SQL expressin. When the surce database accepts SQL filters, set it as execute n surce t minimize netwrk traffic. In the flw diagram f the interface, define the lading strategies (LKM) and specify the integratin strategy (IKM) t match the specificatin. Chse the apprpriate ptins fr the Knwledge Mdule and activate the data quality cntrl when apprpriate. In the cntrl tab f the interface, select the cnstraints t cntrl. If the cnstraints are nt defined n the target datastre, define them in the Mdels view f the Designer Navigatr. After the design f every interface, yu shuld review the cde generated using the Simulatin mde and then test it by executing it. The Operatr Navigatr allws yu t easily fllw the executin f yur interface t get the number f recrds prcessed and many ther useful indicatrs (such as the generated cde, the elapsed time, the number f inserts, updates, deletes etc.) After the interface cmpletes, the resulting data and errrs can be directly selected frm Designer t validate the accuracy f the business rules. Thanks t ODI Cntexts, the executin f the interface will happen n a Develpment envirnment and will nt affect prductin data. Building a Data Quality Framewrk With an apprach based n business rules, ODI is certainly the mst apprpriate tl t help yu build a data quality framewrk t track data incnsistencies. Thanks t the Check Knwledge Mdules, yu simply have t define the cntrl business rules, and incnsistent data is autmatically islated fr yu in errr tables. Hwever, islating errneus data is nt the nly issue in Data Quality. Even if ODI autmatically detects duplicated keys, mandatry fields, missing references, and mre cmplex cnstraints, yu will need t invlve business users in the prcess f qualifying data discrepancies and make the apprpriate decisins. Yu 27

28 shuld have an answer t each f the fllwing questins befre defining yur data quality strategy: What level f data quality is required fr the Data Warehuse? Wh are the business wners f surce data? What shuld we d with rejected recrds? D we need t define an errr recycling strategy? D we need t invlve business wners f the data and reprt rejected recrds? Hw wuld business users mdify errneus surce data? D we prvide a GUI t have them mdify errneus recrds in the errr tables? The dcument Cmprehensive Data Quality with Oracle Data Integratr further discusses Data Quality issues and the recmmended best practices. Develping Additinal Cmpnents Nt all the typical tasks f the lading f the Data Warehuse can be achieved using ODI Interfaces. It is usual t develp additinal cmpnents r jbs that wuld carry n tasks such as, fr example: Receiving and sending s Cpying, mving, cncatenating, renaming files in the file system Cmpressing, decmpressing files Executing web services Writing and executing shell scripts fr a specific perating system Writing and executing java prgrams And s n. These cmpnents can be develped and tested within the Designer Navigatr as Prcedures, Variables, User Functins, r Steps in a package. ODI Prcedures ffer a brad range f pssibilities fr develping these cmpnents. They can include, fr example: An ad-hc SQL statement written fr any database An Operating System call ODI built-in tls and APIs (Send Mail, Read Mail, Wait Fr File etc.) Cde written in any f the scripting languages supprted by the Jakarta Bean Scripting Framewrk. This includes Java, Java Script, Pythn, Perl, NetRexx, Grvy, etc. Of curse, the risk here is t start develping transfrmatin prcesses as Prcedures by hand cding shell and SQL scripts rather than using the pwerful mechanism f ODI Interfaces 28

29 cmbined with Knwledge Mdules. T avid that, try as much as pssible t specify yur transfrmatins as business rules and nt as a technical prcesses. ODI Prcedures shuld always be cnsidered as technical steps that have t be achieved in the verall prcess, but they shuldn t have in-depth business lgic that applies t the data. Typical Data Warehuse prjects wuld have less than 10% f develpment in the frm f Prcedures. Packaging and Releasing Develpment Nw that the Interfaces, Prcedures and Variables are develped and tested, they need t be rdered as steps within Packages. Start thinking abut what shuld happen in case f errr in any f these steps. By default, the ODI Agent will stp the executin f a package at the step that has failed and it will rllback any pen transactin n all the cnnected databases. Even thugh errrs are reprted in ODI Lgs, it is always a gd practice t have an n-errr prcedure that is triggered if sme f the steps fail. This prcedure can fr example send an t an peratr t warn them that the package has failed, reprting the sessin ID. Particular attentin has t be paid t the fllwing step attributes: Next step n failure Number f attempts in case f failure Interval between attempts Try t avid heavy lps inside packages (mre than 50 iteratins). In mst cases, a lp can be avided by simply adding a table as surce t an interface! When the package is successfully tested, yu will have t release it as a Scenari. Think f a scenari as a cmpiled versin f yur surce bjects such as interfaces, prcedures, packages, etc. This Scenari will then be imprted in the Test envirnment befre validatin fr Prductin. Versining Develpment Befre ging t prductin, yu shuld recrd a stable versin f all the bjects invlved in the release f the Scenaris. Creating versins f bjects will allw yu t restre previus released items fr further maintenance in case f prblem. Starting frm ODI 10g, versins f bjects are stred in the Master Repsitry and the dependencies between bjects are maintained in bjects called Slutins. T make sure all the bjects invlved in the release are cnsistently versined, it is thus recmmended that yu create a Slutin bject and add the Prject t this Slutin. Designer calculates all the dependencies between bjects and autmatically creates a new versin fr Prjects, Mdels, Scenaris and glbal bjects used. Restring these bjects frm a previus versin can be dne when needed frm the Designer Navigatr by simply selecting the Restre menu. Sectin Using ODI Versin Management further discusses the best practices fr versin cntrl. 29

30 Scheduling and Operating Scenaris Scheduling and perating scenaris is usually dne in the Test and Prductin envirnments in separate Wrk Repsitries. The Operatr Navigatr prvides a graphical interface t perfrm these tasks. Any scenari can be scheduled by an ODI Agent r by any external scheduler, as scenaris can be invked by an perating system cmmand. When scenaris are running in prductin, agents generate executin lgs in an ODI Wrk Repsitry. These lgs can be mnitred either thrugh the Operatr Navigatr, thrugh any web brwser using the ODI Cnsle r the Oracle Enterprise Manager Plug-in fr ODI. Operating teams need t be trained n the use f the Operatr Navigatr, ODI Cnsle r Oracle Enterprise Manager. They will have t mnitr lgs, restart failing jbs and submit adhc tasks fr executin. Mnitring the Data Quality f the Data Warehuse Business users rely n the accuracy f key indicatrs f the Data Warehuse t make decisins. If these indicatrs are wrng, the decisins are wrthless. Depending n the data quality strategy yu have defined, business users will actively participate in the mnitring f data discrepancies. They will have t help the IT team t better refine the calculatins f the indicatrs as well as the qualificatin f the errneus data. This generally leads t mdificatins f the business rules. These updates are dne in the develpment envirnment and fllw the nrmal release cycle defined fr the prject. Typical prjects reprt data quality errrs t business users thrugh reprts generated frm ODI s errr tables. These reprts are pushed t the data wner in their system r t any wrkflw applicatin used internally. Publishing Metadata t Business Users Business users usually need t have access t ther data quality indicatrs based n metadata such as: When was the last time my table was updated? Hw many recrds were added, remved r updated in my table? What are the rules that calculate a particular indicatr? Where des the data cme frm, and hw is it transfrmed? Where des the data g t, and hw is it transfrmed? Etc. All these questins can be answered if yu give access t the ODI Cnsle t business users. This is dne simply by assigning them a user ID in the Security Navigatr. The web-based interface allws them t see all the data mdels, and the interactins between them. This includes: 30

31 Flw Maps Data Lineage, which is usefully fr understanding the path taken by data and the transfrmatins between applicatins Executin lgs with accurate statistics n the number f recrds prcessed (number f inserts, updates, delete and errrs). Planning fr Next Releases A Data Warehuse evlves as the business need f the enterprise change. This leads t frequent updates f the data mdels. This cnstant changing has an impact n the develpment f new cmpnents and the maintenance f existing nes. Hwever it is imprtant t understand that it shuld always be a cycle driven by business requirements. The general steps listed belw give yu a brief verview f hw this culd be planned: 1. Define r refine Business Rules specificatins 2. Make sure previus releases f the bjects are prperly versined and can be restred safely in a new empty wrk repsitry befre mdifying anything in yur current wrk repsitry. 3. Define new surces r targets in Tplgy Navigatr. 4. Reverse-engineer new and/r existing mdels. Use the ODI crss references t evaluate the impact f the changes t the structure f surce and target fields. Pay special attentin t fields that were remved frm tables as they may still be used in sme transfrmatins. 5. Develp new interfaces and update existing nes. Create r update additinal cmpnents. Test every item separately in the develpment envirnment. 6. Update existing packages r create new nes. Test every package, even thse that haven t changed since the last release, as the underlying schema may have changed. 7. Regenerate the new scenaris fr yur next release. 8. Release the scenaris int each Test envirnment. Crrect any bug that wuld be reprted frm the Test prcess. Invlve business users in the test prcess. 9. Create new Slutins and versin every prject and mdel. Use the change indicatrs that appear n every bject t detect which bjects shuld be versined. 10. When acceptance tests are successful, release yur new scenaris int prductin. 31