Oracle Data Integrator Best Practices for a Data Warehouse

Transcription

1 An Oracle White Paper August 2010 Oracle Data Integratr

2 Preface... 4 Purpse... 4 Audience... 4 Additinal Infrmatin... 4 Intrductin t Oracle Data Integratr (ODI)... 5 Objectives... 5 Business-Rules Driven Apprach... 5 Traditinal ETL versus E-LT Apprach... 6 Understanding Oracle Data Integratr Interfaces... 7 A Business Prblem Case Study... 8 Implementatin using Manual Cding Implementatin using Traditinal ETL tls Implementatin using ODI s E-LT and the Business-rule Driven Apprach 14 Benefits f E-LT Cmbined with a Business-rule Driven Apprach Using ODI in yur Data Warehuse Prject ODI and the Data Warehuse Prject Organizing the Teams Reverse-engineering, Auditing and Prfiling Surce Applicatins Designing and Implementing the Data Warehuse s Schema Specifying and Designing Business Rules Building a Data Quality Framewrk Develping Additinal Cmpnents Packaging and Releasing Develpment Versining Develpment Scheduling and Operating Scenaris Mnitring the Data Quality f the Data Warehuse Publishing Metadata t Business Users Planning fr Next Releases Oracle Data Integratr fr Oracle Best Practices Architecture f ODI Repsitries Reverse-engineering an Oracle Schema Oracle Lading Strategies Using Changed Data Capture Oracle Integratin Strategies Defining a Data Quality Strategy Setting up Agents in an Oracle envirnment Architecture Case Studies

3 Setting up Repsitries Using ODI Versin Management Ging t Prductin Setting up Agents Backing up Repsitries Appendices Appendix I. Oracle Data Integratr fr Teradata Best Practices Architecture f ODI Repsitries Reverse-engineering a Teradata Schema Teradata Lading Strategies Teradata Integratin Strategies Setting up Agents in a Teradata envirnment Appendix II: Additinal Infrmatin Acrnyms used in this dcument

4 Preface Purpse This dcument describes the best practices fr implementing Oracle Data Integratr (ODI) fr a data warehuse slutin. It is designed t help setup a successful envirnment fr data integratin with Enterprise Data Warehuse prjects and Active Data Warehuse prjects. This dcument applies t Oracle Data Integratr 11g. Audience This dcument is intended fr Data Integratin Prfessinal Services, System Integratrs and IT teams that plan t use Oracle Data Integratr (ODI) as the Extract, Lad and Transfrm tl in their Enterprise r Active Data Warehuse prjects. Additinal Infrmatin The fllwing resurces cntain additinal infrmatin: Oracle website: Oracle Data Integratr 11g n-line dcumentatin: Java reference : Jythn reference 4

5 Intrductin t Oracle Data Integratr (ODI) Objectives The bjective f this chapter is t Intrduce the key cncepts f a business-rule driven architecture Intrduce the key cncepts f E-LT Understand what an Oracle Data Integratr (ODI) interface is Thrugh a business prblem case study, understand and evaluate sme different develpment appraches including: Manual cding Traditinal ETL ODI s business-rule driven apprach cmbined with E-LT Business-Rules Driven Apprach Intrductin t Business rules Business rules specify mappings, filters, jins and cnstraints. They ften apply t metadata t transfrm data and are usually described in natural language by business users. In a typical data integratin prject (such as a Data Warehuse prject), these rules are defined during the specificatin phase in dcuments written by business analysts in cnjunctin with prject managers. Business Rules usually define What t d rather than Hw t d it. They can very ften be implemented using SQL expressins, prvided that the metadata they refer t is knwn and qualified in a metadata repsitry. Examples f business rules are given in the table belw: BUSINESS RULE TYPE SQL EXPRESSION Sum f all amunts f items sld during May 2010 multiplied by the item price Prducts that start with CPU and that belng t the hardware categry Custmers with their rders Mapping Filter Jin SUM( CASE WHEN SALES.YEARMONTH= THEN SALES.AMOUNT * PRODUCT.ITEM_PRICE ELSE 0 END ) Upper(PRODUCT.PRODUCT_NAME) like CPU% And PRODUCT.CATEGORY = HARDWARE CUSTOMER.CUSTOMER_ID = ORDER.ORDER_ID And ORDER.ORDER_ID = ORDER_LINE.ORDER_ID 5

6 BUSINESS RULE TYPE SQL EXPRESSION and rder lines Reject duplicate custmer names Reject rders with a link t an nn-existent custmer Unique Key Cnstraint Reference Cnstraint CONSTRAINT CUST_NAME_PK PRIMARY KEY (CUSTOMER_NAME) CONSTRAINT CUSTOMER_FK FOREIGN KEY (CUSTOMER_ID) REFERENCES CUSTOMER(CUSTOMER_ID) Mappings A mapping is a business rule implemented as a SQL expressin. It is a transfrmatin rule that maps surce clumns (r fields) nt ne f the target clumns. It can be executed by a relatinal database server at run-time. This server can be the surce server (when pssible), a middle tier server r the target server. Jins A jin peratin links recrds in several data sets, such as tables r files. Jins are used t link multiple surces. A jin is implemented as a SQL expressin linking the clumns (fields) f tw r mre data sets. Jins can be defined regardless f the physical lcatin f the surce data sets invlved. Fr example, a JMS queue can be jined t a relatinal table. Depending n the technlgy perfrming the jin, it can be expressed as an inner jin, right uter jin, left uter jin and full uter jin. Filters A filter is an expressin applied t surce data sets clumns. Only the recrds matching this filter are prcessed by the data flw. Cnstraints A cnstraint is an bject that defines the rules enfrced n data sets data. A cnstraint ensures the validity f the data in a given data set and the integrity f the data f a mdel. Cnstraints n the target are used t check the validity f the data befre integratin in the target. Traditinal ETL versus E-LT Apprach Traditinal ETL tls perate by first Extracting the data frm varius surces, Transfrming the data n a prprietary, middle-tier ETL engine, and then Lading the transfrmed data nt the target data warehuse r integratin server. Hence the term ETL represents bth the names and the rder f the peratins perfrmed, as shwn in Figure 1 belw. 6

7 Figure 1: Traditinal ETL apprach cmpared t E-LT apprach In respnse t the issues raised by ETL architectures, a new architecture has emerged, which in many ways incrprates the best aspects f manual cding and autmated cde-generatin appraches. Knwn as E-LT, this new apprach changes where and hw data transfrmatin takes place, and leverages existing develper skills, RDBMS engines and server hardware t the greatest extent pssible. In essence, E-LT mves the data transfrmatin step t the target RDBMS, changing the rder f peratins t: Extract the data frm the surce tables, Lad the tables int the destinatin server, and then Transfrm the data n the target RDBMS using native SQL peratrs. Nte, with E-LT there is n need fr a middle-tier engine r server as shwn in Figure 1 abve. Understanding Oracle Data Integratr Interfaces An interface is an ODI bject stred in the ODI Repsitry that enables the lading f ne target datastre with data transfrmed frm ne r mre surce datastres, based n business rules implemented as mappings, jins, filters and cnstraints. A datastre can be: a table stred in a relatinal database an ASCII r EBCDIC file (delimited, r fixed length) a nde frm a XML file a JMS tpic r queue frm a Message Oriented a nde frm a LDAP directry an API that returns data in the frm f an array f recrds Figure 2 shws a screensht f an ODI interface that lads data int the FACT_SALES target table. Surce Data is defined as a hetergeneus query n the CORRECTIONS file, the ORDERS and LINES tables. 7

8 Mappings, jins, filters and cnstraints are defined within this windw. Figure 2: Example f an ODI Interface Wherever pssible, ODI interfaces generate E-LT peratins that relegate transfrmatins t the target RDBMS servers. A Business Prblem Case Study Figure 3 describes an example f a business prblem t extract, transfrm and lad data frm a Micrsft SQL Server database and a file int a target Oracle table. Data is cming frm tw Micrsft SQL Server tables (ORDERS jined t LINES) and is cmbined with data frm the CORRECTIONS file. The target SALES Oracle table must match sme cnstraints such as the uniqueness f the ID clumn and valid reference t the SALES_REP table. Data must be transfrmed and aggregated accrding t sme mappings as shwn in Figure 3. 8

9 Figure 3: Example f a business prblem Translating these business rules frm natural language t SQL expressins is usually straightfrward. In ur example, the rules that appear in the figure culd be translated as fllws: TYPE RULE SQL EXPRESSION / CONSTRAINT Filter Only ORDERS marked as clsed ORDERS.STATUS = CLOSED Jin Mapping A rw frm LINES has a matching ORDER_ID in ORDERS Target s SALE is the sum f the rder lines AMOUNT gruped by sales rep., with the crrectins applied. ORDERS.ORDER_ID = LINES.ORDER_ID SUM(LINES.AMOUNT + CORRECTIONS.VALUE) Mapping Sales Rep = Sales Rep ID frm ORDERS ORDERS.SALES_REP_ID Cnstraint ID must nt be null ID is set t nt null in the data mdel Cnstraint ID must be unique A Primary Key is added t the data mdel with (ID) as set f clumns Cnstraint The Sales Rep. ID shuld exist in the Target sales Rep table A Reference (Freign Key) is added in the data mdel n SALES.SALES_REP = SALES_REP.SALES_REP_ID 9

10 Implementatin using Manual Cding When implementing such a data flw using, manual cding, ne wuld prbably use several steps, several languages, and several scripting tls r utilities. Figure 4 gives an verview f the different steps needed t achieve such an extract, transfrm and lad prcess. Figure 4: Sequence f Operatins fr the Prcess There are, f curse, several technical slutins fr implementing such a prcess. One f them (prbably the mst efficient, as it uses an Oracle data warehuse as a transfrmatin engine) is detailed in the fllwing table: STEP DESCRIPTION EXAMPLE OF CODE 1 Execute the jin between ORDERS and LINES as well as the filters n the surce Micrsft SQL Server database using a database view. Create view C$_SALES As select... frm ORDERS, LINES where ORDERS.STATUS = CLOSED and ORDERS.ORDER_ID = LINES.ORDER_ID Extract the cntent f the view int a flat file using the BCP utility. bcp C$_SALES ut c$_sales_extract.bcp c -S... -U... -P... t\b Use the SQL*Lader utility t lad the temprary BCP file int the TEMP_1 Oracle table. 2 Use the SQL*Lader utility t lad the CORRECTIONS ASCII file int the TEMP_2 Oracle table. 3 Jin, transfrm and aggregate the 2 temprary tables TEMP_1 and TEMP_2 and lad the results int a 3 rd table (TEMP_SALES) using SQL sqlldr cntrl=temp_1.ctl lg=lgfile.lg userid=.../... sqlldr cntrl=temp_2.ctl lg=lgfile.lg userid=.../... insert int TEMP_SALES (...) select SUM(TEMP_1.AMOUNT+TEMP_2.VALUE), TEMP1.SALES_REP_ID,... frm TEMP_1, TEMP_2 where TEMP_1.LINEID = TEMP_2.CORR_ID) 10

11 STEP DESCRIPTION EXAMPLE OF CODE... 4 Check Unique cnstraints using SQL and insert the errrs int the Errrs table insert int Errrs(...) select... frm TEMP_SALES where ID in (select ID frm TEMP_SALES grup by ID having cunt(*) > 1) Check Reference cnstraints using SQL and insert the errrs int the Errr table insert int Errrs(...) select... frm TEMP_SALES where SALES_REP nt in (select SALES_REP_ID frm SALES_REP) 5 Finally, use SQL lgic t insert / update int the target SALES table using a query n TEMP_SALES update SALES set... frm... where ID in (select ID frm TEMP_SALES where IND_UPDATE= U ) insert int SALES (...) select... frm TEMP_SALES where IND_UPDATE= I... The benefits f this apprach are: High perfrmance: Cde flexibility: Uses pure set-riented SQL t avid rw-by-rw peratins Uses Oracle as a transfrmatin engine t leverage the pwer f the RDBMS Uses in-place utilities such as External Tables Leverages the latest features f Oracle such as the built-in transfrmatin functins 11

12 Hwever, this apprach raises several issues that becme painful as the Enterprise Data Warehuse prjects grw, and mre develpers get invlved. These issues are: Pr prductivity Every lad prcess needs t be develped as a set f scripts and prgrams, within different envirnments and with several languages. Business rules ( what happens t the data SUM(AMOUNT + VALUE)) are mixed with technical lgic ( hw t mve / lad the data SQL*Lader, External Table, insert etc.) Mving t prductin is ften difficult when develpers haven t designed envirnment variables, r variable qualified names fr their bjects. High cst f data quality implementatin Hard maintenance N prject flexibility Data cleansing / data quality accrding t predefined cnstraints is usually avided due t the cst f its implementatin Every prject has its wn definitin f the data quality withut any centralized framewrk (default structure f the errr tables, errr recycling etc.) Even when prject managers have set up a framewrk, every script may reinvent the wheel and cntain specific rutines that make it hard t understand Develpments are spread in several machines and flders, withut a central repsitry Impact analysis is impssible as there is n metadata management and n crss references mechanism The cst f a change t the data mdels r t the business rules becmes such a cnstraint that IT teams refuse it, leading t frustratin amngst the business users. Implementatin using Traditinal ETL tls Traditinal ETL tls perfrm all the transfrmatins in a prprietary engine. They ften require additinal hardware t stage the data fr the transfrmatins. Nne f them really leverages the pwer f the RDBMS. A typical ETL architecture is shwn in Figure 5. 12

13 Figure 5: Implementatin Using an ETL Every transfrmatin step requires a specific cnnectr r transfrmer. ETL tls are ften knwn fr the fllwing advantages: Centralized develpment and administratin Easier maintenance Single graphical user interface Centralized repsitry Impact analysis (fr certain tls) Unfrtunately this ETL apprach presents several drawbacks: Pr perfrmance Bad prductivity As the data needs t be prcessed in the engine, it is ften prcessed rw by rw When data frm the target database is referenced - table lkups fr example - it needs t be extracted frm the database, int the engine and then mved back again t the target database. Very few mappings, jins, aggregatins and filters are given t the pwerful engine f the RDBMS Every lad prcess needs t be develped as a set f steps that mix business rules ( what happens t the data SUM(AMOUNT + VALUE)) with technical lgic ( hw t lad the data cnnectr 1, cnnectr 2 etc.) Mving t prductin is ften difficult when develpers haven t designed envirnment variables, r variable qualified names within their queries Sme f them still require the use f heavy manual cding t achieve certain particular tasks, r t leverage the RDBMS pwerful transfrmatin functins 13

14 High Cst ETL tls require additinal hardware ETL tls require specific skills Implementatin using ODI s E-LT and the Business-rule Driven Apprach Implementing a business prblem using ODI is a very easy and straightfrward exercise. It is dne by simply translating the business rules int an interface. Every business rule remains accessible frm the Diagram panel f the interface s windw. Specifying the Business Rules in the Interface Figure 6 gives an verview f hw the business prblem is translated int an ODI interface: The ORDERS, LINES and CORRECTION datastres are dragged and drpped int the Surce panel f the interface The Target SALES datastre is drpped in the Target Datastre panel Jins and filters are defined by dragging and drpping clumns in the Surce panel Mappings are defined by selecting every target clumn and by dragging and drpping clumns r by using the advanced expressin editr. Cnstraints are defined in the Cntrl tab f the interface. They define hw flw data is ging t be checked and rejected int the Errrs table. Figure 6: Implementatin using Oracle Data Integratr 14

15 Business Rules are Cnverted int a Prcess Business rules defined in the interface need t be split int a prcess that will carry ut the jins, filters, mappings and cnstraints frm surce data t target tables. Figure 7 defines the prblem t be slved. Figure 7: Hw t cnvert business rules int a prcess? By default, Oracle Data Integratr will use the RDBMS as a staging area fr lading surce data int temprary tables and applying all the required mappings, staging filters, jins and cnstraints. The staging area is a separate area in the RDBMS (a user/database) where ODI creates its temprary bjects and executes sme f the rules (mapping, jins, final filters, aggregatins etc). When perfrming the peratins this way, ODI leverages the E-LT architecture as it first extracts and lads the temprary tables and then finishes the transfrmatins in the target RDBMS. In sme particular cases, when surce vlumes are small (less than 500,000 recrds), this staging area can be lcated in memry in ODI S in-memry relatinal database ODI Memry Engine. ODI wuld then behave like a traditinal ETL tl. Figure 8 shws the prcess autmatically generated by Oracle Data Integratr t lad the final SALES table. The business rules, as defined in Figure 7 will be transfrmed int cde by the Knwledge Mdules (KM). The cde prduced will generate several steps. Sme f these steps will extract and lad the data frm the surces t the staging area (Lading Knwledge Mdules - LKM). Others will transfrm and integrate the data frm the staging area t the target table (Integratin Knwledge Mdule - IKM). T ensure data quality, the Check Knwledge Mdule (CKM) will apply the user defined cnstraints t the staging data t islate errneus recrds in the Errrs table. 15

16 Figure 8: ODI Knwledge Mdules in actin ODI Knwledge Mdules cntain the actual cde that will be executed by the varius servers f the infrastructure. Sme f the cde cntained in the Knwledge Mdules is generic. It makes calls t the ODI Substitutin API that will be bund at run-time t the business-rules and generates the final cde that will be executed. Figure 9 illustrates this mechanism. During design time, rules are defined in the interfaces and Knwledge Mdules are selected. During run-time, cde is generated and every API call in the Knwledge Mdules (enclsed by <% and %>) is replaced with its crrespnding bject name r expressin, with respect t the metadata prvided in the Repsitry. Fr example, a call t <%=diref.gettable( TARG_NAME )%> will return the name f the target table f the interface with the apprpriate qualifier accrding t cntext infrmatin, tplgy setup etc. A typical SQL INSERT statement wuld be cded in a Knwledge Mdule as fllws: INSERT INTO <%=diref.gettable( TARG_NAME )%>... This template f cde will f curse generate different SQL statements depending n the target table ( INSERT INTO MyDB1.SALES... when the target is the SALES table, INSERT INTO DWH_DB.PRODUCT when the target is the PRODUCT table etc.). This is als particularly useful when migrating ODI prcesses frm an envirnment t anther (prmting prcesses frm Develpment t QA fr example) as ODI will autmatically substitute the crrect 16

17 schema infrmatin based n the specified envirnment withut requiring any cde mdificatins. Figure 9: Hw Knwledge Mdules generate native cde Once the cde is generated, it is submitted t an ODI Agent, which will either redirect it t the apprpriate database engines and perating systems, r will execute it when needed (memry engine transfrmatin, java r jythn cde etc.) In mst cases, the agent is simply a cnductr that des nt tuch the data. Benefits f E-LT Cmbined with a Business-rule Driven Apprach Cmpared t ther architectures (manual cding and traditinal ETL), ODI mixes the best f bth wrlds: Prductivity / Maintenance The business-rules driven apprach delivers greater prductivity as develpers simply need t cncentrate n the What withut caring abut the Hw. They define SQL expressins fr the business rules, and ODI Knwledge Mdules generate the entire set f SQL peratins needed t achieve these rules. When a change needs t be made in peratinal lgic (such as creating a backup cpy f every target table befre lading the new recrds ), it is 17

18 High Perfrmance: Lw Cst: simply applied in the apprpriate Knwledge Mdule and it autmatically impacts the hundreds f interfaces already develped. With a traditinal ETL apprach, such a change wuld have necessitated pening every jb and manually adding the new steps, increasing the risk f mistakes and incnsistency. Flexibility and a shallw learning curve are ensured by leveraging the RDBMS latest features. With a centralized repsitry that describes all the metadata f the surces and targets and a single unified and cmprehensive graphical interface, maintenance is greatly ptimized as crss-references between bjects can be queried at any time. This gives the develpers and the business users a single entry pint fr impact analysis and data lineage ( What is used where?, Which surces ppulate which targets? etc.) In the ODI repsitry, the tplgy f the infrastructure is defined in detail, and mving bjects between different executin cntexts (Develpment, Testing, QA, Prductin, etc.) is straightfrward. With a pwerful versin cntrl repsitry, several teams can wrk n the same prject within different release stages, with guaranteed cnsistency f deliverables. With a centralized framewrk fr Data Quality, develpers spend less time n defining technical steps, and mre time n the specificatin f data quality rules. This helps t build a cnsistent and standardized Data Warehuse. The E-LT architecture leverages the pwer f all the features f in-place databases engines. ODI generates pure set-riented SQL ptimized fr each RDBMS which can take advantage f advanced features such as parallel prcessing r ther advanced features. Native database utilities can be invked by the ODI Knwledge Mdules prvided. When data frm the target database is referenced - table lkups fr example, it desn t need t be extracted frm the database, int an engine. It remains where it is, and it is prcessed by database engine. Oracle Data Integratr desn t require a dedicated server. The lads and transfrmatins are carried ut by the RDBMS. In cnclusin, with its business-rule driven E-LT architecture, Oracle Data Integratr is the best slutin fr taking advantage f bth manual cding and traditinal ETL wrlds. 18

19 Using ODI in yur Data Warehuse Prject ODI and the Data Warehuse Prject The main gal f a Data Warehuse is t cnslidate and deliver accurate indicatrs t business users t help them make decisins regarding their everyday business. A typical prject is cmpsed f several steps and milestnes. Sme f these are: Defining business needs (Key Indicatrs) Identifying surce data that cncerns key indicatrs; specifying business rules t transfrm surce infrmatin int key indicatrs Mdeling the data structure f the target warehuse t stre the key indicatrs Ppulating the indicatrs by implementing business rules Measuring the verall accuracy f the data by setting up data quality rules Develping reprts n key indicatrs Making key indicatrs and metadata available t business users thrugh ad-hc query tls r predefined reprts Measuring business users satisfactin and adding/mdifying key indicatrs Oracle Data Integratr will help yu cver mst f these steps, frm surce data investigatin t metadata lineage and thrugh lading and data quality audit. With its repsitry, ODI will centralize the specificatin and develpment effrts and prvide a unique architecture n which the prject can rely t succeed. Organizing the Teams As Oracle Data Integratr relies n a centralized repsitry, different types f users may need t access it. The list belw describes hw ODI may be used by yur teams. PROFILE DESCRIPTION ODI MODULES USED Business User Business Analyst Develper Business users have access t the final calculated key indicatrs thrugh reprts r ad-hc queries. In sme cases, they need t understand what the definitin f the indicatrs is, hw they are calculated and when they were updated. Alternatively, they need t be aware f any data quality issue regarding the accuracy f their indicatrs. Business Analysts define key indicatrs. They knw the surce applicatins and specify business rules t transfrm surce data int meaningful target indicatrs. They are in charge f maintaining translatin data frm peratinal semantics t the unified data warehuse semantic. Develpers are in charge f implementing the business rules in respect t the specificatins described by the Business Analysts. They release their wrk by prviding executable scenaris t the prductin team. Develpers must have ODI Cnsle Designer Navigatr (limited access) ODI Cnsle Tplgy Navigatr (read nly access) Designer Navigatr: 19

20 PROFILE DESCRIPTION ODI MODULES USED bth technical skills regarding the infrastructure and business knwledge f the surce applicatins. Limited access t Mdels Metadata Administratr Metadata Administratrs are in charge f reverse-engineering surce and target applicatins. They guarantee the verall cnsistency f Metadata in the ODI Repsitry. They have an excellent knwledge f the structure f the surces and targets and they have participated in the data mdeling f key indicatrs. In cnjunctin with Business Analysts, they enrich the metadata by adding cmments, descriptins and even integrity rules (such as cnstraints). Metadata Administratrs are respnsible fr versin management. Full access t Prjects Operatr Navigatr ODI Cnsle Tplgy Navigatr (limited access) Designer Navigatr: Full access t Mdels Restre access t Prjects ODI Cnsle Database Administratr System Administratr Security Administratr Operatr Database Administratrs are in charge f defining the technical database infrastructure that supprts ODI. They create the database prfiles t let ODI access the data. They create separate schemas and databases t stre the Staging Areas. They make the envirnments accessible by describing them in the Tplgy System Administratrs are in charge f maintaining technical resurces and infrastructure fr the prject. Fr example, they may install and mnitr Scheduler Agents backup / restre Repsitries install and mnitr ODI Cnsle Setup envirnments (develpment, test, maintenance etc.) The Security Administratr is in charge f defining the security plicy fr the ODI Repsitry. He r she creates ODI users and grants them rights n mdels, prjects and cntexts. Operatrs are in charge f imprting released and tested scenaris int the prductin envirnment. They schedule the executin f these scenaris. They mnitr executin lgs and restart failed sessins when needed. Tplgy Navigatr (full access) Designer Navigatr (full access) Operatr Navigatr (full access) ODI Cnsle Agents Tplgy Navigatr (limited access) ODI Cnsle Security Navigatr (full access) Designer Navigatr (read access) Tplgy Navigatr (read access) ODI Cnsle Operatr Navigatr ODI Cnsle Oracle Enterprise Manager Plug-in Fr ODI 20

21 The ODI Master Repsitry cntains built-in default prfiles that can be assigned t users. The fllwing table suggests hw t use these built-in prfiles: PROFILE Business User Business Analyst Develper Metadata Administratr Database Administratr System Administratr Security Administratr Operatr BUILT-IN PROFILE IN ORACLE DATA INTEGRATOR CONNECT, NG REPOSITORY EXPLORER CONNECT, NG REPOSITORY EXPLORER, NG DESIGNER CONNECT, DESIGNER CONNECT, METDATA ADMIN, VERSION ADMIN CONNECT, DESIGNER, METADATA ADMIN, TOPOLOGY ADMIN CONNECT, OPERATOR CONNECT, SECURITY ADMIN CONNECT, OPERATOR Reverse-engineering, Auditing and Prfiling Surce Applicatins A gd starting pint fr prjects is t understand the cntents and the structure f surce applicatins. Rather than having paper-based dcumentatin n surce applicatins, a gd practice is t cnnect t these surce applicatins using ODI and t capture their metadata. Once this is achieved, it is usually helpful t define sme data quality business rules in ODI Repsitry. This helps check the data cnsistency in the surce and validate yur understanding f the data mdels. At this phase f the prject it is imprtant t start having an answer t the fllwing questins: Hw many different surces d we have t take in accunt t calculate ur indicatrs? Is the data needed fr ur indicatrs present in the surce systems? What data quality challenges will we have t address t ensure the accuracy f the target warehuse? What surce system(s) will prvide the master data (dimensins) that will be referenced by ur indicatrs? What data vlumes are we ging t manipulate? And s frth. In sme cases, surce applicatins are nt accessible directly. Only ASCII r binary files extracts frm these applicatins are prvided. Starting t wrk with these surce files befre implementing the data warehuse mdel is recmmended, as they represent the visin f the truth that the prductin surce systems want t give yu. The metadata f these files shuld typically be described in the repsitry. Samples f the files can als be initially laded int temprary target tables t validate their structure and cntent. 21

22 All this can be implemented in Oracle Data Integratr as fllws: Cnnect surce applicatins r files in Tplgy Navigatr Define a lgical architecture in Tplgy Navigatr Create ne mdel per lgical schema in Designer Navigatr Reverse Engineer mdels when pssible r manually describe datastres Use standard JDBC Reverse-engineering t get database metadata Use Custmized Reverse-engineering strategies (Reverse Knwledge Mdules) when standard JDBC reverse is nt applicable (r nt accurate) Use Cbl Cpy Bk imprt fr ASCII r binary files if available Use Delimited File Reverse fr ASCII delimited files Enrich the metadata by adding infrmatin when it is nt available: Datastres and Clumns descriptins and titles Unique keys infrmatin (Primary Keys, Alternate Keys) Referential Integrity assumptins between datastres (Freign Keys) Check cnstraints (fr example t check the dmain f values fr a clumn) When the surce data is stred in files, develp simple interfaces t lad these files int a temprary area in the target database t better evaluate their level f data quality. Lcate the imprtant entities in the surce data and use the fllwing features t prfile the cntents: View Data n a datastre View distributin f values fr a clumn f a datastre Cunt the number f recrds Run ad-hc SQL queries frm Designer Navigatr Validate the cnstraints that yu have defined in the mdels by perfrming data quality cntrl: Schedule r interactively run data quality static cntrl n the datastres and mdels Access and understand the cntents f the errr tables Prpse alternatives fr handling data discrepancies in the future Define the acceptable level f data quality fr the target warehuse Of curse, these actin items shuld be dne in cnjunctin with business analysts. Nrmally, in parallel t this phase, yu will be designing the data mdel f the target Data Warehuse. 22

23 Data quality audit errrs and data prfiling will help yu refine yur understanding f the truth, and cnsequently, will lead yu t a better mdeling f the target warehuse. Designing and Implementing the Data Warehuse s Schema This sectin culd be a bk n its wn. The gal here is nt t prvide yu with guidelines in Data Warehuse mdeling, but rather t give sme general hints and tips that may affect yur ODI develpment. It is always a gd practice t have an Operatinal Data Stre (ODS) t stre raw peratinal data. Usually, the data mdel f the ODS is very clse t the OLTP surce applicatins data mdel. Any surce data shuld be accepted in the ODS and almst n data quality rule shuld be implemented. This ensures t have a stre representing all the data-f-the-day frm the peratinal systems. When designing the Data Warehuse schema, the fllwing general tips may be helpful: Where pssible, describe clumns inside the data dictinary f the database (fr example use COMMENT ON TABLE and COMMENT ON COLUMN SQL statements). By ding s, yu allw ODI t retrieve these cmments and descriptins int its metadata repsitry. Design a strage space fr the Staging Area where ODI will create its necessary temprary tables and data quality errr tables. D nt use primary keys f the surce systems as primary key fr yur target tables. Use cunters r identity clumns whenever it is pssible. This makes a flexible data mdel that will easily evlve ver time. Design referential integrity (RI) and reverse engineer freign keys in ODI mdels. D nt implement these freign keys in the target database as they may lead t perfrmance issues. With the autmatic data quality checks, ODI will guarantee the cnsistency f data accrding t these RI rules. Standardize yur bject naming cnventins. Fr example, use three letters t prefix table names by subject area. Avid using very lng names as ODI may add a prefix t yur table names when creating temprary bjects (fr example E$_Custmer will be the errr table fr table Custmer). Use f either 3 rd Nrmal Frm mdeling (3NF) r dimensinal mdeling ( Snw Flakes r Star Schemas ) desn t have any impact n ODI, but rather n the way yu will design yur future business rules develpment and business users reprts. Once the ODS and DW are designed, yu will have t create the apprpriate mdels in Designer and perfrm reverse-engineering. ODI cmes with mdeling capabilities in the frm f a feature called Cmmn Frmat Designer. It can help yu design the structure f the ODS and the DW frm the existing structures f the surce systems by simply drag and drpping tables clumns and cnstraints. 23

24 The generated Data Descriptin Language statements (DDL) autmatically include specific features f the target database. It als keeps track f the rigin f surce clumns t autmatically generate the interfaces fr lading the target, thus resulting in significant time savings. Best practices n the usage f Cmmn Frmat Designer are ut f the scpe f this dcument. Specifying and Designing Business Rules Usually this step starts almst at the same time as the design f the target schema. Understanding the business needs is clearly a key factr in the success f the verall prject. This influences the key indicatrs selected fr the warehuse as well as the rules used t transfrm surce t target data. The mre accurate the specificatin f a business rule, the easier the design f this rule in ODI. Several successful prjects have used ODI Designer t specify the business rules, thus aviding having discrepancies between a Wrd dcument and the final develped rules. The fllwing table summarizes what needs t be specified fr every target table f the ODS r the DW befre designing business rules. TARGET DATASTORE: DESCRIPTION OF THE TRANSFORMATION: INTEGRATION STRATEGY: Give the name f the target datastre, pssibly preceded by a qualificatin prefix (such as the database r schema name) Give a shrt descriptin f the purpse f the transfrmatin Define hw data shuld be written t the target. Examples: Replace cntent with new data frm surces Append incming data nt the target Update existing recrds and insert new nes accrding t an update key (specify the key here) Use a Slwly Changing Dimensin strategy. Specify the surrgate key, the slwly changing attributes, the updatable nes, the clumns fr start and end date etc. A hme-made strategy (fr example: BASEL II cmpliance, SOX histrical audit trail etc.) Every strategy specified here will crrespnd t an ODI Integratin Knwledge Mdule. REFRESH FREQUENCY: DEPENDENCIES: SOURCE DATASTORES: Specify when this datastre shuld be laded (every night, every mnth, every 2 hurs etc) Give all dependencies in the lad prcess (e.g. what datastres have t be laded prir t the current ne; what specific jbs need t be executed prir t this ne etc.) Give the list f surce datastres invlved fr lading the target table. This list shuld include all the lkup tables as well. Fr each surce datastre, include the fllwing infrmatin. SOURCE ODI MODEL Give the name f the surce mdel as it appears in Designer DATASTORE NAME Give the name f the datastre 24

25 PURPOSE/DESCRIPTION FIELD MAPPINGS AND TRANSFORMATIONS: TARGET COLUMN Indicate whether it is the main surce set r a lkup ne. If Change Data Capture is t be used, it shuld be indicated here. Fr every target field (r clumn), indicate what transfrmatins need t be applied t the surce fields. These transfrmatins shuld be written as expressins r frmulas as ften as pssible. Give the target clumn name MAPPING DESCRIPTION Describe the purpse f the mapping MAPPING EXPRESSION Give the expressin using the surce clumn names. Try t express it in pseud cde. LINKS OR JOIN CRITERIA DATASTORE 1 Fr every pair f surce datastres, specify the criteria t lkup matching recrds. This is ften knwn as an SQL jin. Name f the first datastre DATASTORE 2 Name f the secnd datastre LINK EXPRESSION Give the expressin used t link the 2 datastres in pseud cde DESCRIPTION Describe this link by specifying if it is a left, right r full uter jin. FILTERS: FILTER DESCRIPTION Give the list f filters that apply t surce data. Filters shuld be expressed in natural language and in pseud cde whenever pssible. Describe this Filter. FILTER DESCRIPTION Give the expressin used t implement the filter in pseud cde. DATA QUALITY REQUIREMENTS: CONSTRAINT NAME DESCRIPTION CONSTRAINT EXPRESSION List here all the data quality requirements including errr recycling if apprpriate. The requirements shuld be expressed as cnstraints whenever pssible. Name r shrt descriptin f the cnstraint t validate. Purpse f the data quality check Expressin in pseud cde required t cntrl the data T illustrate the use f such a frm, the fllwing table gives an example fr the business prblem case study defined in the chapter Intrductin t Oracle Data Integratr (ODI). TARGET DATASTORE: DESCRIPTION OF THE TRANSFORMATION: INTEGRATION STRATEGY: REFRESH FREQUENCY: DEPENDENCIES: Oracle Warehuse.SALES Aggregate rders and rder lines frm the Micrsft SQL Server surce prductin system. Append new recrds f the day Every night The fllwing tables need t be laded prir t this ne fr referential integrity: PRODUCT SALES_REP SOURCE DATASTORES: 25

26 SOURCE ODI MODEL DATASTORE NAME PURPOSE/DESCRIPTION Micrsft SQL Server Surce ORDERS Orders frm the prductin system Micrsft SQL Server Surce LINES Order lines. Main surce fr aggregatin User Data Files CORRECTIONS File cntaining the value t add t the amunt sld if any manual crrectin has ccurred fr a particular rder line. FIELD MAPPINGS AND TRANSFORMATIONS: TARGET COLUMN MAPPING DESCRIPTION MAPPING EXPRESSION ID Order Id ORDERS.ORDER_ID PRODUCT_PK Prduct Id as it appears in the rder lines LINES.PRODUCT_ID SALES_REP Sales Rep Id as it appears in the rder ORDERS.SALES_REP_ID SALE Amunt sld. If a crrectin fr this amunt exists in the crrectins file, the value shuld be added. The ttal value needs t be summed. SUM ( LINES.AMOUNT + (CORRECTIONS.VALUE when it exists) ) QUANTITY Ttal quantity f prduct sld. SUM ( LINES.QUANTITY ) LINKS OR JOIN CRITERIA DATASTORE 1 DATASTORE 2 LINK EXPRESSION DESCRIPTION ORDERS LINES Link rders and rder lines n the rder ID. Every rder line must match an existing rder LINES CORRECTIONS Lkup a crrectin value frm the crrectin file if it exists given the line item ID (left jin) ORDERS.ORDER_ID = LINES.ORDER_ID LINES.LINE_ITEM_ID = CORRECTIONS.CORR_ID Filters: FILTER DESCRIPTION Orders f the day Orders validated FILTER EXPRESSION ORDER.ORDER_DATE between yesterday and nw ORDER.STATUS = Clsed DATA QUALITY REQUIREMENTS: CONSTRAINT DESCRIPTION CONSTRAINT EXPRESSION Reference t SALES_REP Reference t PRODUCT Ensure that every SALES_REP ID exists in the SALES_REP reference table Ensure that every PRODUCT_PK exists in the PRODUCT reference table SALES_REP references SALES_REP(ID) PRODUCT_PK references PRODUCT(ID) ID Nt null ID Clumn is mandatry Check ID nt null Quantity greater than 0 The quantity sld must always be psitive QUANTITY > 0 Uniqueness f sale A sales rep can t sell the same prduct twice inside the same rder (PRODUCT_PK, SALES_REP) is unique 26

27 The next step is t design these business rules in the Designer Navigatr. The translatin frm the specificatin t the design is straightfrward. Every specificatin f the lading f a target table will be cnverted int an interface. This prcess is even faster if the specificatin is made directly using the Designer Navigatr. The steps t implement the interface are typically as fllws. Drag and drp the target datastre Drag and drp surce datastres Fr every target field, translate the pseud cde f the transfrmatin int SQL expressins. When pssible, decide where t execute the expressin (surce, staging area, target) Fr every jin, define the jin in the surce panel f the interface. Depending n the specificatin, define the jin as inner, left, right, natural r full uter jin. When pssible, try t execute jins n the surces t minimize netwrk traffic. Fr every filter, translate the pseud cde int an SQL expressin. When the surce database accepts SQL filters, set it as execute n surce t minimize netwrk traffic. In the flw diagram f the interface, define the lading strategies (LKM) and specify the integratin strategy (IKM) t match the specificatin. Chse the apprpriate ptins fr the Knwledge Mdule and activate the data quality cntrl when apprpriate. In the cntrl tab f the interface, select the cnstraints t cntrl. If the cnstraints are nt defined n the target datastre, define them in the Mdels view f the Designer Navigatr. After the design f every interface, yu shuld review the cde generated using the Simulatin mde and then test it by executing it. The Operatr Navigatr allws yu t easily fllw the executin f yur interface t get the number f recrds prcessed and many ther useful indicatrs (such as the generated cde, the elapsed time, the number f inserts, updates, deletes etc.) After the interface cmpletes, the resulting data and errrs can be directly selected frm Designer t validate the accuracy f the business rules. Thanks t ODI Cntexts, the executin f the interface will happen n a Develpment envirnment and will nt affect prductin data. Building a Data Quality Framewrk With an apprach based n business rules, ODI is certainly the mst apprpriate tl t help yu build a data quality framewrk t track data incnsistencies. Thanks t the Check Knwledge Mdules, yu simply have t define the cntrl business rules, and incnsistent data is autmatically islated fr yu in errr tables. Hwever, islating errneus data is nt the nly issue in Data Quality. Even if ODI autmatically detects duplicated keys, mandatry fields, missing references, and mre cmplex cnstraints, yu will need t invlve business users in the prcess f qualifying data discrepancies and make the apprpriate decisins. Yu 27

28 shuld have an answer t each f the fllwing questins befre defining yur data quality strategy: What level f data quality is required fr the Data Warehuse? Wh are the business wners f surce data? What shuld we d with rejected recrds? D we need t define an errr recycling strategy? D we need t invlve business wners f the data and reprt rejected recrds? Hw wuld business users mdify errneus surce data? D we prvide a GUI t have them mdify errneus recrds in the errr tables? The dcument Cmprehensive Data Quality with Oracle Data Integratr further discusses Data Quality issues and the recmmended best practices. Develping Additinal Cmpnents Nt all the typical tasks f the lading f the Data Warehuse can be achieved using ODI Interfaces. It is usual t develp additinal cmpnents r jbs that wuld carry n tasks such as, fr example: Receiving and sending s Cpying, mving, cncatenating, renaming files in the file system Cmpressing, decmpressing files Executing web services Writing and executing shell scripts fr a specific perating system Writing and executing java prgrams And s n. These cmpnents can be develped and tested within the Designer Navigatr as Prcedures, Variables, User Functins, r Steps in a package. ODI Prcedures ffer a brad range f pssibilities fr develping these cmpnents. They can include, fr example: An ad-hc SQL statement written fr any database An Operating System call ODI built-in tls and APIs (Send Mail, Read Mail, Wait Fr File etc.) Cde written in any f the scripting languages supprted by the Jakarta Bean Scripting Framewrk. This includes Java, Java Script, Pythn, Perl, NetRexx, Grvy, etc. Of curse, the risk here is t start develping transfrmatin prcesses as Prcedures by hand cding shell and SQL scripts rather than using the pwerful mechanism f ODI Interfaces 28

29 cmbined with Knwledge Mdules. T avid that, try as much as pssible t specify yur transfrmatins as business rules and nt as a technical prcesses. ODI Prcedures shuld always be cnsidered as technical steps that have t be achieved in the verall prcess, but they shuldn t have in-depth business lgic that applies t the data. Typical Data Warehuse prjects wuld have less than 10% f develpment in the frm f Prcedures. Packaging and Releasing Develpment Nw that the Interfaces, Prcedures and Variables are develped and tested, they need t be rdered as steps within Packages. Start thinking abut what shuld happen in case f errr in any f these steps. By default, the ODI Agent will stp the executin f a package at the step that has failed and it will rllback any pen transactin n all the cnnected databases. Even thugh errrs are reprted in ODI Lgs, it is always a gd practice t have an n-errr prcedure that is triggered if sme f the steps fail. This prcedure can fr example send an t an peratr t warn them that the package has failed, reprting the sessin ID. Particular attentin has t be paid t the fllwing step attributes: Next step n failure Number f attempts in case f failure Interval between attempts Try t avid heavy lps inside packages (mre than 50 iteratins). In mst cases, a lp can be avided by simply adding a table as surce t an interface! When the package is successfully tested, yu will have t release it as a Scenari. Think f a scenari as a cmpiled versin f yur surce bjects such as interfaces, prcedures, packages, etc. This Scenari will then be imprted in the Test envirnment befre validatin fr Prductin. Versining Develpment Befre ging t prductin, yu shuld recrd a stable versin f all the bjects invlved in the release f the Scenaris. Creating versins f bjects will allw yu t restre previus released items fr further maintenance in case f prblem. Starting frm ODI 10g, versins f bjects are stred in the Master Repsitry and the dependencies between bjects are maintained in bjects called Slutins. T make sure all the bjects invlved in the release are cnsistently versined, it is thus recmmended that yu create a Slutin bject and add the Prject t this Slutin. Designer calculates all the dependencies between bjects and autmatically creates a new versin fr Prjects, Mdels, Scenaris and glbal bjects used. Restring these bjects frm a previus versin can be dne when needed frm the Designer Navigatr by simply selecting the Restre menu. Sectin Using ODI Versin Management further discusses the best practices fr versin cntrl. 29

30 Scheduling and Operating Scenaris Scheduling and perating scenaris is usually dne in the Test and Prductin envirnments in separate Wrk Repsitries. The Operatr Navigatr prvides a graphical interface t perfrm these tasks. Any scenari can be scheduled by an ODI Agent r by any external scheduler, as scenaris can be invked by an perating system cmmand. When scenaris are running in prductin, agents generate executin lgs in an ODI Wrk Repsitry. These lgs can be mnitred either thrugh the Operatr Navigatr, thrugh any web brwser using the ODI Cnsle r the Oracle Enterprise Manager Plug-in fr ODI. Operating teams need t be trained n the use f the Operatr Navigatr, ODI Cnsle r Oracle Enterprise Manager. They will have t mnitr lgs, restart failing jbs and submit adhc tasks fr executin. Mnitring the Data Quality f the Data Warehuse Business users rely n the accuracy f key indicatrs f the Data Warehuse t make decisins. If these indicatrs are wrng, the decisins are wrthless. Depending n the data quality strategy yu have defined, business users will actively participate in the mnitring f data discrepancies. They will have t help the IT team t better refine the calculatins f the indicatrs as well as the qualificatin f the errneus data. This generally leads t mdificatins f the business rules. These updates are dne in the develpment envirnment and fllw the nrmal release cycle defined fr the prject. Typical prjects reprt data quality errrs t business users thrugh reprts generated frm ODI s errr tables. These reprts are pushed t the data wner in their system r t any wrkflw applicatin used internally. Publishing Metadata t Business Users Business users usually need t have access t ther data quality indicatrs based n metadata such as: When was the last time my table was updated? Hw many recrds were added, remved r updated in my table? What are the rules that calculate a particular indicatr? Where des the data cme frm, and hw is it transfrmed? Where des the data g t, and hw is it transfrmed? Etc. All these questins can be answered if yu give access t the ODI Cnsle t business users. This is dne simply by assigning them a user ID in the Security Navigatr. The web-based interface allws them t see all the data mdels, and the interactins between them. This includes: 30

31 Flw Maps Data Lineage, which is usefully fr understanding the path taken by data and the transfrmatins between applicatins Executin lgs with accurate statistics n the number f recrds prcessed (number f inserts, updates, delete and errrs). Planning fr Next Releases A Data Warehuse evlves as the business need f the enterprise change. This leads t frequent updates f the data mdels. This cnstant changing has an impact n the develpment f new cmpnents and the maintenance f existing nes. Hwever it is imprtant t understand that it shuld always be a cycle driven by business requirements. The general steps listed belw give yu a brief verview f hw this culd be planned: 1. Define r refine Business Rules specificatins 2. Make sure previus releases f the bjects are prperly versined and can be restred safely in a new empty wrk repsitry befre mdifying anything in yur current wrk repsitry. 3. Define new surces r targets in Tplgy Navigatr. 4. Reverse-engineer new and/r existing mdels. Use the ODI crss references t evaluate the impact f the changes t the structure f surce and target fields. Pay special attentin t fields that were remved frm tables as they may still be used in sme transfrmatins. 5. Develp new interfaces and update existing nes. Create r update additinal cmpnents. Test every item separately in the develpment envirnment. 6. Update existing packages r create new nes. Test every package, even thse that haven t changed since the last release, as the underlying schema may have changed. 7. Regenerate the new scenaris fr yur next release. 8. Release the scenaris int each Test envirnment. Crrect any bug that wuld be reprted frm the Test prcess. Invlve business users in the test prcess. 9. Create new Slutins and versin every prject and mdel. Use the change indicatrs that appear n every bject t detect which bjects shuld be versined. 10. When acceptance tests are successful, release yur new scenaris int prductin. 31

32 Oracle Data Integratr fr Oracle Best Practices Architecture f ODI Repsitries It is recmmended that yu install the ODI Master and Wrk repsitries in an OLTP database distinct frm the surce r target applicatins. Reverse-engineering an Oracle Schema The Oracle JDBC driver implements mst metadata APIs. Yu can therefre use the standard JDBC reverse-engineering with ODI. Using this methd, yu can capture metadata fr: Tables, Views etc. including table cmments, Clumns including data types, length and scale, and cmments Primary Keys Freign Keys Check Cnstraints In sme cases a custm reverse-engineering apprach using the RKM Oracle may be necessary. Fr example ODI 11g natively supprts database partitins which can nly be reverse-engineered using the RKM Oracle. RKM RKM Oracle DESCRIPTION Reverse-engineering Knwledge Mdule fr Oracle This RKM uses the Oracle system tables knwn as the Oracle Data Dictinary t extract the metadata definitin f a table and its related artifacts (clumns, keys, partitins etc.). Oracle Lading Strategies When lading an Oracle data warehuse the gal is always t lad the data int the database in the mst efficient manner and Oracle Data Integratr prvides several Lading Knwledge Mdules ptimized fr Oracle that will help yu achieve this. Chsing the right Knwledge Mdule fr each use case will greatly imprve the efficiency f yur data integratin prcesses. 32

33 Yu can refer t the fllwing table fr a list f the Lading Knwledge Mdules ptimized fr Oracle: LKM LKM File t Oracle (EXTERNAL TABLE) LKM File t Oracle (SQLLDR) LKM SQL t Oracle DESCRIPTION Lads data frm a file t an Oracle staging area using the EXTERNAL TABLE SQL Cmmand. Lads data frm a file t an Oracle staging area using the SQL*Lader cmmand line utility. The Oracle client must be installed n the machine hsting the ODI Agent. Lads data thrugh the agent frm any SQL RDBMS t an Oracle staging area. LKM MSSQL t Oracle (BCP SQLLDR) LKM Oracle t Oracle (DBLINK) LKM Oracle t Oracle (datapump) Lads data frm a Micrsft SQL Server database t an Oracle staging area using the BCP and SQL*Lader utilities. The utilities must be installed n the machine hsting the ODI Agent. Lads data frm an Oracle surce database t an Oracle staging area database using database links. Lads data frm an Oracle surce database t an Oracle staging area database using external tables in the datapump frmat. There are als several additinal Knwledge Mdules which can be used t extract data frm SAP ERP, SAP BW r Oracle BI and lad the recrds int an Oracle database. Using Laders fr Flat Files When yur interface cntains a flat file as a surce, yu may want t use a strategy that leverages the mst efficient lading utility available fr yur staging area technlgy rather than the standard LKM File t SQL which uses the ODI Agent t lad the flat file data. The agent uses JDBC t write t the target, using this methd wuld lead t significant perfrmance issues cmpared t lading int Oracle using the External Table SQL cmmand r SQL*Lader. We advise against using the ODI Agent when lading large files and recmmend the LKM File t Oracle (EXTERNAL TABLE) t transfer large files int an Oracle database. Using Unlad/Lad fr Remte Servers When the surce result set is lcated n a remte database server, an alternative t using the agent t transfer the data wuld be t unlad the date int a file and then lad it int the staging area. The LKM MSSQL t Oracle (BCP SQLLDR) is an example f a Knwledge Mdule that fllws these steps, it uses the BCP utility t bulk unlad data frm a Micrsft SQL Server database int a file and SQL*Lader t lad the data back int an Oracle staging area. This Lading Knwledge Mdule prvides the mst ptimal apprach t extracting data ut f Micrsft SQL Server and lading it int an Oracle database. 33

34 Several Knwledge Mdules use a similar apprach including the SAP ERP and BW LKMs. Oracle t Oracle There are use cases in which there is a need t mve data between Oracle systems and ODI prvides several ways t achieve this. One f the best methds when it cmes t lading data in such a scenari is t use Oracle Data Pump and Oracle Data Integratr prvides the LKM Oracle t Oracle (datapump) which leverages Oracle Data Pump t exprt and lad the data efficiently thrugh the creatin f External Tables using Oracle Data Pump n bth the surce and the target databases. ODI als prvides the LKM Oracle t Oracle (DBLINK) which has the ability t create a database link between the surce and the target Oracle database t lad the data. Using Changed Data Capture Applying a Change Data Capture (CDC) apprach can significantly imprve the perfrmance f yur ODI prcesses. Fr example if yu are lading yur data warehuse nightly yu nly need t extract the data that has changed in the past 24 hurs, lading the entire surce data sets will result in a significant degradatin f the efficiency f yur ODI jbs. Using CDC ensures that the extract frm yur varius surce systems is dne incrementally. This reduces the amunt f data transferred frm yur surce systems int yur target database as well as the amunt f data transfrmed by ODI. The creatin f the CDC prcess can be challenging and ODI simplifies it by prviding Knwledge Mdules which leverage many CDC mechanisms including the native database CDC capabilities. Change Data Capture n Oracle When it cmes t detecting changes ccurring in an Oracle database ODI ffers several ptins: CDC using Oracle GldenGate CDC using Oracle Streams CDC using database triggers The Oracle Jurnalizatin Knwledge Mdules (JKMs) autmatically create the underlying infrastructure required by the ODI CDC framewrk s that end users d nt have t wrry abut it. Fr example, the Oracle JKMs have the ability t cnfigure Oracle GldenGate r the Oracle Streams infrastructure when required. 34

35 Yu can refer t the fllwing table fr a list f the JKMs fr Oracle prvided with ODI: JKM JKM Oracle t Oracle Cnsistent (OGG) JKM Oracle 10g Cnsistent (Streams) DESCRIPTION Creates and manages the ODI CDC framewrk infrastructure when using Oracle GldenGate fr CDC. Creates the jurnalizing infrastructure fr cnsistent set jurnalizing n Oracle 10g tables, using Oracle Streams. JKM Oracle 11g Cnsistent (Streams) JKM Oracle Cnsistent JKM Oracle Cnsistent (Update Date) JKM Oracle Simple Creates the jurnalizing infrastructure fr cnsistent set jurnalizing n Oracle 11g tables, using Oracle Streams. Creates the jurnalizing infrastructure fr cnsistent set jurnalizing n Oracle tables using triggers Creates the jurnalizing infrastructure fr cnsistent set jurnalizing n Oracle tables using triggers based n a Last Update Date clumn n the surce tables Creates the jurnalizing infrastructure fr simple jurnalizing n Oracle tables using triggers It is als pssible t leverage ther mechanism in ODI t detect changes like timestamps, dates r sequence numbers. There are many mre JKMs prvided with ODI, yu can refer t the ODI Dcumentatin fr a cmplete list f the JKMs. Oracle Integratin Strategies The Integratin Knwledge Mdules ptimized fr the Oracle database are implementing many f the best practices t insert data int an Oracle data warehuse: - Direct Path lading is applied whenever pssible using the /*+ APPEND */ hint - Additinal ptimizer hints such as PARALLEL which specifies the degree f parallelism fr a given database peratin can be specified via the OPTIMIZER_HINT ptin f the IKMs - The ODI Staging Area tables are created by default using NOLOGGING t ensure best perfrmance - The MERGE SQL statement can be leveraged using the IKM Oracle Incremental Update (MERGE). MERGE cmbines an INSERT and an UPDATE in ne single ptimized SQL cmmand - ODI gathers tables statistics t ensure that the Oracle database Cst Based Optimizer (CBO) selects the mst ptimal executin plan ver time - Indexes are created t speed up the jins and filters executin. End users can chse frm several ptins including the creatin f bitmap indexes which can greatly 35

36 imprve the efficiency f jins between large fact tables and smaller dimensin tables in a star schema. Yu can refer t the fllwing table fr a list f the IKMs that can be used with an Oracle database: IKM IKM SQL Cntrl Append IKM Oracle Incremental Update IKM Oracle Slwly Changing Dimensin IKM Oracle Incremental Update (MERGE) IKM Oracle Multi Table Insert IKM Oracle AW Incremental Update DESCRIPTION Integrates data in any ISO-92 cmpliant database target table in TRUNCATE/INSERT (append) mde.) Data quality can be checked. Invalid data is rejected in the E$ errr table and can be recycled. Set-based incremental update fr Oracle Slwly Changing Dimensin Type II fr Oracle Integrates data in an Oracle target table in incremental update mde, using a MERGE statement Integrates data frm ne surce int ne r many Oracle target tables in append mde, using a multi-table insert statement (MTI). Integrates data in an Oracle target table in incremental update mde and is able t refresh a Cube in an Analytical Wrkspace. It is als pssible t extend the ODI Knwledge Mdules with yur wn best practices when needed. Thanks t the reusability f the KMs it is simple t ensure that best practices are enfrced and utilized by each develper. Defining a Data Quality Strategy Lading data that is nt clean in yur data warehuse can have a majr impact n the verall perfrmance f yur ETL prcesses. Primary key r freign key vilatins, business rules that are nt met, null values that are nt expected can waste time and resurces if the apprpriate Data Quality framewrk is nt in place within yur data integratin prcess. Oracle Data Integratr prvides an ut f the bx framewrk t handle Data Quality errrs ccurring while data is being laded int a database. Business rules and cnstraints are defined at the metadata level and thse validatin rules are checked against the data prir t being integrated int the target tables. The cnstraints can be enabled r disabled when required. The errrs that are captured are ruted int errr tables managed by ODI that can be reviewed thrugh the Designer Navigatr as well as the ODI Cnsle. CKM Oracle is the recmmended Check Knwledge Mdule fr checking data integrity cnstraints defined n a given Oracle table. 36

37 Setting up Agents in an Oracle envirnment Where t Install the Agent(s)? A typical data warehuse implementatin can have ne r mre ODI agent in prductin. In an Oracle envirnment, the agent is usually installed n the hst machine that is used t lad data in the data warehuse. The agent requires a cnnectin t the surce databases r files, and triggers the apprpriate lad utilities. Starting frm ODI 11g it is als pssible t install the ODI agents in WebLgic Server t achieve High-Availability (HA) thrugh a clustered deplyment. 37

38 Architecture Case Studies Setting up Repsitries General Architecture fr ODI Repsitries In a typical envirnment fr a data warehuse prject, yu wuld create the fllwing repsitries: A single master repsitry that hlds all the tplgy and security infrmatin. All the wrk repsitries are registered in this master repsitry. This single master repsitry cntains all the versins f bjects that are cmmitted by the designers. A Develpment wrk repsitry shared by all ODI designers. This repsitry hlds all the prjects and mdels under develpment. A Testing wrk repsitry shared by the IT testing team. This repsitry cntains all the prjects and mdels being tested fr future release. A User Acceptance Tests wrk repsitry shared by the IT testing team and the business analysts. This repsitry cntains all the prjects and mdels abut t be released. Business analysts will use the ODI Cnsle n tp f this repsitry t validate the scenaris and transfrmatins befre releasing them t prductin. A Prductin wrk repsitry shared by the prductin team, the peratrs and the business analysts. This repsitry cntains all the prjects and mdels in read-nly mde fr metadata lineage, as well as all the released scenaris. A Ht fix wrk repsitry shared by the maintenance team and the develpment team. This wrk repsitry is usually empty. Whenever a critical errr happens in prductin, the maintenance team restres the crrespnding prjects and mdels in this repsitry and perfrms the crrectins with the help f the develpment team. Once the prblems are slved, the scenaris are released directly t the prductin repsitry and the new mdels and prjects are versined in the master repsitry. 38

39 This recmmended architecture is described in the figure belw: Develpment Team Maintenance Team IT Testing Team Business Analysts Prductin Team Develpment Ht Fix Testing User Acceptance Tests Prductin Wrk Repsitries Prjects Prjects Prjects Prjects Prjects Mdels Mdels Mdels Mdels Mdels Scenaris and Lgs Scenaris and Lgs Scenaris and Lgs Scenaris and Lgs Scenaris and Lgs Master Repsitry Create and archive versins f mdels, prjects and scenaris Imprt versins fr Ht Fix and release scenari patches Imprt released versins f mdels, prjects and scenaris fr testing Master Repsitry Tplgy Imprt released and tested versins f prjects, mdels and scenaris fr prductin Security Versins Figure 10: ODI Repsitries and Teams Organizatin The master repsitry and all wrk repsitries are usually created in the same OLTP database instance in separate schemas r catalgs. When develpers have finished wrking n a prject and decide t release it, they create a versin fr their prjects and mdels and stre it in the master repsitry. This versin is then restred by the IT testing team in the testing repsitry. After the technical tests have cmpleted, the testing team initializes the user acceptance tests repsitry fr business analysts. They restre the same versin they were wrking n t have it tested by business users. Once this versin is functinally accepted, it is restred by the prductin team in the prductin repsitry. When a critical bug is discvered in prductin, the develpers are usually already wrking n the next release. Therefre they are usually nt able t stp their develpment and restre the previus versin fr crrectins. The maintenance team is then in charge f restring the versin used in prductin int a separate empty wrk repsitry called Ht Fix and applying the apprpriate fixes. Once dne, the maintenance team releases its mdified prjects, mdels and scenaris int the master repsitry s that the prductin team can restre them as a patch in the prductin repsitry. Creating a Separate Master Repsitry fr Prductin Fr sme particular security requirements, yu may nt want t share the same master repsitry between develpment and prductin. In this case, the slutin is t duplicate the master repsitry and t islate the prductin envirnment frm ther envirnments as shwn belw: 39

40 Develpment Team Maintenance Team IT Testing Team Business Analysts Prductin Team Develpment Envirnment Prductin Envirnment Develpment Ht Fix Testing User Acceptance Tests Prductin Wrk Repsitries Prjects Prjects Prjects Prjects Prjects Mdels Mdels Mdels Mdels Mdels Scenaris and Lgs Scenaris and Lgs Scenaris and Lgs Scenaris and Lgs Scenaris and Lgs Master Repsitry Create and archive versins f mdels, prjects and scenaris Imprt versins fr Ht Fix and release scenari patches Develpment Master Tplgy Security Versins Imprt released versins f mdels, prjects and scenaris fr testing XML XML Exprt/Imprt released versins fr prductin using XML exprt/imprt Manually replicate the Tplgy Prductin Master Tplgy Security Figure 11: Several Master Repsitries T create a new master repsitry fr yur prductin envirnment, it is recmmended that yu use an exprt f yur master repsitry frm yur Tplgy Navigatr and then use the Master Repsitry Imprt Wizard. Yu can find the Master Repsitry Imprt Wizard in the ODI Studi. When creating the new master repsitry, yu shuld assign a new ID t it, different frm the ID used by the develpment master repsitry. Once created, d the fllwing t set up the prductin envirnment: Create the prductin cntext Create all the prductin data servers and physical schemas in the physical architecture f the Tplgy Link yur prductin physical schemas t the existing lgical schemas defined by the designers in the prductin cntext. D nt change r remve existing cntexts and lgical schemas. Update the Security s that nly prductin users and business analysts can access the repsitry. Create the prductin wrk repsitry and give it an ID different frm any f the IDs used by the ther wrk repsitries (Develpment, Ht Fix, Testing and User Acceptance Tests). See Understanding the Impact f Wrk Repsitry IDs. Every time the develpment master repsitry is updated, manually replicate the changes in the prductin master repsitry. 40

41 Exprt prjects, mdels and scenaris that are ready fr prductin frm the develpment master repsitry int XML files. Yu can use the Versin Brwser t d s. Alternatively, if using Slutins, exprt yur slutins int cmpressed files Cnnect t the prductin wrk repsitry with Designer and imprt the XML files r the slutin s cmpressed files. Understanding the Impact f Wrk Repsitry IDs When creating a master repsitry r a wrk repsitry, ODI asks yu fr a 3 digit ID fr this repsitry. Yu shuld select this ID by fllwing the rules belw: Every master repsitry created in yur enterprise shuld have a unique ID. Every wrk repsitry created in yur enterprise shuld have a unique ID even if it belngs t a different master repsitry. Every type f bject in an ODI repsitry has a unique ID calculated accrding t the fllwing rule: <aut number> cncatenated with the 3 digits f the repsitry ID. Fr example, if the internal ID f an interface is , yu can autmatically guess that it was first created in the wrk repsitry ID 023. The main gal f this rule is t enable exprt and imprt in Synnym mde f bjects acrss wrk repsitries withut any risk f ID cllisin. If 2 wrk repsitries have the same ID, there is a chance that 2 different bjects within these 2 repsitries have the same ID. Therefre, imprting the first bject frm the first repsitry t the secnd may verwrite the secnd bject! The nly way t avid that is, f curse, t have 2 different wrk repsitry IDs. Using ODI Versin Management Hw Versin Management Wrks Versin management in ODI is designed t allw yu t wrk n different versins f bjects acrss wrk repsitries as shwn belw: 41

42 Several versins Archived Develpment Master Tplgy Security Versins Slutin V1.0 Slutin V2.0 Slutin V2.1 Slutin V2.0.5 Created versins: 1.0, 2.0 and 2.1. Is abut t release versin 2.2 Imprted released versin 2.1 fr testing. Imprted validated versin 2.0 fr prductin. Discvered a critical bug in this versin. Imprted validated versin 2.0 fr critical maintenance. Is abut t release versin that fixes the critical bug. Develpment Testing Prductin Ht Fix Prjects Prjects Prjects Prjects Mdels Mdels Mdels Mdels Scenaris and Lgs Scenaris and Lgs Scenaris and Lgs Scenaris and Lgs Open versins Currently under develpment Versins being tested Versins in prductin Versins being fixed fr a critical patch Slutin V2.2 Slutin V2.1 Figure 12: Versin Management with ODI Nte: The user acceptance tests repsitry des nt appear in this diagram as the mechanism fr ppulating it with new versins is similar t the ne used fr the testing repsitry. The develpers have already released versins 1.0, 2.0 and 2.1 f their develpment in the master repsitry. Every time they have released a versin they have used a Slutin. See Using Slutins fr Cnfiguratin Management fr details. They are nw wrking n the next release as they are abut t release slutin versin 2.2. Meanwhile, the IT testing team is testing versin 2.1 in its wrk repsitry. The prductin team is still wrking n versin 2.0 and it has discvered a critical bug in this versin. Therefre, the maintenance team has restred versin 2.0 in the Ht Fix wrk repsitry and is wrking n a patch release, v As sn as this patch release is cmmitted t the master repsitry, the prductin team will have t restre it int the prductin repsitry. Develpers will have t manually update versins 2.1 and 2.2 s they reflect the changes made by the maintenance team t versin Nte: Every wrk repsitry can nly have a single versin f an bject. Creating and Restring Versins f Objects Slutin V2.0 Slutin V2.0.5 T create a versin f an bject, users simply have t right-click n the bject and select Versin->Create Versin. 42

43 When the versin f the bject is created, it becmes available in the Versin Brwser and in the Versin tab f the bject. ODI als updates all the I and U flags relating t this bject t indicate that it is up t date with the versin in the master repsitry. If yu change yur bject in the wrk repsitry after cmmitting it in the master, yu will ntice that its icn is changed and a small U appears besides it t indicate that its status is nw Updated. This feature is very helpful as it shws yu all the differences between the current versin and the last versin cmmitted in the master repsitry. When yu create a versin f an bject, the bject is exprted t XML in memry, cmpressed and stred in the master repsitry as a binary large bject. Therefre, yu shuld cnsider creating a versin f an bject nly when yu are abut t cmmit it definitely fr a release. T restre an bject t ne f its previus versins, users simply have t right-click n the bject and select Versin->Restre. Then select frm the list f available versins which ne t restre. Yu shuld use cautin fr the fllwing reasns: All updates that yu have dne since the last time yu have created a versin f yur current bject will be lst The bject that yu are abut t restre may reference ther bjects that were remved frm yur wrk repsitry. The typical situatin is restring an interface that references a clumn that desn t exist anymre in yur mdels. If this happens, the restred bject will be marked as invalid with a red exclamatin mark and yu will have t edit it t crrect all missing references. Using Slutins fr Cnfiguratin Management During design time yu will prbably create versins f yur bjects fr backup. Fr example, yu will create versins f yur mdels, flders, interfaces, variables, Knwledge Mdules, etc. Mst f these bjects are designed t wrk tgether as they may depend n ne anther. Fr example, interface versin requires tables frm mdel and Knwledge Mdules versin 1.2 and 4.5. Maintaining the list f dependencies can be very tedius and time cnsuming. Therefre, when yu are abut t release what has been develped fr the testing team, yu wuld prefer t have a single bject that manages these dependencies. ODI slutin bjects are designed t manage these dependencies between a prject and all mdels, glbal bjects and scenaris it references. When yu are abut t release yur develpment, yu simply need t create a new slutin and drag and drp yur prject int this slutin. ODI will then create a new versin f yur prject (if required) and a new versin f every mdel, scenari, ther prject r glbal bject referenced by this prject. Therefre, instead f releasing several bjects, yu simply release the slutin. Ntes: Fr perfrmance reasns, yu shuld cnsider having small prjects t imprve the efficiency f the versin creatin mechanism. It is recmmended that yu split yur develpment int several small prjects cntaining less than 300 bjects. 43

44 When calculating the dependencies fr a prject in rder t build the slutin, ODI creates a versin f every mdel referenced by this prject. Then it lks at every mdel t see what Knwledge Mdules it references (RKM, JKM and CKM). And finally it creates a new versin f every prject t which these Knwledge Mdules belng. S if yur mdels reference Knwledge Mdules frm 3 different prjects, the slutin will reference these 3 prjects as well. A best practice t avid this is t create a prject called Metadata Knwledge Mdules and t have in this prject all the RKMs, CKMs and JKMs that yu will reference in yur mdels. Therefre every time yu create a slutin, this prject will be added as a dependency, instead f 3 r 4 different prjects. Ging t Prductin Releasing Scenaris Sectin Using ODI Versin Management gives an verview n hw t handle versins f bjects t release them t the prductin envirnment. Hwever, smetimes yu will nly want yur scenaris in the prductin wrk repsitry withut their crrespnding prjects and mdels. Fr that, yu wuld create an Executin wrk repsitry rather than a Develpment wrk repsitry. When yur scenaris are released int the master repsitry r as simple XML files, yu wuld either restre them t yur wrk repsitry r imprt them in synnym r duplicatin mde frm the Operatr Navigatr. Executing Scenaris Executing a Scenari Manually Yu can execute yur scenari manually frm the Operatr Navigatr. In this case if yur scenari requires variables t be set, yu wuld edit every variable used by the scenari and update its default value. T execute the scenari, yu als need t indicate the cntext fr the executin as well as the lgical agent in charge f this executin. An alternative way t execute a scenari is t use the ODI Cnsle t start the scenari frm yur web brwser. This apprach can be useful if yu are cnsidering having scenaris that will be executed n-demand by yur business users and analysts. Shuld yur scenari require variables, the user wuld be asked t fill in the variable values accrdingly. Executing a Scenari Using an Operating System Cmmand Yu can als start a scenari frm an perating system cmmand using the startscen shell script. Yu wuld d that if yu plan t use an external scheduler t schedule yur jbs. Refer t the ODI Dcumentatin fr details n this perating system cmmand When starting a scenari frm the perating system, yu d nt need a running agent. The startscen cmmand has its wn built-in agent that executes the scenari and stps at the end f the sessin. 44

45 Assigning Sessin Keywrds fr a Scenari Yu may ccasinally find it difficult t brwse the Operatr Navigatr lg as it may cntain a large number f sessins. T vercme this limitatin, ODI allws keywrds t be assigned t a sessin. This sessin will be stred autmatically in the apprpriate sessin flder created in the Operatr Navigatr. In the figure belw, the flder Prduct Dimensins was created fr the PDIM keywrd. When starting the LOAD_FAMILY_HIERARCHY and LOAD_PRODUCTS scenaris, the -KEWORDS parameter was set t -KEYWORDS=PDIM. Therefre when using the peratr t access the executed sessins, they are autmatically available under the Prduct Dimensins flder. Figure 13: Sessin Flders and Keywrds Yu can use this feature t distribute yur sessins acrss yur business areas. Using the Operatr Navigatr in Prductin The Operatr Navigatr in the ODI Studi is the mdule that is used the mst in a prductin envirnment. Frm this mdule, peratrs have the ability t: Imprt scenaris int the prductin wrk repsitry Execute scenaris manually and assign default values t scenari variables Fllw executin lgs with advanced statistics including the number f rw prcessed and the executin elapsed time Abrt a currently-running sessin. Restart a sessin that has failed and pssibly redefine the restart pint by updating the 45

46 task statuses. Create sessin flders t hld sessins with pre-defined keywrds Access and define scheduling infrmatin Using the ODI Cnsle in Prductin The ODI Cnsle can be used in prductin fr 2 purpses: T give business users, analysts and peratrs access t metadata lineage, metadata searching, scenari executin, and lg mnitring. In this situatin, the prductin wrk repsitry shuld be a develpment repsitry, and all the released prjects, mdels and scenaris shuld be restred in this repsitry. T give peratrs access nly t scenaris and lgs. In this case, the prductin wrk repsitry wuld nly be an executin wrk repsitry, and wuld nly cntain the released scenaris. Setting up Agents Where t Install the Agent(s)? A typical data warehuse implementatin usually requires ne r mre ODI agents in prductin. In a data warehusing envirnment an ODI agent is usually installed n the hst machine that is used t lad data in the data warehuse. The agent requires a cnnectin t surce databases r files, and triggers the apprpriate lad utilities. ODI 11g intrduces Java EE agents which are deplyed in a Java EE applicatin server and can leverage its features, such agents are recmmended when ne f yur requirements is t centralize the deplyment and management f yur agents r when there is a need fr high availability. The netwrk bandwidth between the machine hsting the agents and the data warehuse shuld be large enugh t handle the vlume f data that will be laded by the utilities in the warehuse database. On the ther hand, as the agents may need t access ther surce servers, clse attentin shuld be paid t the netwrk bandwidth t and frm these surce servers. If yur Knwledge Mdules generate perating-system-specific cmmands, these must match the perating system n which the agent is installed. In a nrmal envirnment, yu will set up: 1 physical agent fr the develpment team, preferably n the same perating system as the prductin agent 1 physical agent shared by the testing team, the maintenance team and the user acceptance team. Again, try t use the same perating system as the agent in prductin. 46

47 1 physical agent fr the prductin Using Lad Balancing There are cases when a single agent can becme a bttleneck, especially when it has t deal with large numbers f sessins started in parallel. Fr example, suppse yu want t retrieve surce data frm 300 stres. If yu attempt t d this in parallel n a single agent, it may lead t excessive verhead as it will require pening 300 threads (1 fr every sessin). A way t avid that is t set up lad balancing acrss several agents. T set up lad balancing in ODI, yu can fllw these steps in the Tplgy Navigatr: Define the agent that will be in charge f distributing the wrklad. In the fllwing, this agent will be referred t as the master agent. The master agent usually has nly 2 cncurrent sessins. All the scenaris will be executed by this single agent by using the AGENT parameter. Define several child agents that will be in charge f actually executing the scenaris. Set the maximum number f sessins f each agent t a value between 2 and 20. Edit the master agent in the Tplgy Navigatr, and in the Lad Balancing tab, select all yur child agents. The diagram belw gives yu an example f hw yu can setup this architecture: Machine 2 4 sessins AGENT1 Prt: Sessins: 3 3 Running Sessins 1 Pending Sessin Queued 14 incming sessins Machine 1 Master Agent Prt: Sessins: 2 4 sessins AGENT2 Prt: Sessins: 3 Machine 3 3 Running Sessins 1 Pending Sessin Queued 3 sessins AGENT3 Prt: Sessins: 3 3 Running Sessins 3 sessins AGENT4 Prt: Sessins: 3 3 Running Sessins Figure 15: Example f Lad Balancing In this example, 4 agents are installed n 2 different machines. Each f them accepts a maximum f 3 sessins in parallel. The master agent is installed n anther machine. 47

48 The master agent receives 14 requests t start several scenaris in parallel. It distributes these 14 requests t the available agents. Each agent will start 3 scenaris in parallel, and the first 2 agents will simply queue the remaining sessins fr future executin. As sn as ne f the agents becmes available, it will pick ne f the queued sessins and begin executing it. The master agent initially calculates the wrklad per agent accrding t the rati given by the number f sessins currently being executed versus the maximum number f sessins allwed fr this agent. It distributes incming sessins ne by ne t every agent and then recalculates the wrklad s rati. As the queued sessins fr an agent can be picked up by anther agent, the entire system becmes balanced. This flexible architecture allws yu t add as many agents as needed t reach the maximum scalability fr yur system. If yu plan t start several scenaris in parallel, yu can refer t sectin Using ODI Versin Management fr a detailed example n hw t achieve that elegantly. Ntes: The number f agents required and the maximum number f sessins that they may supprt will depend n yur envirnment settings. It shuld be tested carefully befre ging t prductin. The main bttleneck in this architecture is bviusly the RDBMS hsting the ODI repsitries as all agents access the same database tables. It is recmmended that yu dedicate a database server t the ODI repsitry if yu plan t setup such architecture. Yu wuld therefre have mre flexibility t tune this server in the case f lcks r perfrmance issues. Backing up Repsitries ODI recmmends that yu backup yur master and wrk repsitries n a daily basis t avid any lss f yur wrk. T backup yur repsitries, yu simply need t backup every schema n which yu have installed a master r wrk repsitry, using yur standard database backup prcedure. Refer t yur specific database dcumentatin fr infrmatin n hw t backup a schema r catalg. 48

49 Appendices Appendix I. Oracle Data Integratr fr Teradata Best Practices Architecture f ODI Repsitries It is recmmended that yu install the master and wrk repsitries in an OLTP database distinct frm the surce r target applicatins. The Teradata database is nt recmmended fr hsting ODI repsitries. Reverse-engineering a Teradata Schema The Teradata JDBC driver implements mst metadata APIs. Yu can therefre use the standard JDBC reverse-engineering with ODI. Using this methd, yu can capture metadata fr: Tables and Views including table cmments, Clumns including data types, length and scale, and cmments Primary Keys when they are defined using a PRIMARY KEY cnstraint statement in the databases Hwever, yu will prbably face the fllwing limitatins: Freign Key metadata is nt implemented in the JDBC driver. Therefre ODI will nt retrieve freign keys. Unique Primary Indexes (UPI) and Nn Unique Primary Indexes (NUPI) are nt imprted. T have a UPI imprted as a primary key in ODI, yu need t define it as a PRIMARY KEY cnstraint at the database level. Other indexes are nt imprted Check cnstraints are nt imprted Yu can bypass sme f these limits by using the Reverse Knwledge Mdule fr Teradata prvided with Oracle Data Integratr. This RKM is based n the DBC catalgue tables (DBC.Tables, DBC.Clumns etc). Yu may als enhance this Knwledge Mdule and adapt it t yur needs. RKM RKM Teradata DESCRIPTION Retrieves metadata frm the Teradata database using the DBC system views. This RKM supprts UNICODE clumns. 49

50 Teradata Lading Strategies Yu can refer t the fllwing table fr Lading Knwledge Mdules ptimized fr Teradata: LKM LKM SQL t SQL LKM File t Teradata (TTU) LKM SQL t Teradata (TTU) DESCRIPTION Lads data thrugh the agent frm any SQL RDBMS t any SQL RDBMS staging area Lads data frm a File t a Teradata staging area database using the Teradata bulk utilities. The utilities must be installed n the machine hsting the ODI Agent. Lads data frm a SQL cmpliant surce database t a Teradata staging area database using a native Teradata bulk utility. The utilities must be installed n the Unix machine hsting the ODI Agent. Using Laders fr Flat Files When yur interface cntains a flat file as a surce, yu may want t use a strategy that leverages the mst efficient lading utility available fr the staging area technlgy, rather than the standard LKM File t SQL. The ODI Agent accepts binary files and supprts several native binary data types such as Binary Big and Little Endian, Ebcdic, Ebcdic zned decimals, Packed decimals, etc. Hwever, we recmmend against using ODI Agent t lad large binary files int the Teradata staging area. As the agent uses JDBC t write t the target, using this methd wuld lead t significant perfrmance issues cmpared t lading using the native Teradata Tls and Utilities (TTU) which include FastLad, MulitLad, TPump, Teradata Parallel Transprter r FastExprt. Yu can refer t the TTU dcumentatin fr a way t lad binary files in a Teradata table. The LKM File t Teradata (TTU) shuld meet yur needs fr generating and executing the apprpriate TTU scripts. Using Unlad/Lad fr Remte Servers When the surce result set is n a remte database server, an alternative t using the agent t transfer the data wuld be t unlad it t a file and then lad that int the staging area. The LKM SQL t Teradata (TTU) fllws these steps and uses the OdiSqlUnlad tl t unlad data frm any remte RDBMS. Of curse, this KM can be ptimized if the surce RDBMS is knwn t have a fast unlad utility. Using Piped Unlad/Lad When using an unlad/lad strategy, data needs t be staged twice: nce in the temprary file and a secnd time in the C$ table, resulting in extra disk space usage and ptential efficiency issues. A mre efficient alternative wuld be t use pipelines between the unlad and the lad utility. Unfrtunately, nt all the perating systems supprt file-based pipelines (FIFOs). 50

51 ODI prvides the LKM SQL t Teradata (TTU) that uses this strategy. T have a better cntrl n the behavir f every detached prcess (r thread), this KM was written using Jythn. The OdiSqlUnlad tl is als available as a callable bject in Jythn. Teradata Integratin Strategies The fllwing table lists sme f the Integratin Knwledge Mdules ptimized fr Teradata: IKM IKM Teradata Cntrl Append IKM Teradata Incremental Update IKM Teradata Slwly Changing Dimensin IKM File t Teradata (TTU) IKM SQL t SQL Append IKM SQL t Teradata (TTU) IKM Teradata Multi Statement IKM Teradata t File (TTU) DESCRIPTION Integrates data in a Teradata target table in replace/append mde. Integrates data in a Teradata target table in incremental update mde. Integrates data in a Teradata target table used as a Type II Slwly Changing Dimensin in yur Data Warehuse. This IKM is designed t leverage the pwer f the Teradata utilities fr lading files directly t the target. Integrates data int an ANSI-SQL92 target database frm any remte ANSI- SQL92 cmpliant staging area in replace r append mde. Integrates data frm a SQL cmpliant database t a Teradata database target table using Teradata Utilities TPUMP, FASTLOAD OR MULTILOAD. Integrates data in Teradata database target table using multi statement requests, managed in ne SQL transactin. Integrates data in a target file frm a Teradata staging area in replace mde. IKMs with Staging Area Different frm Target File t Server Append There are sme cases when yur surce is cmpsed f a single file that yu want t lad directly int the target table using the mst efficient methd. By default, ODI will suggest putting the staging area n the target server and perfrming such a jb using an LKM t stage the file in a C$ table and an IKM t apply the surce data f the C$ table t the target table. The recmmended Knwledge Mdule when a file needs t laded int Teradata withut staging the data is the IKM File t Teradata (TTU), which allws the generatin f apprpriate scripts fr each f these Teradata utilities depending n yur integratin strategy. Server t Server Append When using a staging area different frm the target and when setting this staging area t an RDBMS, yu can use an IKM that will mve the transfrmed data frm the staging area t the remte target. This kind f IKM is very clse t an LKM and fllws almst the same rules. The IKM SQL t Teradata (TTU) uses this strategy. 51

52 Server t File r JMS Append When the target datastre is a file r JMS queue r tpic yu will bviusly need t set the staging area t a different place than the target. Therefre, if yu want t target a file r a queue datastre yu will have t use a specific Multi-Cnnectin IKM that will exprt the transfrmed data frm yur staging area t this target. The way that the data is exprted t the file r queue will depend n the IKM. Fr example, yu can chse t use the agent t have it select recrds frm the staging area and write them t the file r queue using standard ODI features. Or yu can use specific unlad utilities such as Teradata FastExprt if the target is nt JMS based. Yu can refer t the fllwing IKMs fr further details: IKM IKM Teradata t File (TTU) IKM SQL t File Append DESCRIPTION Integrates data in a target file frm a Teradata staging area in replace mde. Exprts data frm any SQL cmpliant staging area t an ASCII r BINARY file using the ODI File driver Setting up Agents in a Teradata envirnment Where t Install the Agent(s)? A typical data warehuse implementatin usually requires ne r mre ODI agents in prductin. In a Teradata envirnment, the agent is usually installed n the hst machine that is used t lad data in the data warehuse. The agent requires a cnnectin t surce databases r files, and triggers the apprpriate lad utilities (fastlad, multilad, tpump). Starting frm ODI 11g it is als pssible t install the ODI agents in WebLgic Server t achieve High-Availability (HA) thrugh a clustered deplyment. 52

53 Appendix II: Additinal Infrmatin Acrnyms used in this dcument 3NF API ASCII CKM DDL DML DW EBCDIC E-LT ETL GUI HTTP IKM IT JEE JDBC JKM JMS JNDI JVM KM LDAP LKM ODS OLTP 3 rd Nrmal Frm Applicatin Prgramming Interface American Standard Cde fr Infrmatin Interchange Check Knwledge Mdule Data Descriptin Language Data Manipulatin Language Data Warehuse Extended Binary-Cded Decimal Interchange Cde Extract, Lad and Transfrm Extract, Transfrm and Lad Graphical User Interface Hypertext Transprt Prtcl Integratin Knwledge Mdule Infrmatin Technlgy Java Enterprise Editin Java Database Cnnectivity Jurnalizing Knwledge Mdule Java Message Services Java Naming Directry Interface Java Virtual Machine Knwledge Mdule Lightweight Directry Access Prtcl Lading Knwledge Mdule Operatinal Data Stre Online Transactinal Prcessing 53

54 RDBMS RI RKM SOX SQL URL XML Relatin Database Management System Referential Integrity Reverse-engineering Knwledge Mdule Sarbanes-Oxley Simple Query Language Unique Resurce Lcatr Extended Markup Language 54

55 Best Practices fr a Data Warehuse August 2010 Authr: ODI Prduct Management Cntributing Authrs: ODI Prduct Develpment Oracle Crpratin Wrld Headquarters 500 Oracle Parkway Redwd Shres, CA U.S.A. Wrldwide Inquiries: Phne: Fax: racle.cm Cpyright 2010, Oracle and/r its affiliates. All rights reserved. This dcument is prvided fr infrmatin purpses nly and the cntents heref are subject t change withut ntice. This dcument is nt warranted t be errr-free, nr subject t any ther warranties r cnditins, whether expressed rally r implied in law, including implied warranties and cnditins f merchantability r fitness fr a particular purpse. We specifically disclaim any liability with respect t this dcument and n cntractual bligatins are frmed either directly r indirectly by this dcument. This dcument may nt be reprduced r transmitted in any frm r by any means, electrnic r mechanical, fr any purpse, withut ur prir written permissin. Oracle and Java are registered trademarks f Oracle and/r its affiliates. Other names may be trademarks f their respective wners. AMD, Optern, the AMD lg, and the AMD Optern lg are trademarks r registered trademarks f Advanced Micr Devices. Intel and Intel Xen are trademarks r registered trademarks f Intel Crpratin. All SPARC trademarks are used under license and are trademarks r registered trademarks f SPARC Internatinal, Inc. UNIX is a registered trademark licensed thrugh X/Open Cmpany, Ltd. 0410