Oracle Apps and Data Warehousing An Oxymoron?

Transcription

1 Oracle Apps and Data Warehousing An Oxymoron? David Fuston Director, Business Intelligence The IRUS Group Website: Cell: : 2: Background Technical and functional consultant since 1989 with experience in applications development, IT management, and financial controller positions for Fortune 500 and World 200 companies. Oracle Beta Tester and Early Adopter since 1997 for numerous programs, such as Applications 11i (OFA, EPF & EPB), 9iAS Discoverer and Drake 10g, Warehouse Builder 9i, and 9i/10g OLAP. Officer, speaker, or sponsor in OAUG, BI/DW SIG, OracleWorld, AppsWorld, OpenWorld, IOUG, ODTUG, NCOAUG, RMOUG, SCOUG, KCOUG IRUS Group is certified consulting firm, with practices in Data Warehousing and Business Intelligence, and Oracle Applications since 1996 Celebrating 10 years, still privately held, with steady growth year over year, currently 50+ consultants on 2 continents Partner Status with Oracle HyperRoll Cognos Microsoft 1

2 Agenda What BI is and Terminology Marketplace Assessing BI/DW Readiness Oracle s DW choices Oracle EDW* Teradata Decision Experts (Formerly DecisionPoint Applications**) Jaros Analytics Noetix NETS Cognos Performance Applications DIY with OWB and Discoverer Questions *these products are in maintenance mode **Acquired by Teradata (NCR) in 11/2005 Business Intelligence Definition Business Intelligence The processes, technologies, and tools needed to turn data into information, information into knowledge, and knowledge into plans that drive profitable business action. Business intelligence encompasses data warehousing, business analytic tools, and content/knowledge management. THE DATA WAREHOUSING INSTITUTE Data Mart Data Warehouse Extract, Transform, and Load (ETL) Operational Data Store (ODS) Star Schema versus Snowflake Analytical Applications 2

3 Typical BI/DW Analytical App Architecture The Data Warehouse Institute Administration & Operations (135) : Performance & Usage Management, Database Management, Capacity Planning, More Analytic Applications & Development Tools (298) : Development Tools, Visualization Tools, Business Performance Management, More... Business Analytics (245) : Data Mining, Query & Reporting, OLAP, Business Analytics Suites, More... Business Intelligence Services (123) : Consultants/Systems Integrators, Analytic Service Providers for Data Warehouse, More... Data Integration (184) : Data Quality and Cleansing, DW Mapping & Transformation, Meta Data Management, More... Data Warehouse Design (31) : Data Modeling & Analysis, Data Warehousing Toolsets, More... Information Delivery (98) : Enterprise Information Portals, Broadcasting, Wireless Data Analysis, More... Infrastructure (89) : Relational Database Management Systems, Multidimensional Databases, More... 3

4 ETL tools Infrastructure: Types of data sources (Oracle, non Oracle) Open platform support (NT, Linux, Unix) Data volumes, scalability, and performance Scheduling, Debugging, Concurrent Processing Common metadata shared among applications Business: Complexity of transformation Complexity of integration Resource/Skill Levels: Available resources Skill mix ETL Effort in DW project ETL is the heart and soul of business intelligence (BI) According to most practitioners, ETL design and development work consumes 60 to 80 percent of an entire BI project Developers are five to six times more productive once they learn an ETL tool compared to using a command line interface.. Peter Mimno 4

5 Why a Separate BI Tool? [End User] Empowers end-users to do own analysis Frees up IS/IT backlog of report requests Ease of use easy selection of data Drill-down No knowledge of SQL or tables required Exception Analysis Variance Analysis Easy rotation Formula calculations Aggregate data Why Consider a Data Warehouse? [IT] Standard Oracle Reports don t meet business requirements Visibility and Simplification of Reports Custom reports take too long to produce Too many resources tied up in reporting Data manipulation is required, extensive use of Excel No tools or time to do detailed analysis Data is not accurate Multiple data sources, complex table structures Views are not performing well Multiple versions of the truth in meetings 5

6 Assessing BI/DW Readiness Strong Business Management Sponsor Business Vision IS/Business Partnership Current Analytic Culture Feasibility --Ralph Kimball, The Data Warehouse Lifecycle Toolkit Oracle s Embedded Data Warehouse Content: GL, AP, AR, PO, HR, OM--partial GA in 2001, now in maintenance mode with approximately 30 customers Integration of multiple instances / versions of Oracle Apps (10.7, 11.0.x, 11i) via API Leverage 9i data warehousing features, primarily SQL Analytic commands and Materialized Views Conforms to Common Warehouse MetaModel (CWM), specifications adopted by Object Management Group (OMG--composed of IBM, Oracle, Microsoft, and Informatica) Report using Discoverer only 6

7 Oracle Enterprise DW Requirements Support for non-oracle Apps data is difficult as must load data via API into Oracle Apps tables first. If you use OWB and load non-oracle data directly into EDW tables, you are in a custom non-supported solution Modules not supported yet WIP, BOM, MRP, FA Separate security from apps security due to separate db instance on separate box EDW 4.1 (applicable to apps ) requires 9iAS , OWB , and Discoverer Requires Oracle VP approval to implement via Oracle Consulting as EDW is in maintenance mode BIS, EDW, and DBI timeline 1.2 (11.0.3) 11i.1 11i.2 11i.3 11i.4 11i.5 11i.6 11i.7 11i.8 11i.9 BIS 11i.10 Maintenance only EDW Maintenance only 5.0 DBI Ongoing Development Apr/98 Feb/01 Feb/03 Dec/03 Present 7

8 Oracle EDW/BIS/DBI Architecture Oracle Business Intelligence Oracle Business Intelligence DBI Management Reporting BI Portal Tables, Graphs, KPI s Materialized Views BIS PHP Tables, Graphs, KPI s BIS Views BIS Ad hoc analysis and reporting Discoverer Workbooks End User Layer EDW Facts, Dimensions Custom Facts, Dimensions Base Summary DBI Business View Generator Generated Business Views Custom Business Views EDW e-business Suite transaction tables Other Data Sources Teradata Decision Experts (Formerly DecisionPoint Software) First to Market with Packaged Data Warehouse (Sequent) Established in 1996, headquartered in San Mateo, CA. with approximately 60 employees. Acquired by Teradata (NCR) on 11/30/ Oracle Coverage i in approximately 50 companies. Financials: GL, AP, AR, FA, PA, Grants PO, INV, OE/OM, BOM, WIP HR, Payroll, Benefits Unique Features Sarbanes-Oxley Financial Compliance Dashboard Claim is fast implementation (8 weeks for GL/AP/AR with 2 years of historical data) with 5 report templates Proprietary PL/SQL generator for ETL is bundled with product, or can interface to Informatica PowerCenter Support for FSG(s) imported into Financial Statement Analysis (FSA) Reports to run against the DW Supports DecisionPoint DecisionCast, Brio, MicroStrategy, Business Objects, Cognos, Hyperion System 9, Discoverer, Crystal, etc. as front-end tools 8

9 Teradata Decision Experts (Formerly DecisionPoint Software 90+ star (federated) schemas with conforming and nonconforming dimensions, organized into 14 subject areas Configurations include both 2 and 3 tier environments Proprietary Source Experts extract key flex field segments in Oracle object library and auto generate columns in DW fact tables (flexfields for Oracle Apps, and trees for PeopleSoft) Data scrubbing in DW data loading is via custom PL/SQL scripts or UNIX scripts High end analytics via proprietary DecisionCast web client tool using.net or analysis using Java SDK 1.4+; no modeling or forecasting directly supported Teradata Decision Experts (Formerly DecisionPoint Software) Cross Industry Approach, with large implementations at Alcoa, Imation, BT, Sony, Echostar, Aetna, Kerr-McGee, AMC Theatres. Teradata is targeting manufacturing, retail, and insurance. Support Kimball model of slowly changing dimensions type 1 or 2, but not 3 DW has no archive and purge facility in administrator, must be done manually by DBA 100% of implementations have non-oracle source data in DW via customizations Recent ERP source system additions are PeopleSoft and SAP; recent target system addition is Teradata 9

10 DecisionPoint Jaros Analytics ( Founded 1999, Headquartered in Granite City, IL with approximately 25 employees Beta in 2002, GA in January 2003 Oracle Coverage i and MFG/PRO in 10 companies Financials: GL, AP, AR, FA, HR PO, OE/OM, INV MRP, BOM, WIP, MRP Projects, Service, Contracts under development Unique Features Near-Real Time Analytics (Operational Data Store--ODS) Deliver pre-built Discoverer EUL with package 6 week implementation for entire financial suite Informatica PowerMart ETL and metadata repository with package Supports Oracle 8i/9i/10g as target DB Supports Oracle and non-oracle data sources natively (any Informatica data source relational, mainframe, AS/400, flat files, MS SQL Server, Teradata) Supports Discoverer, Cognos, Informatica PowerAnalyzer, Information Builders, MicroStrategy, etc. as front-end client tools 10

11 Jaros (continued) Industry Focus: Manufacturing and Distribution Operational Data Store is used for near real-time tactical reporting, such as shipments status at the end of the month and backlog statistics Data Warehouse is used for strategic reporting, such as comparisons to last season, trends, profitability analysis, ISO reporting Support for slowly changing dimensions is an out-of-the-box feature, which include Kimball s type 1 (restated) and type 2 dimensions Platforms supported include all platforms supported by Informatica and Oracle, which includes but not limited to Linux, AIX, HPUX, Solaris, and WinXX Jaros (continued) ODS supports loading different areas at different intervals for multiple functional groups Jaros Dashboard is a private label Informatica PowerAnalyzer web client front end tool Configurations include 1,2 and 3 tier approaches Always up-to-date self-documentation via Informatica Metadata Repository, which is native object repository for Business Objects Universes, Discoverer EUL, Cognos catalogs, etc. Star Schema data model supported in CA ERWin or Oracle Designer Software design and implementation based on Six Sigma process methodology for quality assurance. 11

12 Jaros Architecture 9iAS, Any Reporting Tool, OLAP, Other Applications Discoverer EUL: ODS Discoverer EUL: DW Multidimensional, Star Schema Data Warehouse Strategic Reporting Summary, Detail, Snapshot Data Operational Data Store (ODS) High Performance Staging Area, Point of Integration, Lowest Level of Detail, Near Real Time Tactical Reporting ETL Mappings (Complex Transformations) Insulation Views Change Tracking ETL Mappings: Extraction, Transformation, and Load (Near Real Time) Legacy Oracle Applications Other ERP Noetix Enterprise Technology Suite (NETS) ( Founded in 1994, headquartered in Bellevue, WA., with 100+ employees Original NoetixViews launched in 1996 has customers, then extended to NETS in 2001, including large customers like CompUSA, Motorola, Starbucks, Toshiba, Xerox, and Visa Oracle coverage for i in the following modules: Financials AP, AR, FA, GL, DBI, and Grants Supply Chain INV, OM & PO Discrete/Process Manufacturing MRP, BOM, WIP & CST Human Resources HR & PAY, OAB Project Accounting PC & PB 12

13 Noetix Enterprise Technology Suite (NETS) ( NETS composed of NoetixViews basic, value-add, and cross-module views on top of Oracle Apps. Oracle Apps has more that tables condensed into 500+ NoetixViews Noetix Query Server (NQS) query router similar to DNS server on Internet, ODBC drivers, NQS Hub, NQS repository for routing and security data Noetix WebQuery (NWQ) web based reporting tool, gateway to.net web service, and repository for storing ad-hoc reports, folders, and end user information AnswerPoint generates an extensive library (500+) of customizable reports. Noetix Financials Dashboard over 100+KPIs covering GL, AP, AR, and OM presented via a graphical display. Business Objects/Crystal Xcelsius is included as a component of the Noetix Dashboard Foundation NETS (continued) Noetix Query Server (NQS) is a Win2000 SP4 or Win2003 only service that routes queries to and from appropriate data source and requires SQL Server 7/2000 SP4 or later, IIS v5, and MS Reporting Services NQS accepts queries only from WebQuery clients and uses ODBC to access databases NoetixViews reside in a separate schema from Oracle Apps or same Apps schema your choice Both NQS routing and security repository and WebQuery ad-hoc reports and folders repository reside in a SQL Server database LDAP and Oracle Apps security are both supported as setup choices MetaBuilder s auto-discovery of flex fields and pick lists is proprietary and patented Actual case studies confirm that NoetixViews portion can be brought into production in one week. 13

14 NETS (continued) Preferred client tools are Noetix WebQuery, MS Excel, Oracle Discoverer, Business Objects, and Cognos NETS uses existing Oracle Apps security model and user profiles NETS assumes clean data from Oracle Apps and therefore has no transformation or scrubbing capabilities like an ETL tool NETS has no ODS, nor DW, so data presentation is real-time, and therefore summary versus detail data can, in fact, be different due to refresh rates NETS can be connected to the internet via Oracle Portal, Plumtree, or MS Sharepoint. NETS has current coverage for Oracle Apps ( i) and Siebel ( ), and has announced coverage for PeopleSoft (19-Sep-2005 in development for HR, Benefits, and Payroll). Noetix Architecture Applications Query & Reporting Tools Noetix WebQuery with Excel Integration Noetix Dashboards ETL Noetix Platform Query Mgmt Services Metadata Repository Generated Reports (Templates) Oracle ebusiness Generated Views Oracle ebusiness Siebel ebusiness Siebel ebusiness Custom Views Data Oracle ERP PeopleSoft Siebel Legacy App Custom DM/DW 14

15 Cognos Performance Applications Coverage for GL, AP, AR, Projects, OM, INV, PO modules in Oracle Applications 11 and 11i using 9i Database Organized into 3 application areas covering 7 modules, with 100+ KPIs Financial area has 11 dimensions; SCM has 16 dimensions; Sales area has 11 dimensions Additional coverage for SAP 4.0b, 4.6c, and 4.7, PeopleSoft World 7.3+, and EnterpriseOne (formerly OneWorld, XE, ERP8.9). DB2/400, MS SQL Server, and Oracle databases are supported here. Installation and maintenance will require a separate SQL editor of your choice, some IBM client access software (if PeopleSoft), Oracle SQL*Loader or bulk loader utility of your choice, Cognos DecisionStream 7.1 MR2, IE 4.0+ or Netscape 4.0+, Cognos Series 7 v2, Cognos ReportNet v1.1 MR1, and Cognos Metrics Manager v2.2 MR1 Platforms supported are Solaris, HPUX, AIX, WinXX with ETL on one server, target database/apps/reporting tools on another server, and source database on another server. Cognos Performance Applications PowerPlay embedded into ReportNet and vice versa; PowerPlay cubes are still in proprietary format Licensing model expects you to have 80% HTML clients, 10% business authors, and 10% professional authors. Both authoring roles are recommended for ILT training; HTML client training is a CBT. No ODS in design; no persistent staging tables 85% of metrics out-of-box are derived from the raw data in target database in the 8 pre-defined analytic apps. No support for partitioning when using Oracle 9i+; uses sequential updates, not parallel processing Pulls source data directly from both raw tables and source views; timebased, not event based, data movement CWM 1.0 compliant Data model created in Oracle Designer Support for 15 attributes per dimension before customization required; this applies to both key Flexfields and descriptive Flexfields. 15

16 Applications Certification DIY Discoverer 3i, 4i and work against Oracle Applications 11i, 11.0 and rapid install contains Discoverer 4i ( as part of ias ) Customers who use BIS/EDW are certified to use 4i against 11i if ias installed on separate machine (metalink note , ARU ) Customers who use BIS/EDW and want ias in same Apps Oracle Home can use the rapid install (metalink note ) Debugging and Troubleshooting (metalink notes , , , , ) Common Challenges in all DW packages Summary and detail data need to match event based or time based, sync modules or whole app, at what frequency, under what conditions Performance considerations for Apps DB views on same box or separate box copy, multiple years in OLTP versus warehouse, historical data load into OLTP or warehouse at time of upgrade or system conversion Capture the transient detail (ex: net changes to orders) or snapshots for reports such as shipment backlog, inventory turns, AR days sales outstanding, etc. Ability to create summary or aggregate tables and/or cubes and then have these tables (i.e. metadata) be visible to client end-user tool for query performance Partitioning of data for both archive, backup, and data roll-off strategies 16

17 Common Challenges in DW packages continued Metadata shared, such as Common Warehouse MetaModel (CWM), amongst transformation engine, database, and client front-end tool load once, report many times strategy Time/date stamps on different systems in different formats and using different data types need to be merged into a single fact table Data Warehouses are never done, ultimately becoming the historical repository in lieu of upgrading data in Oracle Apps Integration of multiple data sources (Oracle and non-oracle) happens at least 85% of the time in a real-world BI/DW projects No single BI end-user tool will fit with the needs and wants for multiple groups of end-users, with requirements such as simple 2-D reporting, ad-hoc query, traditional 3-D OLAP with drill and rotate, CPM, BAM, data mining, and forecasting what if models Common Best Practices in all DW packages Do not use virtual DW, i.e, do not directly access OLTP tables, use views as insulation Do not code ETL functions by hand use the packaged tools and functions Integrate components of the DW architecture with one central metadata repository Use end-user client tools that integrate with the metadata and pre-computed aggregates Architecture of complete DW includes conforming dimensions for multiple business areas (i.e., data marts) for cross-module drill down 17

18 Differences in DW packages ETL process, non-oracle data, data scrubbing and transformations EDW uses OWB and supports non-oracle data loaded into Oracle Apps via API, or custom loads via OWB into EDW Noetix uses no ETL, has support for both Oracle Apps and Siebel sources, other sources are custom or in development (PeopleSoft) DPA uses a proprietary ETL against Oracle Apps and can customize for non-oracle sources Jaros uses Informatica PowerMart as OEM for any data source outof-the-box, with mappings provided for Oracle Apps Cognos uses its own DecisionStream product for ETL, acknowledges its has weaknesses against market dominate Informatica ETL, but supports all platform databases used in Oracle Apps, PeopleSoft, SAP, and EnterpriseOne (formerly JDEdwards) Differences in DW packages Common Warehouse MetaModel (CWM) EDW has partial CWM with OWB metadata, but not with Discoverer EUL, which is an additional extract and load Noetix uses central repository in MS SQL Server, but it is not CWM compliant, must be loaded into your data modeling tool, and then shares metadata with WebQuery or loads Discoverer EUL DPA uses a proprietary metadata repository, which must loaded into your data modeling tool, and then shares metadata with DecisionCast, or loads Discoverer EUL, Business Object Universe, etc. Jaros uses CA Erwin for data modeling, Informatica PowerMart for central metadata repository, shares metadata natively with Jaros Dashboard, Business Objects, Cognos, Brio, MicroStrategy, Hyperion, or loads Discoverer EUL Cognos uses Oracle Designer for data model, accesses source data directly from raw tables and views, strongly prefers its own reporting tools PowerPlay and ReportNet, and is CWM1.0 compliant 18

19 Common Reasons for Failures in BI/DW Attempted integration of stand-alone packages or tools which do not share common metadata Coding of extraction/transformation/load procedures by hand Populate DW with dirty source data* DBA(s) that have OLTP and normalized table experience, but no dimensional star schema experience Inability to respond to change requests from users Lack of flexibility in query, analysis, and reporting tools Failure to meet the business objectives* * Guaranteed Failure Common Reasons for Failures in BI/DW Lack of ERP data source expertise at table and field level Lack of enterprise wide DW architecture and infrastructure ETL tool, central metadata repository, metadata exchange interface, data modeling tool, and end user presentation tools Lack of defined process steps define business needs, specify architecture, select tools, and then implementation controls Let the ERP package dominate the DW* Did not anticipate performance and/or scalability issues Lost audit trail, data integrity, or ability to reconcile during data transformation/scrubbing* * Guaranteed Failure 19

20 More Questions? Oracle Apps and Data Warehousing An Oxymoron? David Fuston Director, Business Intelligence The IRUS Group Website: Cell: : 2: 20