SAP BI - Data Quality with Business Objects Data Services

SAP BI - Data Quality with Business Objects Data Services SAP NetWeaver BI taps into Data Services November 2008

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 2

Do You Trust Your Information? Survey of information workers* Up to 75% have made wrong business decisions due to flawed data Only 10% always have the information they need to make business decisions They spend up to 30% of their time verifying the accuracy and quality of the data they use to make decisions *Survey of information workers in the US, Great Britain, France, and Germany, commissioned by Business Objects and conducted by Harris Interactive June 2006 SAP 2008 / Page 3

Is your IT organization able to keep up with information demands? Business IT Need timely access to trusted data Changing business requirements Making decisions with knowledge shadows Information Gap Limited capacity to support users Competing priorities Lengthy ETL and data quality development cycles SAP 2008 / Page 4

Data Quality Challenges for Data Warehousing Data Warehouse Challenges in Data Quality General Incorporation of multiple sources of data Completeness of data Credibility of the data Tracking of origin of data (including data lineage) Define strategy based on source data Master data Value check (including plausibility, ranges, etc.) Structure of data (Pattern) Standardizing of the data Elimination of duplicate records Transactional data Datenqualität Referential integrity Checksums on key figures Value check (including plausibility, thresholds, lookups, etc.) SAP 2008 / Page 5

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 6

Business Objects is a market leader in data quality Market Leader Gartner Magic Quadrant for Data Quality Tools, 2008 Source: Gartner Sept., 2008 SAP 2008 / Page 7

SAP Business Objects A Comprehensive solution for EIM Applications ERP, SRM, CRM Performance Management Business Intelligence Enterprise Information Management Data Integration Data Quality MDM Information Lifecycle Management Data Profiling Metadata Management Structured Data Data Layer Unstructured Data ERP DW RDBMS OLAP XML Docs Web Email Notes SAP 2008 / Page 8

SAP Data Services Data Services is the first single tool for data integration and data quality Data Integrator XI R2 Development User Interface Data Services XI 3.0 One Development UI Metadata Repository Runtime Architecture Administration and Connectors One Metadata Repository Data Quality XI R2 Development User Interface Metadata Repository Runtime Architecture One Runtime Architecture Access Profile Cleanse Transform Deliver Administration and Connectors One Administration Environment SAP 2008 / Page 9

One Development User Interface for Data Integration and Quality Integrate heterogeneous data across the enterprise Integrate heterogeneous data across the enterprise Profile and cleanse any type of data anywhere in the enterprise SAP 2008 / Page 10

Data Services Architecture SAP ERP, SAP CRM, SAP MDM, SAP NetWeaver BI, R/3, ERP, NetWeaver BI Oracle, SQL, DB2, etc. PSFT, Oracle Apps, Siebel, etc. Real Time Batch Data profiling SOA Data Services Engine Data Cleansing Data Validation Data Auditing SAP BI Data Migration, Synchronisation, Query, Reporting, Analysis & Dashboards Files, XML, Mainframe, Excel, etc. Impact Analysis Shared Metadata Data Lineage SAP 2008 / Page 11

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 12

Profile And Cleanse Complete and global data quality Data Quality Framework Measure and analyze data through data assessment and continuous monitoring Cleanse and enhance customer and operational data anywhere across the enterprise Match and consolidate data at multiple levels within a single pass for individuals, households, or corporations Improve and automate the delivery of direct mail and goods SAP 2008 / Page 13

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 14

Side-by-Side comparison Requirement Data Services SAP BI Referential Integrity Plausibility Check Pattern matching Lookups BI Master Data attribute lookup Profiling Address cleansing Data Cleansing Matching Formula support Custom routines and functions Auditing SAP 2007 / Page 15

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 16

Data Services for SAP NetWeaver BI - Use Cases Data Services complements the intrinsic BI capabilities Apply 1 Data Quality measures to Non-SAP data Cleanse 2 SAP BI data Call 3 Data Services from BI staging (not scope of this presentation) Any Source SAP 1 SAP NetWeaver BI 2 Address Cleansing 3 WebService UDC Use case Describes where Data Services adds value SAP 2008 / Page 17

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 18

Scenario I: Simple Data Quality by Data Services 1 SAP BI Staging Scenario Describes how Data Services is usedc SAP 2008 / Page 19 Data Data Services XI XI 3.0 3.0 Basic Basic Checks Profiling Profiling of of source source data data Domain Domain value value and and plausibility plausibility check check Pattern Pattern Matching Matching String String Matching Matching

Data Profiling in Data Services - Understand your data - Need to understand the data before creating an ETL process Check for missing values (NULL) Get possible list of values Visualize the data distribution Find patterns Get data ranges (min, max, average) identify data domain outliers Uniqueness of data (distinct values) Can also be used to: Verify results of an ETL load during development Analyze data for system migrations Loading additional data such as potential leads or purchased lists SAP 2008 / Page 20

Example of Profiling SAP 2008 / Page 21

Data Validation check your data Use either Validation or Query Transform Check data in respect to Domain / plausibility check Validity checks Ranges checks (for dates, postal codes, etc.) Examine Data Structures based on patterns for Phone Numbers Dates & Times General Numbers Use Boolean expressions and custom coding for complex requirements Find records by Search strings Wildcard search SAP 2008 / Page 22

Data Validation in Data Services SAP 2008 / Page 23

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 24

Incorporate Data in SAP BI staging Motivation Benefit from Data Services tools and features Easy access to Non-SAP sources Use scheduling feature from SAP BI Trigger process by SAP BI SAP BI Staging Pre-requisites: Based on RFC call Create RFC destination in SAP BI Connect SAP BI and Data Services Data Services as Source System in SAP BI SAP BI as target or source in Object Library of Data Services Start RFC Server for Data Services SAP 2008 / Page 25

Steps for Incorporation of Data Define Data Services as Source System in SAP BI Define BI in Data Services Object Library Import InfoSource Metadata into Data Services Incorporate Info- Source in Data Services Job Export Job execution (for batch execution by SAP BI) Define Info- Package to schedule execution of Data Services Job Check Result in PSA or respective data target SAP 2008 / Page 26

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 27

Scenario II: Data Quality for Experts 1 SAP BI Staging Use case Describes where Data Services adds value SAP 2008 / Page 28 Data Data Services XI XI 3.0 3.0 Complex Checks Address Address // Data Data Cleansing Cleansing Matching Matching Auditing Auditing

Data Cleansing Cleanses and standardizes party data such as names/addresses, emails, phone numbers, SSNs, and dates Manages international data for over 200 countries and reads and writes Unicode data Removes errors to uncover true content of database Improves integrity of data to identify matches and ultimately create a single customer view Parses and standardizes non-party data Such as account numbers, product codes, product descriptions, purchase dates, part numbers, SKUs, etc. Utilizes a rule-based parsing and rule editing architecture for even greater customized results SAP 2008 / Page 29

Data Cleansing (Person record) Input record Maggie.kline@future_electronics.com Margaret Smith-Kline phd FUTURE Electronics 5/23/03 101 6th ave manhattan ny 10012 001124367 Output record Salutation: Ms. First name: Margaret Last name: Smith-Kline Post name: Ph. D. Match standards: Maggie, Peg, Peggy Gender: Strong Female Company name: Future Electronics Address 1: 101 Avenue of the Americas City: New York State: NY ZIP+4: 10013-1933 Email: maggie.kline@ future_electronics.com SSN: 001-12-4367 Date: May 23, 2003 SAP 2008 / Page 30

Data Cleansing (Product Data) Input Parsed output Description Product Dimension Type Form Kallkyle screw screw Kallkyle test steel plate 20 x 35 mm plate 20x35 mm steel test wire 23.33 x 40.50 cm wire 23.33 x 40.50 cm 34 x 60 mm steel plate steel plate 34,0 60 mm 34.0 x 60,0 mm steel plate 34 x 60 mm steel plate? plate plate 34 x 60 mm steel plate 34 x 60 mm steel plate 34 x 60 mm steel plate 34 X 60 mm steel plate steel plate plate steel SAP 2008 / Page 31

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 32

Matching and Consolidation Unlocking the relationships between distinctly different sets of data House-holding data to identify members of same household, corporation or any other hierarchy Identifying snowbirds i.e. individuals or households with multiple residences Creating a panoramic single best record Preventing firms from doing business with entities on government watch lists, Do-Not-Mail, prison lists, etc Providing identity resolution to uncover non-obvious relationships for fraud detection SAP 2008 / Page 33

Matching and Consolidation Input records Ms Margaret Smith-Kline Ph.D. Future Electronics 101 Avenue of the Americas New York NY 10013-1933 maggie.kline@future_electronics.com May 23, 2003 Maggie Smith Future Electronics Co. LLC 101 6th Ave. Manhattan, NY 10012 maggie.kline@future_electronics.com 001-12-4367 Ms. Peg Kline Future Elect. Co. 101 6th Ave. New York NY 10013 001-12-4367 (222) 922-9922 5/23/03 Consolidated record Name: Ms. Margaret Smith-Kline Ph.D. Company name: Future Electronics Co. LLC SSN: 001-12-4367 Purchase date: 5/23/2003 Address: 101 Avenue of the Americas City: New York, NY 10013-1933 Latitude: 40.722970 Longitude: -74.005035 Fed code: 36061 Phone: (222) 922-9922 Email: maggie.kline@ future_electronics.com SAP 2008 / Page 34

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 35

Data Auditing Audit the quality of the ETL process itself Compare data Before (source) During (transformations) and After the ETL process (target) Set audit points to calculate audit values Count records Calculate Checksums, Sum and Averages for numeric columns Define audit rules based on Boolean Expressions Raise alerts or notification via Email when these rules are violated. SAP 2008 / Page 36

Data Auditing in Data Services Audit rule: Check the number of records at the beginning and End of Data Quality process $Count_SAP_CUSTOMER_DATA = $Count_SAP_CUSTOMERS_CLEANSED). Audit action: Send message to administrator and / or write entry in error log SAP 2008 / Page 37

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 38

Scenario III: SAP BI and Data Services Closed-loop 2 SAP BI Staging SAP BI Open Hub Service Use case Describes where Data Services adds value SAP 2008 / Page 39 Data Data Services XI XI 3.0 3.0 -- Address Cleansing SAP SAP BI BI customer customer address address data data in in BI BI Hand Hand over over SAP SAP BI BI customer customer addresses addresses to to Data Data Services Services using using existing existing OpenHub OpenHub API API Load Load cleansed cleansed data data back back to to SAP SAP BI BI

Agenda 1. Motivation 1.1. Data Quality as an issues 2. Business Objects Data Services in Detail 2.1. Introduction 2.2. Features and Functions 2.3. Which product for which requirement 3. The SAP BI <-> Data Services Use Cases 4. I - Data Quality Made Easy 3.1. Easy to consume Data Service features 3.2. Include Data into SAP NetWeaver BI staging 5. II - Data Quality for Experts 4.1. Address / Data Cleansing 4.2. Matching 4.3. Auditing 6. III Closed Loop Scenario 7. Summary SAP 2008 / Page 40

Key Points to Take Home Integration & Data Quality SAP BI & Data Services Extend the SAP BI capabilities Access to Non-SAP sources enriched with Data Quality in one tool Data Quality all-around Delivered Dictionaries Use country-specific information for address cleansing and matching From simple validation to complex cleansing and matching operations SAP 2008 / Page 41

Further Information SAP Public Web: SAP Developer Network (SDN): www.sdn.sap.com SDN Business Object area: https://www.sdn.sap.com/irj/boc Related Links: Blog https://weblogs.sdn.sap.com/pub/wlg/12040 HowTo Guides (https://www.sdn.sap.com/irj/sdn/howtoguides) How To Use Data Services I - Data Quality Made Easy https://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/b0e611 92-b296-2b10-ca90-a21eea43f569 How To Use Data Services II - Data Quality For Experts https://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/c0f79d 98-b396-2b10-9098-db6b2890d190 SAP 2008 / Page 42

Thank you! SAP 2008 / Page 43

Copyright 2008 SAP AG All rights reserved No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. SAP, R/3, xapps, xapp, SAP NetWeaver, Duet, SAP Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned and associated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence. The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte können Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind. SAP, R/3, xapps, xapp, SAP NetWeaver, Duet, SAP Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwähnte SAP-Produkte und Services sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Ländern weltweit. Alle anderen in diesem Dokument erwähnten Namen von Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen. Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderen Vereinbarung mit SAP. Dieses Dokument enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP -Produkts und ist für SAP nicht bindend, einen bestimmten Geschäftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP übernimmt keine Verantwortung für Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht die Richtigkeit oder Vollständigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewähr, weder ausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zweck sowie für die Gewährleistung der Nichtverletzung geltenden Rechts. SAP übernimmt keine Haftung für Schäden jeglicher Art, einschließlich und ohne Einschränkung für direkte, spezielle, indirekte oder Folgeschäden im Zusammenhang mit der Verwendung dieser Unterlagen. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit. Die gesetzliche Haftung bei Personenschäden oder die Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinks zugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen über Internetseiten Dritter ab. Alle Rechte vorbehalten. SAP 2008 / Page 44