Building a multi-source universe

September 10-13, 2012 Orlando, Florida Building a multi-source universe Pierpaolo Vezzosi 2011 SAP AG. All rights reserved. Public 1

Learning points Learn the details of the multi-source universe technology See how to build a simple multi-source universe Discover how to optimize a multi-source universe 2011 SAP AG. All rights reserved. Public 2

Agenda Definitions and architecture What is the multi-source universe (MSU) technology Architecture Differences with Data Federator XI3.0 Building a universe Connections for MSU Data foundations for MSU Business Layers for MSU Optimizing a multi-source universe Optimization algorithms The data federation administration tool 2011 SAP AG. All rights reserved. Public 3

Definitions and architecture

The multi-source universe value proposition The new BI4.0 multi-source universe (MSU) technology provides the ability to create a universe federating data across multiple data sources. Built on the great technology of Data Federator XI3.0, it was designed for No additional license: the functionality is included in the BI platform licenses No additional servers: the technology is installed with the BI plaftorm and administered along with other servers in the CMC No additional desing tools: Information design tool replaces the workflow of Data Federator designer. Multisource is just a radio button away 2011 SAP AG. All rights reserved. Public 5

MSU architecture DB DB DB 2011 SAP AG. All rights reserved. Public 6

MSU architecture DB DB DB Information design tool WebIntelligence Crystal Reports Enterprise Dashboards Explorer Query server 2011 SAP AG. All rights reserved. Public 7

MSU architecture DB CMS session DB DB 2011 SAP AG. All rights reserved. Public 8

Architecture DB CMS session DB DB Information design tool Query server WebIntelligence Crystal Reports Enterprise Dashboards Explorer 2011 SAP AG. All rights reserved. Public 9

Differences with Data Federator XI3.0

Data Federator XI3.0 deployment Universe SAP BOBJ tools DB DB DB Other data integration needs 2011 SAP AG. All rights reserved. Public 11

BI 4.0 Multi source universe deployment BI 4.0 platform UNX Universe SAP BOBJ tools BI 4.0 DB DB DB MSU= multi-source Universe 2011 SAP AG. All rights reserved. 12 Public

Available deployments with BI4.0 and DF XI3.0 Data Federator XI 3.0 is still shipped for existing customers and specific use cases not yet covered by MSU SAP BOBJ BI 4.0 platform tools BI 4.0 UNX Universe UNV Universe Direct SQL access Other data integration needs MSU= multi-source Universe 2011 SAP AG. All rights reserved. 13 Public

Data Federator XI3.0 and MSU BI 4.0 differences Technical details DF 3.0 Separate license Yes No Separate, standalone install Yes No Access with SQL from external applications Yes No Connect XI3.x universes (UNV) Yes No Use web services or to XML files as data sources Yes No MSU Use complex mapping rules Yes Limited* Use native database syntax No Yes** Create BI 4.0 universes (UNX) No Yes Design workflow in the information design tool No Yes * Mapping rules, data cleaning and lookup mechanisms in MSU are limited to SQL statements which can be inserted in derived tables or calculated columns ** Native database syntax can be used in derived tables defined over a single source in an MSU universe Migration constraints: In BI4.0 it is not possible to migrate an existing DF XI3.0 project to the new technology It is not possible to build UNX universes on top of a DF XI3.0 project 2011 SAP AG. All rights reserved. Public 14

Building a universe

Create a connection for a multisource universe LIVE DEMO OR VIDEO 2011 SAP AG. All rights reserved. Public 16

Create a data foundation for a multisource universe LIVE DEMO OR VIDEO 2011 SAP AG. All rights reserved. Public 17

Create a business layer for a multisource universe LIVE DEMO OR VIDEO 2011 SAP AG. All rights reserved. Public 18

The data federation administration tool

The data federation administration tool (DFAT) DFAT is installed in the BI4.0 client toolset. You can use it to: Find bottlenecks Which database server is running slow? Is there something I can do at the database server level to improve performance? Which query is taking too much time to execute? Why? Is there something I can do to improve it? 2011 SAP AG. All rights reserved. Public 20

The data federation administration tool (DFAT) DFAT is installed in the BI4.0 client toolset. You can use it to: Find bottlenecks Understand the query plan Can I push more computation down to the database engine? Where is the query executed? What database statistics will improve the query plan definition? 2011 SAP AG. All rights reserved. Public 21

The data federation administration tool (DFAT) DFAT is installed in the BI4.0 client toolset. You can use it to: Find bottlenecks Understand the query plan Fine tune the query server settings Can I optimize the memory of my machine for MSU? 2011 SAP AG. All rights reserved. Public 22

The data federation administration tool (DFAT) DFAT is installed in the BI4.0 client toolset. You can use it to: Find bottlenecks Understand the query plan Fine tune the query server settings Perform trial and error tests What happens to the query execution if I change my SQL or some parameters? 2011 SAP AG. All rights reserved. Public 23

Fine tuning MSU with the Data federation administration tool LIVE DEMO OR VIDEO 2011 SAP AG. All rights reserved. Public 24

Optimizing a universe

The MSU optimization algorithms The MSU technology provides multiple built-in optimization algorithms Two goals Push execution to databases as much as possible Reduce data transfer as much as possible 2011 SAP AG. All rights reserved. Public 26

Techniques to optimize MSU execution Gather database statistics Fine tune connectors and the query server Define an optimized data foundation 2011 SAP AG. All rights reserved. Public 27

Techniques to optimize MSU execution Gather database statistics Fine tune connectors and the query server Define an optimized data foundation 2011 SAP AG. All rights reserved. Public 28

Gathering statistics to let the optimization algorithm understand how to work Statistics can enable semi join optimizations LIVE DEMO OR VIDEO 2011 SAP AG. All rights reserved. Public 29

Techniques to optimize MSU execution Gather database statistics Fine tune connectors and the query server Define an optimized data foundation 2011 SAP AG. All rights reserved. Public 30

DFAT parameter example MAX_CONCURRENT_MEMORY_CONSUMING_QUERIES Defines the number of queries that consume memory that can run concurrently If you have many large queries: Set a low value memory will be dedicated to few queries; no disk memory swapping will be necessary If you have many small queries: Set a high value more queries will be able to execute in fast memory 2011 SAP AG. All rights reserved. Public 31

Techniques to optimize MSU execution Gather database statistics Fine tune connectors and the query server Define an optimized data foundation 2011 SAP AG. All rights reserved. Public 32

Optimizing the data foundation to set up intelligent connections Intelligent connections use a table field to chose on which database the query has to be run. It improves performance as only the needed database are queried. 1. Define a derived table retrieving data from all the necessary sources 2. Add a field with a fixed, discriminating value by source 3. When querying, use the discriminating field as a filter Multi-source universe US JP 2011 SAP AG. All rights reserved. Public 33

Setting up intelligent queries Example Define a derived table as: Select *, US as country from Source1.US_Customers Union Select *, JP as country from Source2.JP_Customers JP JP US US US Kaori Nobuko Tim Jack Mary Then define an object on the Country field and use it as a filter. When filtering for US, only source1 will be queried, source2 will be skipped. If no filter is applied, both sources will be used. 2011 SAP AG. All rights reserved. Public 34

Where to find more information In the data federation administration guide: 20+ pages of optimization techniques 20+ pages explaining how to set server and connector parameters 2011 SAP AG. All rights reserved. Public 35

Learning points In this session we have: Learned the details of the multi-source universe technology Seen how to build a simple multi-source universe Discovered how to optimize a multi-source universe Now it is time for QUESTIONS 2011 SAP AG. All rights reserved. Public 36

Thank You! 0610 Building a multisource universe Contact information: Pierpaolo Vezzosi pierpaolo.vezzosi@sap.com

Thank you for participating. Please provide feedback on this session by completing a short survey via the event mobile application. SESSION CODE: 0610 Learn more year-round at www.asug.com

2011 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iseries, pseries, xseries, zseries, eserver, z/vm, z/os, i5/os, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company. Sybase and Adaptive Server, ianywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase, Inc. Sybase is an SAP company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG. 2011 SAP AG. All rights reserved. Public 39

2011 SAP AG. Alle Rechte vorbehalten. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Die von SAP AG oder deren Vertriebsfirmen angebotenen Softwareprodukte können Softwarekomponenten auch anderer Softwarehersteller enthalten. Microsoft, Windows, Excel, Outlook, und PowerPoint sind eingetragene Marken der Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iseries, pseries, xseries, zseries, eserver, z/vm, z/os, i5/os, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli und Informix sind Marken oder eingetragene Marken der IBM Corporation. Linux ist eine eingetragene Marke von Linus Torvalds in den USA und anderen Ländern. Adobe, das Adobe-Logo, Acrobat, PostScript und Reader sind Marken oder eingetragene Marken von Adobe Systems Incorporated in den USA und/oder anderen Ländern. Oracle und Java sind eingetragene Marken von Oracle und/oder ihrer Tochtergesellschaften. UNIX, X/Open, OSF/1 und Motif sind eingetragene Marken der Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame und MultiWin sind Marken oder eingetragene Marken von Citrix Systems, Inc. HTML, XML, XHTML und W3C sind Marken oder eingetragene Marken des W3C, World Wide Web Consortium, Massachusetts Institute of Technology. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork und weitere im Text erwähnte SAP-Produkte und - Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und anderen Ländern. Business Objects und das Business-Objects-Logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius und andere im Text erwähnte Business-Objects-Produkte und Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der Business Objects Software Ltd. Business Objects ist ein Unternehmen der SAP AG. Sybase und Adaptive Server, ianywhere, Sybase 365, SQL Anywhere und weitere im Text erwähnte Sybase-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der Sybase Inc. Sybase ist ein Unternehmen der SAP AG. Alle anderen Namen von Produkten und Dienstleistungen sind Marken der jeweiligen Firmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen. Die in dieser Publikation enthaltene Information ist Eigentum der SAP. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, nur mit ausdrücklicher schriftlicher Genehmigung durch SAP AG gestattet. 2011 SAP AG. All rights reserved. Public 40

The Grid Headline area White space Drawing area 2011 SAP AG. All rights reserved. Public 41