Integrating SAP and non-sap data for comprehensive Business Intelligence

WHITE PAPER Integrating SAP and non-sap data for comprehensive Business Intelligence www.barc.de/en Business Application Research Center

2 Integrating SAP and non-sap data Authors Timm Grosser Senior Analyst BARC Carsten Bange CEO BARC Business Application Research Center BARC GmbH Steinbachtal 2b 97082 Würzburg Germany +(49) 931 880651-0 info@barc.de This research note was conducted and written independently by BARC, an unbiased market analyst. It can be distributed free of charge thanks to sponsorship from Composite Software.

Integrating SAP and non-sap data 3 Leveraging SAP data for Business Intelligence During the recent economic crisis, many companies realized how important it is to quickly deliver information on emerging business and market developments. Business Intelligence not only helps companies get the insight they need on these and other important trends, it also delivers the necessary analysis and forecasting tools to generate key information and manage business performance. Yet, the fast fluctuations in external demand clearly showed that many Business Intelligence systems can no longer meet today s growing requirements for managing performance. In general, one question has gained central importance: How can companies access data from their numerous operational systems and integrate it with internal and external information from different transactional and decision support systems for analysis? In this context, accessing data from SAP sources is a prime focus of interest. SAP is the global market leader for ERP systems and offers various operational and decision support systems. The options for leveraging SAP data, however, vary from system to system. This difference is clearly evident between its operational systems (e.g. ERP, CRM or SCM) and SAP BW, which was designed for decision support. Further, large enterprises often have multiple SAP versions and instances, adding further complexity. As a result, companies need to seriously contemplate how they can integrate their SAP data. both directly from the source systems and consolidated for decision-making and analysis. This study, therefore, will provide an overview of the different possibilities for accessing and integrating SAP data as well as their advantages and disadvantages in light of the growing demands on Business Intelligence in modern organizations. Growing demands on Business Intelligence The requirements for Business Intelligence systems have increased dramatically in recent years. Although each company s needs vary by industry, users, internal usage as well as the level of cooperation with partners, certain requirements generally apply in most cases: Fast query responses: Users often become frustrated when they have to wait too long to receive queries from a Business Intelligence system. This factor alone can result in the failure of a BI project. According to the BI Survey 9, the largest user survey on Business Intelligence, poor query performance has been named one of the top three problems in BI projects in recent years. Analyses across different systems: Companies often store their data in many different systems for specific departments and tasks. Multiple modules (e.g. ERP, CRM or SCM), multiple versions, and multiple instances create even more complexity. This means that they have no comprehensive, consistent view of their business activities. Gaining a single view of data is becoming more and more important, for example, in order to identify opportunities. Integration of up-to-the-minute data: Business Intelligence is moving closer to operational processes. In fact, it already supports them with analytics. In order to create a comprehensive view across the entire enterprise, companies need to incorporate historical, highly aggregated KPIs as well as detailed, operational data in their analyses. This requires a fast, easy integration of the most current data.

4 Integrating SAP and non-sap data Open, easy-to-integrate architecture: Analyzing data from various internal and external sources such as SAP systems and beyond is a key requirement for any BI application. In addition, the architecture should support multiple BI systems used for different tasks Validated pool of data: Poor data quality is a major problem in BI projects. The BI Survey 9 shows that poor data quality and its effects are becoming more and more important in companies. Validated, tested data forms the foundation of any beneficial BI system. Scalability: New studies show that there are many other scenarios for using BI. Due to these new possibilities, BI systems are growing in terms of users, data sources, data volumes as well as the complexity of heterogeneous queries. As a result, BI systems must have a scalable technological architecture. Agility: Companies must be able to implement new business, technical as well as functional requirements in a fast, uncomplicated manner. To ensure that they can deliver important information to decision-makers on all levels of the organization, companies should be able to quickly change existing objects as well as add new data sources, KPIs or dimensions in a timely manner. Access to SAP data varies by system SAP ERP systems process critical business data from operational processes and, therefore, often form the core of a company s IT systems. Accordingly, SAP data is generally an important part if not the sole base of data for reporting, analysis, planning and other Business Intelligence tasks. Nevertheless, accessing and integrating SAP data whether alone or with additional data sources poses a major challenge. In general, there is a difference between accessing operational SAP data directly from the transactional system or consolidated, analytical data from SAP s data warehouse application, SAP BW. Data in operational SAP applications Many companies directly access the data in their operational SAP systems to monitor and report these processes. SAP offers built-in functions for data analysis or reporting within its operational ERP, CRM, PLM, SCM and SRM applications. Alternatively, users can utilize SAP BAPI and other SAP proprietary interfaces to export data or to directly access SAP data for queries in external analytical tools such as Microsoft Excel or a standard Business Intelligence tool. Reporting directly from the operational system, however, is often insufficient. If a company has more advanced requirements, there are clear drawbacks with regards to: Integrating external data Analyzing a combination of data from different modules Building new KPIs or analysis structures (especially hierarchies) Creating a history of data and allocating it into analytical structures Delivering sophisticated options for formatting or visualizing data (e.g. in management dashboards) Supporting BI tasks that go beyond operational reporting such as advanced or flexible data analysis, forecasting and planning

Integrating SAP and non-sap data 5 The BI tools that are available in operational SAP systems are traditionally limited to simple reporting tasks. If companies have requirements that go beyond that, they almost immediately need to use complementary SAP and non-sap BI solutions to analyze, visualize and distribute SAP data in a suitable manner. For BusinessObjects users we expect these possibilities will improve in the mid to long term through the stronger integration of BusinessObjects tools with operational SAP systems. The drawbacks described above arise from the general character of operational systems. For starters, these systems can only process their own data for analysis. Since there is generally no possibility to integrate external data, creating a complete analysis across different operational SAP modules and systems can be very difficult. Many companies, too, have implemented their various ERP installations in many different ways and the data models of these applications for example, SAP ERP and CRM can differ as well. In addition to requiring unique structures and KPIs, Business Intelligence generally needs to access data for longer periods of time as well as transform it into analytical data models. This functionality, however, is not available or highly limited in operational systems. These operational systems limitations led to the development of SAP BW, SAP s data warehousing solution. According to SAP, 10,000 customers a third of SAP s estimated 25,000-30,000 ERP customers use this system. Data in SAP BW SAP Business Warehouse (SAP BW) provides access to information formatted for analysis in the form of a data warehouse that runs complementary to the SAP operational systems. Its specialized access mechanisms for extracting data from operational SAP systems as well as transferring it into data structures that are optimized for analysis generate a consolidated warehouse for business decision making. As a classical data warehouse, SAP BW supplies this information through interfaces for further processing. Yet, accessing data in SAP BW is subject to a few limitations: BI Consumer Services (BI-CS), the most powerful interface for user tools, only works in combination with SAP BusinessObjects. In addition, the write-back interface for data entry and planning applications is generally not open to other applications. OLAP BAPI, MDX, XMLA, the open front-end interfaces of SAP BW, are known to cause various usage problems including functional limitations, performance issues and restrictions in the amount of readable data. If a company wants to use a tool which accesses data that is exported from SAP BW and not from direct queries to that data warehouse, it will have to purchase a data export interface which has a list price of 250,000 a major licensing hurdle. In addition to these limitations with regards to access, other attributes of SAP BW often provoke companies to maintain separate tools and additional data warehouses and marts for Business Intelligence.

6 Integrating SAP and non-sap data Diagram 1: This parallel data warehouse approach uses SAP BW and a separate analytical database which combines and integrates SAP data with information from heterogeneous source systems for analysis. There are many different reasons for this development: There are limited options for integrating data from non-sap systems in SAP BW because connecting standard applications to the SAP system or extracting changed data from external sources requires a significant amount of specialized programming or third-party data integration tools. Typical SAP BW applications generally use data from operational SAP systems and little data from other sources. End users often experience poor query times if they use SAP BW without BW Accelerator. Optimizing query speed requires a significant amount of administrative work and is rather difficult in environments with heterogeneous query profiles. Purchasing BW Accelerator to speed up queries is not an option for many companies due to the high investment costs. There are few possibilities for exchanging metadata between SAP BW and the semantic view of BI user tools. In SAP BW, it is very difficult to enrich historical reports with up-to-the-minute data. Adding data sources or changing existing models is relatively complex in SAP BW. Analysts must have MDX expertise, whereas SQL domain knowledge is more widespread. Some Business Intelligence tools have certain requirements on the database technology. Especially in reporting, planning and data mining, users need to create real-time calculations, manipulate data or simply access information using standard SQL queries. Sometimes, the tools and applications for these tasks only work in combination with certain databases, which then are used in combination with SAP BW. As a result, many companies have a growing number of systems that contain SAP data in various formats and need to create a common strategy to access this data as well as other relevant information to make decisions. The technical solution to support this type of strategy is an integrative layer that enables users to get a complete view of business information from SAP systems and other data sources.

Integrating SAP and non-sap data 7 Solution architectures for integrating SAP and non-sap data for enterprisewide Business Intelligence SAP data is located in SAP BW as well as many different operational systems from SAP. In addition, users often need to access information from other operational systems, other data warehouses and data marts as well as additional internal and external databases for comprehensive Business Intelligence. Increasingly, companies are seeking a unified information architecture that integrates all of these data sources and systems completely and consistently across their enterprise. To do this, companies can generally apply one of three approaches. They can: Physically integrate the data in an all-inclusive, enterprisewide data warehouse, Create an enterprisewide, shared data layer that allows virtual integration without physically moving the data from the source systems, or Use a hybrid combination of physical and virtual that maximizes the benefits and minimizes the limitations of each approach. Enterprise data warehouse via physical data consolidation The principles and business value of an all-inclusive, enterprisewide data warehouse for enabling business decision-making are well understood and illustrated in diagram 1. The advantages of this approach are: Consolidating and reorganizing large volumes of data from heterogeneous systems Building an analytical view by creating multidimensional models of relevant objects for the decision-making process Saving data and analytical structures over a longer period of time (5-10 years) Decoupling analysis from the operational systems Supplying data for many different analytical scenarios with optimal performance Creating complex calculations within the database as well as for data mining applications The disadvantages of this approach are: Creating a second system requires additional resources and software licenses Companies need an Open Hub service license to integrate SAP BW data and build a data warehouse or analyze the data using third-party tools Integrating data from standard applications (e.g. SAP) requires special interfaces and functions The key criteria for evaluating enterprise data warehouse and supporting data integration infrastructure are also well known. Companies can use SAP BW as the all-inclusive enterprise data warehouse. Due to the restrictions described above, however, most companies use only SAP BW as a data mart for executing specific tasks or evaluating SAP data in combination with other data warehouse and reporting solutions.

8 Integrating SAP and non-sap data Enterprise data layer via data virtualization An enterprisewide data layer provides an alternative to the traditional enterprise data warehouse. A virtual layer provides a simple, flexible way to integrate and provide access to data from many different systems. Enterprise-scale data virtualization platforms have emerged from earlier Data Federation or Enterprise Information Integration (EII) tools to provide the functionality required to create such a layer. These offerings include data discovery, modeling and design capabilities to prebuild the integrated views of the data required from across the entire range of source systems. These views can be traditional database views, data services or both as appropriate. When a user s BI solution requests information, the data virtualization server accesses, transforms, federates and presents the data in the defined target structure at the moment a query is made. Diagram 2: In the integration layer data from heterogeneous sources (including SAP systems) is combined and provided for Business Intelligence. The advantages of this approach are: Fast, easy integrations of new or temporary data sources Synchronized access to decision support and operational systems High flexibility for making changes since they don t need to be made to physical structures and data No need for transferring data since it doesn t need to be replicated from the operational or decision support systems (or their operational data store equivalents) Simple integration of up-to-the-minute data and new data sources This approach, however, also has its disadvantages: Despite optimization and caching measures, companies still need to check if performance meets service level requirements.

Integrating SAP and non-sap data 9 Ensuring data quality can be a challenge if complex, multi-step data cleansing is required. As a result, companies must ensure that their data is valid before the integration process takes place. Large analytic queries whose datasets require significant re-dimensioning or time-series summations, etc. are not appropriate. Building an integration infrastructure with data virtualization capabilities is a new approach for many companies. To evaluate vendors and products for this innovative technology we recommend including these criteria: Ease-of-use and productivity in modeling and development Performance & scalability Connectivity to a wide range of data formats, databases and applications, especially certified SAP ERP and BW specific data access functionality Integration with security concepts, e.g. SAP security and implementation of security concepts in the virtual data integration layer. Mechanism to ensure reliable, enterprise-scale operation and maintenance Hybrid Architecture combining enterprise data warehouse and enterprise data layer As a common information architecture that integrates SAP operational, SAP BW, and various non SAP sources for comprehensive Business Intelligence, both the enterprise data warehouse and enterprisewide data layer have advantages and limitations. However, by effectively mixing and matching these approaches, companies can continue to benefit from the advantages, while mitigating many of the disadvantages. Diagram 3: In the integration layer data is provided by combining the virtual and the enterprise data warehouse approach for Business Intelligence.

10 Integrating SAP and non-sap data The mix and match approach obviously means adding a system to the existing architecture that needs upfront investment and resources to run. But the advantages are often compelling in situations where these requirements exist today: High flexibility in implementing new business requirements using an easily adaptable data model Fast integration of new data sources, up-to-the-minute or missed data to extend the data available in the data warehouse Combine operational and analytical master data from master data management, data warehouse and operational systems Create a common view on data stored in disparate data marts or warehouses Adding to the capabilities of existing data warehouses without having to fundamentally change the systems Summary and recommendations The demands on Business Intelligence tools have increased dramatically in recent years. One increasingly important requirement is the creation of complete views of business information by integrating data from different systems across and around an enterprise. In this regard a prime area of interest is data from the operational and decision support systems of SAP. Although SAP systems offer functions and interfaces for Business Intelligence, they have their shortfalls. This results in multiple customized SAP systems as well as a multitude of systems containing SAP data. In order to create a holistic view on relevant information for making decisions, companies need to consolidate data from various systems into unified, enterprisewide integration architecture. Currently, they can choose one of three different architecture approaches: an all-inclusive, enterprisewide data warehouse, an enterprisewide virtual data layer, or a hybrid combination of the two. In all three solution approaches, the open interfaces and support for standards increase the interoperability with other systems and create new potential (e.g. by integrating specialized BI tools) that SAP tools alone could not offer. This way, companies can also secure their SAP investments because they react to changing requirements without having to completely rebuild or replace their current systems. The following aspects are often cited as advantages of physically integrating information in an enterprisewide data warehouse: If query speed is a very important factor, different database technologies including multidimensional and column-based relational databases or innovations such as in-memory data storage provide a broader range of options using physical data integration. A physical integration into an all-inclusive data warehouse is more suitable than a virtual integration when it comes to creating complex calculations and analyses, transferring large volumes of data and mapping histories or deeper dimensions. If a company needs to improve the data quality between its operational systems and queries, they can implement data quality management tools during the transformation process between source systems and the data warehouse.

Integrating SAP and non-sap data 11 The enterprise wide data layer, in contrast, has advantages in other scenarios: By building and deploying virtualized views and data services, companies can meet their growing requirements for agile BI. Enterprises can generally change existing view and services, as well as integrate different data sources on a regular basis more quickly using this approach. Data virtualization allows companies to combine many different types of data sources more flexibly. It also allows for running test scenarios on the fly and applying other analytical methods faster. The virtual approach is particularly useful for integrating up-to-the-minute data because companies can directly access source data without waiting for ETL batch updates. Thus, for companies with a wide range of requirements, a hybrid approach combining the two approaches shows multiple advantages: The hybrid approach allows the computing power of parallel data warehouse systems and analytical databases to be used for complex analytical queries and processing high volumes of data. Using the virtual approach at the same time increases the system flexibility and agility, for example by making it easier to integrate data sources or to implement new business requirements. The hybrid approach offers an infrastructure that makes it easy to combine historical and validated data of a data warehouse with new or up-to-the-minute data. This allows comprehensive BI analysis and meets growing business requirements. The hybrid approach can facilitate the BI roll-out. Even when covering a wide range of business needs preliminary results can be returned very quickly. Fast implementation is a key factor for project success and BI marketing within companies to win over a broader range of business users and sponsors. We therefore recommend that all companies identify their requirements and pain points in the situations described in this paper to evaluate the new possibilities of a virtual data integration infrastructure either as an add-on or an alternative to existing systems for integrating data from SAP systems and other data sources.

12 Integrating SAP and non-sap data BARC stands for neutrality, competence and quality Neutrality The Business Application Research Center (BARC) was founded in 1994 as a spin-off from the University of Wurzburg s Chair for Information Science. BARC is an independent institute and treats all software vendors with the same objectivity. It does not accept fees for coverage in software evaluations or commission for software recommendations. BARC also does not offer software implementation services to prevent internal conflicts of interest. Competencies BARC employees have assessed Business Intelligence and Data Management products and consulted companies in this field since 1994. BARC analysts have vast market, product and implementation insight. Their comprehensive, yet detailed knowledge of the latest market developments and the offerings of all relevant software vendors is backed by years of dedicated market research and hands-on product evaluations. Quality BARC consulting projects are highly efficient and offer the highest level of security for software selections. BARC reports provide competent market overviews for all applied areas of information management. BARC conferences and seminars offer a concentrated overview of key players in a specific segment of the software market.