SAP HANA Using In-Memory Data Fabric Architecture from SAP to Create Your Data Advantage Deep analysis of data is making businesses like yours more competitive every day. We ve all heard the reasons: the ability to make better decisions, get closer to customers, discover new opportunities, create operational excellence, and so on. It s a constant race to improve your business, and the opportunities seem almost limitless to get ahead. So what s keeping businesses from making better use of the oceans of data available? Often, it has to do with the architecture of how you manage your data.
Analytics: A Key to Competitive Advantage WHY THE ENTERPRISE DATA WAREHOUSE NEEDS TO EVOLVE Part of the reason some businesses are slow to utilize their data is that the traditional architecture for storing data is inadequate for today s tasks. When enterprise data warehouses were first envisioned, it was assumed that the primary type of data to be stored would be transactional and not much else the kind of data we typically see on an order form, shipping receipt, or bank statement. But today there is much more data, and more kinds of data, than previously thought. The old-style enterprise data warehouse, which was little more than a second database filled with the data from other application databases, is showing its weaknesses. The process of moving data from many sources into one database, rearranging that data, summarizing it, adding accelerators, and so forth breeds an endless cycle of increasing complexity. Moreover, why did businesses need to put all their data in separate systems in the first place? The answer to how enterprise data warehouses became separated from the source data systems involves the architecture of the data management systems used to store the data. In earlier days, the most readily available data management systems were optimized to speed up transaction processing. Considerable design and data science went into optimizing the use of traditional drives to allow the database managers to process the maximum number of transactions each and every hour of the business day. And these systems were good at it writing hundreds of thousands of records per hour. But for analytics, they were not so good. Analytics is not typically about writing down a single record like John bought a blue T-shirt today. Instead it looks at questions like How many customers bought how many T-shirts in how many stores around the world today? And to answer that type of question, you need to look at a lot of records, often very quickly. However, when these questions were asked of traditional transaction processing data managers, it slowed down the processing of transactions. The solution at the time was to separate the transaction processing system from the data warehouse. Tools like extract, transform, load (ETL) were employed to copy data from transactional systems to the data warehouse, which was usually just another instance of the same data manager used for the transactional systems. And this is where the trouble began. As more sources of data were added from different applications and systems to the data warehouse, more ETLs and more structural changes to the data were required. Data engineers came up with various patches, such as adding more indexes to some of the data and even precalculating some summary data in the attempt to anticipate what questions would be asked. It started to become a losing battle as new questions and ideas and data sources were constantly added. Pretty soon, the systems started to look like Figure 1. 2 / 6
Figure 1: Traditional Data Management Transactions Traditional: Online transaction processing (OLTP) and online analytical processing (OLAP) separate Extract, transform, load (ETL) 48-hour-old data Multiple data sources Aggregate Streams Staging DB Many separate ETL processes 48 hours elapsed Data was separated from sources, processed through many ETLs, reorganized, and aggregated, eventually arriving at a state where it was somewhat usable, for a while. All this took time and delayed access to data. Complexity was creeping in everywhere as separate databases for separate applications and analytics were promoted by vendors who were in the business of selling more and more database licenses. The problems of complexity that this whole process created drove innovators to look for better approaches. REVOLUTIONIZING THE DATA MANAGEMENT SYSTEM FOR DATA WAREHOUSES At SAP, we saw the problems that the old architecture of separate transaction and analytics systems was causing. This architecture was creating too much complexity. It required lots of ETL processes to get the data from one system to the next. It demanded that businesses know many of the questions ahead of time so the data could be summarized in advance. It couldn t respond in real time. And it wasn t ready for 21st-century challenges like Big Data. 3 / 6
So SAP questioned whether the status quo was good enough. Sure, it worked. But could we make it better? The answer from our data scientists and engineers was clearly yes. We could take advantage of all the changes in data processing that have occurred since the 1970s. We were no longer bound to traditional disk drives; we had lots of memory at our disposal. Processors that once could only address a megabyte of data now could address many gigabytes, if not terabytes, of memory. We observed that organizing the stored data exactly as it was received (that is, as records or rows) wasn t always the best way to use the data later. Using more sophisticated storage techniques, we could store the data as columns and dramatically speed up data queries. And what if every column was its own index? No longer would we be forced to guess what question would be asked next. Thus, we centered our new data architecture on an in-memory, columnar system using the SAP HANA database, shown in Figure 2. Figure 2: Data Management Using SAP HANA Transactions Streams OLTP + OLAP in the SAP HANA database Current data SAP HANA Simple data access Multiple data sources available with live access Immediate OLTP = Online transaction processing OLAP = Online analytical processing 4 / 6
Building our data warehouse architecture around an in-memory database has allowed us to do some amazing things. We can virtually eliminate aggregates and precomputations. Because everything is indexed, we don t need to worry so much about creating specialized organizations of the data or schemas to presolve anticipated questions. But most important, we can run business applications on the same database as our analysis. ETL processes are virtually eliminated. Staging databases and intermediate stores are a thing of the past. SAP HANA is so fast we can feed fast-moving streams of data like pointof-sale, radio-frequency identification (RFID), and stock tickers straight to the database itself. But what about other sources of data outside of applications, perhaps stored in legacy systems? What about large amounts of data in archives? Or heaps of new data resting in Hadoop? That s where the idea of creating a data fabric in SAP architecture comes into play. Making use of technologies pioneered by Sybase, now an SAP company, and refined in the demanding environment of Wall Street, SAP HANA employs realtime data integration we call smart data access. With smart data access, SAP HANA can directly integrate data from other databases such as those from Oracle or Teradata in real time. The data doesn t need to be stored in SAP HANA. Smart data access enables the query optimizer in SAP HANA to draw out the data from these sources as needed, automatically. And Hadoop? Getting to your Hadoop data is no problem with SAP HANA. Smart data access technology has built-in support for MapReduce to automatically generate Hadoop jobs on the fly. COMPLETING THE DATA FABRIC SAP HANA is integrated with other key SAP technologies to form a comprehensive data fabric that can solve all manner of data warehousing problems. The SAP Business Warehouse (SAP BW) application provides tools for orchestration and modeling in the data warehouse as well as comprehensive semantic intelligence of SAP applications. SAP BW opens your data warehouse environment to all information so you have a 360-degree view into your business. The tools simplify administration tasks and reduce IT workloads for a lower total cost of ownership. SAP Event Stream Processor provides robust support for streaming data applications. With our award-winning complex event processing platform, you can develop and deploy businesscritical applications that give you the agility you need to make quick, profitable decisions. SAP IQ database software provides an extensive system for managing petabytes of near-line data in a columnar format. In 2014 SAP HANA and SAP IQ achieved the Guinness World Record for largest data warehouse, at 12.1 petabytes (12,100 terabytes) of data. SAP IQ also holds the record for the fastest loading of data warehouse data, at over 34 terabytes an hour.* Together, SAP HANA and SAP IQ can take on your largest data warehousing needs. And because SAP IQ is a live data management system, you never need to put data in offline storage again. * www.guinnessworldrecords.com/world-records/5000 /largest-data-warehouse 5 / 6
LOOKING AHEAD TO SIMPLICITY BY DESIGN Today, some data architects may try clinging to the old, complex ways. In fact, to augment the traditional approach of managing data warehouses, some in the industry are now trying to accelerate their 1970s-based technologies by adding another in-memory database, subdatabase, or cache to their existing disk-oriented systems. But this only adds to the complexity. Simplicity and its attendant speed are not achieved by adding more and more to existing systems. Simplicity happens by design. It happens when a company boldly makes architectural decisions that reduce the number of parts to be managed and then accelerates and integrates the parts that remain into a uniform system that works as a whole. That s why SAP designed its inmemory data fabric architecture: to simplify your data management and maximize your business success with a competitive edge. TO LEARN MORE To find out more about how in-memory data fabric architecture can help your company perform at its best by simplifying data management, contact your SAP representative or visit us online at www.sap.com/imdf. Studio SAP 32207enUS (14/07) 6 / 6
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. Please see http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark for additional trademark information and notices. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE s or its affiliated companies strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.