Breaking News! Big Data is Solved. What Is In-Memory Computing and What Does It Mean to U.S. Leaders? EXECUTIVE WHITE PAPER

Transcription

1 Breaking News! Big Data is Solved. What Is In-Memory Computing and What Does It Mean to U.S. Leaders? EXECUTIVE WHITE PAPER

2 There is a revolution happening in information technology, and it s not just the newest app for your smartphone. There is a sea change occurring in how enterprises can manage their data, breaking a technology paradigm that has been in place since the midpoint of the last century. The potential benefits include radically better computing performance at less cost in US national security missions. Without much risk of contradiction, it can be said the requirements to ingest and store the various types of Big Data have been addressed by both demand-driven, commercial-off-the-shelf (COTS) software manufacturers and by the voluntary software development efforts of participants in the free and open source (FOSS) consortia. However, business and government leaders may not be satisfied that their investments in Big Data solutions have produced meaningful improvements in their mission and business processes. The sticking point has been that the visualization, analysis, and exploitation of Big Data pools to date have been largely batch-oriented, not interactive; awkward, not agile; and reliant on having very highly skilled personnel. In this paper we discuss a new paradigm, In-Memory Computing, and how it can help you fully exploit the value of Big Data and leverage the investment already made in your data infrastructure. To understand how this new paradigm can deliver value to your organization, let s put our nerd on and review how we reached this revolution in data management. A WALK DOWN MEMORY LANE Since 1944, mainstream computer design has been based on the architecture created by pioneering computer scientist John Von Neumann. His architecture is based on a control unit taking chunks of data from external memory into main memory, where it is operated upon by the logic processor. The control unit takes in the data that is needed for the program to operate in manageable chunks; the logic processor performs its operations on each chunk of data; and then the system fetches the next chunk of data for processing; and so on, over and over again, until the processing is completed. The essence of the Von Neumann architecture was to bring the data to the logic, given the limitations that existed at the time in logic processor speed and main memory capacity. Figure 1: The Von Neumann Architecture SAP National Security Services TM SAP NS2 TM 2

3 A few years ago, the Hadoop Distributed File System (HDFS or simply Hadoop ) was developed at Yahoo, with inspiration from a paper published by Google. 1 Hadoop has a fundamental difference in its design. It splits up very large files of data into small chunks spread across hundreds or thousands of computers; then it splits up and sends the business logic to be processed to each of the CPUs and memories on all of those computers. This is what makes Hadoop work well on very large Big Data files; the processing logic that the data scientist writes (called a MapReduce program) is run simultaneously on all of the small chunks of the big file. So a fundamental paradigm shift occurs when Hadoop is used to process data. Instead of bringing the data to the logic processor in small chunks, Hadoop distributes and brings the logic to the data. Figure 2: The Hadoop Distributed File System Architecture SOFTWARE BUILT TO LEVERAGE THE NEW CAPABILITIES OF PROCESSORS AND MEMORY Most recently, SAP has pioneered a new approach to computing that is another fundamental paradigm shift. It is called the High Performance Analytical Appliance, or SAP HANA TM, and it leverages recent advancements in both logic processors and memory chips to deliver a different architecture from the Von Neumann machine, and from Hadoop. 2 Co-innovated with Intel Corporation 3, the architecture of SAP HANA makes it possible to radically accelerate Big Data analyses in the enterprise, including data stored in the Hadoop Distributed File System. 1 Doug Cutting and others at Yahoo wrote Hadoop during , based on Google s Big Table architecture. See F. Chang, J. Dean, et. al, Bigtable: A Distributed Storage System for Structured Data, Proceedings of the 7th Symposium on Operating Systems Design and Implementation, SAP HANA-certified servers based on the Intel chipset are available from Cisco, Dell, Fujitsu, Hitachi, HP, and IBM. 3 Co-innovation with Intel allows the SAP in-memory solution to leverage HANA-specific instruction sets in the Xeon Ivory Bridge and Hayworth chipsets. See the Intel white paper, Analyzing Business as it Happens, Version 1.0, April 2011, available at SAP National Security Services TM SAP NS2 TM 3

4 In the SAP HANA architecture, all of the data that will be subject to logical processing is contained in main memory at all times, even very large amounts of data. External memory -- either in solid state or physical disc drives -- is no longer necessary, except for back-up and disaster recovery purposes. There is no chunking of the data or sequential delivery to the processor as in the Von Neumann architecture. Nor is there any splitting up and distribution of the logic algorithms as in the Hadoop architecture. With SAP HANA, the logic and all of the data reside and work together in main memory. Figure 3: SAP HANA Architecture The practical result of this is a remarkable increase in analytical processing speed, even on very large data sets. Therefore, SAP HANA can meet users expectations of very fast response times for transactions, analyses, and visualizations, even with Big Data sets. Freed from the limitations of the older generation of hardware (including CPUs, memory and discs), SAP HANA does not require the extensive administration and constant tuning that database administrators have had to do with traditional database management systems which were first written in the 1970s. SAP HANA can be deployed as a stand-alone in-memory database accommodating 2 to 8 terabytes of memory per server node. Alternatively, SAP HANA can be combined with Hadoop s on-disc data storage in a hybrid deployment in which the hot, frequently used data is stored in the SAP HANA in-memory columnar database, while the warm, less frequently used data is stored in a Hadoop system. In short, SAP HANA delivers speed-of-thought analysis, whether data is sourced from traditional row-based data warehouses, transactional business applications, streaming sensor data, or from the Hadoop system. SAP calls this architecture the Real-Time Data Platform. SAP National Security Services TM SAP NS2 TM 4

5 REAL-WORLD BENEFITS The real-world mission and business benefits that leaders can expect from employing the SAP Real-Time Data Platform include: Accelerated analyses of complex queries against granular (not summary level) Big Data. HANA s near-real-time analyses can help support your decisions, improve planning, and optimize mission execution. A single in-memory data repository can be used both for transactional applications and for analyses simultaneously. This cuts the Operation and Maintenance (O&M) costs of having to maintain a separate transactional database for mission applications; another data warehouse optimized for reporting and analysis; and multiple other replications into data marts for specific users mission needs. Integrating highly compressed in-memory data storage in SAP HANA with bulk on-disc data storage in Hadoop allows organizations to optimize the use of their infrastructure resources and save money. Integrating data from other systems inside or outside the enterprise multiplies the value of data sets by unlocking previously undiscoverable analytical insights and making them easier to access in a self-service way. The SAP Real-Time Data Platform allows for a reduction in specialized skills needed to operate and maintain the overall system, because data integration with other systems in the enterprise uses standard interfaces which are commonly available skill sets (such as SQL, MDX, XML, ODATA, JSON, and JDBC). Now engineers at SAP National Security Services (SAP NS2) are working with partners and customers to develop specific applications that exploit this game-changing technology to advance the missions of the US Intelligence Community, Department of Defense, and homeland security customers. We would welcome the opportunity to discuss how SAP NS2 can help your organization advance your mission with the SAP HANA Real-Time Data Platform. SAP National Security Services TM SAP NS2 TM 5

6 FOR MORE INFORMATION Contact your account manager or call us at SAPNS2 ( ) Website: AUTHOR Bob Palmer Senior Director, Solutions About SAP National Security Services (SAP NS2 ) SAP National Security Services (SAP NS2) offers a full suite of enterprise applications, analytics, database, and mobility software solutions from SAP with specialized levels of security and support to meet the unique mission requirements of US national security and critical infrastructure customers. SAP National Security Services and SAP NS2 are trademarks owned by SAP Government Support and Services (SAP GSS). For more information, please visit Copyright 2013 by SAP Government Support and Services. All rights reserved. May not be copied or redistributed without permission