Harness Insight from Hadoop with MapReduce and Text Data Processing Using SAP Data Services and SAP HANA

Transcription

1 SAP Co-Innovation Lab Harness Insight from Hadoop with MapReduce and Text Data Processing Using SAP Data Services and SAP HANA

2 Table of Contents 3 Abstract 4 Hadoop, MapReduce, NoSQL, and the Enterprise 5 Solution Architecture The Project at SAP Co-Innovation Lab The Deployment at SAP Co-Innovation Lab The Scenario 8 The Role of SAP Data Services in Processing Text Data in Hadoop Using Text Data Processing to Process Customer Reviews 13 Summary Learn More Acknowledgements This document is the work of a virtual project team at SAP Co-Innovation Lab. Team members include Vish Agashe (SAP), Christopher Y. Chung* (IBM), David Cruickshank* (SAP), Jim Dilley (IBM), Sougata Dutta (SAP), Marie Goodell* (SAP), Kevin Liu* (SAP), Justin Martinson (SAP), Ralph C. Nissler (IBM), Yuvaraj Athur Raghuvir* (SAP), Awez Syed (SAP), Anthony Waite (SAP), Sue Waite (SAP), Kevin Wright* (SAP), and many other colleagues from SAP and IBM who helped with this project. *Contributing authors

3 Extracting Insight from Large Volumes of Structured and Unstructured Data Abstract Almost every company today is faced with managing vast amounts of data from a variety of data sources. These range from traditional databases that manage transactional data generated by business applications, to enterprise data warehouses that facilitate deep analysis, to the Apache Hadoop Distributed File System that stores large volumes of mostly unstructured data that can be mined for new insights. There is value to be found in this data. The challenge for IT organizations is how to access and integrate relevant data for deeper insight. Today, IT organizations are taking advantage of new technologies such as Hadoop, MapReduce, text data processing, and NoSQL (not only Structured Query Language) operations to mine large volumes of data located on low-cost storage. By extracting and loading relevant data from these large data sets into an in-memory database, organizations can combine unstructured and structured data for insights never seen before. In this paper, we explore how IT can enable business users to leverage meaningful information in real time by using solutions from SAP and our partners to: Harness the value of large volumes of data stored in Hadoop systems Identify salient entities from unstructured textual data Combine unstructured and structured data in the SAP HANA database New technologies enable companies to mine large volumes of data. By extracting and loading relevant data from large data sets into an in-memory database, organizations can combine unstructured and structured data for insights never seen before. Harness Insight from Hadoop with MapReduce and Text Data Processing 3

4 Identifying the Solution Components Hadoop, MapReduce, NoSQL, and the Enterprise Apache Hadoop is an open-source software framework for distributed data processing. It enables organizations to process large volumes of data from terabytes to petabytes on low-cost, commodity hardware. Data is stored in a Hadoop Distributed File System (HDFS) that can scale across a cluster of servers. MapReduce is a programming model that enables developers to write applications that rapidly process vast amounts of data in parallel. It splits up a problem, sends subproblems to different servers, and lets each server solve its own subproblem in parallel. Then it merges all the subresponses to form the output the answer to the original problem it was trying to resolve. While this process can often appear inefficient compared to algorithms that are more sequential, MapReduce can process queries as background batch jobs against a server farm and sort through petabytes of data in only a few hours. Another benefit is that the data can remain in its original location, which eliminates the cost and time associated with data movement. NoSQL databases such as Apache HBase or Apache Hive that support queries on HDFS data enable organizations to execute very large volumes of reads and simple updates against very large data sets. NoSQL databases are designed for processing and analyzing large lists of elements from millions of online users, such as Twitter posts. Hadoop, MapReduce, and NoSQL can all offer value to an enterprise on their own. But organizations can gain competitive advantage by uncovering useful nuggets of information from a Hadoop or NoSQL database, extracting relevant content, ensuring its quality, and then loading it into an in-memory database to combine it with structured data for unprecedented insight. The challenge of data integration software, therefore, is to generate value not only in moving data passively from or to the Hadoop system but to also enable intelligent data processing by extending and utilizing the MapReduce programming framework. Using data stores to handle data transformation even before the extraction process begins is a well-known approach to data integration when sources conform to the regular SQL world. Using MapReduce to act as a vehicle for the visitor pattern opens up the Hadoop and NoSQL data sources by distributing known processing techniques across the distributed file system. SAP Data Services software leverages this essential principle to bring NoSQL value to an enterprise. The software combines data integration, text data processing, and data quality management in a single solution. With SAP Data Services, organizations can extract, transform, and load data from almost any source to any target. In addition, it delivers behind-the-scenes extensions to push down operations into HDFS directly or through Hive, where operations can leverage the power of MapReduce for parallel processing to increase performance. This enables business analysts to gain insight from Hadoop without having to write code. Using an intuitive design tool, a data architect or business analyst can create models that represent the target and source systems and the desired processing. Queries submitted by SAP Data Services through the Hive Query Language (HQL) can support simple joins, order data, filter data, and apply functions (such as average, minimum, maximum, and so on). When SAP Data Services generates Apache Pig scripts against files in HDFS, it can push down text data processing for semantic extraction based on linguistic markup through distributed MapReduce processing and then apply pattern matching to relate entities. Once semantic entities and patterns are extracted, SAP Data Services can rapidly load relevant data into a highperformance database, such as SAP HANA, to combine unstructured and structured data for contextual analysis in real time. Deep integration between SAP HANA and the SAP BusinessObjects Business Intelligence suite enables organizations to deliver results quickly to business users for competitive advantage. Through this integration, SAP offers quick time to value on the information that is mined out of large unstructured data. The goal is to extend the current practice in enterprises and embrace Hadoop, MapReduce, and NoSQL approaches to augment business decision making models in a streamlined way that offers rapid time to value. The solution would be incomplete without mention of the hardware that powers these systems and the Hadoop distribution that enables enterprise adoption. In this paper, we explore the solution architecture developed at SAP Co-Innovation Lab in close collaboration with IBM as the hardware provider. Let us now look into the components of the solution architecture in more detail. 4 Harness Insight from Hadoop with MapReduce and Text Data Processing

5 Building a Big Data Solution at SAP Co-Innovation Lab Solution Architecture In this paper, we explore the comprehensive solution developed at SAP Co-Innovation Lab for mining large volumes of unstructured data from social media and structured data from enterprise applications for real-time analysis. The solution is based on software functionality from SAP and its partners. The Project at SAP Co-Innovation Lab As part of the SAP Next Business and Technology organization, SAP Co-Innovation Lab is one of the newest communities in the SAP ecosystem. It enables like-minded companies to forge new ideas and new solutions by harnessing the collective power of SAP experts, partners, and customers. One of the unique features of SAP Co-Innovation Lab is a simulated but full-featured data center built using hardware and software contributed by SAP and lab sponsors. Participants at the lab have ready access to all the SAP applications, servers, databases, network technology, tools, and storage devices they need to create sophisticated IT landscapes. In this project, we worked closely with IBM Corporation. The Deployment at SAP Co-Innovation Lab Figure 1 illustrates the key components of the solution architecture deployed at SAP Co-Innovation Lab. The project team developed this architecture to support a scenario that demonstrates how the solution smoothly merges unstructured data from social media and structured data from enterprise applications. Figure 1: Solution Architecture SAP BusinessObjects BI platform JDBC SAP HANA ODBC SAP ERP RFC SAP Data Services ODBC Apache Hadoop Hive Pig MapReduce Hadoop Distributed File System JDBC = Java Database Connectivity; ODBC = Open Database Connectivity; RFC = Remote function call Harness Insight from Hadoop with MapReduce and Text Data Processing 5

6 SAP Solutions The SAP solutions used in this project include the following offerings: The SAP ERP application enables organizations to streamline and automate business processes that support financial operations, human capital management, sales and services, procurement and logistics, product development and manufacturing, and corporate services. SAP HANA is an in-memory database that enables organizations to analyze operational, analytical, and text data in real time. It delivers a fundamentally new approach to data processing that allows organizations to supercharge core business processes or custom applications with deep business insight delivered with near-zero latency. SAP Data Services is a single solution for data integration, text data processing, and data quality management enabling organizations to gain a complete and accurate view of enterprise information. The SAP BusinessObjects Business Intelligence (BI) platform provides a flexible foundation for delivering BI tools across the enterprise. In this project, SAP BusinessObjects Web Intelligence software was used to deliver the BI results. IBM Hardware The SAP HANA database in this project is running on a powerful IBM ex5 enterprise server with the Intel Xeon E7 processor, combining the speed and efficiency of in-memory processing with the ability to analyze massive amounts of enterprise data. Specifically, SAP HANA uses an IBM System x3690 X5 2U rack-mount server a powerful, two-socket server with 10 cores per socket and 256 GB RAM. With an Intel Xeon CPU E running at 2.40 GHz, it is one of two IBM Intel-based high-end servers that represent a workload-optimized solution for SAP HANA and are certified by SAP. The IBM System x3690 X5 was preconfigured and preinstalled on key software components. It runs the SUSE Linux Enterprise Server (SLES) operating system and IBM General Parallel File System (GPFS) to help accelerate the delivery and deployment of the solution. By using solutions from SAP and our partners, you can harness the value of Big Data stored in Hadoop, identify salient entities from unstructured textual data, and combine unstructured and structured data in the SAP HANA database. 6 Harness Insight from Hadoop with MapReduce and Text Data Processing

7 The x3690 X5 features IBM exflash internal storage using solid-state drives to maximize the number of input and output (I/O) operations per second (IOPS). All configurations for SAP HANA based on x3690 X5 use exflash internal storage for high IOPS log storage or for both data and log storage. IBM also provided a Hadoop cluster consisting of six IBM System X x3620 X3 with Intel Xeon CPU x5620 running at 2.40GHz a two-way server with between four and six cores per socket. The Scenario The scenario illustrated in Figure 2 shows how product data from SAP ERP and relevant unstructured data (for example, sentiment analysis about those products) from Web logs stored in Hadoop were loaded into SAP HANA for deeper analysis. Consider the hypothetical case of a large and successful retailer that does business in stores and on the Web. This business needs real-time insights that increase its ability to win customers by ensuring that it offers attractive products at attractive prices. To deliver innovation that provides a competitive advantage, the IT team must be able to leverage all available data. To drive excellence, the team s solution must also help ensure that the company s products are selling at their full potential, day in and day out. This retailer is also interested in optimizing its marketing campaigns by using social media platforms to capture expressed sentiments. To meet such needs, the solution stored the review comments from e-commerce Web sites in Hadoop. Using the text data processing feature of SAP Data Services, relevant customer sentiment (expressed in the review comments) was extracted. The relevant comments were then associated with the product data, thereby providing marketing professionals with insight into the optimal messaging and positioning for the product. The process of extracting valuable data from the reviews was done in the following steps: 1. Extract value from review comments stored in Hadoop Submit a request to Hadoop using SAP Data Services Process the text to extract sentiment from the review comments and associate relevant sentiment to the product list Load relevant sentiment into SAP HANA 2. Replicate business content (product entities in SAP ERP) in SAP HANA 3. Contextualize the information and present it to the business user Join the information extracted from the review comments with the master data in SAP HANA Create views to visualize this information The following sections describe how we used SAP Data Services to gain insight from large volumes of data stored in Hadoop, extract meaningful data, and load it into SAP HANA for deeper analysis. Figure 2: Flow of Structured and Unstructured Data for Analysis SAP ERP SAP Data Services Hadoop SAP BusinessObjects BI platform SAP HANA Harness Insight from Hadoop with MapReduce and Text Data Processing 7

8 Analyzing Big Data for Real-Time Insight The Role of SAP Data Services in Processing Text Data in Hadoop SAP Data Services provides intuitive user interface tools for both the data architect and business user to identify, transform, extract, and load meaningful information from Hadoop into a high-performance database like SAP HANA for contextual analysis of unstructured and structured data in real time. Not only does it facilitate the movement of data, it also supports powerful data transformations, such as text data processing, data cleansing, matching and deduplication, and data enrichment. One of the main benefits of SAP Data Services is that it allows you to connect source and target systems without the need to fully understand them. Using the designer feature of SAP Data Services, you can design data flows by graphically selecting source and target nodes and connecting them together. SAP Data Services then optimizes the data flow by pushing down as much work as possible to the source or target systems. In this manner, the power of the underlying system is leveraged without you having to directly interact with it. SAP Data Services has multiple ways it can interact with NoSQL databases and Hadoop: Hive support To query data in Hive tables, SAP Data Services generates HQL scripts. Hive converts the query into a MapReduce job, and the resulting data files are generated on the HDFS system. SAP Data Services then loads the results into SAP HANA. HDFS and GPFS direct access To directly access files within a distributed file system, SAP Data Services leverages the libhdfs application programming interface and reads those files into memory for processing. HDFS and GPFS access via Pig To perform standard query operations or text data processing, SAP Data Services generates Pig scripts. The Pig scripts are converted into MapReduce jobs to access and process files natively within the Hadoop cluster. HDFS generates the resulting data files. SAP Data Services then loads the results into SAP HANA for deeper analysis. Figure 3: Data Flow from Hadoop File System to Database Table student2.txt (st...) student.txt (stu...) Join f_read_out(repo.k...) The data flow depicted in Figure 3 represents a join between two delimited files sitting in the Hadoop file system with the results being output to a standard database table. During the optimization process, SAP Data Services determines that both source files are from the same Hadoop file system and generates a Pig script to join the two files together. Once the Pig script completes, the results are read in using the HDFS direct file reader and placed into the database table. Here is the Pig script that gets generated: STUDENT = LOAD student.txt USING PigStorage( ) AS (STUDENT::FIRSTNAME:chararray, STUDENT::ID:int, STUDENT::GPA:double); STUDENT2 = LOAD student2.txt USING PigStorage( ) AS (STUDENT2::LASTNAME:chararray, STUDENT2::ID:int, STUDENT2::YEAR:chararray); O1 = JOIN STUDENT BY ( STUDENT::ID ), STUDENT2 BY( STUDENT2::ID ); 8 Harness Insight from Hadoop with MapReduce and Text Data Processing

9 Using Text Data Processing to Process Customer Reviews Based on the process steps identified in the previous section of this document, let us now look into the details of each step. Phase 1: Extract Value from Review Comments The first step is to generate a custom dictionary to use with the Text Data Processing Entity Extraction transform. The text processing tools in SAP Data Services allow users to create custom dictionaries consisting of sets of terms or simple patterns, along with a canonical form for the set. For instance, if you are interested in finding companies that you do business with, you might generate a list of variants of the company name, such as I.B.M., IBM, IBM Corp, International Business Machines, and so on. The canonical form would be specified as International Business Machines Corporation. In the case of customer reviews, a dictionary of product names is useful. In this case, SAP ERP contains the product information. Figure 4 depicts the data flow that creates the custom dictionary. Figure 4: Data Flow to Create Custom Dictionary SelectDistinct- Typ... MARA(SAP _ERP_R63_... SelectDistinct- Typ... FinishedGoods MapToXsd TdpXsd- Dictionary Row_ Generation Harness Insight from Hadoop with MapReduce and Text Data Processing 9

10 Once the dictionary is ready, you can now perform text data processing on the customer reviews. This is done by modeling a data flow that attaches the Hadoop data source to the Text Data Processing Entity Extraction transform and passing the output to a table in SAP HANA. Figure 5 illustrates this data flow. The Text Data Processing Entity Extraction transform has basic entity extraction functionalities such as persons, companies, places, and so on. You can also specify custom dictionaries and additional rules. In this case, the initial text data processing dictionary is specified, along with the voice of the customer sentiment analysis rules. You can see them highlighted in Figure 6. Figure 5: Data Flow to Perform Text Data Processing *.* - (Hadoop) Base_ EntityExtrac... HADOOP_DEMO _OUT_2... Figure 6: Selection of Dictionary and Rules for Text Data Processing 10 Harness Insight from Hadoop with MapReduce and Text Data Processing

11 When the data flow is executed, SAP Data Services detects the Hadoop data source and pushes the Text Data Processing Entity Extraction transform down to Hadoop as a MapReduce job. The process that takes place is as follows: SAP Data Services generates a Pig script that is submitted to Hadoop. The Pig script initiates a MapReduce job for the Text Data Processing Entity Extraction transform. The Text Data Processing Entity Extraction transform requires a number of language modules, rule files, and such. It also requires several native C libraries. All of these dependencies are given to the Hadoop distributed cache object, which distributes them to all nodes where the MapReduce job is run. When the MapReduce job is started, it is handed the list of files to process, as well as all relevant options from the Text Data Processing Entity Extraction transform (including the product dictionary and voice of the customer sentiment analysis rules). The list of files is handed to the Hadoop FileInputFormat framework, which splits them up based on location within the cluster and hands them to MapReduce subtasks that are started on the nodes where the data resides. The output from all tasks is placed in a directory on the distributed file system. Once all MapReduce tasks are complete, SAP Data Services reads the results using the direct file access and loads them into a table in SAP HANA. Phase 2: Replicate Business Context in SAP HANA To analyze the data within a business context, structured data about actual products from SAP ERP is loaded into a table in SAP HANA. The data is then correlated with the customer sentiment in the text data processing output. This is done in two steps (see Figure 7). A special data transport piece speeds up data transfer using a script based on the ABAP programming language. Figure 7: Two-Step Process for Correlating Structured Data with Customer Sentiment T134T(SAP_ERP_R63... MAKT(SAP_ERP_R63_... MARA(SAP_ERP_R63_... Query T023T(SAP_ERP_R63... DataTransport4381 MAT_EXT_HADOOP Query MATERIALEXTRACT(H... Harness Insight from Hadoop with MapReduce and Text Data Processing 11

12 Phase 3: Contextualize and Present the Information SAP HANA was used to rapidly combine and analyze the product data from SAP ERP with the relevant text data extracted from the Web logs in Hadoop. Results were delivered to business users through a report generated by SAP BusinessObjects Web Intelligence. Figure 8 shows the sentiment analysis discovered for the Samsung S860 digital camera and the corresponding feedback in converted form, standard form, and type. Figure 8: Sentiment Analysis and Corresponding Feedback 12 Harness Insight from Hadoop with MapReduce and Text Data Processing

13 Collecting, Processing, Analyzing, Visualizing, and Exploring Big Data Summary Revisiting the case of the large successful retailer wanting to get closer to customers, we have extracted the customer sentiment and correlated that information with particular products. As a result, the business can now get immediate and deep insight into what is being expressed in the social media. Based on this architecture, organizations can start thinking about brand value protection, innovative service models, and customer co-innovation as emerging business models that can improve their competitive edge. The typical concerns that come up when social media needs to be analyzed include: Collecting and storing large volumes of data at speed and scale Processing textual representations using linguistic tools to reliably extract sentiments Contextualizing the insights gained based on enterprise master data to provide actionable information Creating an intuitive visual environment to help users explore, analyze, and act on the insights gained by mining Big Data In this paper, we have demonstrated a solution architecture that is able to cover all of the points above. In particular, the key functionalities in this solution architecture are: Collect and store: Social media data reaches petabytes quickly it grows in size incrementally. Using traditional data stores means capital investment and processing challenges. Furthermore, the addition of structured data stores requires preprocessing. This turns out to be suboptimal for the volumes of data. It is also ineffective because the use of the data is discovered iteratively after the data is collected. Apache Hadoop based systems have been successfully used for storing such large volumes of data in a cost-effective and scalable way. The Apache Hadoop community is vibrant and is constantly improving and innovating in this space. Using Apache Hadoop offers economic scalability that is very valuable for Big Data solution architectures. Process: When data sizes are petabyte scale, moving data to the point of processing becomes extremely inefficient. The MapReduce framework of Apache Hadoop offers a highly scalable distributed processing model. In this project, the processing model contains three parts: MapReduce framework and text data processing: The MapReduce framework is used to distribute text data processing to the various nodes. Data transformation and movement: On top of this distributed computing infrastructure, SAP Data Services orchestrates the data transformation and movement into suitable target systems. Using the designer feature of SAP Data Services, the various transformations required from the different data sources can be modeled from Apache Hadoop to SAP HANA. In this example, the Text Data Processing Entity Extraction transform was pushed down and processed in Hadoop enabling the organization to discover relevant sentiment about products while reducing data movement of low-value data. With its proven highvolume data throughputs, SAP Data Services enables fast delivery of data between end points. Enterprise-ready extract, transform, and load process: With SAP Data Services, data management professionals get a familiar environment with improved functionalities to work against emerging data sources like Apache Hadoop. The need to write Java code to extract value from Hadoop is vastly reduced, if not eliminated, by using SAP Data Services. This is a crucial step in making Hadoop enterprise-ready in terms of use and management. Correlate and analyze: Effective analytics need enterprise master data combined with extracted and suitably characterized insights. Text analytics must also be based on enterprise master data. This is achieved in the processing phase of sentiment extraction as indicated above. The same master data needs to be made available in a fast analytical store to enable ad hoc exploration. With its in-memory data analytics functionality, SAP HANA offers high-speed exploration on extremely large volumes of data. Visualize and explore: The value locked in the data becomes truly useful when presented through an intuitive user interface that enables exploration. The SAP BusinessObjects Business Intelligence suite offers the most comprehensive BI clients that can serve a large community of BI users. Furthermore, with mobile clients enabled, your enterprise can make analytics mobile too. Built for visualizing the largest volumes of data, the SAP BusinessObjects BI suite is Big Data ready right from the start. Harness Insight from Hadoop with MapReduce and Text Data Processing 13

14 To solve a Big Data problem at the scale required for social media analytics, every component in the solution architecture must be Big Data ready. This calls for deep engineering expertise and a proven track record. With the SAP portfolio of products and our vibrant ecosystem of software and hardware partners, enterprises can quickly realize the value of their Big Data. In this paper, we have examined how organizations can implement sentiment analytics efficiently and effectively using SAP HANA, SAP Data Services, the SAP BusinessObjects BI suite, and Apache Hadoop. Enterprises can make use of the solution architecture described in this document to define their social analytics strategy and reap immediate benefits. The solution uses available sources of data in new ways to achieve deeper customer insights primarily focusing on the sentiments expressed. With these insights, you can improve customer engagement across a variety of enterprise functions like brand reputation management, proactive service offerings, and customer co-innovation on product functionality. The team at SAP and IBM was able to build the desired solution architecture relatively easily. A concise video summary of the entire solution is available here. Learn More The following Web sites serve as useful references to supplement the information contained in this paper. SAP Co-Innovation Lab SAP HANA SAP HANA and SAP Data Services /boexir4/en/xi4_ds_reference_en.pdf SAP solutions for enterprise information management -Management-with-SAP.html IBM systems solution for SAP HANA An Introduction to GPFS Version 3.3, IBM white paper ftp://public.dhe.ibm.com/common/ssi/ecm/en /xbw03010usen/xbw03010usen.pdf GPFS Library, IBM Cluster Information Center /index.jsp?topic=/com.ibm.cluster.gpfs.doc/gpfsbooks.html *You need an authorized user ID to access this information. 14 Harness Insight from Hadoop with MapReduce and Text Data Processing

15 CMP24082 (13/05) No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP AG and its affiliated companies ( SAP Group ) for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Please see for additional trademark information and notices.