February 2012 Understing the Value of In-Memory in Sponsored by QlikView
Contents The Many Faces of In-Memory 1 The Meaning of In-Memory 2 The Data Analysis Value Chain Your Goals 3 Mapping Vendors to the Value Chain 4 Selecting the Right Technology Combination 10
1 The Many Faces of In-Memory This paper aims to clear up the meaning of in-memory, a term that describes a vital collection of BI capabilities. If you are a CIO or CTO, you may have noticed the term in-memory popping up in articles vendor announcements. You may think it is something new, but in fact in-memory has been around for a long time. The term in-memory is used to describe many forms of technology related to business intelligence. QlikView introduced the term in-memory to the world of BI in 1993 as a way to empower people to explore data in a new way using an associative architecture. Today, SAP, Oracle, others use the term to describe something quite different. Other companies that make solutions for complex event processing, marketing automation, high volume data processing, visualization all use the term in-memory to describe different ways of helping their customers. Do all these companies mean the same thing when they say in-memory? In one sense, they do. All these companies use in-memory to speed up processing. In the most important sense, though, the answer is no. Speed creates a different kind of value for each company. Does it matter how different companies use the term? In our view, it does. By gaining a sophisticated understing of in-memory, technology leaders gain clarity about the emerging lscape how to select combine technologies from that lscape to meet their needs. In that spirit, CITO Research is writing this paper to answer three questions: What does in-memory mean? How is the term used by different vendors? How can various in-memory capabilities work together?
2 The Meaning of In-Memory First the basics. Remember that all processing in computers takes place in the CPU. Information is moved between the CPU memory as a program runs. Most of the time, the information in RAM is moved from disk or from some other location on a network. An in-memory application works with data that is stored in RAM so that transferring data does not introduce delays. Many in-memory processors also compress data, often by 10 orders of magnitude or more, which greatly speeds processing. s that pull data from disk into memory are slow compared with applications with all the data in memory. Delays present a huge challenge for computationintensive applications, such as financial trading algorithms or scientific calculations, but delays are just as challenging for end-users who have to wait for an answer after they start a query. When memory was expensive, in-memory was impractical. As the cost of memory dropped, it became economical to do calculations in-memory, more vendors are exploiting the technology. That is why there is so much talk about in-memory these days. In-Memory, Speed, Business Value: The Heart of the Matter The second thing to remember about in-memory technology is that it leads to speed. All vendors who offer in-memory solutions provide benefits based on speed. The key to making the best use of in-memory technology is understing how speed creates business value. Think of it in the same way that auto manufacturers highlight vehicles with a V-8 engine, which provides power. The power of a V-8 engine could be used for acceleration in a sports car, but in a completely different application, the power could be used by a construction truck to move heavy loads. The same engine is used, but applied optimized to accomplish different goals. With in-memory technology, speed is used to create value, as shown in Figure 1.
3 Better ways to explore data Faster processing of queries In-Memory Speed visualizations Real-time analytics High volume processing Figure 1. In-memory creates value through speed The Data Analysis Value Chain Your Goals So far, so good. In-memory provides speed, speed provides business value. Now we can address the most important question: How does in-memory technology help me achieve my goals? This question really cannot be answered without a way to express different types of goals. Our solution is to categorize the ways people benefit from BI technology, which we call the Data Analysis Value Chain, shown in Figure 2.
4 Store Search Distill Aggregate Analytical Techniques Real-Time Monitoring Analysis User-Driven Exploration Discovery Visualization Figure 2. The Data Analysis Value Chain In the next section, we map the way that different in-memory technologies provide business value along this value chain how different vendors use in-memory technology. Mapping Vendors to the Value Chain No one vendor satisfies every part of the value chain. In-memory technology can be applied at any of these stages. By combining products, in-memory technology can be applied in all stages using different technologies or combinations of technologies. Each of the following sections examines the way that different vendors use in-memory technology to support various elements of the value chain. Oracle Exalytics In-Memory Technology Oracle recently introduced the Oracle Exalytics In-Memory Machine, a packaged system of hardware software that supports analysis, modeling, planning. The Exalytics In-Memory Machine boasts 40 available computation cores 1 terabyte of memory is capable of replicating an entire data warehouse the workhorse of the traditional BI world into memory. It is optimized to work with the Oracle Exadata database uses columnar compression to exp capacity speed up analytics queries, which are launched through a SQL interface. Developers will tailor Oracle Exalytics to meet specific needs. Distill Aggregate Analytical Techniques
5 Typical user: Developer Optimize the response time of a financial trading or telecom billing application Build new analytical applications in an enterprise heavily invested in Oracle Massively Scalable Spreadsheets In-Memory Technology 1010data provides a large in-memory database that can process, present, analyze data in a way similar to a common spreadsheet. 1010data can hle spreadsheets with hundreds of billions of rows. End users interact directly with a snapshot of an entire dataset rather than interacting with a sample. In-memory technology accelerates analysis of large amounts of data. This, added to the familiar spreadsheet paradigm, dramatically exps the number of end-users who can directly analyze large data sets. Typical user: Analyst Store Search Distill Aggregate Analytical Techniques Reduce the time interval for analysis (days rather than months) Make searching less constrained than traditional database searches Increase business user access to data (no SQL skills needed)
6 SAP HANA In-Memory Technology SAP HANA uses in-memory technology to combine into one the characteristics of relational databases used for transactional applications, data warehouses based on star schemas, 1 data cubes. SAP HANA enables one model to support both transactional analytical use along with many types of common analytical data processing functions in the database. SAP HANA transforms SAP s portfolio of enterprise applications BI technology into a more cohesive offering. SAP HANA is not something that end users will use directly; it will primarily be used by developers. SAP HANA works in conjunction with analytical tools. It can be the foundation for designing custom applications it can feed into business systems such as ERP. In-memory technology is vital because analytics without star schemas requires very fast aggregation. The aggregation of data that happens in star schemas still must occur, but because of the speed of the database, it happens in real time when the query is executed. The ability to serve both transactional analytic applications from one database obviates ETL processes needed to migrate data from transactional databases to data warehouses. Distill Aggregate Analytical Techniques Real-Time Monitoring Analysis Typical user: Developer Reduce analytical tasks from hours or days to seconds Increase the amount of analysis performed Improve the speed of BI tools Process extremely large data sets with billions of rows Include both structured (transactional information) unstructured information (such as email) in one database 1 Star schemas are designed to optimize user ease-of-use retrieval performance by minimizing the number of tables to join to materialize a transaction.
7 Complex Event Processing Systems In-Memory Technology StreamBase Systems similar vendors use in-memory technology to build complex event processing (CEP) applications for financial firms. The object behind Stream- Base is to process streaming data from financial sources or connected devices, putting structured unstructured data in-memory. As the streams of data arrive in-memory, complex rules other analytics are applied to recognize important events that may be used to drive automated reactions. This sort of system is used in program trading other real-time operational systems. Real-Time Monitoring Analysis Typical user: Developer Accelerate improve the performance of real-time financial or manufacturing applications Marketing Automation Systems In-Memory Technology Metamarkets provides real-time analytics for digital marketing applications. Metamarkets in-memory database can sift through billions of marketing records up to 1,000 times faster than conventional databases. Agencies get a clear view into digital marketing activities. In this case, in-memory technology supports a database storage mechanism, along with a specially designed pipeline that allows parallel processing on Hadoop, which is optimized to keep up with huge volumes of data.
8 Store Search Distill Aggregate Analytical Techniques Real-Time Monitoring Analysis Typical user: Analysts Evaluate the effectiveness of digital marketing campaigns Underst the real-time impact of such campaigns Business Discovery In-Memory Technology QlikView uses in-memory to deliver Business Discovery to its customers, a method of analysis that allows everyone to gain insights using intuitive tools work together collaboratively concurrently. Business Discovery helps people make decisions based on multiple sources of insight: data, people, geography. Users can create share knowledge analysis in groups, across organizations. Business Discovery platforms help people ask answer their own stream of questions follow their own path to insight on their own in formal or informal groups. QlikView uses in-memory technology because it is the only way to power an associative architecture. When business users explore data in QlikView, they use associations patterns that are more like those in the human mind than in a traditional database, releasing users from the limitations of databases. In associative architecture, every data point is associated with every other data point. Picture a fiber-optic spider web, where everything is connected. Pulling on one thread lights up the related associations, showing you new paths through the data revealing new kinds of connections.
9 User-Driven Exploration Discovery Visualization Typical users: End users, analysts Collaborative analysis decision-making across a shared data set Building user-driven analytical applications on the fly Determine where sales are underperforming take immediate, corrective action Identify analytical gaps in traditional database queries Visualization Software In-Memory Technology Software vendors that focus on interactive visualization use in-memory technology to provide instantly updated graphics. Tableau Software other vendors use in-memory technology to present interactive visualizations. Data can be presented on a graph or in some other complex visualization. Parameters can be changed to include or exclude data, the visualization changes immediately. Users can manipulate information displays in real time in multiple dimensions using data from predetermined queries. User-Driven Visualization
10 Typical users: End users, analysts Communicating information that is difficult to underst in non-graphical formats Figure 3 provides a visual summary of the vendors products just discussed. Store Search Distill Aggregate Analytical Techniques Real-Time Monitoring Analysis User-Driven Exploration Discovery Visualization Oracle Exalytics Oracle Exalytics 1010data 1010data SAP HANA StreamBase Systems Metamarkets QlikView Tableau Tableau Figure 3. Mapping Vendors to the Value Chain Selecting the Right Technology Combination The best in-memory approach for you may involve a combination of technologies. For example, QlikView s Business Discovery offering the high-speed databases provided by Oracle Exalytics SAP HANA are not mutually exclusive. SAP HANA is extremely powerful but does not inherently render data in a way that business users can exploit. Yet datasets processed in an in-memory appliance delivered to QlikView for presentation in Business Discovery applications, using an associative architecture, could be very effective in completing the last mile to business value because business users can work with data right away. The user-friendly, collaborative nature of
11 QlikView supports gaining new insights. It complements exps the usefulness of heavy-duty appliances such as Exalytics or HANA, which are optimized to work with other Oracle SAP software reach deep into the machinery of a data-rich organization, cutting a lot of the red tape in the process. This way, the investment you ve made in enterprise technology is not only preserved; it s actually enhanced. Another common pattern is to use products for visualization along with heavy duty data crunching systems like Hadoop. In some cases, systems like Hadoop can feed data into SAP HANA or Oracle Exalytics then feed that data into a system like QlikView for interactive processing. The best way to determine which combination of in-memory technology will deliver the best value for you is to first examine the goals of your business. The powerful V-8 engine of in-memory technology is now available in many incarnations to serve those goals. CITO Research CITO Research is a source of news, analysis, research, knowledge for CIOs, CTOs, other IT business professioals. CITO Research engages in a dialogue with its audience to capture technology trends that are harvested, analyzed, communicated in a sophisticated way to help practitioners solve difficult business problems. This paper was sponsored by QlikView created by CITO Research