The Analytic-Transactional Data Platform: Enabling the Real-Time Enterprise

Transcription

1 Industry Developments and Models The Analytic-Transactional Data Platform: Enabling the Real-Time Enterprise Carl W. Olofson IDC OPINION With the ever-increasing volume, variety, and velocity of data coming at enterprises today, there is increasing pressure to exploit that data not only for offline strategic decisions, or for tactical adjustments, but for decisions "in the moment"; these are the decisions that drive the real-time enterprise. This last category of decisions must be driven in operational applications, which means that those applications must be able to blend transactions and analytics against a single data collection with very low response times, because business will not wait for the data to process. In considering the sort of data platform needed to handle such a requirement, the following considerations are key: To achieve the low latencies required, the core database management engine must be memory optimized, keeping the most relevant data in an optimized in-memory format for fast processing, and such an engine must be capable of processing both transactions and analytics on this data. Since it is neither practical nor necessarily desirable to maintain all data relevant to these blended operations in a single database, the platform must bring together multiple databases, perhaps even databases having different core architectures, and feature the ability to draw data dynamically from, or push data to, relevant data sources as required, or to access directly such data where it resides. Metadata, including all data models and schemas in the relevant databases or data collections, must be harmonized, kept current with those databases, and mapped to higherorder constructs, including a business glossary and, for data managed in common, a canonical data model, in order to facilitate the access and management of the data. December 2014, IDC #253100

2 IN THIS STUDY This study examines the need for an integrated transactional and analytical data platform in order to process complex analytic queries as a harmonious data collection. One key capability of such a platform is to enrich transaction processing with the intelligence that comes from immediately relevant business intelligence. Such a platform requires maintaining in one database instance the data that is current and relevant to the transactions in question to facilitate the kind of real-time business intelligence necessary to enrich or even determine business decisions at the point of action. Another requirement is to provide transactional applications with the ability to reference other BI data that may reside in other databases, such as data warehouses, and to permit analytical applications running on data warehouses to enrich their analytics with data from transactional databases and other data collections, including nonrelational data stores. This study describes such a platform, considers the kinds of technology options available in the market today, and projects future developments that will enhance this area of functionality. SITUATION OVERVIEW Introduction The typical enterprise today keeps its data in many databases; sometimes hundreds of them. Most of these are transactional databases specialized to the requirements of specific applications or sets of applications. Some are analytic databases, used for query and reporting, and a few are very large enterprise data warehouses, which collect data from many of the transactional databases, reconcile that data to a common schema, and are optimized for the processing of very complex queries. In addition to these, some enterprises are collecting and managing data in nonrelational databases, including NoSQL databases, and in Hadoop data collections. To enrich operational applications and the transactions they process with up-to-the-moment business intelligence, it is necessary to find ways of bringing data together from a number of these sources, and to perform both analytics and transactions on a single data platform. That platform needs to be able to handle a variety of types of data, and to execute both complex queries and transactions very quickly. The technology that does this is sometimes called "hybrid transactional/analytical processing (HTAP)." Since, however, the analytics are used to drive transaction execution, a better term might be "analytic transaction processing." This study refers to the supporting technology combination as an "analytic transaction data platform." The Business Problem With the increasing pace of business, and the competitive advantages to be realized by executing transactions more intelligently, there is an emerging need for conditioning transaction processing with some level of analytics that enable quicker decisions and more refined actions. Most enterprises use business intelligence software and databases, including data warehouses, to inform business decisions for reporting and planning purposes. Few are in a position to use up-to-the-minute business intelligence to make tactical decisions "in the moment" as business is being transacted IDC #

3 IDC has identified three key levels of decision management: Strategic: Decisions based on a broad range of historical data, sometimes going back years, to help support enterprise-level decisions that affect the next quarter, the next year, or multiple years (e.g., Should we open a new office? Should we close an office? Should we fund a new product line?) Operational: Decisions based on recent business data, reflecting the state of the business over the past day or week, used to make decisions of an immediate nature (e.g., Should we increase our inventory level for a popular product? Should we adjust our floor displays? Should we add or reduce personnel for tomorrow's shifts?) Tactical: Decisions based on current data, cast against contextual data from the past, for making decisions "in the moment" (e.g., This customer has expressed interest in a particular product; based on the customer's behavioral profile, when compared with other similar customer behavior patterns, might s/he be interested in some upgrade accessories? There is heavy traffic ahead for one of our trucks; based on the current traffic pattern, compared with similar traffic patterns in the past, how should it be rerouted?) Sample Analytic-Transaction Scenarios Strategic decisions are usually informed by analytics performed on the enterprise data warehouse, sometimes in conjunction with other BI data sources. Operational decisions may be driven by status reports and tools using a subset data warehouse or operational data store. Tactical decisions need to be driven by analytical queries on current, or nearly current data, some of which is closely related to, or even includes, data in the transactional database. Such combined analytical transaction operations enable decisions that affect current actions. The following are sample scenarios that illustrate this capability; they are generalized and do not exactly reflect any one company or actual operational situation: Retail: An online sales application uses a customer's past pattern of purchases to find a broader pattern of purchases by customers generally, using the customer's profile, and then guides the process of selecting products and offering discounts based on that data. Financial services: A portfolio management application looks at current share price changes, comparing them not only with complex event triggers but also with current and past patterns of price movements and the trading rules for each portfolio to make trading decisions. e-entertainment: An online entertainment app that provides streaming video makes decisions in the moment regarding what films, TV shows, and so forth, to highlight for the user based on current trending data and past user behavior. Bringing the Data Together Since transactions are carried out by operational applications with specific transactional databases, blending transactions and analytics is largely a matter of bringing relevant contextual data for analytics to the transactional database of the application in question. It is not, however, practical, necessary, or even advisable to try to bring all such data into one database. Data that is seldom accessed, or accessed in a simple way, may be best kept in its source and accessed remotely IDC #

4 Platform Data and External Data Sources Operational applications act on transactional databases that are designed to serve the needs of the application. In some cases, multifunction application suites, such as ERP systems, may be built on a common schema foundation. Even in such cases, however, the data is optimized for the specific transactions carried out by the application or applications, not for analytics. So an alternative structure, optimized for analytics, that includes immediate and relevant data, needs to be included together with the transactional data under the control of one platform. But in addition to these, there is a need to include relevant contextual data for BI in the processing to achieve an enriched transactional result. Such contextual data may come from a variety of sources, including data warehouses, nonrelational sources, and other operational application databases. Differences Between Transactional and Analytical Databases There are fundamental differences between transactional and analytic data processing that determine differences in the corresponding databases. Transactions generally involve the retrieval and insertion or update of whole rows. Analytical databases are for complex queries, in which only a subset of columns is involved. For this reason, transactional database management is usually optimized around row management, while analytical database management is frequently optimized around columns. Most relational database management systems (RDBMSs) that are optimized for transactional databases hold the data physically in row format, either resident on disk or in memory. A few use alternative techniques that more easily blend transactional and analytic operations by using a structure that reflects the row structure logically but maps to columns. Transactional applications typically issue a number of queries before performing a data change operation (insert, update, or delete). In general, databases that are held in row format need secondary indexes to ensure that those queries execute with acceptable performance, but secondary index maintenance tends to slow transaction processing, so they are kept to a minimum in most cases. By contrast, RDBMSs that are optimized for analytic processing use a variety of techniques to speed up complex queries involving many nested join operations. Such techniques have, in the past, been necessitated by the row-wise management of analytic data on disk, forcing the use of accelerators and shortcuts, such as bitmap indexes, star schema organization for OLAP, and special intersection structures for specific columns called cubes. Another technique meant to optimize the processing of complex joins is materialized views. The technique that is becoming the standard for optimizing analytic queries in the database is that of managing the relevant data in memory in a query-optimized compressed columnar format. In some cases, this in-memory construct is built from data kept in either row or column format on disk, and in other cases it is always in memory, with replication and snapshots written to disk for recovery in case of database server failure. Differences between transactional and analytical databases are not limited to the types of operations performed and the optimizations that support those operations. They also include the types of data held and the structure of the schema. Transactional databases have tended to hold only the data needed for the application to execute its transactions, although increasingly, transactions also capture data about the historical context of the transaction, which may aid in subsequent analysis. In the case 2014 IDC #

5 of a unified ERP system, a transactional database schema may be quite extensive, yet still omit elements of information necessary for the kinds of analytic queries that might enrich application, or user, decisions at the point of action. Of course, a data warehouse, which is an analytic database intended to give the user a broad view of the state of the business, the customer, and of market opportunity, contains data from multiple transactional databases, optimized for generalized query (often based on a canonical data model in third normal form). For this reason, a data warehouse tends to have a very different schema than that of any transactional database. An Analytic-Transactional Data Platform To integrate transactional operations with analytics, it is necessary to implement an analytictransactional data platform one that is capable of both complex analytic queries and of executing transactions based on relevant data. Such a platform must be capable of optimizing along both transactional and analytic lines. It needs to be able to manage a blended database of the most relevant and frequently accessed data for both kinds of operation, and to be able to access less relevant or less frequently used data in other data sources as needed. An In-Memory Core The analytic-transactional data platform is not made up of just one database, though it should have, at its core, for each application context, a memory optimized database of the most relevant and active data running on a database management system (DBMS) that is optimized for the blending of transactional and analytic data operations. This may be a subset of a larger database, which the platform also manages, that keeps the remaining data in SSD or spinning disk, based on its frequency of access. Constituent Databases In addition to this core database, the platform may, for each transactional context, incorporate a group of databases and other data collections (such as might be managed in Hadoop) relevant to the processing of real-time analytics in support of the transactional application that is active at the time. These databases and data collections require special management and coordination in order to ensure that their data is consistent with the data in the core, and that data from them may be provided to the core immediately on an as-needed basis. Platform Consistency In addition, the analytic-transactional data platform should set the schema for its core database in the context of a larger data model that encompasses all the other databases and data collections that the platform manages. Such a system not only enables the incorporation of real-time BI into transaction processing but enables the incorporation of up-to-the-minute data managed under multiple formats, including relational, document (JSON or XML), graph, or some program-defined (e.g., Hadoop) format, for business analysis. Other Data Sources Outside the platform itself is a whole host of other databases and data collections that are not managed by the platform, but which, using advanced data integration technology, provide data for the platform, and extract data from the platform. These databases and data collections are separate 2014 IDC #

6 because of the specific requirements of the transactional or analytic applications they serve. Also, these data sources may be used by multiple different analytic-transactional data platforms, each of which is governed by its own schema. Why Not One Database? Given that the candidate technologies for the analytic-transactional data platform are increasingly able to blend analytic and transactional processing in a single database, it would seem logical to aim for collapsing all the data for an application or application suite, together with all relevant analytic data, into one database. Emerging memory-optimized database technologies have demonstrated that this is technically possible. Such an approach is, however, neither practical nor particularly desirable, although it may seem so at first blush. Although such a database would necessarily be huge, the issue here is not one of scale. Although such a database, if held mostly in memory, would be expensive to maintain (DRAM is still far more costly than either SSD or spinning disk), the issue is not one of cost. The issue is one of manageability and agility. There is also the matter of efficiency in relation to location. Sometimes, it is necessary for a database to be in the field, or in a store or branch office, close to where the data is being collected. As enterprises seek to more effectively harvest value from the data arising from the Internet of Things (IoT), local or regional databases, acting as collection points, may speed such data collection, performing data cleansing and summarization operations before shipping a reduced, more compact set of data to the central system. Schema Management Challenges As mentioned previously, analytic and transactional database schemas tend to be quite different. Analytic databases include data from non-transactional sources. They blend data from multiple transactional sources. Transactional database schemas organize the data into the fewest number of tables possible, and are often at least partially denormalized, to optimize the whole row nature of their operations. In addition to these structural differences, there are organizational ones. The transactional database schema is managed by the operational applications team, whereas the analytic database schema is managed by a BI (or business analyst) team, usually in concert with a data warehouse. Blending these schemas together would not only result in a very complex and difficult-to-manage schema, but it would force a level of cooperation at such a detailed level that updates would become slow and bureaucratic. The schema would need to be kept entirely in third normal form for manageability, which would eliminate the ability to optimize for transaction processing. This may not be an issue with respect to some RDBMS technologies that optimize the blending of transactional and analytic operations, but it would still result in such complexity that any substantial change could force renormalization and a reorganization of the database. If simplicity is the key to agility, then it can also be said that a certain level of specialization is a key to simplicity IDC #

7 Other Relevant Data Sources: Hadoop (HDFS) and NoSQL It is increasingly necessary to include data from nonrelational sources in the kinds of real-time BI that the analytic-transactional data platform needs to support. This data often resides in Hadoop (HDFS) files, in NoSQL databases, or in Spark. Its organization may change frequently and is not typically governed by a schema. Obviously, such data cannot be merged into a core database, although selected elements could be imported. Current Industry Dynamics If one is looking for a RDBMS technology to serve as the foundation for a analytic-transactional data platform, one needs, in addition to the ability to unify data operations against multiple data sources, the ability to maintain at its core data that is used for both transactions and analytic queries. All the leading RDBMS vendors, and a number of emerging ones as well, are providing support for memory-optimized management of transactional and analytic data, either separately or together. Only a few, however, provide combinations of functionality that enable analytic and transactional data management in the same system. Each of the leading products in this category takes a different approach to this area of functionality. Blended Versus Segregated Data Although multiple, even many databases may serve as components of the analytic-transactional data platform, they interoperate with a core that operates under a single database instance. This core must blend the capability to perform transactions with the ability to perform complex analytic queries at high speed. The main approaches to the problem of providing memory-optimized support for both transactions and analytic queries on a single core database instance are as follows: Two databases (or schemas), one transactional and the other analytical, managed by a single RDBMS instance, with internal synchronization of common data. This approach can be used in cases where the transactional data and analytic data need to be available for combined SQL operations but need not be coordinated. If, for example, one is processing delivery scheduling by analyzing traffic data from an external system, where the traffic data is highly changeable, this approach may be used since the delivery schedule data is not related to the traffic data, but the traffic data is used to set delivery times. This approach is also useful for cases where one wishes to perform near-real-time monitoring and operational reporting, which is the most common use case today. One database, with the data segregated into rows (for transaction optimization) or columns (for analytic query optimization). This approach may be used when the transactional data (assigned to row storage) plays a minor role in analytic queries and where the analytic data (assigned to column storage) is seldom updated (since column stores require more time to update than do row stores). One database, with data simultaneously in both row and column format, synchronized internally. In this approach, data subject to frequent row-wise update (typical transaction updates) would be present in the row store; data required for high-performance complex queries would be kept in the column store; and data that fits both kinds of operations would be present in both stores. This approach may be called for in cases where contextual data that is stored with transaction updates is needed immediately for analytical queries, as might be the 2014 IDC #

8 case in some online retail applications designed to optimize the shopping experience (and upsell opportunity). As is the case with approach 1 (see bullet 1 in this section), this is also useful for cases where one wishes to perform near-real-time monitoring and operational reporting. One database, with transactional data moving through a delta store that is optimized to update the column store, which is the main structure. In this case, all the data is kept in the column store, so unlike approach 3 (see bullet 3), there is no selective assignment here. The delta store is used to ensure fast transaction processing and hold the data for queries until it can be properly blended into the appropriate columns. This approach is needed in cases where the transactional and analytic data cannot be separated cleanly. One database, holding the data in atomic elements that are tagged or prefixed and indexed in such a way that they can be resolved as table structures, optimizing both transactions and analytic queries. This architecture is rare today, but there are a few such cases out there. This approach serves similar use cases to approach 4 (see bullet 4) but involves a very different internal architecture from the norm, which carries inherent risk. These approaches are illustrated in Figure 1. It should be noted that approaches 3, 4, and 5 (see bullets 3, 4, and 5) are especially appropriate in cases where a combination of transaction processing and contextual analytics is required to support processes that are driven by streaming data. While such cases are unusual as of the time of this study, they should become fairly common within five years IDC #

9 FIGURE 1 Forms of Analytic-Transaction Data Engine Source: IDC, 2014 From the perspective of schema flexibility and overall agility, the first approach may be preferable, although the database synchronization approach may introduce an unacceptable level of latency into the picture. The second approach makes sense when the application data can be sensibly segregated in the way described, offering greater performance along both transactional and analytical dimensions. Up to this point, the discussion of data classification has assumed that all data is either transactional or analytic. In fact, data managed for transactions can also be subject to analytic queries. For this reason, approaches 3 and 4 are probably most desirable as the foundation for an analytic-transactional data platform, with 4 getting the edge simply because it is less complicated. Type 5 holds promise for a best-of-both-worlds capability but has not been fully tested against enough real-world examples to be considered a proven general solution to the problem at hand. Integrating the Enterprise Data Warehouse So far, this document has discussed the analytic-transactional data platform as an environment that supports a wide range of data management activities including, at the application level, a blending of real-time BI with transaction processing that may be called "enriched transactions." In addition to the core database in support of such transactions, including a set of data for a blending of analytic and 2014 IDC #

10 transactional database operations, it includes various levels of integration with other data sources including nonrelational databases and relational databases. None is more important in this regard than the enterprise data warehouse, where data that reveals the overall enterprise context is maintained. There are a number of initiatives for optimizing data warehouse analytics, including in-database analytics, which need to be integrated as part of the overall analytic-transactional data platform picture. The Full Scope of the Analytic-Transactional Data Platform Figure 2 shows how the various kinds of enterprise databases and data collections fit together within the scope of the analytic-transactional data platform. The ring marked A represents the domain of data managed by the platform. The ring marked B represents the additional data in various external data sources that provide necessary context for analytic-transactional processing operations and are affiliated with the platform. FIGURE 2 Data Elements of the Analytic-Transactional Data Platform Source: IDC, IDC #

11 The analytic-transactional platform data elements are as follows: Operational BI data: Application transactional data and closely related data in the enterprise data warehouse Extended analytical data: A combination of data warehouse data and closely related data in nonschematic sources such as Hadoop and NoSQL databases Transitory operational data: Transactional data combined with closely related transitory data that may be kept in a NoSQL document based database Core: A subset of the previously mentioned elements that is so continuously required and closely related that it is combined in a single core database IDC has developed a 3 x 3 opportunity map to show how analytic data is used for planning, execution, and prediction to manage money, people, and things. Figure 3 illustrates how analytic-transactional data participates in this matrix. Figure 3 reflects the fact that initial use of this technology will be to incorporate current transactional data for operational planning and decision support, but as transactional applications that are driven by analytics emerge, the analytic-transactional domain will evolve into the tactical, real-time realm. FIGURE 3 The Analytic-Transactional Domain in the 3 x 3 Opportunity Map Source: IDC, IDC #

12 FUTURE OUTLOOK Trying to make definitive statements about the state of analytic-transaction data platforms going forward is challenging, because both the database kernel technology and the hardware on which it runs are evolving at a rapid pace. In addition to this, new workloads and mounting performance requirements add even more to the pace of development. It is safe to say that all the technology described in this study, admittedly in a very abstract manner, may be described as transitional technology that is evolving quickly. New approaches to data structures, new optimizations for transactional data once it is fully freed from the constraints of disk optimization, new ways of organizing processors and memory, and the introduction of non-volatile dual in-line memory modules (NVDIMMs) all will no doubt result in technologies within 10 years that are very different from what is described here. Nonetheless, the current products are a starting point, and with them, enterprises themselves will evolve, adding speed and agility to business activity to such an extent that within a few years the very term "business process" will require redefinition. Industry Impact How will all this impact the IT industry? For sure, there will be a rise in demand for transactional applications that are driven by "real-time" analytics. There will be closer connections between analytical and transactional database products, resulting in mergers and alliances. New vendors will become very significant, at least until they are acquired by an industry leader. Established vendors may well sink or swim depending on how well they can embrace this trend. Technology Assessment New ways of organizing data (or, at least, new implementations of established, alternative ways of organizing data) will combine with new hardware architectures and new application architectures to redefine the way IT is done, both on-premise and in the cloud. In the meantime, there are vendors and technologies available today that can help users get started on the road to an integrated analytictransactional data platform. Established Vendors and Products An analytic transactional data platform requires the integration of in-memory database technology with full transaction support in the core, as well as metadata-driven data integration capability that includes dynamic data movement for both the platform-managed databases and data collections as well as the external affiliated data sources. Table 1 provides a series of thumbnail sketches of products offered by the leading established DBMS technology providers (in alphabetical order) that address the analytictransactional requirement IDC #

13 TABLE 1 Vendors and Products Supporting an Analytic-Transaction Data Platform Vendor DBMS Products Data Integration Products IBM DB2 with BLU Acceleration and BLU Shadow Tables InfoSphere DataWorks Microsoft Microsoft SQL Server with In-Memory OLTP and ColumnStore Data integration bundled with SQL Server Oracle Oracle Database with the In-Memory Option Oracle Data Integration with GoldenGate SAP HANA SAP Replication Server Source: IDC, 2014 Emerging Analytic-Transaction Data Engine Technologies There are also new vendors with emerging technologies that address the requirement for the platform core, especially as it pertains to new cloud-based and device-driven workloads. It is possible to purchase the component parts, such as a dynamic data integration platform and a memory-optimized DBMS with analytic-transaction processing capability, and assemble an analytic-transaction data platform in-house. For those willing to do so, a sample list of relevant DBMS vendors and products follows: Aerospike (Aerospike DB) Altibase (Altibase HDB) JustOne (JustOne Database) memsql (memsql) nuodb (nuodb) VoltDB (VoltDB) ESSENTIAL GUIDANCE For any IT buyer, if ability to condition transaction processing with the results of analytic queries against current data is a requirement, adopting an analytic-transactional data platform may be necessary. To choose the right platform, and vendor, for the task at hand, it is important to know the answers to the following questions: Do we need to include in our analytical queries the same data that is used for transaction processing? If so, does that data need to be current as of the last transaction commit? 2014 IDC #

14 If not, what is the acceptable latency? How tolerant is the enterprise to the use of new or experimental technology? Does the analytic-transactional data platform need to be based upon the technology of the existing transactional database, or is conversion to another database platform a possibility? Does the database vendor offering an analytic-transactional data platform have a vision that fits with the enterprise IT vision long term? LEARN MORE Related Research Worldwide RDBMS 2013 Vendor Analysis: Top 10 Vendor License Revenue by Operating Environment (IDC #251648, October 2014) Oracle Announces General Availability for the Oracle Database In-Memory Option (IDC #249561, June 2014) Worldwide Database Management Systems Forecast and 2013 Vendor Shares (IDC #248952, June 2014) IDC's Software Taxonomy, 2014 (IDC #249238, June 2014) Move Over Hadoop: Emerging Database Technologies for Tackling Big Data Problems (IDC #248553, May 2014) The In-Memory Database Bandwagon: SAP, IBM, Oracle, and Microsoft Are Onboard (IDC #lcus , December 2013) Oracle Database In-Memory Option: IMDB or Not IMDB? (IDC #lcus , December 2013) Synopsis This IDC study examines the emerging technology in support of an analytic-transactional data platform, enabling transaction processing to be driven, and conditioned, by the results of complex analytic queries, based on technology that can bring together both transaction and complex query processing in a single database platform. "With the ever-increasing volume, variety, and velocity of data coming at enterprises today, there is increasing pressure to exploit that data not only for offline strategic decisions or for tactical adjustments but for decisions 'in the moment'; these are the decisions that drive the real-time enterprise," says Carl Olofson, research vice president for Database Management and Data Integration Software research at IDC. "This last category of decisions must be driven in operational applications, which means that those applications must be able to blend transactions and analytics against a single data collection with very low response times, because business will not wait for the data to process." 2014 IDC #

15 About IDC International Data Corporation (IDC) is the premier global provider of market intelligence, advisory services, and events for the information technology, telecommunications and consumer technology markets. IDC helps IT professionals, business executives, and the investment community make factbased decisions on technology purchases and business strategy. More than 1,100 IDC analysts provide global, regional, and local expertise on technology and industry opportunities and trends in over 110 countries worldwide. For 50 years, IDC has provided strategic insights to help our clients achieve their key business objectives. IDC is a subsidiary of IDG, the world's leading technology media, research, and events company. Global Headquarters 5 Speen Street Framingham, MA USA idc-insights-community.com Copyright Notice This IDC research document was published as part of an IDC continuous intelligence service, providing written research, analyst interactions, telebriefings, and conferences. Visit to learn more about IDC subscription and consulting services. To view a list of IDC offices worldwide, visit Please contact the IDC Hotline at , ext (or ) or sales@idc.com for information on applying the price of this document toward the purchase of an IDC service or for information on additional copies or Web rights. Copyright 2014 IDC. Reproduction is forbidden unless authorized. All rights reserved.