Smart Consolidation for Smarter Warehousing. A Key IBM Strategy for Data Warehousing and Analytics

Transcription

1 Smart Consolidation for Smarter Warehousing A Key IBM Strategy for Warehousing and Analytics

2 Table of Contents Overview...go to 3 Introduction...go to 4 Executive Summary: Smart Consolidation in a Nutshell...go to 4 The Centralized Enterprise Warehouse...go to 5 Rethinking Warehousing and Analytics: The Logical Warehouse...go to 8 Smart Consolidation s Guiding Principles... go to 8 Foundational Technology Requirements... go to 9 Next Steps... go to 9 Tenet One: Consolidate Infrastructure to Simplify Analytics...go to 10 Technology Requirement... go to 11 Technology Proof Points... go to 11 Use Cases... go to 12 Tenet Two: Process Workloads on Fit for Purpose Nodes...go to 13 Technology Requirements... go to 15 Technology Proof Points... go to 15 Use Cases... go to 16 Tenet Three: Coordinate Management and Across the Logical Warehouse...go to 17 Technology Requirements... go to 18 Technology Proof Points... go to 18 Use Case... go to 19 Technology Requirements...go to 20 Workload-Optimized Systems and Appliances... go to 20 Seamless Flow... go to 20 Virtualization and Query Redirection... go to 21 Sophisticated Enterprise Management Tools... go to 22 Performance... go to 22 The Components of a Logical Warehouse...go to 23 Smart Consolidation Entry Points...go to 27 Conclusion...go to 28 Written by: George Davies Jr., IBM Netezza Strategic Marketing, IBM Software Group, Information Management

3 Overview High-Performance Analytics and the Logical Warehouse Business intelligence is built on data warehouses. These systems commonly integrate data from internal data sources, which are often transactional systems that record an organization s interaction with customers, prospective customers, suppliers, business partners, competitors and regulators. An enterprise data warehouse is commonly implemented as a centralized system: a single data-management technology running on a single computer system. This approach leads to very large databases the warehouse is commonly an organization s largest database. warehouses are usually built on database management systems that were originally designed for processing online transactions, not online or offline analyses of data generated by web applications and social media. Online transaction processing systems typically manage much smaller data sets than those used in analytic processing, which suggests that a database management system optimized for processing online transactions is not always the best choice for analyzing data. The quest for information and competitive advantage extends business intelligence beyond reports and dashboards to analytic applications that deliver predictions and insight to guide decision making. Simple reporting and advanced analytic applications create very different workloads, the former fulfilled by relatively straightforward read operations expressed in SQL, while analytic applications demand heavy computation from languages such as C, C++, Java and others. New data sources (such as , log files, sensor data, and user-generated content on social media websites) and new data types (such as images and audio), unconsidered at the inception of data warehouses in the 1990s, are now driving growth opportunities for business intelligence and analytics. 3

4 Introduction At its BI Summit on May 2, 2011, Gartner observed that the traditional enterprise data warehouse vision has, in general, not been achieved. These industry analysts refer instead to a logical data warehouse. In June, IBM announced a strategy, Smart Consolidation for Smarter Warehousing, that adopts Gartner s terminology and recommends an evolutionary change in direction from a single, centralized physical system to a distributed architecture, where computation is provided by individual systems, with each node optimized for specific workloads. In this paper, we elaborate on the Smart Consolidation strategy, emphasizing fresh use cases and proof points along the way. Executive Summary: Smart Consolidation in a Nutshell As organizations grow their business intelligence portfolios, deploying analytic applications to derive greater value from their expanding data stores, their data warehouse systems assume ever more importance. These systems, originally conceived to support offline reporting and rudimentary analytic queries, are now expected to support enormous and growing data volumes, new unstructured and semi-structured data sources, and expanding communities of knowledge workers running a wide range of analytic workloads. For many organizations, the traditional model of a single, centralized enterprise data warehouse (EDW) system has become too rigid, unable to keep pace with modern demands for mixed workloads, extreme performance, and advanced analytic applications. This paper describes an evolutionary strategy, a smarter vision for harvesting business advantage from large and varied data stores with advanced analytics: Smart Consolidation and the logical data warehouse. Smart Consolidation s Guiding Principles 1. Consolidate infrastructure to simplify analytics. 2. Process workloads on fit for purpose platforms. 3. Coordinate system management and data governance across the enterprise. Smart Consolidation Technology Requirements Workload-optimized systems and appliances Seamless data flow virtualization and query redirection Sophisticated enterprise data management tools Performance 4

5 A Sample Road Map to Smart Consolidation Consolidate sprawling data marts by offloading analytics workloads from the EDW to workload-optimized systems. Introduce queryable archiving to provide cost-effective analytics on massive data sets. Accommodate new particularly Big sources into the analytic infrastructure. Consolidate enterprise data management and logical warehouse management. Before expanding on these topics, let s review the case for change. The Centralized Enterprise Warehouse Many large enterprises have adopted centralized enterprise data warehouses as their analytic infrastructure. While this model served reasonably well into the mid 2000s, the growth in data volume, variety, and complexity has combined with the exploding demand for analytics to severely limit the EDW s utility as an enterprise solution. Figure 1: The EDW as Originally Envisioned Vision: All enterprise data storage, analytic and operational processing takes place in one central data warehouse. Reality: Many single EDWs cannot handle today s volume, velocity and variety, of data and workloads. Lack of agility, increasing latency. Business needs are not being met. Sources CRM Integration Traditional Centralized Enterprise Warehouse ERP External Sources 5

6 While this depiction of a comparatively bleak reality does not describe all installations many organizations have performed quite nicely with a monolithic warehouse in place the accelerating future of business analytics requirements, and their effects on the single-store vision, is not in doubt. In 2010, Gartner estimated that over 70% of EDWs had performance issues ( Warehouse Magic Quadrant, 2010). In a November 2010 global database survey, Forrester reported that 65% of enterprises found it difficult to deliver performance with their existing architectures. With these levels of performance dissatisfaction as a starting point, adding complex, high-volume unstructured big data handling, while satisfying the growing demand for still more sophisticated analytics, would appear to deliver an insurmountable challenge to the centralized EDW model. Note: It is worth emphasizing that if your workloads and data stores can indeed be satisfied by a single, high-performance warehouse system, you have already achieved Smart Consolidation. That is, some organizations will find that they can satisfy all of Smart Consolidation s guiding principles on a single system. As data volumes, variety, and complexity continue to grow, and analytic workloads multiply, a single computer system, running a single database management system, may begin failing to meet expected service levels. Workload performance declines. When the centralized data warehouse does not deliver, line-of-business users take predictable action. The typical first response to long-running workloads and underperforming queries is to tune and partition the system. These actions may work for a time for selected workloads but inevitably, they draw valuable technical staff into an endless cycle of warehouse care and feeding. It is hard to overstate the negative business impact of diverting highly skilled technical resources away from business-driving innovation and applying them instead to ineffective system maintenance. When tuning fails to bring EDW performance in line, frustrated user communities react quite logically: they begin to extract data subsets and move them to secondary systems or data marts. But attempting to solve one problem creates multiple new ones: data silos limit enterprise-wide analytics, creating blind spots; governance becomes impossible; data extract-and-offload operations create additional load on the already teetering EDW; and costs and complexity escalate. 6

7 Figure 2: The Return of Sprawl A single, centralized EDW is simply unable to handle today s volume and variety of data. Lines of business resort to ad hoc solutions, creating data mart sprawl resulting in: Limitations to enterprise-wide analytics and visibility; A lack of true governance; Increased strain on the EDW, shortening its lifespan; An inability to scale; and Escalating cost and complexity. Ultimately, the complexity and cost of a single EDW outweigh the business benefits. Sources CRM Integration Traditional Centralized Enterprise Warehouse ERP External Sources The spreadmart warehouse topology depicted above is too complex to administer, too reliant on tuning, too inefficient at analytics, and too costly to maintain. governance is impossible, analytic performance is unreliable, and analytic innovation is effectively derailed. In short, the costs of this installation have come to outweigh its value to the business. Unfortunately, at many sites, with large IT investments in play, this realization does not come quickly or easily. The sprawl-makers ad hoc efforts at distributed processing are not entirely mistaken, but while a distributed approach does have merit, enterprise-level success requires some important architectural adjustments and investments in new software tools. We call the required tools and adjustments Smart Consolidation, and the resulting infrastructure a logical data warehouse (LDW). 7

8 Figure 3: Evolving to a Logical Warehouse Key Tenets 1. Consolidate infrastructure to simplify analytics. 2. Process workloads on fit for purpose platforms. 3. Coordinate system management and data governance across the enterprise. Traditional Centralized Enterprise Warehouse Evolution Logical Enterprise Warehouse A very nice picture, but how do we get there? Rethinking Warehousing and Analytics: The Logical Warehouse Smart Consolidation s Guiding Principles This paper suggests a way forward from a single physical system to a logical data warehouse, implemented as a distributed-computing infrastructure integrated by software utilities. This evolution is guided by three architectural tenets, which illuminate the evolutionary pathway, and suggest a roadmap for stepwise action: 1. Consolidate infrastructure to simplify analytics. Appliances and specialized systems reduce complexity by consolidating sprawling data marts into a small number of workload-optimized systems. 2. Process workloads on fit for purpose platforms. Computation is mapped to appliances and systems specifically designed for well understood workloads. These specialized systems offer optimal performance at affordable prices, their simplicity 8

9 accelerates time-to-value, and their deployment frees the EDW to assume a more focused role as orchestration engine and data management hub. 3. Coordinate system management and data governance across the enterprise. Centralize data management, not data and compute resources. IBM s industry-leading software portfolio of data management, governance, replication, and integration tools makes logical data warehouse management easy and affordable. Foundational Technology Requirements Each of these guiding principles implies several additional design principles, or technology requirements: Workload-Optimized Systems and Appliances Consolidated infrastructure and distributed data/compute nodes demand high-performance, cost-effective processing platforms to handle the assigned workloads. Seamless Flow flows smoothly between warehouse nodes. Synchronization operations replication, change data capture (CDC) updates, etc. are integrated, automated, and reliable. Virtualization and Query Redirection Transparent distribution of systems and data. Warehouse topology is invisible to business users and applications. virtualization and automated query redirection hide system complexity, and data is accessed through a discrete set of well-defined access points browsers, application clients, and APIs. Sophisticated Enterprise Management Tools integration, data governance, and their related disciplines and sub-disciplines (master data management, changed data capture, data quality, data cleansing, and so on) require (a) an orchestration platform, and (b) enterprise-level applications with cross-system visibility. Performance Everything has to be fast. Next Steps Later, we will describe a variety of entry points, or adoption strategies, for Smart Consolidation. Here is just one practical pathway, and the step order is not required: Step: Consolidate sprawling data marts, and offload analytic workloads from the EDW to workload optimized systems. The EDW is now the central locus for data governance and metadata management. Use simple, effective tools for data flow planning, data movement, and governance. This step delivers high performance analytics and simplifies infrastructure and its management. 9

10 Step: Introduce queryable archiving to provide cost-effective analytics on massive data sets at an economical price point. IBM Netezza has announced the C1000 family of High Capacity Appliances, which minimize the per-terabyte cost to store large historical data sets, while keeping the data accessible for on-demand user queries. Step: Accommodate new particularly Big sources into the analytic infrastructure, using systems with real-time streaming analytic engines and Hadoop platforms to undertake pre-processing and initial analysis, and to ingest data into the logical warehouse. can then flow from those platforms to other analytics appliances and systems for further downstream processing. Step: Consolidate enterprise data management and logical warehouse management, incorporating data integration, cleansing, governance, metadata management, and distribution of data flows to the appropriate analytical platforms. (This longer-term step is actually a series of steps that, while highly desirable, can be viewed as an ideal end state rather than a strict requirement.) The remainder of this paper expands on Smart Consolidation s guiding principles, technology requirements, processing nodes, and adoption strategies. Tenet One: Consolidate Infrastructure to Simplify Analytics Because unsatisfied analytic demands drive the rise and proliferation of data marts, consolidating this portion of your architecture may provide the quickest, highest value for business users. Appliances and specialized systems reduce complexity by consolidating sprawling marts to a small number of workload-optimized systems. That is, offload analytics from the EDW to an environment optimized for analytics the data warehouse appliance. Figure 4: Evolving to a Logical Warehouse: Consolidate Infrastructure Smart Consolidation is an evolutionary strategy, not a disruptive one, particularly for clients who have already built EDW-centric architectures. Consolidate infrastructure with purpose-built appliances and systems. Reduce data mart sprawl. Offload analytics from the EDW to appliances optimized for performance. Achieve true data governance. Reduce stress on EDW. Lower total cost of ownership. Simplify queries and analytics against historical data. (figure continued on following page) 10

11 Sources CRM Integration Traditional Centralized Enterprise Warehouse ERP External Sources BI + Ad Hoc Analytics Operational Analytics Queryable Archive Governance, Security + Lifecycle Management By consolidating a sprawl of ungovernable data marts into far fewer purpose-built analytic appliances, IT teams can deliver the best price-performance for analytical queries, while streamlining administrative effort. This frees valuable technical staff to develop and deploy new BI and analytics applications. Technology Requirement Cost-effective, high-performance, workload-optimized systems and appliances Technology Proof Points IBM Netezza Warehouse Appliances IBM Netezza data warehouse appliances are well suited to the Smart Consolidation model. Each appliance is purpose-built for advanced analytics the ultimate workload optimized system. IBM Smart Analytics System for Operational Intelligence IBM Smart Analytics Systems are designed for warehouses that support mixed workloads, including analytics and operational decision support. Starting at under $50K including integrated Cognos reporting Smart Analytics System models are available on Power Systems, System z, or System x. 11

12 IBM DB2 Analytics Accelerator (IDAA) The System z mainframe accelerator comprises one or more IBM Netezza data warehouse appliances, to which (a) selected data are synchronized, and (b) deep analytic queries are redirected automatically, and invisibly to end users. Users have no direct interaction with the IBM Netezza node(s). synchronization and query redirection are fully automated. Installation and deployment are quick and easy, requiring no professional services. No other vendor can bring the same level of directed hardware innovation or the same software connectivity portfolio to its logical warehouse vision. Use Cases Banking A major U.S. bank deployed the IBM Smart Analytics System for credit risk analysis, operational BI, custom reporting, and data cleansing and management cost-effective, high-performance support for multiple workloads. Healthcare A large healthcare alliance must capture, integrate, manage, and analyze diverse high-volume data sources clinical, financial, and operational and share the resulting data stores and analysis with clients, partners, and practitioners. The customer uses a variety of IBM software and systems. Key components include IBM Smart Analytics and DB2 Warehouse Edition software for data integration, consolidation, and operational BI, and IBM Netezza data warehouse appliances for complex analytic workloads an excellent example of consolidating workloads on fit for purpose processing nodes. Financial Services A financial institution must calculate value-at-risk for an equity options desk. The IBM Netezza platform was able to run a Monte Carlo simulation on 200,000 positions with 1,000 underlying stocks (2.5 billion simulations) in under three minutes. Leveraging an in-database analytics approach allowed the financial institution to analyze the data where it resides, rather than build a parallel data-processing platform to run the simulation. Faster query response time and eliminating the time required to move data between two platforms allowed the company to add variables to investment strategy simulations, and to run the risk analysis more frequently. Yale/FINRA Limit Rules Backtesting Yale researchers evaluated an IBM Netezza data warehouse appliance against cloud-based (Amazon EC2) data storage while analyzing 24 billion historical stock transactions. Bringing computation to the data with IBM Netezza In-base Analytics yielded a 43% performance gain over the cloud-based solution with no system tuning. 12

13 Tenet Two: Process Workloads on Fit for Purpose Nodes and compute resources are assigned to appliances and other systems specifically designed for well-understood workloads. These specialized systems offer optimal performance at affordable prices, their simplicity accelerates time-to-value, and their deployment frees the EDW to assume a more focused role as orchestration engine and data management hub. Unburdened of the analytic processing for which it was not designed, the central data warehouse regains computational resources that can be focused on operational and orchestration activities, including data integration and data quality oversight. Eventually, this shift sees the enterprise data warehouse evolving into a new role as the enterprise data hub, mediating data flow, coordinating data integration, and distributing data to the appropriate analytics engines with a simple, appliance-based approach. Centralizing data management reduces complexity and costs and simplifies the pursuit of rigorous data governance. Further opportunities to offload data management and computation include an appliance for operational analytics; an appliance as a queryable archive; a stream-processing system for real-time analysis of data on-the-wire (for example, from digital sensors or an network); and a grid running the Hadoop Distributed File System for analyses of big data such as web-click streams and call-detail records. The logical data warehouse is a dynamic system: nodes may come and go, and inter-node connections permit the results of big data analyses to be moved downstream to analytic appliances for deeper, more advanced analytic processing. Obvious first steps include offloading existing analytic workloads, absorbing rogue data marts back into the logical warehouse, and deploying a queryable archive. At many sites, however, there is a new strategic priority: Move to accommodate new (predominantly Big ) sources into the analytic infrastructure, using systems with real-time streaming analytic engines, time series data processing, and Hadoop platforms to undertake preprocessing and initial analysis, and to ingest data into the logical warehouse. can then flow from those platforms to other analytics appliances and systems for further downstream processing. from new sources may, by design, pass initially through very different data integration and governance filters. Once data flows beyond the ingestion platform, the EDW applies governance, data flow, and life cycle rules. As new data paradigms and analytic platforms emerge, they, too, can be similarly integrated. 13

14 Figure 5: Evolving to a Logical Warehouse: Distribute and Computation Distribute data and compute to the LDW node that best meets the requirements of the application or workload price-performance, availability, data sensitivity, etc. Agile architecture for introducing new data types and analytic models: Offload data and analytics from the EDW to workload-optimized nodes Extend the data warehouse by adding big data and real-time analytic processing Add analytics for new data types New Sources Sensor + Meter Big Processing Real-time Analytics Time Series Processing Event Internet/Social Media Integration Traditional Centralized Enterprise Warehouse Traditional Sources CRM, ERP, External Sources BI + Ad Hoc Analytics Operational Analytics Queryable Archive Governance, Security + Lifecycle Management 14

15 First, identify workloads that can be isolated and offloaded cleanly analytic applications against discrete data sets make good candidates, as do the Big workloads mentioned above. Whether offloaded from the central warehouse (structured data analytics) or deployed on purpose-built nodes before ever having been deployed on the central warehouse (Big and queryable archive workloads), the key benefit is that these workloads can now run on more efficient, less expensive platforms than the subset of EDW resources that would be required to serve these workloads. Technology Requirements Cost-effective workload-optimized systems and appliances Seamless data flow virtualization and query redirection Enterprise-level orchestration engine Technology Proof Points IBM DB2 Analytics Accelerator The IBM Netezza accelerator for System z demonstrates automated inter-system data synchronization and query redirection. Advanced analytic workloads are offloaded from the mainframe to a fast, efficient appliance purpose-built for analytics. IBM InfoSphere BigInsights IBM developed InfoSphere BigInsights to analyze unstructured data such as text, video, audio, and social media. The software, developed in part by IBM Research, is based on Hadoop and more than 50 IBM patents. IBM is also rolling out 20 new analytic services, including: - Server and storage optimization tools for faster implementation time. - center life cycle cost analysis to cut expenses and bolster sustainability efforts. - Security analytics to automate handling of critical events. IBM has also developed the Jaql query language and released it to the open source community. Jaql is a high level declarative language for processing both structured and nontraditional data. Its SQL-like interface means quick ramp-up for developers familiar with SQL and also makes it easier to operate on relational databases. Jaql is highly extensible, and IBM has used this capability to include pre-built Jaql modules in BigInsights that enable integration with IBM Netezza data warehouse appliances and text analytics systems such as IBM Content Analytics. IBM InfoSphere Streams InfoSphere Streams is designed for real-time capture and analysis of unstructured data streams such as Tweets, video, sensor data, stock market data, blog posts, video frames, EKGs, and GPS data. Software advancements have improved previous-generation performance by roughly 350%. Note that BigInsights complements Streams by applying analytics to massive historical data at some time after the first-level, real-time analysis performed by Streams. 15

16 IBM InfoSphere Warehouse The DB2-powered data warehouse software at the heart of the IBM Smart Analytics System is designed for warehouses that support mixed workloads, including analytics and operational decision support. It is available across a broad set of operating systems and hardware platforms to support a wide range of custom solutions. IBM Netezza High Capacity Appliance Introduce queryable archiving to provide cost-effective analytics on massive data sets at an economical price point. Use Cases Insurance A global insurance company uses DB2 Analytics Accelerator to offload claims analysis queries from DB2 for z/os to a tightly integrated IBM Netezza-based accelerator system. In addition to accelerating complex analytic query speeds, this transparent solution also reduces load and improves performance on the System z mainframe. Banking A large international bank uses InfoSphere Streams to capture Internet and ATM data streams, while a high-performance central warehouse stores consolidated, integrated data for deep analytics to drive targeted marketing efforts and derive new insights from customer data, behavior, and credit card transactions. BlueKai and Intuit: Hadoop and IBM Netezza Stream tens of billions of clickstream data points to a Hadoop grid to perform first-level analysis. Then forward selected data subset(s) to an IBM Netezza data warehouse appliance and run advanced analytics to decipher consumer behavior for ad targeting and market campaign optimizations. Department of Energy, Pacific Northwest National Laboratory InfoSphere Streams grid captures high-speed, high-volume log streams and then discards uninteresting status messages before forwarding exceptions and other outliers to IBM Netezza for advanced analytics. BigInsights and IBM Netezza A large retailer performs web log analysis with BigInsights for site optimization, followed by segmentation analysis on per-customer clickstreams. As expected, using Streams and/or BigInsights to process extreme data, and then uploading filtered data or data subsets to IBM Netezza data warehouse appliances and IBM Smart Analytics Systems for advanced analytics, is rapidly becoming a common use case. Note that stream-captured and other big data can also be supplemented or enriched before it is forwarded to an advanced analytics appliance. For example, by adding information to stock market ticks, such as industry class or interested trader lists, even more sophisticated analysis and fine-tuned alerting can be achieved. Furthermore, in real time, Streams can help determine which bits of the incoming data stream should be routed to longer-term database storage and subsequent analysis. As a result, gains in analytic insight are compounded by reduced storage and administration costs. 16

17 Tenet Three: Coordinate Management and Across the Logical Warehouse Crucially, this data warehouse requirement to serve as a data management control point does not fall away as we evolve from traditional EDW to logical data warehouse. Fortunately, IBM s exceptional portfolio of software systems for enterprise management and data governance makes logical data warehouse management simple and cost-effective. These tools and policies manage metadata, provide data governance, distribute/replicate data, and manage the data life cycle, and they do so across heterogeneous systems, including those from multiple vendors. Figure 6: Evolving to a Logical Warehouse: Coordinated Management and Enterprise Smart Consolidation allows clients to evolve from a complex monolithic architecture to a more agile logical and decentralized architecture. Computation is managed centrally but it is executed on LDW nodes optimized for specific workloads, which improves performance, governance, scalability, and agility. New Sources Sensor + Meter Big Processing Real-time Analytics Time Series Processing Event Enterprise Internet/Social Media Traditional Sources CRM, ERP, External Sources BI + Ad Hoc Analytics Operational Analytics Queryable Archive Governance, Security + Lifecycle Management 17

18 Rather than shackling data in a central repository in the name of governance, we want to centralize the management not the distribution and processing of enterprise data. The goal is to simplify the administration, provisioning, and scalability of the extended analytics infrastructure by centralizing those functions. Move data management into the hub, and move analytics out to workload-optimized nodes. Technology Requirements Seamless data flow Enterprise-level orchestration engine Sophisticated enterprise data management tools Technology Proof Points IBM InfoSphere Change Capture (CDC) InfoSphere CDC offers cross-system change data replication for focused data synchronization between disparate systems, and the supported platform list continues to grow. IBM Netezza Replication Module, IBM InfoSphere Replication Server More key pieces of the enterprise data puzzle. Reliable, full-scale replication options are required to meet high availability and disaster recovery standards, and also to enable queryable archiving, for which demand is rising dramatically. Stage-BigInsights Integration This combination streamlines bulk data integration between HDFS and a data warehouse system. Users can launch a BigInsights analytics job from the Stage user console. IBM InfoSphere Blueprint Director Blueprint Director is positioned to take a larger role in logical data warehouse configuration. 18

19 Figure 7: A Conceptual View of Proposed Logical Warehouse Configuration Tools Blueprint Director Warehouse Replication Rules Analytic Analytics IBM InfoSphere Warehouse IBM Change Capture IBM Netezza 1000 IBM Cognos Business Intelligence Optim Management and IBM Netezza High Capacity Appliance Integration Another of IBM s enterprise-level software systems, optimized to manage the new IBM Netezza C1000 series of High Capacity Appliances. Use Case InfoSphere CDC A large financial institution synchronizes several million transactions per day between internal financial data systems and customer-facing query and transactional systems, in near real time. InfoSphere CDC performs real-time data integration automatically, with high availability and single-console setup and management. 19

20 Technology Requirements Previous sections referred to some core technology requirements associated with the Smart Consolidation model s three main tenets. Here, we ll describe these requirements in a little more detail. They provide a good basis for tracking and evaluating new and existing technologies, from IBM and other vendors, and IBM s product portfolio is evolving rapidly in these critical areas. Note that these requirements overlap, as do the domains of the hardware, software, and networking components designed to address them. Workload-Optimized Systems and Appliances Consolidated infrastructure and distributed data/compute nodes demand high-performance, cost-effective processing platforms to handle the assigned workloads. This premise is self-evident, and IBM s Netezza acquisition, workload-optimized Smart Analytics System designs, Informix TimeSeries, and InfoSphere Streams and BigInsights Big platforms all testify to full commitment in this domain. Simplicity To make adding a logical warehouse node simple and cost-effective, ease of use and roughly linear scaling become non-negotiable requirements. Weeks or months of tuning and load testing a new processing node diminish the value of a distributed system, where agility and adaptability should be primary benefits. Seamless Flow The requirement is fast, reliable data flow. This requirement applies to the various data load, synchronization, and distribution tasks, and to the orchestration hub itself the various forms of glue that bind the logical data warehouse nodes into a coherent system. movement enablers and optimizations span the InfoSphere Software portfolio and include the BigInsights IBM Netezza connector, Stage ETL/ELT connectors, CDC, and the IBM Netezza Replication Module. These critical components exist today, all under the same roof at IBM, and they are being enhanced, refined, and recombined to serve the Smart Consolidation model. The latest Stage IBM Netezza and Stage BigInsights optimizations are prime examples, and others will follow. Keep in mind that in a multi-node logical warehouse, data flow traffic may not be fully predictable, nor is it one-way. Some nodes may serve primarily as pre-analysis data sources or repositories, while others function primarily as offload targets for data to be analyzed. However, data synchronization or multi-level analysis may involve data flow into, out of, or around the hub. In some cases, this may include back-and-forth traffic, or variable node-to-node flow patterns driven by dynamic rules and policies, or by data virtualization and run-time query redirection. The newly announced IBM DB2 Analytics Accelerator exemplifies the seamless data movement principle, binding IBM System z and IBM Netezza data warehouse appliances transparently with a 10GbE Private Service Network. 20