SAND CDBMS Nearline for SAP BW

Transcription

1 Accelerating the largest user populations, the biggest data, and the most complex analytics SAND CDBMS Nearline for SAP BW Make business warehouse work

2 Contents Executive Summary... 5 Data Growth: The Challenges for Data Warehousing... 6 Why Adding Hardware is not the Solution... 6 Introducing the Concept of Information Life Cycle Management... 8 Maintaining System Availability with SAND CDBMS Nearline for SAP BW... 9 Introducing the Concept of the SAP NetWeaver BW Data Warehouse Layer... 9 Implementing a Multi-Layered SAP NetWeaver BW Data Warehouse Layer to Serve as a Corporate Memory... 9 Benefits of the SAP NetWeaver BW Enterprise Data Warehouse Layer and Enterprise Corporate Memory Enhancing the SAP NetWeaver BW Data Warehouse Layer and Corporate Memory with SAND CDBMS A Nearline Repository for One or More SAP NetWeaver BW Instances Benefits of Using SAND CDBMS Nearline as Part of the SAP NetWeaver BW Enterprise Data Warehouse Layer Enabling Access to Data from More Time Periods with SAND CDBMS Nearline Integrating Archiving Solutions into SAP NetWeaver BW to Complete the Information Life Cycle Conclusion Appendix A: SAND CDBMS: A Nearline Storage Solution for SAP NetWeaver BW Data Archiving Process (DAP): Saving SAP NetWeaver BW Data to Nearline Storage Managing the Nearline Process in SAP NetWeaver BW: Process Chains Delete Data Restoring Data From Nearline Storage Copying Selected Data From Nearline Storage Handling Structural Changes to InfoProviders, InfoObjects and Master Reference Tables Accessing Nearline Data Using BEx Direct Drill-Through from InfoCubes to Nearline DSO Data Transparent Nearline Lookup API Appendix B: SAND CDBMS Technical Overview Introduction Column-oriented Data Management Tokenization Compression Architecture Scalability Management About SAND Technology... 22

3 Copyright 2010 SAND Technology. All rights Reserved. SAND CDBMS, SAND CDBMS Nearline, SAND CDBMS, SAND Analytic Server, and SAND Searchable Archive are trademarks of SAND Technology. SAP, the SAP logo, and the SAP Partner logo are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. Other trademarks remain the property of their respective owners. Visit SAND Technology on the Web at 10/10/21

4 SAND CDBMS Nearline for SAP BW: Helping Meet the Challenge of Enterprise-Wide Data Explosion Executive Summary Every day, new challenges are threatening the ability of business warehouses to deliver the performance and service levels that enterprises need. Data volumes are increasing and requirements for retention of historical data are becoming more extensive. At the same time, users analytic requests are more demanding than ever before. The need to store ever greater amounts of data, and still have it readily accessible for analysis, data reconciliation, data audits, reconstruction of Key Performance Indicators (KPIs), and so on, means that traditional data management strategies need to be reconsidered. CIO s who are under pressure to meet the obligations of Service Level Agreements need an alternative to the usual options of either making expensive investments in hardware and consulting, or excluding data from the warehouse and thereby compromising the quality of decisions or the power of Business Performance Management (BPM) processes. Traditional solutions that try to address these challenges actually increase load times, lengthen batch cycles, multiply warehouse com-plexity, drive up hardware costs, strain administrative resources all for diminishing returns. The business warehouse still doesn t work. If physical warehouses stopped working, we would get someone in to understand the problem and free up the movement of goods. The same is required for our business warehouses: the information that sits inside them must be allowed to flow fast and freely. SAND understands the problem, and SAND CDBMS enables the business warehouse to work. This white paper discusses how SAND Technology s SAND CDBMS solution can be implemented in accordance with SAP s recommended ILM strategy as a highly efficient nearline solution for any SAP NetWeaver BW installation, allowing for any level of data growth while providing the ability to meet SLA s within available timeframes. SAND CDBMS Nearline for SAP BW is a flexible and cost-effectively scalable database extension for SAP NetWeaver BW, allowing massive and growing volumes of warehoused data to be kept easily accessible, and enabling data warehouse managers to maintain required levels of service while meeting the constantly changing analytical demands associated with increasingly complex business processes. 5

5 SAND Technology White Paper Data Growth: The Challenges for Data Warehousing The data warehouse is an essential tool for any organization seeking to create a single consistent view of their business to help guide critical decisions. Success in building such a view depends on an ability to access data from as many relevant sources as possible. However, the world of data warehousing is currently facing some daunting new challenges that threaten to hinder analytic access to the full range of useful corporate data. The recent explosive growth in the volumes of data generated by organizations in the course of normal business operations has been widely noted. Analysts have estimated that data volumes are growing in excess of 100 percent of one year s data annually, even before factoring in new legal requirements for data retention. Beyond the simple growth in generated operational data, however, increasingly complex analytical demands are also contributing to the expansion in the amounts of data that must be handled by the data warehouse, and are becoming difficult to satisfy using traditional data management approaches. Business users are recognizing that in order to gain a better competitive edge, their data warehouses need to become more powerful in a variety of ways: They must offer a more comprehensive view of the business and the competitive environment, incorporating information from across the enterprise and beyond The information they contain must be more up-to-date and delivered more quickly to users They must be more accessible to a greater number of users around the world, and around the clock A greater amount of detail data must be available for historical analyses and reports Detail data must be more readily available just-in-case, for example to rebuild the analytical environment or to respond to requests requiring construction of new InfoCubes/Data Marts. Also, the increasingly stringent regulatory requirements imposed by the European Data Retention Directive, the SEC, the US FDA, HIPAA, GDPdU, Basel II, and so on mean that the warehouse needs to offer: Finer granularity of information; that is, easy access to detail data A deeper historical record, based on long-term compliance-ready storage for older data in accordance with a data retention policy. These demands are posing major challenges for CIOs responsible for data warehouse operations. Larger data volumes typically have an adverse effect on the performance of traditional relational database systems in areas such as: Query execution and data loading Change runs, rollups, and so on Backup and recovery Migration and upgrade. All of these factors combine to make it increasingly difficult to deliver the levels of system availability and responsiveness demanded by organizational Service Level Agreements. Why Adding Hardware is not the Solution Some might argue that, since the costs of disk storage are decreasing while performance increases, an organization can simply add more disk storage to deal with this problem and maintain levels of data warehouse service to users. But data volumes are in fact growing faster than the price/performance ratios of disk storage technology. Furthermore, there are many more factors involved in managing data environments than simple disk storage capacity. Consider the following facts: In order to meet agreed service levels for data availability, integrity and performance, multiple copies of the data are typically maintained on mirrored devices, off-site disaster recovery facilities and development systems. Storage devices for industrial use require rugged and/or redundant components, with far greater performance and reliability than is typically provided by low-cost drives. 6

6 SAND CDBMS Nearline for SAP BW: Helping Meet the Challenge of Enterprise-Wide Data Explosion Large amounts of expensive cache memory are required to get adequate performance from large storage arrays. Significant processing power is required to perform I/O operations and processes to ensure data availability (backups, for example). Storage farms and the associated processors require expenditures on energy, space and other environmental factors. This includes both the cost of managing the stored data itself and the associated expenses for air conditioning, sophisticated Uninterruptible Power Supply (UPS) equipment, and so on. Offsite hot sites require network bandwidth to maintain data currency. Considerable administrative work (for example, index building) is involved in retrieving data and making it ready for analysis. Data backup and long-term data retention typically require tape subsystems that can meet specific performance levels in order to guarantee the availability of production data Data warehousing best practices often call for implementation of development or staging systems containing full replicas of production data to support pre-production validation of performance and service levels. Figure 1: Breakdown of Data Storage Costs The result is that for every terabyte of required production data, as many as six or seven terabytes need to be kept online and under management across the enterprise, with associated infrastructure costs to maintain adequate service levels. For these reasons, when a mid-tier service level agreement (SLA) is in place for data, the cost of provisioning a terabyte of data can easily exceed $200,000 per annum. 1 terabyte of data in our SAP NetWeaver BW production environment generates 5 terabytes for failover and backup processes. Adrian Bourcevet, Volkswagen Bank GmbH Germany Even more significant is the impact on system response resulting from the pressures of large amounts of data: this cannot be addressed simply by purchasing additional disk and memory, since the operational bottlenecks (for example, protracted backup and recovery times) imposed by traditional database management systems will continue to pose problems as data volumes increase. Recent technological developments, however, have made it possible to overcome these challenges or in the cases where the data explosion has not yet hit, to proactively avoid them. Specifically, SAND Technology has teamed with SAP as a software and development partner to integrate SAND CDBMS with SAP NetWeaver BW as a nearline storage component, within SAP s recommended ILM architecture. This integration enables SAP NetWeaver BW customers to implement an efficient nearline repository that can keep massive volumes of data in a tiny footprint, in a format that permits rapid, easy access. This not only reduces (or prevents a major increase) in both the financial and administration-related costs of maintaining a very large data warehouse capable of satisfying the stringent analytic requirements of today s business users, but also guarantees that the data warehouse manager or CIO will have the crucial ability to provide users with the levels of service they demand, no matter how much data needs to be processed and made available in the warehouse. Implementing the SAP-certified SAND CDBMS architecture can be cost-effective from the outset, thereby positioning the SAP NetWeaver BW environment for ongoing scalability without increasing the cost of ownership of the original infrastructure. Figure 2: Comparison of the Cumulative Costs of Data Storage 7

7 SAND Technology White Paper What is SAND CDBMS? A software-based solution, fully integrated into the SAP NetWeaver BW infrastructure Seamlessly and transparently conforms to standard SAP-defined operating procedures Provides compression of at least 85%, frequently as high as 95% (depending on the data) Does not require indexes to be built, but still allows any data field to be queried with BEx or any SAP-certified analytical front-end tool Data can be searched in its compressed state: no need for staging Retains read-only data with automatic versioning and self-contained metadata for auditability Runs on most popular operating systems (Tru64, Solaris, AIX, HP-UX, Linux, Windows) No special hardware required Can run on the same server as SAP NetWeaver BW, or on a different server Integrated with major archiving solutions to enable full Information Life Cycle Management in accordance with SAP s recommended approach Introducing the Concept of Information Life Cycle Management SAP NetWeaver BW (along with its precursor, SAP BW) is widely recognized as providing a scalable, robust data warehouse infrastructure that can help retain and cost-effectively manage the large volumes of online data needed to support critical business decision-making. Nevertheless, as with any data warehouse system, SAP NetWeaver BW implementations are not immune to the data management challenges posed by the data explosion. In particular, data growth makes it increasingly difficult for the traditional database management systems underlying SAP NetWeaver BW data warehouses to continue to satisfy Service Level Agreements, as more and more data needs to be processed within the strictly defined windows allocated for standard data warehouse operations such as data preparation and backup not to mention the frequent requirement to create new aggregations and KPIs in accordance with changing analytic needs. Based on the recognition that a given data set will lose its current relevance over time and therefore be accessed less and less frequently, SAP recommends implementation of an ILM strategy based on frequency of data access, correlated with the age of the information. This approach is designed to enable data warehouse managers to ensure the availability of a large data warehouse without jeopardizing the delicate balance between financial savings and analytical benefits. SAP s recommendation is to divide data between three repositories: Online, Nearline, and Archived, as shown in figure 3: depending on the organization s specific business values and the frequency with which data is accessed, less frequently used data is moved from the online SAP NetWeaver BW database into a nearline repository, where it is stored economically while still remaining easily accessible; nearline data that is accessed very rarely is in turn moved into an archiving solution, where it can be retained as long as required. Figure 3: Balancing Data Access / Storage Requirements Based On a Data Aging Strategy This approach is designed to relieve the main SAP NetWeaver BW relational database of the burden represented by growing quantities of less frequently used data, thereby streamlining the online database and optimizing its performance. Using this concept of slimming the data warehouse without diminishing access to any required information, the ILM strategy for SAP NetWeaver BW provides the following benefits: Increased volume of data available for analysis Increased system availability Optimized system responsiveness. 8

8 SAND CDBMS Nearline for SAP BW: Helping Meet the Challenge of Enterprise-Wide Data Explosion Maintaining System Availability with SAND CDBMS Nearline for SAP BW To understand how to reap the greatest benefits from the proposed ILM architecture incorporating the SAND CDBMS nearline solution, it is worth taking a closer look at the key data elements in SAP NetWeaver BW: DataStore Objects and InfoCubes. DataStore Objects SAP NetWeaver BW uses DataStore Object technology to build the DataStore layer (formerly known as the ODS layer) as well as the data warehouse layer. A DataStore Object is a consolidated dataset from one or several InfoSources. In contrast to the multidimensional data models of InfoCubes (see below), data in DataStore Objects is stored in flat, transparent database tables. DataStore Object data can be updated in InfoCubes or other DataStore Objects using a delta update. Data in a DataStore Object can be analyzed using SAP Business Explorer (BEx), the Business Intelligence front-end tool of SAP NetWeaver BW. InfoCubes InfoCubes are containers that organize data in terms of business dimensions. This means that users can analyze information from various business perspectives, such as geographic region or type of sales channel. InfoCubes can be accessed by the BEx tool for reporting and OLAP (Online Analytical Processing)-type analysis. As we will see in the following discussion, each of these key data warehouse elements or layers has a role to play in overcoming the challenges posed by the data explosion, and each is part of SAND s nearline strategy for efficient maintenance of overall system availability. Introducing the Concept of the SAP NetWeaver BW Data Warehouse Layer Until recently, the analytical infrastructure of an SAP NetWeaver BW implementation has normally been based on construction of an SAP NetWeaver BW Data Mart Layer. The data marts consist of InfoCubes, which are containers that organize data into multidimensional structures to enable analysis from various business perspectives. While very powerful, this approach will always require some degree of calculation and aggregation in order to make the data useful for analytical purposes, and so will involve pitfalls like the following: Data is separated from its historical context Data that has been transformed through aggregation, manipulation, and overwriting has limited utility for other purposes Since keeping large amounts of data available for just-in-time applications would quickly lead to a negative impact on performance, only the data that is currently needed is retained in the data mart Because the same data is held in various InfoCubes, often with different degrees of granularity, data ends up being stored redundantly There is limited access to historical and detail data drawn from operational sources. Furthermore, an organization may have to contend with a heterogeneous landscape of disparate data mart and data warehouse implementations developed for different projects over time, which leads to the following issues: No single version of the truth is available, making it very difficult to run regional or global reports and analyses and thereby slowing down the decision-making process The quality of decisions is compromised due to missing or wrong data Aggregated and calculated Key Performance Indicators (KPIs) are difficult to understand because of nontransparent business rules Building new regional or global views in response to new requests becomes very difficult Costs are higher for maintenance and administration of multiple Business Intelligence technologies. Implementing a Multi-Layered SAP NetWeaver BW Data Warehouse Layer to Serve as a Corporate Memory An effective solution to these problems is to build a comprehensive foundation for Business Intelligence in the form of a separate data layer, where the integrated results of the data transformation and data quality assurance processes can be stored in a consistent, granular and historical form an SAP NetWeaver BW Data Warehouse Layer. This SAP NetWeaver BW Data Warehouse Layer, maintained using DataStore Objects, is ideally suited for use as the foundation of a unified Enterprise Data Warehouse. This is because it presents corporate data in a relatively detailed 9

9 SAND Technology White Paper form that can serve as the single source for any subsequent analytical projects (for example, architected Data Marts built with InfoCubes to meet specific analytical needs). This typically involves a Multi-Layer architecture where DataStore Objects in each layer are designed for specific purposes, such as: Fast data acquisition Partitioning Real time reporting. At the same time, the Enterprise Data Warehouse Layer acts as a kind of Corporate Memory that contains all the organization s data in a pure granular form, providing a repository of information to serve as the comprehensive data store of record for the enterprise. Benefits of the SAP NetWeaver BW Enterprise Data Warehouse Layer and Enterprise Corporate Memory Reliability and Reproducibility When a single version of the truth is in place, the SAP NetWeaver BW Enterprise Data Warehouse Layer is centrally administered and not subject to transformation for arbitrary projects. In this way, the Data Warehouse Layer can serve as the common source for all the data used to build Data Marts for access by users enterprise-wide. Consistency Implementing a single central Enterprise Data Warehouse Layer for SAP NetWeaver BW means that all extraction, staging and integration can be managed centrally: data is extracted only once, and redundant staging processes are avoided. This not only contributes to the consistency of the information used within the organization but reduces the complexity and costs of administration as well. Business Process Harmonization Having a common source of information ensures that broader business processes are supported by globally consistent data and a common KPI framework. Having a single set of numbers provides better support for dashboard views and drill-downs, and enables faster adaptation to new analytic demands as well as increased responsiveness to advanced analytical requirements. However, it is important to note that since the implementation of a comprehensive inventory of DataStore Objects to act as the SAP NetWeaver BW Data Warehouse Layer/Enterprise Corporate Memory involves the management of very large amounts of data, it brings with it the same range of challenges as the data explosion discussed earlier in this paper. For this reason, the architecture discussed above can benefit from implementation of an efficient nearline facility that permits economical, easily manageable storage of detailed data, while keeping it available for quick analytic access and use as a DataSource for deriving new InfoCubes or DataStore Objects. Enhancing the SAP NetWeaver BW Data Warehouse Layer and Corporate Memory with SAND CDBMS A Nearline Repository for One or More SAP NetWeaver BW Instances Administrators can efficiently store designated DataStore Objects from one or more SAP NetWeaver BW Data Warehouses in a single easily accessible repository, controlling the process via the Nearline Integration facility of SAP NetWeaver BW. Once a selected data set is moved to nearline storage, it is deleted in SAP NetWeaver BW. When needed, the data can be reloaded or restored into SAP NetWeaver BW to build new DataStore Objects. It can also be accessed directly by end users via their familiar business intelligence tools - without requiring indexes on the data to be built first. More importantly, the nearline repository can either be used as a DataSource (via remote cube functionality) or employed in the Analysis Process Designer (APD) process to build new Data Layers within a controlled redundancy architecture, permitting derivation of new InfoCubes and DataStore Objects without loading the data back into SAP NetWeaver BW. When multiple SAP NetWeaver BW instances are used as part of the Corporate Memory, nearline data can be exchanged between the instances by means of the RemoteCube functionality. Because of the high degree of data compression it offers, the nearline repository also supports efficient migration from various disparate SAP NetWeaver BW Data Warehouses to a single corporate SAP NetWeaver BW Data Warehouse designed to serve the entire enterprise. In all cases, using SAND CDBMS as the nearline repository can significantly enhance the performance of the warehouse infrastructure (particularly the I/O subsystem) because locating any piece of information involves reading only a small portion of the compressed data. 10

10 SAND CDBMS Nearline for SAP BW: Helping Meet the Challenge of Enterprise-Wide Data Explosion Figure 4: Where nearline fits Benefits of Using SAND CDBMS Nearline as Part of the SAP NetWeaver BW Enterprise Data Warehouse Layer Optimized Service Level Management Using SAND CDBMS as part of the SAP NetWeaver BW Data Warehouse Layer significantly reduces the number of DataStore Objects being stored, resulting in shorter time requirements for backups and data loading and thereby enabling better satisfaction of the obligations of Service Level Agreements. Furthermore, specific information can be located by reading only a small portion of the compressed data without going through a resource-intensive staging process, which significantly streamlines data warehouse operations and improves performance. Complete, Accessible History and Corporate Memory When efficient and secure storage is in place, SAND CDBMS extends the SAP NetWeaver BW Data Warehouse Layer by enabling storage of more detailed data including: Detail in its pure form More levels of granularity Infrequently used data Just-in-time data Aged and historical data Legacy data. A More Efficient DataSource SAND CDBMS can act as a very efficient DataSource for Transfer and Update Rules to build new Data Layers within a controlled redundancy architecture, or to derive new InfoObjects. Faster Data Warehouse Consolidation and Migration Because multiple SAP NetWeaver BW Data Warehouses can store their data in one SAND CDBMS repository, it is possible to migrate and consolidate various data warehouses into a single harmonized layer without significantly affecting storage requirements or imposing additional data management constraints. Greater Flexibility in Responding to New Analytical Requirements Having a high-efficiency nearline repository of integrated, detailed data that can be easily restored back into SAP NetWeaver BW or used as a highly responsive DataSource means that new InfoCubes can be built without delay, even when they involve history or new extractions. Furthermore, by providing direct access to DataStore Objects, SAND CDBMS enables SAP NetWeaver BW to respond quickly to unpredictable, one-off user requests, without having to build or activate persistent structures. Reduced Administration Because the number of DataStore Objects being stored in the SAP NetWeaver BW Data Warehouse Layer is reduced, it becomes easier for the administrator to monitor and control overall warehouse processes. 11

11 SAND Technology White Paper Reduction of Total Cost of Ownership Implementing high-efficiency nearline storage brings a reduction in hardware expenditures on disk, main memory and CPUs. System administration costs are also reduced as a result of faster, easier software and release management and reduced backup and recovery times. Enabling Access to Data from More Time Periods with SAND CDBMS Nearline Most SAP NetWeaver BW reports and analyses are based on InfoCubes that organize data in terms of multiple business dimensions. This means that users can analyze information from various business perspectives, such as geographic region or type of sales channel. To meet the increasingly complex requirements of SAP NetWeaver BW users, however, more time periods need to be made accessible. InfoCubes that previously had to be deleted or archived because of the high cost of keeping large volumes of historical data online can now be moved into SAND CDBMS. SAND CDBMS is ideally suited to act as part of the Analytical InfoCube Layer for SAP NetWeaver BW. The nearline storage process can be controlled directly from SAP NetWeaver BW. Less frequently used InfoCubes can be easily moved into SAND CDBMS, then deleted in SAP NetWeaver BW; when required again, nearline InfoCubes can be reloaded into SAP NetWeaver BW or directly accessed using SAP BEx or any certified BI front-end. Example: Using SAND CDBMS as Part of the Analytical InfoCube Layer As an example of the benefits derived from implementing SAND CDBMS as part of SAP NetWeaver BW, consider the situation of an analyst who needs to see accounting data covering an extended period in order to investigate long-term trends. In a traditional scenario, older data might have been archived offline and then deleted in SAP NetWeaver BW, making it very difficult to access. Now, this data can be stored in SAND CDBMS, where it remains available for direct access, as well as for use as an SAP NetWeaver BW DataSource, without negative effects on total cost of ownership (TCO) or system performance. Figure 5: Nearline Storage of InfoCubes 12

12 SAND CDBMS Nearline for SAP BW: Helping Meet the Challenge of Enterprise-Wide Data Explosion Integrating Archiving Solutions into SAP NetWeaver BW to Complete the Information Life Cycle SAP s recommended Information Life Cycle approach involves moving rarely used nearline data into an archiving solution, in accordance with applicable data retention policies. SAND CDBMS works seamlessly with major archiving solutions including EMC Centera (via a direct XAM interface), Open Text s Livelink ECM - Archive Server, Network Appliance (NetApp) NearStore and StorageTek IntelliStore from Oracle to enable such complete Information Life Cycle Management for SAP BI data. Figure 6: SAND CDBMS in the SAP NetWeaver BW Information Life Cycle Organizations today are facing formidable challenges posed by exponential growth in the amounts of corporate data that must be stored and made accessible to increasingly demanding business users, without diminishing the levels of service the organization has committed to provide. By implementing SAND CDBMS as an efficient nearline repository, organizations can empower themselves to overcome these challenges or avoid in the future by relieving the SAP NetWeaver BW database of the burden represented by large amounts of infrequently used data. SAND CDBMS provides very compact storage for all kinds of SAP NetWeaver BW structured data (DataStore Objects and InfoCubes), while still keeping it available for direct querying, fast restoration into the warehouse, or easy use as a DataSource for new InfoCubes/Data Marts or DataStore Objects in other layers. This provides a number of important benefits, including. 13

13 SAND Technology White Paper Conclusion Organizations today are facing formidable challenges posed by exponential growth in the amounts of corporate data that must be stored and made accessible to increasingly demanding business users, without diminishing the levels of service the organization has committed to provide. By implementing SAND CDBMS as an efficient nearline repository, organizations can empower themselves to overcome or avoid in the future these challenges by relieving the SAP NetWeaver BW database of the burden represented by large amounts of infrequently used data. SAND CDBMS provides very compact storage for all kinds of SAP NetWeaver BW structured data (DataStore Objects and InfoCubes), while still keeping it available for direct querying, fast restoration into the warehouse, or easy use as a DataSource for new InfoCubes/Data Marts or DataStore Objects in other layers. This provides a number of important benefits, including: Ability to efficiently meet Service Level Agreements Reduced backup and recovery times Faster data loading processes in SAP NetWeaver BW Greatly improved I/O subsystem performance, speeding query processing for reporting and analytics Quicker, simpler software and release management in SAP NetWeaver BW More History and Greater Granularity, Enabling Construction of a Corporate Memory More information available, enabling more intensive Business Performance Management (BPM) More historical information available for ad hoc analyses and rebuilding of Key Performance Indicators (KPIs) Better compliance with regulatory obligations More information available for Data Reconciliation and Data Audits Optimal use of hardware resources Reduction of hardware costs for SAP NetWeaver BW Lower memory and CPU requirements Reduced system administration costs Better Compliance with Data Retention Requirements Integrated with major archiving solution providers Unlike other Information Lifecycle Management products that focus simply on managing the storage of data on different media, SAND CDBMS is specifically designed to provision data for analytic use, enabling organizations to implement an efficient, intelligent ILM strategy, and to meet the obligations represented by their Service Level Agreements, without increasing administrative workload or data management complexity. 14

14 Appendix A: SAND CDBMS: A Nearline Storage Solution for SAP NetWeaver BW SAND CDBMS Nearline for SAP BW: Helping Meet the Challenge of Enterprise-Wide Data Explosion This appendix provides an overview of the functionality and procedures involved in moving data from SAP NetWeaver BW into SAND CDBMS, where infrequently used, lower-value data and/or granular data contained in InfoCubes or DataStore Objects can be kept economically in a highly compact format while still remaining accessible for querying or usage as a DataSource. Features of SAND CDBMS and the Nearline Interface of SAP NetWeaver BW SAND CDBMS offers a broad range of unique features that set it apart from conventional archiving solutions and nearline architectures Efficient storage of DataStore Objects and InfoCubes from SAP NetWeaver BW, with at least 85% compression Direct access to nearline DataStore Objects and InfoCubes via BEx or any other certified Business Intelligence tool Transparent access to combined data from SAP NetWeaver BW and SAND CDBMS Direct drill-down from InfoCubes into nearline detailed data in DataStore Objects Ability to feed nearline DataStore Objects and InfoCubes back into the SAP NetWeaver BW process when they are required to fulfill new analytic requirements Ability to use nearline DataStore Objects and InfoCubes as DataSources for Data Transfer Processes that derive new InfoCubes, DataStore Objects or Data Layers Integration of all operations into SAP NetWeaver BW Data Archiving Processes (DAP) and Process Chains Can be used in APD processes to derive new InfoProviders or to feed Data Mining processes Supports the remodeling functionality of SAP NetWeaver BW Reduced administration requirements: Automatic storage of designated tables No need to create indexes Any table/column can be queried Automatic creation of SAND CDBMS data structures Automatic adaptation to structural changes in SAP NetWeaver BW Data Archiving Process (DAP): Saving SAP NetWeaver BW Data to Nearline Storage Moving SAP NetWeaver BW data into SAND CDBMS is a straightforward process involving the following steps: In SAP NetWeaver BW: Selection of DataStore Objects or InfoCubes to be saved to the nearline repository within a DAP. The entire data set or a specified slice can be selected. Data can also be partitioned based on the time characteristic, if this is selected as a key field in the structure. Enables: Efficient Storage of Infrequently Used or Aged Data in SAND CDBMS Say SAP NetWeaver BW contains data from the years 2000 to To reduce the size of the database and lower TCO, the data from 2000 to 2003 can be stored nearline in SAND CDBMS, which still allows it to be directly accessed. Tips & Tricks: Depending on the structure of the data model, any dimensions of an InfoCube or the key and value fields of an ODS Object can be used to cut data out of SAP NetWeaver BW. For example, an InfoCube containing 12 months worth of data can be reduced by storing six months nearline, or six InfoCubes each containing just one month of data might be stored nearline without cutting any data out. Partitioning can be defined on more than one Characteristic. The maximum size of Data Packets transferred to nearline storage can also be defined. Managing the Nearline Process in SAP NetWeaver BW: The nearline process can be managed from the InfoProvider Administration screen. Administrators can use the Archiving button to manage nearline requests. 15

15 SAND Technology White Paper The nearline process is executed in multiple steps: locking the selected data area to prevent changes, copying the data to nearline, verifying the data, and deleting the data from SAP NetWeaver BW. Overall status is set to green only after all the steps are successfully executed. Tips & Tricks: The Archiving button is available on the InfoProvider Administration screen only after the DAP (Data Archiving Process) has been successfully created and activated. Before creating the nearline request, the cube must be compressed using the Collapse button on the InfoProvider Administration screen. The locked data area can be released only after the request is restored to SAP NetWeaver BW. The Data Archiving Process can also be run in Test Mode to simulate the results. Process Chains Process Chains are very useful for automating tasks and controlling long-running jobs. It is possible to include the DAP in Process Chains, by adding the Archiving the data from an InfoProvider process type. Enables: Simplified Administration Since the DAP function is completely integrated into existing SAP NetWeaver BW operations, the effort required to administer and monitor these processes is greatly reduced. Tips & Tricks: To avoid spending excessive time creating and maintaining nearline process chains, process chain variants should be stored with rolling time parameters. For example, imagine that a slice of data in an InfoProvider is to be stored nearline after being kept active in SAP NetWeaver BW for 12 months. The rolling mechanism can be used, so that on the first day of the thirteenth month, the appropriate data is automatically moved to nearline storage in SAND CDBMS and afterwards deleted in SAP NetWeaver BW. Delete Data Based on the status setting in the Process Flow Control, InfoProvider data can be deleted automatically after the copy-to-nearline and verification steps are successfully completed. 16

16 SAND CDBMS Nearline for SAP BW: Helping Meet the Challenge of Enterprise-Wide Data Explosion Tips & Tricks: If the nearline request has been scheduled via process chains, automatic deletion of the data from SAP NetWeaver BW can also be achieved by setting the status to No Suspension in the DAP Variant. The explicit data deletion process type is not required to delete the data from SAP NetWeaver BW. APD (Analytic Process Designer) Nearline InfoProviders can be used within an Analysis Process Design to support Data Mining processes or to build new InfoCubes and DataStore objects. Tips & Tricks: To avoid data inconsistency, use of the Query functionality in the APD process is recommended, since data consistency in SAP NetWeaver BW is ensured by means of SAP NetWeaver BW s Query capabilities. SAND CDBMS can also be accessed via JDBC. In this case an OpenHub needs to be licensed from SAP, and data consistency cannot be ensured. Restoring Data From Nearline Storage Restoring data from SAND CDBMS nearline repository into SAP NetWeaver BW is a similarly simple procedure, which can be performed using the Archive button on the InfoProvider Administration screen. Enables: Fast Adaptation to New Analytical Requirements A portion of the SAP NetWeaver BW Data Warehouse Layer will be just-in-time data, efficiently stored in SAND CDBMS. When a new InfoProvider needs to be built using this data, to respond to a new and unanticipated analytical requirement, the required InfoProvider is restored. Tips & Tricks: When restoring nearline data into an InfoProvider, it is not possible to select a subset of the data. Copying Selected Data From Nearline Storage Data can also be copied from nearline storage back into SAP NetWeaver BW (for example, to build an InfoCube and dependent aggregates based on a subset of the nearline data stored just-in-case ). This process is performed using a Data Transfer Process (DTP). Enables: Fast Adaptation to New Analytical Requirements In some cases, only a portion of an InfoProvider is required to build a new InfoProvider to respond to an unanticipated analytical requirement. This functionality makes it possible to restore a selected portion of an InfoProvider. Tips & Tricks: Depending on the structure of the data model, any dimensions of an InfoCube, or key fields and value fields of an DataStore Object, can be used to copy data back into an InfoProvider. 17

17 SAND Technology White Paper The data from nearline storage can be used as a Data Source for SAP NetWeaver BW, via creation of a DTP (Data Transfer Process). Handling Structural Changes to InfoProviders, InfoObjects and Master Reference Tables Since the Virtual Provider always accesses the actual Master Reference Data tables during a query, any changes to values in the Master Data tables will be reflected in the query results. Changes to Data Types of InfoObjects: When the length of a data field is increased, the length of the corresponding field in SAND CDBMS is automatically increased as well. Changes to Characteristics and Key Figures: All changes to Characteristics and Key Figures will be automatically reflected the next time data is saved to nearline storage During queries, any missing Characteristics and Key Figures will be filled with nulls to ensure correct query processing. Accessing Nearline Data Using BEx Data stored in SAND CDBMS can be accessed with SAP s BEx front end or any other certified Business Intelligence tool, in the following way: Select the query against the InfoProvider that contains nearline data. On the Query Properties dialog box, choose Nearline Storage Should Be Read As well Save and execute the query Example: The Sales department wants to see margins per product for the years Data from 2003 can be efficiently stored in SAND CDBMS, while data from is maintained in SAP NetWeaver BW. SAP NetWeaver BW analytical engine provides transparent access to combined data from both data sources, ensuring a consistent view. Direct Drill-Through from InfoCubes to Nearline DSO Data Transparent access to nearline data enables BEx users to drill down from InfoCubes to see the underlying detail data in nearline Data Store Objects (DSO s): Define the query against the InfoCube. At the RSBBS transaction, specify the RRI (Report Report Interface), which allows movement from the InfoCube to the DSO data in nearline storage. On the Query Properties dialog box, ensure that the Nearline Storage Should be Read As Well option is set for both DSO and InfoCube queries. Use the commands on the BEx context (right-click) menu to drill down and navigate through the DSO data. 18

18 SAND CDBMS Nearline for SAP BW: Helping Meet the Challenge of Enterprise-Wide Data Explosion Example: Drilling Down to the Details Data Underlying Aggregations in an InfoCube Consider a scenario where item-level details are stored in a DSO that has been moved to nearline storage, while aggregate values are contained in an online InfoCube. If a user analyzing the sales for one month discovers that the sum value for a particular customer is out of the ordinary, they can investigate by drilling through to the details in the nearline DSO. Transparent Nearline Lookup API The active table of a DataStore object is frequently used as the basis for look-up operations. When this is done (for example, within Data Transfer Processes), direct SELECT statements are used to change or enrich the data within a SAP BI data flow. When nearline storage is implemented, the DataStore object used in a lookup operation will very likely be at least partially resident in nearline storage. In this case, the SELECT statements would normally target only the online portion of the data. Therefore, a Nearline Lookup API is required to transparently combine active DataStore object data from the online and nearline components. The Nearline Lookup API delivered by SAP, outlined in Note , meets these requirements by implementing transparent access to the totality of the active data of a DataStore object. 19

19 SAND Technology White Paper Appendix B: SAND CDBMS Technical Overview Introduction Unlike conventional relational databases, the SAND approach to data management is based on the principle of storing data in columns rather than in rows. Furthermore, in a SAND repository the data is not stored as is ; rather it is first tokenized, and data manipulation works with the tokens instead of the actual values. Understanding how SAND CDBMS works therefore requires a discussion of the column-oriented approach to data management as well as of tokenization. Column-oriented Data Management As an illustration of how a column-oriented approach works, first consider the load process. Instead of loading each customer record one after the other, all the customer IDs would be loaded, then all the customer names, then all the customer addresses and so on. Consider the implications of this for a query environment. It is now very easy to add an index at the same time as the database is loaded. In fact, this can be done automatically. In some cases, the column itself will be the index, which can significantly reduce the need for indexing in the first place. Moreover, this doesn t just apply to data loading but also to retrieval. When data needs to be retrieved, only those columns containing the required information need to be read, rather than each row in its entirety. Suppose, for example, that a query retrieves just 20 characters from a customer record (row) that totals 500 characters: the read requirement will be 25 times smaller when using a column-oriented approach. Moreover, whereas conventional relational databases typically are around three times larger than the raw data size, the reduced need for indexes when using a column-oriented approach means that the resulting data store is much smaller, and therefore requires a more modest hardware investment. Another feature of the column-oriented approach is that it typically requires much less administrative effort. There are two main reasons for this: first, the whole column-oriented approach makes index handling very much simpler; there is no need for the sort of index tuning that is required in row-oriented environments. Second, row-oriented databases use horizontal data partitioning (that is, partitioning across disks, by row), which means that the database can become unbalanced if more records are added to a partition on one disk than others, and this will require significant management (and down-) time to re-balance. By contrast, column-oriented products use vertical partitioning: since each column always has exactly the same number of entries, this form of partitioning never becomes unbalanced, with a consequent reduction in administration requirements. Tokenization Another important characteristic of the column-oriented approach is the use of tokenization. Put simply, tokenization means replacing data values with numeric symbols, which are much less complicated to manipulate. Of course, a value list that translates the tokens to the original values must be maintained, but if there are many instances of a particular value in a column this can result in significant space savings. This in turn will result in substantial reductions in I/O and, therefore, performance improvements. In practice, not all columns have to be tokenized in this way: columns that consist of integers are already, in effect, tokenized, meaning that a value list does not need to be maintained for reference purposes. Compression Perhaps the most important distinguishing feature of SAND CDBMS is the high degree of compression that is applied to data. Consider a set of data stored in the standard fashion, as rows. If the data is to be compressed, this must be done row-by-row. Typically, any one row will contain a mix of alphabetic, numeric and other data, and if this is compressed as a whole, an approach that is most suitable for the average row will need to be used in other words, a lowest-common denominator. WinZip, for example, typically achieves compression rates of around 40%. Contrastingly, when a data set is decomposed into columns, each column can be compressed separately using the best available algorithm for the data type in that column. In other words, compression can be optimized by column. In fact, SAND CDBMS Nearline uses a number of differ- 20

20 SAND CDBMS Nearline for SAP BW: Helping Meet the Challenge of Enterprise-Wide Data Explosion ent algorithms that result in much greater compression rates than would otherwise be possible: the system automatically selects the algorithm best suited to the data in a given column, which has already been converted to a compact format via the tokenization process. The final result is an extremely high degree of compression at least 85% when compared to conventional relational storage. In some instances, compression rates in excess of 95% have been achieved. Architecture SAND Compacted Tables The key feature of the SAND CDBMS architecture is the SAND Compacted Table or SCT. This is a compressed table that contains the data in tokenized form, along with metadata about that table and mapping information indicating where the tokenized data appears in the table. The data can come from any source, as long as it is in tabular format. In contrast to other data compression technologies, SAND CDBMS SCT data can be queried without decompressing it first. Data Partitioning Typically, multiple SCTs will be created for a given table, usually partitioned by date or geographic region or according to some other range-based approach. Each SCT can be loaded independently in parallel, via a separate agent. Moreover, agents can attach to multiple SCTs, enabling an architecture that is not only parallel, but actually grid-based. SCTs may be stored either nearline or offline as required. Metadata The Metadata Repository enables the SCTs to be queried by clients: it stores details about the location of each SCT, when each SCT was created, and a copy of the metadata from each SCT, in XML format. The metadata also includes details such as the number of records in the table, the names of the columns in the table, and the highest and lowest value per field. Thus certain queries can be answered directly on the basis of the metadata, without having to access the data tables at all. When these do need to be addressed, all queries are performed in parallel against the compressed data based on the partitioning mentioned earlier, and the results are merged automatically. Process Automation The SSA Service provides interaction between users and the SAND CDBMS environment. Users can register for data to be moved into the compressed repository or, conversely, restored. The SSA Service has its own scheduling facilities, allowing users to set rules for the automatic storage or restoration of specified data. Platforms SAND CDBMS runs on most popular operating system platforms, including: Windows Linux HP Tru64 UNIX HP-UX IBM AIX Sun Solaris 21