Agile Information Life-Cycle Management: Controlling Data Growth and Managing Complexity with Virtual Databases

Transcription

1 I D C V E N D O R S P O T L I G H T Agile Information Life-Cycle Management: Controlling Data Growth and Managing Complexity with Virtual Databases April 2011 Adapted from The Critical Role of Database Archiving in Information Life-Cycle Management by Carl Olofson, IDC # Sponsored by Delphix When IT organizations think about information life-cycle management (ILM) and storage, they are usually referring to the use of tiered storage for inactive data such as backups or archives, or subsetting or masking test data. ILM should be concerned with all stages of the information life cycle; yet most enterprises, while seeking to manage storage for the end of the life cycle, do not have a strategic approach to managing storage for data at the beginning or in the middle of that life cycle. Instead, they spend inordinate amounts of storage resources (up to 50% of all replicated storage capacity deployed) and staff time in managing development and test databases. Often, the challenges in dealing with storage assets in these projects can significantly delay project timelines and threaten budgets. This Vendor Spotlight examines the seldom discussed dimension of ILM that focuses on the improved use and reduced management of storage assets in development and test database environments. This paper also looks at the role of virtualized data solution vendor Delphix in enabling the creation of private database clouds to boost the effectiveness of data management in the early and middle stages of the application life cycle. Data Inflation Raises Risk in Test and Development Environments Most enterprises suffer from something that might be called "data inflation." Put simply, this means that they replicate data for development and test purposes and use more storage than is required by their production systems sometimes as much as eight times more. Specific causes of data inflation tied to test and development include:! Application upgrades, updates, and patches, such as SAP, Oracle E-Business Suite, PeopleSoft, Siebel, and so forth! Database upgrades, updates, and patches, such as Oracle, Microsoft SQL Server, and so forth! Custom application development! Break/fix, troubleshooting, or tuning to deal with regular failures and maintenance and performance issues! Operational reporting to offload production transaction-oriented databases The replicated data overload, in turn, can lead to a kind of "risk inflation." The organization takes on the inherent risk associated with managing these proliferating and often untracked copies of data sets. It's also hard for an organization to keep the data optimally refreshed, introducing risks in production rollouts. For example, in test/development environments, using old or nonrepresentative data sets can jeopardize the timing and success of an overall project. IDC 1116

2 In addition, every replica database must be set up or refreshed with current data by a database administrator (DBA), often together with system and storage administrators, every time such a database is allocated or data sets are refreshed, which is a drain on valuable staff time. Because the replicas are deployed temporarily and then taken down so the same physical resources can be used for other projects, this activity can represent the expenditure of hundreds of staff person hours performing almost the exact same set of tasks dozens of times every year. Finally, faster business cycles and increasing reliance on applications are driving an increase in the number of database provisioning and refresh tasks each quarter. Cost and Complexity Associated with Excessive Storage Management In a world where customers and governmental organizations expect companies to quickly deliver the right information to the right person in a timely fashion, control of information assets is critical. Organizations need to know that information is secure from inappropriate use, that information is protected from permanent loss or extended unavailability, and that information is easy to rediscover and reanalyze. ILM products generally include data masking software to help address this problem. With regard to data security, organizations address concerns through the use of data masking, but this can create additional management and storage use problems, if not handled correctly. The underlying storage technologies used by IT teams to meet these goals are quite diverse. Replicated data (also known as copies of active data) for data protection and test/development is an excellent example of both the challenges and the opportunities facing IT executives with regard to storage systems. Between 2005 and 2010, the annual amount of new disk storage capacity deployed in datacenters to store replicated data sets grew from just over 918 petabytes (PB) to almost 4.3 exabytes (EB). By 2014, annual deployments will surpass 10EB (see Figure 1). This growth represents all kinds of storage (DAS, SAN, and NAS). While the shift to disk-based backup accounted for some of this growth, data inflation in test/development environments was just as important a contributor. In fact, from a manageability standpoint, it is even more critical as test and development environments often drive proliferation of data set copies (increasing storage inefficiency) and storage administrator workloads (slowing test/development cycles) IDC

3 Figure 1 Impact of Replicated Data on Storage Capacity Demand 9.0 Enterprise Disk Storage Used for Replicated Data Disk storage capacity shipped (exabytes) Source: IDC's Enterprise Disk Storage Consumption Model, December 2010 IT organizations want solutions that work with existing storage environments while balancing security, cost, and business requirements. Such solutions must focus on the information life cycle of data in test/development environments. They must enable more centralized and reliable replication of test/development data, yet they must also ensure that DBAs and developers have timely access to data sets without the need for constant intervention by storage administrators. What Is ILM? ILM, according to the Data Management Forum of the Storage Networking Industry Association (SNIA), is: The policies, processes, practices, services and tools used to align the business value of information with the most appropriate and cost effective infrastructure from the time information is created through its final disposition. Information is aligned with business requirements through management policies and service levels associated with applications, metadata, and data. [Source: ILM Definition and Scope: An ILM Framework SNIA DMF 28 July 2004] ILM is realized through both business and technical staff policies and through the use of technology that can track and govern the creation, modification, distribution, and disposal of data of all kinds, including both structured and unstructured data. Key elements of that technology include data security, auditing, cost-effective test/development data management, and archiving software IDC 3

4 In the area of test/development data management, most ILM efforts have focused on the isolation of inactive data into an archive or low-tier storage to improve database performance while reducing cost or on the data itself: preserving security rules and providing data masking. Another aspect of this problem exists, however, that has to do with the cost of managing the storage for development and test databases. Dynamic Database Provisioning The storage industry has long developed products in the area of ILM, but most of the industry focused on stewardship of the physical data, generally at the file level. This file-level perspective makes it difficult for disk storage systems to provide ILM where database data is concerned. If an organization uses file-based NAS, the finely articulated elements within the database can't be seen or discretely managed. In effect, the database is opaque from the point of view of any software that does not operate within that database's schematic context. If an organization use block-level storage, the challenges are even worse. Block-level storage provides even less visibility and requires much greater intervention by the storage administrator each time developers need more capacity. Both storage approaches also suffer from the replication problem. Developers often create/require multiple versions of the same data sets (as well as more with small modifications). In most file- and block-based storage systems, these copies consume massive amounts of space, driving down effective utilization levels for storage capacity to less than 20%. To provide an effective means of managing and controlling the staffing and hardware costs associated with development and test databases, the storage facility must be designed specifically for this purpose and must "understand" how the database works. A virtualized data facility provides a database management system (DBMS) with storage functionality that is entirely consistent with normal physical storage operations from the database server's point of view but actually stores and retrieves that data in ways that are designed to optimize storage for a particular use case. It includes technology that is designed to minimize storage use by taking advantage of the fact that the development and test databases are really just variants of the production database. Knowing that all development or test versions of a database are about 90 95% duplicated enables the virtualized data system to dramatically reduce the storage footprint by making just one copy of the database, then recording the variant elements or "deltas" for each test and development virtual instance. It's also possible to add compression, enabled by the relatively lower performance requirement of a development and test database. One example of the potential benefits of virtualized data is a test/development environment that needs to create eight different versions of a database over the life of the project. Instead of having to set up and maintain eight full-scale databases on a storage system, the storage administrators have to set up only one pool that is just 10% to 15% larger (not 800% larger) than the original database. Equally important, once the base copy is established, each instance of a development or test database can be established and reset instantly by the development team with no storage administrator involvement, saving untold hours of IT staff time and speeding the development cycle. Laying the Foundation for Database Life-Cycle Management The database industry and its users are evolving toward even more manageable, efficient, and reliable methods of data management. Part of this trend is embodied in the maturation of ILM to include methodologies and techniques that ensure responsible, secure, and efficient management of data at each stage of its life cycle, including database development and testing, regular database operations, and management of inactive (or archival) data. Up to now, most approaches to managing test data sizes and content have included data subsetting, which involves using a software tool that selects subsets of production data to use for test/development to reduce the total storage size of the IDC

5 test/development database; test data generation, which involves generating imaginary data based on some criteria, again as a way to streamline the test/development database setup and reduce size and cost; and storage system based snapshot technologies, which involve using single-instance snapshot functions supported in select disk storage arrays. These systems logistically store many different copies of the data sets but physically store only one copy plus any variations. However, these approaches have limitations:! Data subsetting and test data generation in general yield databases that do not truly reflect the distribution of various sorts of data, resulting in an unrepresentative test that neglects to expose application operations that can fail when applied to the real database.! Storage system based snapshot technologies are challenging to use in the test/development environment because they operate at the block or file level rather than the database object level. IT organizations must look at solutions such as virtualized data for databases as the pace of test/development activities accelerates and the pressure to reduce underlying infrastructure and management costs grows. Considering Delphix Delphix Corp. of Menlo Park, California, provides virtualized data solutions to solve the problem of optimizing development and test database deployment for Oracle Databases. Because virtualized data solutions must interact with the database server in precisely the manner expected by the DBMS, such technology must be designed specifically for a particular DBMS. Developing such software requires an intimate knowledge of the inner workings of the DBMS in question. The Delphix technical team includes engineers who have worked for Oracle on the Real Application Clusters (RAC) and Active Data Guard features of Oracle Databases and are very familiar with Oracle internals. Delphix provides virtualized data software that is specifically designed to optimize the process of standing up and standing down development and test databases as well as the ability to easily refresh data while minimizing the underlying storage resources required to support such databases. Delphix does this by building a base of data from the production database. Such development and test data for an existing system is an exact copy of a production database at a point in time. Delphix can also reduce the burden associated with the data security issue by linking to a source database and then running a script to mask data as part of the virtual data provisioning process. According to the company, Delphix continues to update its server with change blocks and logs to ensure that customers can open virtual copies from any point in time and refresh with as much frequency as needed without impact to production databases. Multiple parallel development and testing tasks often mean that such data may be replicated many times. Knowing this can enable the virtualized data system to optimize such storage use to reduce the overall storage footprint as well as the time it takes to stand up and stand down a development database. With Delphix, developers and QA no longer need to share a single database instance; rather, they can have their own dedicated instance. In addition, more copies of databases can be created to ensure that all testing requirements are met without exhausting storage infrastructure. The company is now working to expand its reach by offering versions of the Delphix solution that serve other DBMS products besides Oracle Databases IDC 5

6 Challenges Delphix faces little competition in this area right now, but competitors could well emerge. It is also possible that changes in storage architectures and approaches could alter the economics of the Delphix solution. The major challenge that Delphix faces, however, is in educating the user community regarding the company's approach and its benefits. Conclusion The provisioning and management of development and test databases remains an area of astonishing waste in terms of both storage and staffing costs at many organizations. Standing up and standing down many copies of the same database, each with its own storage, is unnecessarily expensive in terms of both storage cost and staff time, and often the resulting inflexibility inhibits organizations from doing the level of development and testing they need to do to ensure high-quality databases and applications. Data refresh has also been difficult to keep up without taking away resources from production database maintenance. While companies have evaluated a number of approaches to alleviate this problem, none of the approaches have wide applicability. The virtualized data approach of Delphix provides a simple and effective way to overcome these obstacles and enable rapid, nimble deployment of multiple development and test databases against a single production system with minimal storage use and effort. A B O U T T H I S P U B L I C A T I O N This publication was produced by IDC Go-to-Market Services. The opinion, analysis, and research results presented herein are drawn from more detailed research and analysis independently conducted and published by IDC, unless specific vendor sponsorship is noted. IDC Go-to-Market Services makes IDC content available in a wide range of formats for distribution by various companies. A license to distribute IDC content does not imply endorsement of or opinion about the licensee. C O P Y R I G H T A N D R E S T R I C T I O N S Any IDC information or reference to IDC that is to be used in advertising, press releases, or promotional materials requires prior written approval from IDC. For permission requests, contact the GMS information line at or gms@idc.com. Translation and/or localization of this document requires an additional license from IDC. For more information on IDC, visit For more information on IDC GMS, visit Global Headquarters: 5 Speen Street Framingham, MA USA P F IDC