A Better Approach to Managing File System Data

Transcription

1 EMC White Paper A Better Approach to Managing File System Data Lowering Costs, Reducing Risk, and Managing Data Growth Abstract: For most large organizations, file system data is growing at a staggering rate. This growth strains the resources of even the most robust IT departments who must juggle the competing priorities of data access, cost effective storage, and legal and regulatory obligations. This white paper explores these challenges and the comprehensive storage management solutions provided by EMC DiskXtender. Date 8/3/2005

2 Copyright 2005 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC 2, EMC, and where information lives are trademarks of EMC. EMC DiskXtender, EMC Xtender, and EMC DatabaseXtender are trademarks or registered trademarks of EMC Corporation. All other trademarks used herein are the property of their respective owners. All other brand names are trademarks or registered trademarks of their respective owners V3 EMC DiskXtender A Better Approach 2

3 Table of Contents Executive summary...4 File system data: What is it? Where is it?...4 Explosive storage growth for file system data... 4 The ongoing tug of war with end users... 5 Compliance requirements for file system data... 6 Data in litigation discovery: Can you find it?... 6 Drawbacks of current approaches...7 Just add more disk: Increasingly impractical... 7 Inefficient use of high-performance, high-cost storage... 7 Increasing cost of manual processes... 7 Inability to protect and recover within shrinking windows... 7 Backups as archives limit access... 8 Requirements of a practical solution...8 Embracing data growth... 8 Ensuring retention and compliance... 8 Optimizing storage costs by leveraging tiered storage... 8 Putting the right data in the right place at the right time... 9 EMC DiskXtender software solutions...10 Data movement policy framework: Four key factors EMC DiskXtender products...11 Heterogeneous support Application integration Reducing storage TCO with EMC DiskXtender solutions...13 Savings in primary storage costs Savings in backup time, recovery time and tertiary storage costs Conclusion...15 EMC DiskXtender A Better Approach 3

4 Executive summary Data growth continues to be a challenge for IT and storage administrators. Data is often subject to longterm retention and special storage requirements driven by regulatory compliance, litigation, corporate governance, and operational business needs. These factors are compelling many companies to retain complete and accurate business records, governed by strict retention rules, for long periods of time. In turn, changes to retention policy are substantially increasing data storage complexity. Organizations face significant practical challenges in managing the rapid growth of data including shrinking backup and recovery windows, growing management overhead costs, and risks of noncompliance. IT managers are constantly faced with decisions about what files to save and where. These challenges create huge operational headaches for IT, stress the storage infrastructure, and increase costs and risks. Enterprises need a better solution for managing all their data. The way to reduce costs, reduce risks, and ensure compliant retention for file system data is to leverage a tiered storage infrastructure in tandem with EMC DiskXtender software solutions. EMC DiskXtender for Windows and EMC DiskXtender for Unix/Linux provide automated, policybased data movement with transparent access for Microsoft Windows, UNIX, and Linux-based file systems, as well as for data residing on network attached storage (NAS). By using DiskXtender products within a tiered storage architecture, including content addressable storage (CAS) and optical or tape media, enterprises can implement effective information lifecycle management (ILM) strategies. EMC DiskXtender solutions enable organizations to meet data retention requirements, fulfill access demands, and reduce storage total cost of ownership (TCO) through lower primary and secondary storage costs, faster backup and recovery, and decreased complexity and management overhead. File system data: What is it? Where is it? File system data includes all data that is not managed by databases and other applications such as . It commonly includes Microsoft Word, Excel and PowerPoint files, as well as hundreds of other file types such as HTM, JPEG, and PDF files. This data exists in file systems located on desktop and mobile computers and corporate file servers. It is typically created and managed by end users and is commonly copied and moved within the IT network for sharing purposes. EMC DiskXtender addresses the needs of file system data movement and archiving. In addition, organizations typically maintain structured data in databases and semi-structured data sets managed by applications such as and messaging software. EMC provides a full range of ILM solutions for all these data types, including EMC Xtender for , EMC Documentum products for document and content management, and EMC DatabaseXtender for Oracle application data management. While many organizations use EMC DiskXtender (2000) software in combination with other EMC software for semistructured data, this white paper focuses on the benefits of DiskXtender software solutions for managing unstructured data. Explosive storage growth for file system data Many organizations are experiencing an explosive growth of file system data, creating massive challenges in terms of storage cost, business risk, and IT management complexity. Three main business drivers are propelling this rapid growth: End-user productivity needs for data retention and retrieval Regulatory compliance requirements for record preservation and access Litigation discovery demands and litigation-readiness strategies EMC DiskXtender A Better Approach 4

5 The following sections outline some of the challenges and implications of these three business drivers, the strengths and weaknesses of traditional solutions, and the superiority of DiskXtender as a way to manage this data over its lifecycle. The ongoing tug of war with end users End-user productivity is a major business driver of data growth. Employees are accustomed to managing their day-to-day work with messages, reports, presentations, and spreadsheets. In a typical day, an employee creates dozens of new files. When this behavior is multiplied by thousands of users in an organization, the daily growth of data is staggering and the demand it places on the IT infrastructure is enormous. One common solution is to restrict file storage on corporate servers by imposing storage quotas on users, effectively pushing these data files back onto the employee desktop. Unfortunately, data files on personal and local file systems are less likely to be backed up or preserved for productivity and organizational memory. Many IT organizations are engaged in a constant tug of war with end users over storage of data in the corporate infrastructure. End users want to have unlimited storage for their data, while IT managers want to reduce the burden on storage infrastructures and budgets. This problem is magnified by the fact that often over 80% of data is inactive. Older files in particular are rarely accessed, but they continue to consume expensive disk storage resources. (See Figure 1). Figure 1. Typical Enterprise Data Growth This 80/20 ratio for data is easy to understand. We are all familiar with how quickly we stop using files from one day to the next. We store these files on servers or our desktop machines for future reference, and do not delete them because they represent vital information such as customer correspondence, contracts, company reports, meeting presentations, personnel records, budget forecasts, and financial audits. Many of these files may never be opened after they are stored, but we cannot readily predict which ones will be needed later. As a consequence, we save them all fueling the enormous growth of data stored on the corporate IT infrastructure. IT organizations have traditionally attempted to solve the data growth problem by deleting older files to make room for new ones. Administrators may delete files or folders from file servers, with or without user approval, and may also force users to delete files by imposing storage quotas. Unfortunately, simple deletion-oriented approaches are likely to backfire. Deletion policies reduce productivity as they force users to manually copy their files to alternative locations. Deletion policies can also cause regulatory compliance failure and increase the company's risks and costs during legal proceedings. The fact is that users continue to save their data while deletion-oriented policies EMC DiskXtender A Better Approach 5

6 simply force this behavior underground. Users simply move their files to local disks or offline media. In a regulatory investigation or during litigation discovery, an organization can be forced to search for, locate, recover, and organize those files an expensive process. And the company may face fines, penalties, and adverse judgments if the resulting compilation proves incomplete or inconsistent. Compliance requirements for file system data Regulatory compliance is a major factor forcing longer retention of electronic documents and data. Heightened government scrutiny of business processes and records, and new rules for data retention, access, and security are increasing the amount of data that companies must keep. Under these rules, companies must retain files, such as purchase orders and contracts, customer complaints and claims, production records, personnel forms and reports, medical image files, and many others. In the US, the Sarbanes-Oxley Act of 2002 and the Consumer Protection Act and the Health Insurance Portability and Accountability Act (HIPAA) are examples of laws and regulations that require organizations to preserve documents and data for specified periods of time. Other industry-specific examples include the Gramm-Leach-Bliley Act for protection of customer data in the financial services industry and SEC rules for record keeping in securities trading firms. Regulatory compliance, combined with business productivity, leads end users and organizations to retain more data. Compliance requirements can also prevent near-term deletion of affected data files, once they are created. The required retention periods vary from months to years or even decades, with seven years being a very common standard. Many regulations also require companies to implement technical capabilities to ensure the integrity, accessibility, and/or privacy of these records during the required retention period. Thus, by expanding the scope and duration of record retention requirements, compliance has become a major factor in data growth and IT spending. Data in litigation discovery: Can you find it? Litigation discovery is a third driver for longer retention requirements. Electronic records are increasingly targeted by court subpoenas during discovery, and the cost of a complying with a single discovery request can quickly run into many thousands of dollars if the records must be retrieved from backup tapes and desktop file systems. Also, companies are required to suspend any manual or automatic deletion of potentially relevant files, as soon as they become aware of a possible lawsuit or investigation. A deletion hold can further stress the storage infrastructure, especially if all the data has been maintained on high-cost disk storage. The adverse consequences of failing to comply with a discovery order can be even more severe. Consequences can include fines, adverse jury instructions, loss of a lawsuit, or even criminal penalties. Failure includes inability to locate all the documents required by a court order, especially if potentially relevant files were deleted after the company became aware of an investigation or likely litigation. Even slow response, resulting from inefficient storage and retrieval processes, can be construed as obstruction or failure to comply. To reduce the chances of a smoking gun document, some companies want to dispose of electronic records as soon as the laws and regulations permit. Other companies and their legal advisors believe that maintaining a complete business record is the best defense. Since it is extremely difficult to guarantee that all copies of an electronic document have been found and deleted at expiration, the legal team would prefer to have a complete record that frames "smoking gun" documents in a more benign context. If someone else is likely to have a copy, the legal team wants to have one too. A complete record enables them to prepare their case without fear of surprises and it provides a more credible body of evidence for presentation at trial. Whatever policies a company chooses regarding retention and deletion, managing a massive volume of data in compliance with those policies requires effective technology solutions. EMC DiskXtender A Better Approach 6

7 Drawbacks of current approaches The current practices for managing file system data takes many forms, none of which are 100 percent effective. These traditional approaches generally reflect policies developed for handling paper records or much smaller volumes of electronic files and are insufficient for today's regulatory, legal, and business environment. Just add more disk: Increasingly impractical As file system data is created and stored on file servers, file systems must be enlarged and more disk space must be provisioned and allocated, increasing storage costs. In many organizations, data files are kept on relatively high-cost disk storage. But while the cost of disk has dropped significantly, the apparent simplicity of a "just add more disk" approach can lead to overwhelming complexity and management overhead costs that go well beyond the cost of physical storage. And then there's backup. When historical files are kept on primary storage, they are generally included in full backups along with more recent files even though they have not changed and have been backed up many times before. This leads to ever increasing backup times and associated costs, as well as longer recovery times and increased risk of recovery failures. Inefficient use of high-performance, high-cost storage All data is not equal, so it does not all belong on the same class of storage. Infrequently accessed files cause inefficient use of high-performance disk storage. These files generally do not require high-end storage capabilities such as fast access and retrieval, snapshot copies, and continuous replication to remote data centers for disaster recovery. Without processes and technology to manage and control the location of unstructured data files, these files can end up just about anywhere. Network file servers that have disk space allocated on a corporate storage area network (SAN) are a prime example. End users typically specify a file system and directory when files are first created, but do not have the time and skills to manually move these files to less costly storage locations as they age. Increasing cost of manual processes Administrators can take manual steps to reduce the total number of files by deleting files or moving them to tape. These manual processes are time consuming and costly in terms of IT resources. Often, administrators must consult with the end users who own the files, further consuming resources and reducing employee productivity. Moving files manually is also error prone files may be mislocated, mislabeled, or deleted. The net result of unmanaged data growth is a ripple effect that starts with increased disk consumption and ultimately impacts backup operations and other IT processes, putting greater stress on the IT infrastructure. Inability to protect and recover within shrinking windows Common backup practice is to perform a weekly full backup and daily incremental backups. Individual files or entire file systems can be recovered from a combination of the full backup and incremental backup tapes. This process is constantly challenged by file servers that contain growing amounts of data. As new files are created, backup processes take longer and longer. For many organizations the limit has already been reached the backup window does not allow enough time to complete full backups. The result is incomplete backups and forces the IT staff to backup servers selectively based on perceived priority. It also increases risk of data loss with potentially severe consequences. The resulting backup data sets are larger, more expensive to store, and more difficult and time consuming to restore when needed. Longer recovery times can result in longer application downtime, lower service levels for employees and customers, and substantial cost to the organization. EMC DiskXtender A Better Approach 7

8 Backups as archives limit access Organizations have business, regulatory, and legal needs for well-organized data archives, where specific records can be located and retrieved quickly and efficiently. Backup applications are designed for a different purpose: data recovery from unplanned events such as device failure or accidental file deletion. When using a backup application for archival purposes, data is copied to tape and then deleted from primary storage, thus removing the data from a users normal file system view. To access archived data a user must: 1. Be aware the data they need is stored on a backup tape 2. Engage the assistance of a backup administrator Often users must also sort through a large restore volume to obtain the files needed. This leads to long access delays, unhappy and unproductive users, and a drain on IT staff resources. Because backup applications are not intended to fulfill the increasing anytime, anywhere data access needs of the business, more data is left on primary storage and storage management problems grow. Requirements of a practical solution A practical solution to growing data problems will embrace and enable data growth, meeting retention and access requirements while optimizing storage costs. Embracing data growth Since end users are attached to the desktop applications that drive the creation of new data, the growth of data cannot be stopped. These users also require access to network file servers to allow file sharing. Such sharing can increase employee productivity and reduce the number of files that travel through the company server as attachments. A practical solution will therefore work seamlessly with existing file servers and the resident file systems to enable easy assess to data. The most effective strategy is to give users what they want unlimited storage capacity, easy access, and content preservation. Ensuring retention and compliance Laws and regulations require enterprises to preserve important business documents for months or even years. The IT challenge is to manage the many thousands of files that employees create and to preserve each of them for the proper retention period. In most situations, employees are not aware that the files they create must be preserved. A practical solution will assist in the preservation of these files by keeping the files in a protected central repository while maintaining full access for authorized users, administrators, and regulators. Where regulations or business needs require tamper-proof records, a compliant solution will preserve the data files on appropriate storage such as WORM media or content-addressed storage (CAS) systems. Optimizing storage costs by leveraging tiered storage The value of data and its inherent access requirements often change over time, in predictable and lesspredictable ways. Figure 2 illustrates an example of the changing access demands for data files associated with a new purchase order. At the time of the transaction, sales and customer service staff create new documents that are frequently accessed while the order is being processed. Once the product ships, these documents must remain readily accessible for a few weeks in the event of questions about the order, shipment problems, etc. After that time, the documents will be kept for the length of the warranty period (at minimum) but access activity will be low (just the occasional call if something breaks). As long as a customer support person can pull up order information within in a few minutes, the response time is acceptable to the business user and customer. What happens, however, to this access pattern if a significant product flaw is found, necessitating a recall? It may be advisable to keep data beyond the warranty period; in specific industries this may be legally EMC DiskXtender A Better Approach 8

9 required. When an investigation or recall commences, the customer order information is once again in high demand, with frequent access by customer service and legal staff. Figure 2. Data Lifecycle Example for Customer Order Records Putting the right data in the right place at the right time The product order retention example illustrates the different access requirements that apply to a file during its retention period. It also underlines the inefficiencies and unnecessary overhead costs that often exist by not storing data appropriately over its lifecycle. A practical solution will effectively leverage a tiered storage architecture. It will keep files resident on high-performance RAID (using DAS, NAS, or a SAN) while access needs are high and move files seamlessly to secondary storage when access needs decrease. Multiple types of disk storage (Fibre Channel, SCSI, ATA, and CAS), tape, and optical storage can make up a tiered storage infrastructure. Each storage type offers a different combination of cost, performance, reliability, and content preservation. The process of matching the right data to the right storage at the right time is a core component of an information lifecycle management (ILM) strategy. Automated, policy-based data movement capabilities are central to a tiered storage architecture and facilitate file movement to more cost-effective and/or compliant storage, and provide direct read from or movement back to primary storage when archived files become active again. Figure 3 illustrates the movement of data among tiers of storage as driven by cost and/or storage performance. EMC DiskXtender A Better Approach 9

10 Figure 3. Data Movement across Tiered Storage EMC DiskXtender software solutions EMC DiskXtender software solutions provide the required capabilities for managing unstructured data files efficiently. They leverage tiered storage to enable organizations to reduce primary storage costs as well as the related costs of replication, management, and backup. DiskXtender puts the right data on the right storage at the right time through automated, policy-based data movement, while maintaining transparent access. It provides efficient, transparent access wherever data resides at a given time to optimize end-user productivity, reduce legal and regulatory risks, and minimize administrative overhead. DiskXtender preserves data integrity and accessibility, and helps organizations align their storage infrastructures with business requirements. EMC DiskXtender enables the enterprise to apply ILM strategies to unstructured data files. (See Figure 4.) Figure 4. Information Lifecycle Management EMC DiskXtender A Better Approach 10

11 Data movement policy framework: Four key factors DiskXtender solution strategies reflect a data movement policy framework based upon four key factors, as illustrated in Figure 5. These factors include business/it demands, data retention requirements, data access needs, and the capabilities of the storage tiers. Business/IT demands include the policies and objectives of business and IT management, which reflect the challenges of managing data files through the entire data lifecycle. These challenges include reducing storage costs, backing up and recovering faster, ensuring storage compliance, and expanding protection. Data retention requirements are imposed by external regulations and internal policy and define what data is saved and for how long. They may also include special requirements to protect the data from alteration, such as tamper-proof storage. Data access needs reflect the access patterns and service levels required by users and applications, including the frequency of access and the required speed of response to an access request at each stage of the information lifecycle. Capabilities of the storage tiers include the performance and cost parameters of the many types of disk, tape, and optical storage. Figure 5. Data Movement Policy Framework EMC DiskXtender products EMC archiving solutions for file systems consist principally of EMC DiskXtender for Windows for Microsoft Windows file systems and EMC DiskXtender for UNIX/Linux for UNIX based and Linux file systems. DiskXtender software products also provide solutions for data residing on EMC Celerra. DiskXtender products perform automated, policy-based data movement of data files across tiers of storage, using rules that can be created and executed quickly. No longer are manual processes required to retire EMC DiskXtender A Better Approach 11

12 inactive files to secondary storage or place data on special secondary devices to meet compliance requirements. DiskXtender s file movement rules engine based on user, file size, age, type, date of last access, or other file attributes allow for flexible data classification and heterogeneous storage alignment. (Figure 6). For example, age-based rules may be most appropriate for general-purpose files that are not being used but are consuming valuable primary storage resources. In contrast, rules based on file type can be implemented for financial data that must be preserved for compliance and stored on tamper-proof CAS, write-once tape, or optical media. Administrators can define combinations of rules to address various situations that require access to unstructured data files, enabling the organization to effectively manage data movement and access requirements, storage costs, and capacity needs. Figure 6. EMC DiskXtender Software DiskXtender accomplishes data movement in a manner that is transparent to end users. Files are viewed in the same file system location and are accessed in the same manner. After migration, files on primary storage are replaced with small file stubs that point to the entire file at its new location. (See Figure 7.) As a result, file systems are effectively relieved from the growing burden of inactive data files consuming high performance disk resources, thereby reducing storage costs and improving backup and recovery speeds, while maintaining end-user productivity. EMC DiskXtender A Better Approach 12

13 Figure 7. Reclaim High Performance Disk Capacity and Reduce Backup Volume Heterogeneous support DiskXtender products provide automated data migration for Windows, UNIX and Linux operating systems as well as for data residing on NAS servers such as EMC Celerra. IT environments also have the flexibility to utilize heterogeneous disk, tape, and optical devices from a broad range of storage vendors. Application integration EMC recognizes the importance of application integration for data movement in tiered storage and has developed APIs that allow applications such as archiving, content management, third-party healthcare applications (such as PACS and HIS), and third-party financial applications to interface directly with DiskXtender. These integration capabilities further enhance the benefits of DiskXtender products as key enablers for enterprise ILM. Reducing storage TCO with EMC DiskXtender solutions EMC DiskXtender enables enterprises to reduce overall storage total cost of ownership by migrating inactive data files from high-cost primary storage to more cost-effective secondary storage. To realize the full benefits of DiskXtender solutions, IT organizations need to understand the physical cost savings associated with alternative storage tiers, savings in backup and recovery time, and overall savings in storage management overhead. Savings in primary storage costs Figure 8, illustrates the relative costs of several storage tiers, arranged in a traditional storage hierarchy. In a well-managed storage infrastructure, only a small percentage of the data is maintained on the highestperformance, highest-cost storage. Less critical data is allocated or migrated to lower-cost storage as appropriate. EMC DiskXtender A Better Approach 13

14 Figure 8. Tiered Storage Costs Primary disk storage can cost $50 per GB, or more for high-end RAID storage systems when each storage volume is mirrored and replicated for high availability and disaster recovery. Each file stored on such a volume will participate in the cost structure for that tier, even when it no longer needs high performance and rapid recovery capabilities. In contrast, low-performance disk costs as little as $10 per GB for ATA disk storage. Tape storage remains the least expensive and costs as little as $1-4 per GB. Enterprises can reduce storage costs by using DiskXtender to move data files from high-end to low-cost storage. For example, if 5 TB of data are placed on high-end storage at a cost of $50 per GB, it is consuming $250,000 of storage. If 80 percent of the data (4 TB) is inactive, the IT organization can save $160,000 in storage investment by moving that 4 TB to low-end disk at a cost of $10 per GB. Savings in backup time, recovery time and tertiary storage costs Considerable savings in backup and recovery time can be achieved using DiskXtender. Using the example above, assume that it normally takes 10 hours to backup the 5 TBs of data from high-end storage. However, if 4 TBs of inactive data is moved to secondary storage, the primary backup volume will be reduced to approximately 1 TB (active data plus stub files). This reduces the backup time from ten hours to two hours. Conversely, if it took 14 hours to restore the 5 TB file system, moving the inactive data to secondary storage will reduce the restore time to 2.8 hours. In the event of a restore, only active files and stub files need to be rewritten to primary storage. Tape costs are driven directly by the number and size of the files in each backup data set. For example, a full backup of 5 TBs of data would consume $10,000 worth of tape media ($2 per GB). For routine weekly backups with a standard six-month retention schedule, the annual tape cost would be $260,000. If 80 percent of the data files that are inactive are migrated, the full backups of the active data (1 TB) would require less tape media and cost only $52,000 per year, a cost saving of $204,000 per year depending upon tape recycle/reuse policies. These examples illustrate the immediate and measurable return on investment that organizations can realize using DiskXtender software solutions in a tiered storage environment. EMC DiskXtender A Better Approach 14

15 Conclusion EMC DiskXtender solutions within a tiered-storage infrastructure offer an effective, comprehensive strategy for managing growing volumes of data. It enables organizations to fulfill ever increasing access requirements and meet legal and regulatory data-retention demands while gaining significant savings in storage TCO. DiskXtender for Windows and DiskXtender Unix/Linux products provide the required capabilities for managing file system data files efficiently in terabyte to petabyte-size environments. Through automated, policy-based data movement, DiskXtender puts the right data on the right storage at the right time. Data integrity and transparent accessibility is preserved as IT organizations efficiently align heterogeneous storage resources with changes in the value of data. EMC DiskXtender solutions enable enterprises to effectively apply ILM strategies to file system data. Businesses can ensure compliance with regulatory, legal, or internal requirements for electronic record retention. File systems can be relieved of the growing burden from inactive file storage, thereby lowering storage costs, improving backup and recovery speeds, and reducing management overhead. EMC DiskXtender A Better Approach 15

16 About EMC EMC Corporation (NYSE: EMC) is the world leader in information storage systems, software, networks, and services, providing automated networked storage solutions to help organizations get the maximum value from their information, at the lowest total cost, across every point in the information lifecycle. Information about EMC s products and services can be found at For more information about EMC storage management products, visit or call (outside the U.S.: ). EMC DiskXtender A Better Approach 16