Delivering an Effective Backup and Recovery Service: How a Proactive Approach Can Be a Real Life Saver

Transcription

1 Symantec Global Services Delivering an Effective Backup and Recovery Service: How a Proactive Approach Can Be a Real Life Saver Who should read this guide: CIOs, CTOs, and IT directors who are managing backup and recovery operations Operational budget holders Advice offered about: Why backup and recovery is critical to the business The most common mistakes Getting it right, with case examples Comparison of in-house and outsourced approaches Confidence in a connected world.

2 This guide examines why backup and recovery is critical to the business and how a proactive approach can be the key to backup success.

3 How a Proactive Approach Can Be a Real Life Saver Contents Introduction...5 Why is backup and recovery critical to the business?...7 Tackling common problems with backup...9 Tackling common problems with recovery Winning board backing for investment Stick or twist? What to consider when outsourcing Consultants have been there and done that Symantec Managed Backup Services Case example What our clients say About Symantec Global Services About Symantec

4 Our contributors Technical infrastructure manager, leading financial services firm We spoke to a technical infrastructure manager working for a leading financial services firm. For corporate reasons, his contributions are anonymous. Service director, managed service provider, public sector program We interviewed a service director working on a high-profile public sector engagement. For corporate reasons, his contributions are anonymous. Archie Maddocks, Senior Service Delivery Manager, Symantec Global Services An IT professional with 18 years experience working in operations, Archie has worked on FTSE 100 and government contracts and has specialized in storage-related service delivery for the past eight years. When it comes to running backup services, Archie has been there and done it. Adrian Spink, Head of Technical Architecture & Managed Service, Symantec Global Services After nearly 20 years of running IT infrastructure services in the automotive, oil and gas, telecom, and investment banking sectors, Adrian brings broad experience to this guide. A track record of delivering for clients like JPMorganChase and Sun Microsystems, combined with inside knowledge of organizations like CSC and IBM Global Services, helps ensure that his insights are from both sides of the outsourcing fence. 4

5 Introduction What you will get from reading this guide An understanding of the business-critical nature of backup and recovery services An overview of the most common mistakes Advice on getting it right and case examples An introduction to services Awareness of the key benefits and ROI Like many homeowners, businesses have often been satisfied with just minimal provision and preparation for disasters. It is only when a disaster arrives that it becomes clear how inadequate this approach is. What this guide is not This guide is not another explanation of backup technologies or the future protection technologies that will revolutionize the customer experience. Neither is it an analysis of the benefits or shortcomings of online vs. offline protection. Instead, this guide aims to examine why backup and recovery is critical to the business and how a proactive approach can be the key to backup success. 5

6 A useful analogy Businesses tend to view backup and recovery in the same way that property owners view homeowner s insurance. Cast your mind back to the devastating floods that hit Britain in the summer of 2007 or the fires that ravaged Southern California the same year. You will recall that many of the affected homeowners had insufficient insurance. Only when houses were flooded or burned, belongings destroyed, and lives turned upside down did homeowners come to realize the importance of adequate insurance policies and to profoundly regret having allowed homeowner s insurance to sink to the bottom of their priority lists. Historically, many IT departments have approached backup and recovery in a similar way. Too often the budget is spent on the very latest technologies at the expense of protecting the critical data that allows the business to function. Like many homeowners, businesses are often satisfied with minimal provision and preparation for disasters. It is only when a disaster arrives that it becomes clear how inadequate this approach is. Who should read this guide? This guide is intended for CIOs, CTOs, and IT directors faced with the challenge of managing backup and recovery operations. Operational budget holders will also benefit from some of the financial messages in the guide. We believe that a reactive attitude should be replaced with a proactive and assertive strategy, which attaches much greater significance to the role of backup and recovery. I commissioned this guide to help organizations make the right decisions about how best to prevent data loss and potentially catastrophic IT failure. Archie Maddocks, Senior Service Delivery Manager, Symantec Global Services 6

7 Why is backup and recovery critical to the business? The sad truth is that many businesses do not much care about backup and recovery. The majority of businesses that do care are often either unaware of the best way to address the issue, or under the impression that their existing measures are enough. The end result is that vital services provided by the IT group are often ignored or misunderstood. Indeed, because too many businesses still see backup as a burdensome drain on their time, organizations that have well-run backup operations are very much in the minority. A significant proportion of the problems associated with backup and recovery can be attributed to this basic reluctance or inability to take a firm grip of it. As a result, very few businesses are adequately organized, and there tends to be a lack of effective planning behind their strategies. Correct processes and procedures are often missing, and there is little evidence of root cause analysis when things go wrong. So it is not surprising that many businesses operate with a false sense of confidence that everything will turn out alright. In truth, however, they are risking a cycle of perennial firefighting. I think in the past, businesses didn t always give backup and recovery the attention it deserved. But increasingly, businesses are taking it much more seriously because we all know the potential repercussions. Technical infrastructure manager, leading financial services firm The whole issue of backup and recovery is too low on the list of priorities, says Symantec s Archie Maddocks. Too many businesses are only interested in storage when they can t recover data they desperately need, and it costs them money. Data backup and recovery is very much a reactive part of the overall end-to-end service. The benefits of a proactive approach What businesses do care about is making profit and the quality of the service they provide to their customers. While backup and recovery operations are often treated as necessary in an IT organization, they only become of strategic importance during an IT failure in recovering missioncritical information. The outbreak of a blended threat, a natural disaster, or a regional catastrophe such as an oil refinery explosion, could leave a business without the resources to locate, restore, and recover each data instance. Yet any interruption to service whether IT-related or not will affect profits, lower customer satisfaction, and even threaten the long-term prospects of the business. In Symantec s experience, those businesses that respond positively with a robust strategy for dealing with worst-case scenarios can reap substantial business benefits. 7

8 Sarbanes-Oxley, Basel II, and SAS 70 are three of the biggest compliance concerns at present and they all place significant demands on the recovery of data. The pressure for compliance Recent high-profile incidents of data loss demonstrate that companies and governments must understand the procedures and policies they implement and ensure compliance with legal standards. Businesses are also under pressure for a variety of other legal reasons. If, for example, a business is asked to produce data in litigation, it is essential that the data be located quickly and efficiently. Organizations tend to store data on backup tapes for many years in the belief that it can be retrieved when necessary. With tape-based backups, however, time can severely compromise the readability of data. There are at least two reasons. First, tapes begin to degrade over time. Second, the recording technology becomes obsolete and is replaced, leaving no playback capability. Moreover, few organizations catalog their data-storage tapes, relying instead on timestamps to identify likely locations. Many businesses are therefore unable to quickly identify the tape that contains the data they need which means that their organizations do not meet current compliance laws. Not only can a robust approach to data backup and recovery help lessen the likelihood of such problems, but it can also be a powerful tool for winning and retaining the trust of investors and customers. 8

9 Tackling common problems with backup Returning momentarily to the homeowner s insurance analogy, assuming that nothing very bad will happen is the most common mistake associated with backup and recovery. For too many businesses, it seems that only when they stare data loss in the face will backup and recovery receive the attention and investment it demands. But by that time, it is usually too late. Understanding what you are backing up and why The fundamental problem is that businesses do not always know the actual value of their data, so they do not know what to back up or why. It is not uncommon to find a business that is failing to backup all of the data it really needs, says Symantec s Archie Maddocks. Backup success rates may appear to be impressive, but the business may be missing critical data entirely. Moreover, there may have been little thought about how the data will be recovered and where it will be recovered to. There are businesses using snapshot techniques and off-host backup for data sets that aren t the product of high-volume, mission-critical applications and that don t need such stringent and expensive backup and recovery technology. Managing business and data growth Business growth is a thorny area, particularly when it comes to justifying costs. As a business grows, it is almost inevitable that data volumes will also increase especially if growth is rapid. In these circumstances, convincing the board to invest will require appropriate and persuasive reports and metrics.. 9

10 Absence of a dedicated storage function The responsibility for ensuring data backup and recovery often falls to platform teams. However, they generally believe they are employed to perform other tasks. For example, the Wintel platform team might be asked to manage backups even though their main focus is on the day-to-day patches and updates of other Wintel services. As a result, data storage becomes a secondary consideration. By its very nature, this leads to a reactive environment. Because businesses are unable to formalize responsibilities, it is difficult for them to prevent, diagnose, or repair problems with backup. Companies may be confident that they can adequately respond to problems, but they will rarely be able to keep a problem from occurring Disk storage (TB) H 2H 1H 2H 1H 2H 1H 2H E Growth in SAN disk storage in the average Fortune 1000 data center Figure 1. Storage capacity, utilization, and anticipated growth for Fortune 1000 companies. 10

11 Getting processes right Another potential pitfall is the general trend to focus too narrowly on technology. When problems are encountered in the backup and recovery process, organizations often conclude that software or hardware must be to blame. However, flawed processes and procedures are often the real issues. More often than not, poor practice in root cause analysis is the real problem that makes it nearly impossible to get the most out of the IT infrastructure. Poor communication is another contributing factor to many backup and recovery failures. If the appropriate people are not communicating effectively, services are bound to suffer. For example, if a backup schedule is looking for a particular application but somebody has taken that application offline for testing, the backup software will not be able to connect to the target files. This will be logged as a failure, and it is unlikely that anybody will know why. Often it is the absence of a dedicated storage team that leaves businesses most exposed. Beyond this, the fundamental problem is that businesses don t always know what, or why, they are backing up or what the value of their data actually is. Approaching the problem with the right attitude Successful backup and recovery require considerable tenacity. It is too easy to accept a 90 percent backup success rate as adequate. A truly committed business will be less interested in the 90 percent than in the 10 percent that is missed. This tenacious approach to backup and recovery will scrutinize the remaining 10 percent carefully and analyze exactly what has been overlooked. Shrewd businesses will work out why 10 percent of the data could not be backed up, then come up with a solution to the problem. And why the need for this intense focus? If it is the same 10 percent that has failed every night for two years, you are leaving yourself open to considerable data loss, says Symantec s Adrian Spink. Even with a good backup success rate of 97 percent, an organization may leave itself exposed. This is especially true if the 3 percent failure rate includes repeated failures for essential data, such as a customer database that is accessed and modified on a daily basis. For instance, imagine that a corruption occurred, and the database had to be restored from tape or disk backup images. Then imagine that consecutive backup failures meant the backup images from the past 12 to 24 hours were unavailable. It does not take much imagination to realize that the company could face significant challenges. 11

12 How to back up the right data in the right way Translate business requirements and align them with backup and recovery requirements. If the business does not have clear objectives, try to define some of your own and seek agreement. Develop a tiered list of backup and recovery requirements (service catalog) that fits the organization s business needs. Ensure that this is reviewed and revisited regularly. Work out the true respective costs of the organization s backup and recovery operations and apportion them to the various options. Predict the impact of data loss and build this into a budget proposal. Build or modify the infrastructure to deliver against the service catalog. Establish service-level agreements (SLAs) to define the expected level of service. Test the recovery process regularly, and perform analysis of the recovery times and recovery points to ensure that they fit with the business needs. Proactively manage and analyze backup logs to determine whether jobs are completed successfully. Perform effective incident management on backup failures or storage-related issues. Perform root cause analysis on all problems and issues found. Perform service-level management to the agreed service catalog. 12

13 Tackling common problems with recovery It is important to remember that without the ability to recover data, backing up data is futile. Too many organizations relax when they have all their data backed up. Inexplicably, many fail to test their ability to recover the data, This is typically the difference between success and failure. Disaster preparedness checklist Consider all the stages of recovery and all the components required. Investigate what will happen if a Bare Metal Restore (BMR) is needed i.e., if a complete hardware failure demands reinstallation of the operating system and applications. Too many organizations relax when they have all their data backed up. Inexplicably, many fail to test their ability to recover this data. This is often the difference between success and failure. Perform recovery exercises for critical applications to ensure that backups are consistent and that technologies and processes are in place to recover data to the required level. Consider when recovery may be required and what skills and resources will be available to work on the problem. Define and document recovery procedures. Develop competency in process adherence and improvement. Test and then test again. Ensure that backup and recovery are part of the IT service continuity strategy, underpinning your organization s business continuity plan. 13

14 The ability to recover data is not necessarily an area of high focus until something goes wrong. Unless something is glaringly wrong, it is easy to take things for granted. Service director, managed service provider, public sector program Make testing central to recovery A failure to test recovery on a regular basis can even disguise the fact that backups themselves are not working. In a worst-case scenario, it could become apparent that the entire backup process is flawed only when the business is required to recover critical data or applications. Testing the recovery process would highlight such a problem at an early stage, allowing the business to limit the damage. Working under the assumption that data recovery will be straightforward is extremely dangerous. When dealing with recoverability, it is particularly important to understand the business requirements. If these are not known, there is no way of working out how quickly the business will need to recover the data. RTOs and RPOs The recovery time objective (RTO) refers to how much time a business can afford to lose as it waits for data to be recovered. The recovery point objective (RPO) refers to how much data the business can afford to lose. Together, RTOs and RPOs are vital in a corporate environment. Therefore, it is essential that they be intelligently aligned with genuine business requirements. This is particularly important in today s litigious environment, where the inability to recover data can have significant legal repercussions. Many businesses believe that they are addressing data security by taking their tapes offsite. How the data will be recovered and when and to where it will be recovered often remain a mystery. People forget that in extreme cases, the whole recovery cycle could take weeks, not hours. More and more, we are discovering that businesses do not focus enough on recovery because they are too obsessed with backup. Wks Days Hrs Mins Secs Secs Mins Hrs Days Wks Data Loss (Recovery point objective) Downtime (Recovery time objective) Figure 2. Determining your data recovery needs. 14

15 Winning board backing for investment It is often said that IT experts and business experts speak very different languages. This makes it extremely difficult for IT experts to impress upon their boards of directors the importance of effective backup and recovery. With this in mind, there are ten basic steps that an IT manager or IT director can take to ensure the board s agreement to, and support for, a backup and recovery plan. If you follow these steps and report regularly to the board, they will soon begin to understand how critical backup and recovery are to the organization. The secret is to avoid technical jargon and communicate your concerns in real, understandable business terms Find out what regulations affect your business, and assess the impact they will have on your data retention and protection policies. Perform a risk and impact assessment of the business from a data loss perspective. Identify the critical business processes and ensure that these receive appropriate protection. Define a recovery strategy based on the criticality of the business processes. Establish a set of information and backup and recovery policies explaining exactly what needs to be done. Develop a full audit and reporting framework where backup and recovery operations are tracked and audited for process adherence. Develop an operational reporting framework for backup and storage. Make it visible to the organization, highlighting the importance of backup and recovery to the business. Educate all areas of the organization including end users on the importance of good information management. Ensure that the backup and recovery strategy effectively underpins a comprehensive security program. Interestingly, 40 percent of data compromise happens inside the organization. Rolebased access should be used. Develop a separate, usable archiving policy where information can be searched and recovered easily. Practice recovering data. 15

16 The media are always looking for things to go wrong. Any mistakes can lead to damning frontpage headlines. Service director, managed service provider, public sector program Signs that your backup and recovery operations needs to be improved Spiraling or unknown costs or total cost of ownership (TCO) Unknown operational and business risks Unknown or low level of backup success rates Struggling to meet audit compliance (e.g., Sarbanes-Oxley, Statement on Auditing Standards [SAS] No. 70) No clear service-level agreements (SLAs) Undefined roles and responsibilities (i.e., lack of clear ownership) Lack of tiered storage model Undefined operational processes Recovery issues due to failed backups Communicating risk At all times, you need to identify the reduction in risk that your backup and recovery strategy would provide. If the organization is presented with the end result of the improved service in terms of productivity or other metrics, skeptics will always take an interest. The secret is to avoid technical jargon and to communicate your concerns in real, understandable business terms. For example, if a bank loses encrypted tapes containing customer data, what exactly will the impact be? Similarly, what if your own business should lose confidential customer data? It is important to put this information into some form of cost/risk profile. This can then be used to inform the board that more investment is needed in order to improve the backup and recovery service. 16

17 Stick or twist? Managing backup and recovery operations can be a daunting task. Businesses are often left with a simple choice: continue to manage backup and recovery operations in-house, or seek the assistance of an outsourcer. The appeal of an in-house solution In some circumstances, an in-house backup and recovery team will be attractive for a variety of reasons: Invariably, businesses are left with a simple choice: continue to manage backup and recovery operations in-house, or seek the assistance of an outsourcer. The perception that it will be cheaper. Without conducting a thorough investigation of costs, there will be understandable concerns that obtaining outside assistance is more expensive than doing the job completely in-house. Concerns over security and confidentiality. Organizations are sometimes reluctant to hand access to their data over to someone else. Security is one of several considerations that may persuade a firm to keep storage in-house or simply avoid obtaining planning assistance. Basic reluctance to pursue outsourcing. Cultural resistance to outsourcing is commonplace. A desire to remain independent. There is an element of risk attached to the formation of any business relationship. Some firms will be reluctant to accept this potential danger. Anxieties over flexibility. Some businesses may also fear a loss of flexibility because an outsourcer will assume some degree of operational control. There may be concerns that relinquishing even a small amount of control will inhibit the organization s ability to react to changing business conditions. The benefits of outsourcing A strong, reliable outsourcer will bring a range of benefits which may resolve the concerns listed above: Greater visibility of TCO and the assurance of a fixed price. Typically, the budget manager will be unaware of the total cost of ownership (TCO) of an in-house solution. Consequently, the manager will be unable to forecast ongoing costs. However, an outsourcer can set a pricing schedule based on fixed monthly costs. Even better, this may well prove less expensive than existing methods. Guarantees on service levels. It is often difficult to establish, adhere to, or even measure operational levels internally. Strict SLAs provided by an outsourcer will enable management to learn exactly how effectively the organization s backup and recovery operations are performing. 17

18 In March 2007, Symantec research 1 found that bestin-class organizations are monitoring and measuring controls and procedures to protect sensitive data on a weekly basis. The average business only examines these controls and procedures every 176 days. Greater focus on your core business. An outsourcer will allow an organization to focus on its core competencies, thus allowing resources to be redeployed to more central concerns of the business. Stronger expertise in supporting your backup process. Many organizations do not have full knowledge of service management processes. Neither do they have a deep understanding of backup architecture and technology. An outsourcer will be able to improve service levels by efficiently applying best-of-breed solutions with strong service management processes. A more structured approach to critical business data. The financial repercussions of data loss or downtime can be devastating. An experienced outsourcer can minimize the risk by helping the business identify and effectively back up its critical business data. In March 2007, Symantec research 1 found that best-in-class organizations are monitoring and measuring controls and procedures to protect sensitive data on a weekly basis. In contrast, the average business only examines these controls and procedures every 176 days. A heads-up on RTOs and RPOs. The board will gain reassurance from an outsourcer with in-depth experience working out RTOs and RPOs. An understanding of RTOs and RPOs will allow the business to choose between gradual recovery, intermediate recovery, and immediate recovery. An understanding of the importance of compliance. Compliance demands will only increase in the years ahead. It is important to have experts onboard to deal with these growing concerns. Symantec s Archie Maddocks comments: Organizations are usually aware of the legislation, but they do not necessarily know how to configure or manage storage to meet the various requirements. Assistance with the auditing process. When an audit is conducted, it is crucial that everything is in place and up to date. A third-party outsource provider can offer the skills and experience needed to help ensure that audits run smoothly. Greater accountability and help in the decision-making process. Reporting capabilities of in-house systems are generally poor, which can limit damage accountability. An outsourcer, however, should be able to produce effective reporting of their services. This should be scalable from operations to business owners, and it should relay pertinent information based upon roles. However, an outsourcer s customers should ensure that they have a method of independently verifying the reported information Symantec: Taking Action to Protect Sensitive Data report, March 2007.

19 What to consider when outsourcing If you are thinking about outsourcing your data backup and recovery operations to a third party, what factors should you consider? Finding a trusted third party There should always be a strong professional relationship between the business and the outsourcer. The client must be able to trust the third party to do the job more efficiently and reliably than was previously possible. Clients want relationships, and they want to know that they can trust the people with whom they work. They want to know that the third party will deliver on its promises and step up if something goes wrong. They want a hands-on relationship. And they want to know that the outsourcer will deliver on its promises. If you are going to let somebody else run something important for you, it is crucial that you trust them implicitly. Technical infrastructure manager, leading financial services firm Getting the right range of services Because all businesses have specific motivations when outsourcing backup and recovery operations, it is important that tailored services be available to you. For those looking for sophisticated metrics and status information as well as trending and historical mapping reporting tools may be the only service they will need. Some third-party outsource providers also offer a range of onsite and offsite backup and recovery services. These include providing onsite specialists who work as members of your team for an extended period of time. These resident specialists can perform services under a statement of work or under an arrangement where the outsourcer takes responsibility for day-to-day administration and management under a strict SLA. Alternatively, managed backup services can also use remote teams and offsite infrastructure investments to perform 24x7 remote monitoring and remote management, analysis, and reporting under SLAs. These services can also work together as a managed solution, combining best practices onsite with remote management capabilities offsite. Through the use of onsite resident specialists or remote experts, existing internal resources can be freed to focus on strategic projects and core activities.. 19

20 Backup should be a chief priority for all firms and there should be a dedicated team taking control of it. Outsourcers live and die by their success rates, which can give them an edge over in-house teams. They will not be afraid to ask the difficult questions to achieve the desired results. Adrian Spink, Symantec Global Services Ensuring cost-effectiveness Choosing an outsourcing partner with a strong industry reputation should guarantee an improved service. Customers rarely know the total cost of ownership of running backup and recovery in-house, says Symantec s Archie Maddocks. There is a misconception that the cost of managing backup and recovery is limited to the salaries of the people actually employed for these purposes. In reality, you also need to consider the cost of upgrading, the cost of training, and the cost of managing high-impact failures or even data loss. In many cases, businesses taking control of their own backup and recovery services are hugely wasteful of their resources, simply because they lack the necessary processes and best practices. This inefficiency invariably has an impact on cost. Before agreeing to work with an outsourcer, it is important to understand how much you are currently spending on your backup and recovery operations. In our experience, organizations spend much more on backup and recovery than they realize and, in many cases, much more than necessary. Setting service levels When engaging with a business over backup and recovery, outsourcers will typically produce a proposal based on five pre-defined SLAs: Backup success rates Restore success rates Response time to incidents Repeat backup failure Reporting By establishing firm SLAs in this way, organizations are able to keep a closer eye on the quality of services provided. Metrics are made available to test all five SLAs. This means businesses can expect first-class management of the entire backup and recovery process. 20