The Good, Bad and Ugly Parts of DR Testing

Transcription

1 The Good, Bad and Ugly Parts of DR Testing Almost everyone would agree that disaster recovery testing is smart, but many still don t test. This guide looks at why many don t run DR tests, and for those who want to take them on, we ll tell you what to do and what not to do to make them smooth and efficient exercises. SMOOTH SAILING

2 EDITOR S NOTE It s Best to Test Your Disaster Recovery Plan Testing, testing! It s one of the most important elements of disaster recovery planning testing to make sure your plan works. And while it s universally recognized that DR testing is crucial, many don t do it or don t do it enough. In this Drill Down, you ll discover common reasons why people don t perform DR tests, learn about technologies that can help you with your testing, and find out frequent test mistakes. Surveys are showing that while some organizations are good about testing often, some don t even have a schedule. According to a recent TechTarget survey, 28% don t have a set schedule for running DR plan tests and 4% never test. Reasons for shunning testing include lack of resources, complexity of the process and cost. But there are ways around the excuses, as you ll find in this Drill Down. With the technology of today, there are methods for all kinds of businesses to test their DR plan. As an example, look at disaster recovery as a service (DRaaS), which has become a popular option. Your DRaaS provider may be able to help you prepare a DR test plan. A good DR test does require a lot of work, so we ve provided tips and guidelines to help you through it, as well as mistakes to avoid before, during and after testing. What s the point of having a DR plan if it doesn t work? One surefire way to confirm that it works is to test it. Read on for guidance on a critical element to keeping your data safe. n Paul Crocetti Site Editor SearchDisasterRecovery 2 THE GOOD, BAD AND UGLY PARTS OF

3 INADEQUATE TESTING Why Firms Don t Conduct DR Tests Despite all the talk about disaster recovery testing, most organizations still don t do it enough. And recovery point objectives (RPOs) are still way too high to facilitate adequate DR, according to a survey conducted by cloud vendor CloudVelox. CloudVelox, which offers automated disaster recovery in the cloud, interviewed 343 IT executives responsible for DR in their organizations from nine vertical markets. The surveyed organizations ranged from less than 100 employees to more than 1,000. The survey found 58 percent of the respondents ran DR tests once a year or less. Another 33 percent tested their DR infrequently or never, while 26 percent tested it quarterly and 16 percent did it monthly. These results should not be surprising because other recent surveys have had similar results, including one conducted by our parent company TechTarget. WHY ISN T MORE WIDESPREAD? So why aren t people testing more often? Fiftysix percent of the CloudVelox respondents said their DR testing was infrequent because they didn t have adequate internal resources. Another 34 percent found the process complex, while 19 percent did not find it to be a priority and 12 percent said it costs too much. Respondents also say their traditional DR products don t offer adequate RPOs. One-third said their RPO was more than 12 hours, with only 21 percent claiming it is two hours or less and 46 percent said it is between two hours and 12 hours. The fact that RTO and RPO in this day and age is still in the two-to-12-hour range shows that disaster recovery is broken, said Vasu Subbiah, CloudVelox s vice president of products. And IT does not have the resources. The average IT spend for disaster 3 THE GOOD, BAD AND UGLY PARTS OF

4 INADEQUATE TESTING recovery is between five to seven percent. If they test less frequently, then mistakes are compounded when they try to recover in the future. Cloud Velox, formerly called CloudVelocity, offers cloud-based disaster recovery, cloud data migration and testing and development in the cloud. The July 2015 survey included the following verticals: oil and gas, basic materials, industrial, consumer goods and services, healthcare, telecommunications, utilities and finance. WHICH INDUSTRIES TEST THEIR DR PLANS MOST FREQUENTLY? The survey also found variations based on the vertical. For instance, the survey found the oil and gas industry has the highest average RPO, with 70 percent stating that it took 12 hours or more, and they had the lowest test frequency, with 80 percent of those surveyed saying they test once a year or less. Thirty percent of all the industries included in the survey stated they had an RPO of 12 or more hours. In healthcare, 69 percent tested once a year or less. Consumer services and healthcare were most willing to embrace cloud-based DR if they could automate network and security controls to the cloud. Sixty-five percent of respondents in consumer services and 64 percent of healthcare would do cloud DR if they had the option of automation. One in four of the respondents said they experience failure or delays over half of the time when they tested their secondary data center. Fifty-three percent said network connectivity was the common cause of failure when testing their DR environment. Another thirty-seven percent cited wrong configuration and 33 percent cited missing patches. Network and security concerns often are singled out as barriers to cloud adoption. CloudVelox s survey found that 55 percent of respondents would use cloud DR if they could automate their on-premises network and security controls in the cloud, while the other 45 percent would not consider the cloud even if they had on-premises network and security controls. Sonia Lelii 4 THE GOOD, BAD AND UGLY PARTS OF

5 DR TECHNOLOGY How to Make DR Testing Smooth Sailing Today s disaster recovery technologies can make the process of testing the recoverability of critical systems a snap. A traditional DR test plan involves many people; extensive preparations to ensure the availability of systems, networks, databases and other resources; and thorough documentation of procedures. And that s before the test even starts. By contrast, advances in DR technology now make it possible to perform a system failover to a backup environment in seconds and then fail the recovered system back to its normal production status in the same amount of time. The old-fashioned way can often take hours, and if the DR test is unsuccessful, additional hours of forensic work may be needed to determine what went wrong and conduct a followup test. Here are some available technology tools and resources that can make a DR test plan a more pleasant and less time-consuming experience. HIGH-AVAILABILITY ENVIRONMENTS Migrate your existing systems and infrastructure to a high-availability (HA) environment to build in additional layers of fault tolerance or load balancing. This hardens your IT infrastructure and makes it more resistant to disruptions, and can also increase your ability to prevent a disaster from happening. However, if one or more HA platforms, data, applications or infrastructure elements are disrupted, available technology offerings can greatly simplify your ability to quickly recover data and move critical systems to alternate platforms, where they can maintain virtually uninterrupted operations. An example of this kind of DR software is Double-Take from Vision Solutions. Once the preparations have been made, and the recovery environment is operational, launching Double-Take is simple and failover of the primary system to the backup can take a matter of minutes. Once 5 THE GOOD, BAD AND UGLY PARTS OF

6 DR TECHNOLOGY the emergency has been addressed, failing back to the original platform also occurs quickly. High availability and disaster recovery are not the same. HA ensures the entire platform has sufficient redundancy if one or more service components are disrupted, the overall platform or service remains available. By contrast, DR assumes an alternate site with a suitable infrastructure is available so that data or the disrupted service can be recovered at the standby site. DR PRODUCTS Many system vendors offer disaster recovery to enhance their products and services. It may not be necessary to invest in a third-party offering if your vendor offers its own DR products. For example, VMware offers numerous DR products to protect the many virtual machines your organization will develop. Check for available DR testing support options. Disaster recovery as a service (DRaaS) has quickly become a useful DR option. Cloud service providers that offer DRaaS typically present a suite of cloud-based recovery options for backing up data, hardware and software they can help you prepare a DR test plan. Alternatively, you can elect to plan and manage DR tests yourself, and your DRaaS provider can support those efforts. Despite the many available options for DR testing, you may decide to build your own. Be mindful of the platforms to be recovered and the infrastructure resources available to support a DR test plan, and carefully define your recovery requirements before proceeding. As with any third-party products that generally include their own documentation, thoroughly document your testing procedures. A DR test plan is important for every IT organization. Mission-critical systems, networks, applications, hardware devices and databases as determined in a business impact analysis should have a documented recovery strategy and be tested periodically to ensure they can be recovered. Many options are now available to make DR testing a much easier process, and access to them should hopefully encourage more regular testing. Paul Kirvan 6 THE GOOD, BAD AND UGLY PARTS OF

7 MISTAKES Five DR Test No-No s A great deal of work goes into a disaster recovery test plan. Among these activities are: Identifying the test team. Securing the test team s availability on the planned DR test date. Identifying additional subject matter experts such as database administrators and network engineers. Determining what will be tested and the expected outcomes. During the actual disaster recovery test, the game plan must be ready to use, the testing procedures documented and shared with all players, and the systems and network resources should be in place. But despite all this preparation, here are five mistakes that could happen when planning for and conducting your DR test plan: Mistake No. 1: Not having a disaster recovery test plan. You need to have a documented plan of action that clearly outlines pre-test activities, test procedures, post-test wrap-up activities, test scripts and test team members. This written plan is also important from an audit perspective, as it provides auditors with evidence of how the test was planned and executed. Mistake No. 2: Not having all the proper technology representatives present. While you may think you have all the necessary players identified for the DR test plan, double-check to make sure a key subject matter expert, such as the system s developer, has not been overlooked. Ensure you have all the technology elements represented network, hardware, application, database, utilities and any other special arrangements that are unique to the system being recovered. 7 THE GOOD, BAD AND UGLY PARTS OF

8 MISTAKES Mistake No. 3: Not having a detailed script of the test activities. Aside from the disaster recovery test plan itself, this is one of the most important technical documents you ll have for the test. Scripts are the playbook used to execute tests, and their accuracy is essential. Even transposing two characters in a line of code can cause a system test to fail. If possible, have a subject matter expert review the test script to identify any possible changes or edits. It might also be useful to have a dry run of the test before going live. Mistake No. 4: Not having the test elements in place and ready to use. If your test is for a certain day, you ll be expecting all the components needed for the DR test to be in place. If one or more of your hardware, software or network elements are not ready, cancel and reschedule the test so you won t disrupt production systems. Mistake No. 5: Not checking to see if others have scheduled tests. In a medium- to large-sized IT department, there will likely be many scheduled activities in place, such as new system acceptance testing, network upgrades and software upgrades. This means your test will need to fit in with the other activities, so there are no disruptions to scheduled work. DR testing can often take several hours, so it should be scheduled as far in advance as possible. You should also send out notifications to other IT leads advising them of your planned test. ADDITIONAL DISASTER RECOVERY TESTING PLAN CONSIDERATIONS While the above five mistakes should be avoided before your plan is tested, here are three factors to consider during and after the test. 1. Not being willing to halt the test and reschedule if things are going poorly. If things are not going as planned, stop the test and review what happened up to the point where the test systems were no longer performing as expected. If it s possible to bypass the element that failed and continue with the test, continue with the original script. 8 THE GOOD, BAD AND UGLY PARTS OF

9 MISTAKES 2. Not documenting what happens during the test. This includes recording when steps are completed and identifying modifications to the test script that are developed on the fly. One of the key players in a disaster recovery test is a scribe the person taking notes of what is happening. Another is a timekeeper, who notes the specific start and end times of test activities. 3. Not preparing an after-action report summarizing the test, lessons learned and so on. The scribe s and timekeeper s notes will be used to prepare an after-action report on the disaster recovery test. Auditors will want to review how the test was conducted, actions of the specific test participants, how specific issues were handled, any infrastructure elements that didn t work and the results achieved. When you have a disaster recovery test plan and conduct a system and/or technology test, you can increase the odds of a successful test with sufficient preparation and properly scripted actions. Paul Kirvan 9 THE GOOD, BAD AND UGLY PARTS OF

10 ABOUT THE AUTHORS SONIA LELII is senior news writer with TechTarget s Storage Media Group. her at slelii@techtarget.com. PAUL KIRVAN is an independent consultant, IT auditor, and technical writer, editor and educator with more than 25 years of experience in business continuity, disaster recovery, security, enterprise risk management, telecomm and IT auditing, and over 30 years of experience in technical writing and editing, and technical training and public speaking. him at pkirvan@msn.com. The Good, Bad and Ugly Parts of DR Testing is a SearchDisasterRecovery.com e-publication. Rich Castagna VP Editorial Ed Hannan Senior Managing Editor Cathy Gagne VP Editorial Paul Crocetti Site Editor Dave Raffo Senior News Director Garry Kranz Staff Writer Linda Koury Director of Online Design Jillian Coffin Publisher TechTarget 275 Grove Street, Newton, MA STAY CONNECTED! today TechTarget Inc. No part of this publication may be transmitted or reproduced in any form or by any means without written permission from the publisher. TechTarget reprints are available through The YGS Group. About TechTarget: TechTarget publishes media for information technology professionals. More than 100 focused websites enable quick access to a deep store of news, advice and analysis about the technologies, products and processes crucial to your job. Our live and virtual events give you direct access to independent expert commentary and advice. At IT Knowledge Exchange, our social community, you can get advice and share solutions with peers and experts. COVER: FOTOLIA 10 THE GOOD, BAD AND UGLY PARTS OF