f f How to bring your business systems back to life f f Keeping data from falling into a black hole f f Do-it-yourself considerations and hazards f f Data center options for business continuity 1
DISASTER RECOVERY DRILL About this paper In Part I: Disaster Recovery Planning from Scratch, Peak 10 s experts provided the blueprint to identify a company s most mission-critical data systems and accompanying recovery point objectives in the event of disaster. This information alone will get most companies halfway through a real disaster recovery plan. Now Peak 10 will explain how to actually recover these systems using a simulated disaster recovery scenario. Red alert! Disaster has struck. It s the middle of the night, and you don t know it yet, but your corporate office just experienced a major power outage that the utility company estimates will take at least three days to restore. In just a few hours, employees will report to work, but they won t be able to access the systems they need. Customers will attempt to place orders by phone or online, but won t be able to reach you. If you followed the guidelines outlined in Disaster Recovery Planning from Scratch you at least have a solid handle on which business data and functions to recover as soon as possible before revenue takes a real hit. That s half the battle. Now, what about the other half? STEP ONE: ASSESS THE DAMAGE You won t have much time to get all the information you d like, as employees and customers will soon want to know what is going on. The goal here is to get as full a report as you can in the shortest time possible. At a minimum, you ll need to know: ff What happened? ff Which systems are affected? ff What needs to be done to recover? ff Do you declare a disaster or not? ff Estimated time of recovery everyone will be asking you this. STEP TWO: COMMUNICATE Always remember, over-communication is the art of war when it comes to a disaster. If the key people know what s going on, they can relax and stay off your back for a little while, anyway. But how do you communicate when your corporate phone and email are down? Stick to the K.I.S.S. principle when designing your communication plan: ff Keep an updated contact list stored at a free online file sharing site. ff To get the word out as quickly as possible, assign a communication tree where one person calls three people, who each call three more people, and so on. ff Have alternative email addresses in place, such as Yahoo! or Gmail, and consider a text messaging solution like ReadyAlert.
DISASTER RECOVERY DRILL Of course, realistically, these options won t help all people get in touch with you until they know how. Additionally, you ll need a way to inform your customers of what s going on. If you have thousands of customers, this is going to be essentially impossible without nearby access to some sort of powered facility with working computers. In other words, a do-it-yourself communication plan isn t bulletproof. STEP THREE: RESTORING CONNECTIVITY So how do you get people connected to your company and your systems back online? For that matter, how or where will your employees perform their daily functions during this outage? The short answer is, until power is restored to your facility, you can t reunite your people with their needed business applications. That being said, it s possible to restore the power yourself, if your company has redundancy for all your key systems. Some things to consider are: f f Power Redundancy An Uninterruptable Power Supply (UPS) battery and a generator. f f Internet Redundancy Obtain your business class service from a local fiber provider and your secondary service from a local cable or wireless solution provider. f f Cooling Depending on the size of your IT equipment, if the room the servers are in cannot be cooled, you could have potential damage due to overheating. f f Network Gear Along with your internet connection being redundant, what about the routers, firewalls and switches that connect your infrastructure to your people both inside and out. f f Critical Systems Some often overlooked items that require redundancy are: ff Windows Active Directory Servers aka Domain Controllers, which handle all your login and rights-based authentications to your down stream systems. f f Domain Name System(DNS) These are the servers that map your server s IP address to a ff friendly name such as email or finance. DHCP This is the device that hands out IP Addresses to network devices on the network. f f Database Servers This may be an obvious redundancy, but should not be forgotten! It should be noted that in-house redundancy isn t a guarantee if the power outage stretches out longer than expected. Few companies invest in the kind of long-lasting generator power that a professional data center, for example, has on hand. Something else to consider: Any entity on your network that plays a critical role should be evaluated as to whether or not it needs to have redundancy. One way to decipher between those entities that do and those that don t is to ask yourself the question: If entity X has an issue, what is the impact to the business? STEP FOUR: GETTING YOUR DATA CURRENT Since this is a simulated drill, we have the luxury of considering some different scenarios at this point in your disaster recovery plan. Of course, if your company has no business continuity/disaster recovery
DISASTER RECOVERY DRILL plan and must simply wait until power is restored, then ensure your data is as current as it can be. Please refer back to your recovery point objectives as described in plan and must simply wait until power is restored, then your data is as current as it can be. Please refer back to your recovery point objectives as described in Disaster Recovery Planning from Scratch to determine how long you can go without updated information. Scenario A: Your company has a backup disaster recovery site that allows you to continue to do business. Keep in mind that your offline primary site is now filled with outdated systems and data, growing older by the hour. How will you update this site once it s back online? It s a bit more difficult than you might think. Many vendors say they offer data replication solutions, but fail to produce in the fail back state. Note that fail over is the act of making your DR site your production site; fail back is making your production site once again your primary. As you can see, one site will have more current information than the other. Some tips to make the best of this situation: f f Know in advance which systems to include in the DR stack. Some of these would be your CRM, database, email and ERP systems, while employee archival or document archival are less critical. f f Base these tiered systems on your recovery point objectives. Scenario B: Data backup is outsourced with a free backup service. While there are multiple vendors that offer these services, the space itself is relatively small. Nor do these companies adhere to service level agreements in regard to timeliness or guarantee that your data will be there when needed. They do offer other levels for a fee, but either way, be aware that these services are often sup-ported by a fourth or fifth party infrastructure. In short, your data is still significantly exposed to risk in the event of a real disaster. Is it time to consider working with a professional? Scenario C: Outsource to a managed Services Provider Many companies opt to work with a managed services partner, such as Peak 10, to implement their disaster recovery plans. There are many advantages to working with an industry expert like Peak 10 and we have explored some of these advantages below. With its many data centers strategically located throughout key business regions in the United States, Peak 10 is more than just a place to keep and protect your data, it is a full managed services partner with the flexibility and expertise to tailor solutions precisely to your unique business needs. Benefits of working with Peak 10 include:
DISASTER RECOVERY DRILL f f Fully owned and operated data hosting infrastructure. Customers know who they are working with and where their data resides, two enormous advantages in the event of a disaster. Auditors like this, as well. f f Managed data and virtualization services for companies of all sizes, some with large IT staffs and others with one person who does it all. f f Robust data virtualization, replication, backup, disaster recovery and other solutions that meet full HIPAA/HITECH, PCI and SOX compliance requirements. f f All Peak 10 facilities are SSAE 16 accredited. f f Additional contractual services that guarantee rapid shipment of generators, servers and various equipment that are often hard to find in the aftermath of a disaster. f f Ready suites of offices in nearby Peak 10 physical locations, completely powered with working phones, computers and Internet service. STEP FIVE: IMPLEMENT AND TEST. TEST SOME MORE. THEN SCHEDULE YOUR NEXT TEST! By now, the common thread of this white paper should be obvious: the most important part of having a BC/DR plan is to actually have one. The second most important? Test it! Simply put, if you don t test your BC/DR plan, then you don t have one. Once you implement all the solutions and processes to support a DR strategy, challenge your company and data center partner to find out how well it really works. Some tips: f f Leave an element of surprise for your vendors and staff, as a real disaster will have this in spades. Reveal a general time window for the test run, but not the actual date. f f Set a recovery time goal as short as possible, maybe even beyond what you think is possible. Remember your own service level agreements! f f Don t declare victory after one successful test run. While testing is the last step in your DR plan, it s never really over. What works today may need to be tweaked tomorrow. Still, finally putting the DR plan together is a significant advances in disasterproofing your business. If you would benefit from having a more detailed conversation about BC/DR, a Peak 10 expert would be happy to speak with you. Take the first step toward your own DR plan and get in touch with Peak 10 today!
DISASTER RECOVERY DRILL ABOUT PEAK 10 Peak 10 is the only data center solutions company with rock solid people, rock solid solutions and rock solid results. Its dedicated team of experts is empowered to serve clients and proactively embraces the industry s evolving technologies. Peak 10 s tailored, reliable and secure solutions for hosting and managing complex information technology infrastructure deliver concrete results that are cost-effective and sustainable. The company s responsive team, financial stability, geographic diversity and technical expertise attract market-leading companies such as LendingTree, Global Knowledge, Pergo, Healthways, Churchill Downs and Carnival Cruise Lines. Peak 10 is SSAE 16 audited and helps companies meet the requirements of various regulatory compliance acts such as Sarbanes-Oxley (SOX), HIPAA/HITECH, PCI DSS and Gramm-Leach-Bliley (GLBA). For more information on data center solutions from Peak 10, visit www.peak10.com.