1 The Weakest Link in Disaster Recovery By Alex Bakman, CEO Much of the focus of disaster recovery planning is on creating redundant data sites and backup tapes. Very often, a crucial component is overlooked: that of keeping current documentation for all IT configuration settings. Having such documentation and the original software discs can restore a network 40 percent faster than running backup tapes. Access to the latest detailed configuration settings means faster disaster recovery. This paper demonstrates where having detailed configuration documentation fits in the disaster recovery process and how it aids in the rapid restoration of an IT infrastructure. The characteristics of an ideal tool to solve the problem of collecting and documenting current configuration settings and how such a tool and the information it provides can solve some of the day-to-day challenges of managing an enterprise IT infrastructure, will also be outlined. To be effective, most disaster recovery (sometimes called Business Continuity) plans require extensive testing, skilled personnel, access to vital records, and alternate recovery resources, including backup facilities. Plans are designed to restore an IT infrastructure the business backbone of today s corporation as quickly as possible. For most organizations, information, and the technology that supports it, represents the organization's most valuable assets. Enterprise applications are deployed over multiple systems and databases and across multiple locations making documenting current configuration settings a very difficult task.
2 Time is Money The high cost of business downtime goes beyond lost sales. Failure to perform can lead to contractual penalties. Customers who choose an alternate supplier may never come back and even if they do, your cost of sales can increase due to a new competitive mix. If records such as invoices are lost, you lose revenue on delivered products and services. While you are waiting to restore your IT infrastructure, you still have to pay salaries or suffer a possible public relations disaster. In the case of the September 11 th tragedy, a company s reputation may not suffer, but stock prices, credit ratings, and cash flow may still be impacted. Faced with the events of 9/11, enterprise IT departments are focusing more time and money on disaster recovery plans, equipment, and services. Many enterprises have invested heavily to ensure the survival of the IT infrastructure (e.g. physical data center security and fire prevention equipment) as well as in redundant data centers (e.g. distributed corporate data processing facilities and/or outside services) to ensure business continuity. Most mission-critical applications have their data backed up to tape or other media and these archives (and other critical documents) are best stored in a safe site off the corporate premises. Many disaster recovery plans include some level of IT infrastructure configuration data, collected as a snapshot at a given point in time. Typically, this a hardware and software asset catalogue which contains such information as vendor name, model number, serial number, location, etc. for hardware and vendor name, version number, service pack information, etc. for software. Most enterprises feel they have all the bases covered with these products and services. However, the speed of business restoration is impeded by inadequate IT infrastructure documentation. Detailed knowledge of server, database, and router configurations is essential to re-establishing a working framework in which to restore corporate data and services. The IT disaster recovery plan has, until recently, been viewed as a static document that sits in a three-ring binder on every IT mid-level manager s shelf, doing little more than provide comfort that the IT department is ready to do its part to ensure business continuity. Collecting this information from diverse platforms and massaging it into meaningful information (if that is attempted) takes a tremendous number of hours and most IT departments do not devote resources to keep the information current. As a result, all configuration data collected in these documents rapidly becomes out of date due to the one constant in the IT world: change. Until recently, most disaster recovery plans assumed the existing IT staff would be involved in the restoration process. For a fire in a corporate data center, this might be true. However, in a natural disaster such as a tornado or flood where the area surrounding a data center may also be affected, IT staff will initially be more concerned with their families and homes than with their work responsibilities. Sadly, September 11 th taught us that the unthinkable could occur. Even if IT staff is available to assist recovery, the multitude of IT platforms and the large
3 number of changes that occur on a daily basis limit their effectiveness to support a backup data center s restoration efforts. Thus, the IT disaster recovery plan needs to be continuously updated with the latest configuration settings reported in a clear, consistent manner. All changes should be easily identifiable to preserve IT decisions. 1. There are three main reasons that detailed configuration data is not collected and kept current in many enterprises: Almost no company has enough IT staff. According to the Information Technology Association of America (ITAA), of the current US IT workforce requirement of 10 million, there are over 800,000 vacancies that cannot be filled due to the lack of trained talent. The workload increases, but hiring never keeps up. 2. The technical competence of individual IT talent varies with training and experience. Configuration documentation may seem an entry-level task that most professionals seek to quickly move beyond. Disparate IT staff members often collect different types of information and the quality of their reports varies greatly. The more senior IT people are assigned to "more critical" tasks, deployed by management where they provide the most perceived value for their salaries, which average $85,000 per year ($75 per hour). The hours needed to assemble, verify, and report configuration settings could amount to tens of thousands of dollars in a larger IT shop. 3. IT staff turnover ranges from 8-17 percent, depending on industry and geographic region. The costs of hiring and training new staff to replace lost employees is nearly triple the IT overhead cost (about $225 per hour). And when IT staff leaves, their knowledge of the corporate IT infrastructure leaves with them. Are Backups Enough? One of the most common reasons that detailed configuration information is not recorded is the belief that backups contain everything needed to restore systems into production. The effectiveness of backup tapes depends upon the nature of the disaster. A system that experiences a simple power outage or hardware failure can easily be restored from backups, but restoring following a complete meltdown is another matter. Critical information not contained in backups includes: hardware specification for each system, EEPROM settings, specific boot instructions, SCSI ID manipulation, BIOS versions, virtual memory swap space sizes, disk partition slices, space allocation considerations, recovery/re-installation prerequisite considerations, network services provided, network dependencies required for normal functioning, kernel parameters, initial system installation cluster, and configurations that affect storage devices. Typically, volume management software and RAID software is on the tape, but is useful in disk arrangement prior to reinstallation and restoration. While preserving business data and transactions, most backup tapes contain no configuration data. If you have a hot backup site and don t even need to use tapes, you still need to build a system on which to restore the data.
4 Before you can restore the data, you need to reconfigure your IT infrastructure to support it. Following are the five stages of a typical disaster recovery. 1. Immediate Response: This ensures the safety and evacuation of all employees, notification of appropriate management and continuity personnel, assessment, command center activation, and disaster declaration processes. To demonstrate how immediate a response can be: On September 11 th, the first disaster declaration to reach a backup service came at 9:02 am, 17 minutes after the first plane crashed into the World Trade Center. 2. Environment Restoration: This phase provides alternate space for the people and equipment. Generally people relocate to recovery work areas while computer equipment is sent to a hot site or data recovery center. Operating system software is restored for computer systems, while basic facility preparations are conducted for work areas. 3. Voice and Data Communications Restoration: This phase restores communication to the functional parts of the organization, customers, vendors, and, in some cases, the disabled facility. 4. Functional Restoration: This phase includes restoring systems for computer applications, which can only be done once the infrastructure is properly reconfigured. A critical element is the most recent security settings. You need to ensure that the restored applications do not have any security holes when they are returned to production. 5. Restoration and Synchronization: This final phase includes restoration of data from offsite locations through the use of electronic media and paper. Information that was not backed up may be lost forever. Most often data is protected at different times during the business cycle and synchronizing, validating, and reviewing data from different sources is a critical step in a successful recovery. Once reliable data is established, backlogged transactions that have accumulated during the recovery are processed. Throughout the recovery process, detailed configuration documentation that contains change information allows the original IT staff and the restoration team to easily see, discuss, and alter any changes in configuration settings that occurred from the last safe settings. Ideal solution for collecting and maintaining configuration documentation The answer to the problem of collecting and maintaining detailed configuration documentation is quite simple: automation. Where the process is being done manually (or with tools that provide parts of the required information), automated tools make this task easier by eliminating the work involved and increasing the speed that the information is collected. They also improve the accuracy of the information collected by superceding the human error prone involvement in sorting through and reporting mountains of input data. The information is presented and preserved in a consistent format. Where the process is not being done, automated configuration reporting and change management tools make it possible to actually accomplish the task for the first time.
5 Automatic scheduling of configuration data collection eliminates the manual part of the process. Automation allows the data to be updated on a regular basis to ensure that the most current information is always available for disaster recovery. Cost Savings Offset Purchase Price Unlike an insurance policy where only a disaster realizes the benefits, detailed configuration information and documentation can be used on a daily basis to improve the operations of the IT infrastructure; in troubleshooting, security, auditing, and training. Compliance Reporting is a subset of a larger IT management requirement that is driven by individual industry requirements for security both of the data being managed and of the IT Infrastructure itself. A critical component for being in compliance with these industry-specific mandates is possessing current and historical documentation that provides detailed configuration settings of the IT Infrastructure. For example, the healthcare industry is working towards compliance with the Health Insurance Portability and Accountability Act of 1996 known as HIPAA. Within HIPAA is the requirement for the security and confidentiality protection of electronic health information. Ecora s products contribute toward compliance with the security requirements of HIPAA by providing current (and historical) detailed configuration reports to support auditing, security, and disaster recovery. There are similar requirements in the financial industry, including Gramm-Leach-Bliley, mandated by the Federal Reserve System, which requires recording detailed configuration settings for security and backup. Firms that are ISO-9000-compliant or working towards that certification also require extensive documentation of IT processes and policies. In summary, detailed configuration settings for all the major devices on your network should be kept accessible and up-to-date in order to speed recovery in the event of a disaster. Most companies do not maintain such documentation due to a lack of resources. Automated solutions exist that take the guesswork out of documentation and enable staff to schedule a variety of summary and change reports. Reports on settings are not only useful for restoring a network 40 percent faster in the event of a disaster, but also furnish crucial compliance information. Alex Bakman is founder & CEO of Ecora Software, makers of Configuration Auditor, a comprehensive solution for auditing, security, and disaster recovery.
One Stop Data & Networking Solutions PREVENT DATA LOSS WITH REMOTE ONLINE BACKUP SERVICE Prevent Data Loss with Remote Online Backup Service The U.S. National Archives & Records Administration states that
REMOTE BACKUP-WHY SO VITAL? Any time your company s data or applications become unavailable due to system failure or other disaster, this can quickly translate into lost revenue for your business. Remote
CHAPTER 3 HUMAN RESOURCES MANAGEMENT AND TECHNOLOGY by Julie Bulmash LEARNING OUTCOMES AFTER STUDYING THIS CHAPTER, YOU SHOULD BE ABLE TO Describe how HR technology has evolved. Explain what a human resources
Moving from Legacy Systems to Cloud Computing A Tata Communications White Paper October, 2010 White Paper 2010 Tata Communications Table of Contents 1 Executive Summary... 4 2 Introduction... 5 2.1 Definition
Cyber Security: Guidelines for Backing Up Information A Non-Technical Guide Essential for Executives, Business Managers Administrative & Operations Managers This appendix is a supplement to the Cyber Security:
Problem Management Contents Introduction Overview Goal of Problem Management Components of Problem Management Challenges to Effective Problem Management Difference between Problem and Incident Management
VirtuousIT Ltd 1 Table of Contents 1 Introduction 3 The Data Explosion 3 The Importance of Rapid System Recovery 3 2 The VirtuousIT Solution 4 3 RecoveryShield - Solutions 5 Large Business 5 Small Medium
The Data Recovery Solution A White Paper by ONTRACK Data Recovery, Inc. Contents Data Loss Dangers... one Critical Data Risks... two Challenges and Responsibilities... three Protection, Detection, Correction...
Mi c r os of tgl obalsec ur i t y Mi cr osof tgl obalsecur i t yshowcase Physi calsecur i t yatmi cr osof t Taki ngadvant ageofst r at egi ci TConver gence Techni calwhi t epaper Publ i shed:apr i l2009
White Paper May 2006 Applying Electronic Records Management in the Document Management Environment: An Integrated Approach Written by: Bud Porter-Roth Porter-Roth Associates Table of Contents Introduction
White Paper EMR Infrastructure Readiness Kathleen Gaffney June, 2012 Introduction An often-overlooked aspect to implementing an Electronic Medical Record (EMR) system is the need for a solid, medical-grade
Information Technology Outsourcing GTAG Partners AICPA American Institute of Certified Public Accountants www.aicpa.org CIS Center for Internet Security www.cisecurity.org CMU/SEI Carnegie-Mellon University
The Path to Managed Services in Discovery KJ Kuchta and Mark Walker 2012 2600 N. Central Avenue Suite 550 Phoenix, Arizona 85004 602 992 3600 The Path to Managed Services in Discovery By KJ Kuchta, CEO
Standards for Internal Control in New York State Government October 2007 Thomas P. DiNapoli State Comptroller A MESSAGE FROM STATE COMPTROLLER THOMAS P. DINAPOLI My Fellow Public Servants: For over twenty
Online Backup: A Viable Offsite Data Backup Strategy for Your Business? October 2011 A. Executive Summary Most organizations recognize the need for offsite data backup, but with so many vendors and solutions
Best Practices for Deploying and Managing Linux with Red Hat Network Abstract This technical whitepaper provides a best practices overview for companies deploying and managing their open source environment
Tandberg Data SMB Guide to Backup Best Practices FEATURES Best practices to ensure your data is fully protected! Complete checklists for tape migration and backup Covers both standalone tape drives and
Frequently Asked Questions about Cloud and Online Backup With more companies realizing the importance of protecting their mission-critical data, we know that businesses are also evaluating the resiliency
Online Backup White Paper The Dollars and Sense of Online Backup Table of Contents Executive Summary...3 IT Challenges: It s ugly out there...3 The problem with tape...4 Now is the time for online backup...5
CUNY Business Continuity and Disaster Recovery Task Force Information Technology Subcommittee IT Disaster Recovery/Business Continuity Recommendations Adopted by CUNY IT Steering Committee on September
Best Practices for Cloud-Based Information Governance Autonomy White Paper Index Introduction 1 Evaluating Cloud Deployment 1 Public versus Private Clouds 2 Better Management of Resources 2 Overall Cloud
With hundreds of Help Desk software packages available, how do you choose the best one for your company? When conducting an Internet search, how do you wade through the overwhelming results? The answer
ICC CYBER SECURITY GUIDE FOR BUSINESS ICC CYBER SECURITY GUIDE FOR BUSINESS Acknowledgements The ICC Cyber security guide for business was inspired by the Belgian Cyber security guide, an initiative of