5-04-25 Adding Communications Network Support to Existing Disaster Recovery Plans Leo A. Wrobel Payoff This article reviews the processes that must be documented in a recovery plan for a company's mission-critical communications networks. Policies and procedures help ensure network integrity and prevent disasters. The emphasis is on ensuring continuity between IS, facilities management, LAN management, and other departments, so that a plan can be executed flawlessly. Introduction A broad-based business recovery plan must address three critical components: physical space for employees, connection to data processing systems essential to the conduct of core business operations, and telecommunications facilities that turn these data processing systems into revenue generators for the company. There are several ways to dovetail communications systems into an organization's existing disaster recovery plan for its mainframe computer room. Phase 1: Business Risk Analysis What the Organization Needs to Protect and Why This first phase involves preliminary identification of mission-critical communications systems. It may be necessary to run a series of executives interviews within the company to identify core business systems as well as the communications systems that support those activities. Examples include inbound call centers, customer service lines, engineering or R departments, sales departments, and divisions involved in financial filings for the company. Management may need to be convinced that recovery planning for the communications network is an important and essential component of the overall business recovery plan. A helpful technique is to draft a white paper assessing the risks to the company and presenting them in nontechnical language that management can understand. To be most effective, a white paper to management should outline the big four areas in which communications systems disruptions cause a loss to the company. These include: Lost sales. Lost market share. Lost customer confidence. Loss of productivity. These are all things management can understand and subscribe to. Focusing on these issues will further the cause within the organization.
Phase 2: Updating Procedures The second phase of a successful communications systems recovery planning effort involves becoming up-to-date in disaster recovery planning methodologies for the network. IS may want to consider establishing some type of liaison with service providers geared around the disaster recovery effort. It is also time to talk to related departments within the organization, such as security personnel and facility management, which may already have disaster recovery plans that network support plans can be rolled into. Operating and Security Standards One of the most significant tasks in phase 2 is documenting a set of operating and security standards for communications systems. These standards are essentially the basic operating practices for the network, and they are designed for two reasons. The first reason is to ensure that disasters are prevented before they happen. Policies and procedures help maintain network integrity and prevent disasters. Standards that prevent disasters include policies on the management of combustibles - for example, no smoking policies, training in the use of fire extinguishers, and standards for change management when making software changes to mission-critical systems such as Private Branch exchange or multiplexers. The second reason is to ensure that the emergency procedures dovetail gracefully with the operational environment. By working together, related departments such as IS, computer operations, LAN management, facility management, and others can avoid the perception that they are trying to impose a solution on another department. This approach also ensures continuity between the departments. The following basic security standards should exist Equipment rooms lock and signin logs exist for people entering and leaving the area. PBX class-of-service indicators are backed up daily and stored off site, similar to procedures in the computer room. Passwords are changed frequently for dial-in maintenance access to critical multiplexers, PBXs, and voice-mail systems. Trash is not permitted to accumulate in equipment room. There is a no smoking policy. Basic housekeeping procedures exist within the equipment room. If possible, the equipment room is located in an area other than the basement. Any water problems that develop anywhere within the building will ultimately end up in the basement. There are regular surveys of the cable routes between the organization and the local service provider. Infrared scanning equipment is used to pick up heat sources within computer or telecommunications rooms and thus help avert fire. Such equipment is available from fire protection contractors and other sources.
Power is separated from electrical cables. In addition to being a cause of noise and interference, electrical cables in telephone cable racks are also a safety hazard, sometimes leading to catastrophic fires. Fire-retardant cable is used in equipment rooms. In addition to the traditional Teflon cable that resists burning, there are also newer materials available, such as Halar, Kevlar, and Stolsis. Permanent Virtual Circuit, or polyvinylchloride cable, can burn and produce nauseous fumes. When water is poured on burning Permanent Virtual Circuit cable, it creates acid compounds that can rapidly destroy equipment. Emergency instructions are prominently posted in the PBXs room and adequate command and control exists to send messages rapidly should something go wrong. An additional checklist of standards is presented in Exhibit 1. Other standards are geared specifically toward the recovery process itself. For example, if emergency procedures call for a list of home telephone numbers for employees who need to be called back to work, something must be documented in the operational environment to ensure that list exists in the first place. Responsible people should also be assigned to keep the list up-to-date. Similar policies must be in place for equipment inventories, vendor callout lists, and other components of the emergency plan that rely on the standards to execute properly. Checklist of Communications Systems Standards * Password protection of remote maintenance port dial-in access, DISA, and DATA dial-in. * Fraud protection on DISA through use of caller ID, DISA, and other methods. * Smoking ban in effect in equipment room. * Separate power breakers for sensitive telecommunications equipment. * Instructions posted for human safety and for graceful equipment shutdown in equipment rooms. * Back-up power tested frequently. * Lightning protection where applicable. * Emergency lighting. * Equipment room: locked door, sign-in logs, posted emergency procedures. * Water pipes labeled, under-floor moisture detectors installed, plastic sheeting of drape equipment stored nearby. * Sign-off procedures for major equipment or software changes. * Policy of performing back-up before major telecommunications equipment changes. The last part of Phase 2 involves making long-term recommendations for the network. Because it is usually impossible to scrap equipment that is already installed, much of this equipment may have to be phased out over time to allow for disaster recovery plans. At minimum, specific recommendations on long-term network changes to be executed at an appropriate future date should be made. Phase 3: Documenting the Plan A solid, systematic set of disaster recovery procedures can be summed up using the seven R's of a successful recovery planning process. Recognition
If a night security guard sees water coming under the door of the equipment room, who does this guard notify, and how, precisely, would the emergency call be routed through an organization? Instructions should be displayed prominently within the room with callout numbers for key technologists who may have to respond immediately to a disaster. Procedures might exist, for example, whereby the director of facilities would call the director of technical services in such an event and request an on-site representative. The facilities department must know what steps to take for human safety, such as shutting off power if the equipment appears wet. These and dozens of other issues have to be addressed to ensure that everyone is called quickly and can respond as quickly as possible to any type of facility disaster affecting communications systems. Response Once key personnel have been called, what exactly are they needed to do when they arrive on site? One suggested approach is to immediately open a critical-events log. A critical-events log need not be more complicated than a small notebook or a handheld voice recorder. It is important, however, because many command decisions are going to be made in rapid succession and need to be tracked. This permanent record of command decisions will be useful later, either for assessing liability or for reassessing what went right and what went wrong in the recovery plan. The name of the game in the response phase is to arrive on-site, execute a successful callout of key personnel and vendors, and make a report to management within 90 minutes or some prespecified time of the disaster, explaining how serious the disaster is, whether it will involve other departments to recover, and providing some estimate of how long it will take to recover, as well as whether a companywide recovery plan should be activated because of the communications system disaster. Recovery Getting back to business as soon as possible is the objective. This recovery process should be documented to a level where it involves technical personnel, such as LAN or mainframe personnel, to execute the plan in the event communications personnel are unavailable to effect the recovery process or are injured in the disruption itself. It is important to note that recovery does not mean restoration of the original equipment; rather, it means restoration of the business process that the equipment provides, even if it is in some type of degraded mode. For example, a large department may have 50 telephones. In a disaster, the plan may be to provide only 25 telephones, but to add a second shift. Not everyone within the organization needs to work 8 to 5. This is why an understanding of the core business is so important to create a flexible and workable recovery plan. Telecommunications personnel will also have to be dispatched at this time to commercial computer recovery or business recovery centers to which the company subscribes. Restoral Close interdepartmental coordination is important during the restoral phase of a recovery process. For example, the communications systems manager has certain responsibilities for wiring, but a LAN manager has others, and the facility manager, responsible for electrical power, for example, has still others. These responsibilities should be carefully documented and delineated to ensure the correct type of wiring is installed. Return to Normal Operations
When the emergency is over, it is then time to tear down any emergency configuration and go back to business as usual. If the recovery center is stable and operating, and the revenue stream of the company is firmly established, all new configurations still must be adequately tested before migrating back to the original site. This includes documenting in the recovery plan what constitutes a successful test before going back to the original network configuration. Rest and Relax Needless to say, after responding to a disaster, employees will be tired and stressed out and probably at their wit's end. Therefore, it is important to schedule compensatory timeoff so the staff can get some rest after what could have been several days or weeks of 12-hour shifts. Regroup and Reassess After any execution of the communications systems recovery plan, whether it is a test or a full-blown recovery implementation, it is important to go back and reassess how effectively the procedures worked and make adjustments within the plan. This is part of the reason for the critical-events log during the recognition phase of the recovery effort. Adjustments that are made after tests or activation of the plan strengthen the plan in the long run, so that it can be expected to execute more flawlessly the next time. Other considerations in a successful communications systems recovery plan include: Defining a meeting place to coordinate recovery activity. This could be any suitable real estate located off-site. It should also be equipped with a small contention of telephones, fax machines, and supplies, and serve as the focal point for command and control for recovery activity. It may also house the emergency management team (EMT) that coordinates the overall disaster response. Defining an emergency management team of executives for communications systems disasters, and appropriate recovery teams for both the on-site and offsite recovery processes. Teams and their designed back-ups should be defined for: Dispatch to a recovery facility. Coordination of on-site recovery activities. Retrieval of off-site magnetic media. Administrative functions. Keeping employee callout lists and home telephone numbers current. The best way to do this is to import them, perhaps over a LAN, from known reliable sources, such as human resources. Establishing procedures for maintaining human life and safety when reentering damaged facilities. These would be procedures such as immediately shutting off power and other precautions before entering a damaged facility. Keeping an inventory of all equipment that will be required for the recovery process and all equipment installed on-site. One way of doing this is to establish a liaison with the accounting department. Whenever new equipment is purchased and accounting receives a copy of the contract for the equipment purchased, accounting could be asked to update a data base with such information as the equipment's serial number, software revision
number, date of purchase, and number of months the equipment is amortized. In a disaster, this list can be created quickly and used as the basis for fast command decisions on whether to scrap or attempt to save damaged equipment, depending on when it was purchased and what the original price was. Lastly, be sure the plan adequately defines the roles between communications systems personnel and those from other departments, such as LAN management, operations, and facilities, to ensure coordination during a recovery implementation. Procedures on where to get cash, how to arrange travel, and how to purchase new equipment, for example, may be documented already within the organization by one of these other groups; these procedures can be adopted in the communications systems plan. Conclusion This article has reviewed the processes that must be documented in a successful communications systems recovery plan. The most important component of the plan is its ability to bring various departments within the organization together to ensure a seamless recovery process and a flawless execution of a companywide recovery plan. Whether the disaster is confined to the communications systems (in which case IS must recover on its own) or is a companywide disaster (in which case the department becomes a supporting player), the level of detail in the recovery plan directly influences how well it executes and how well protected the assets of the company are. In short, a detailed communications systems recovery plan equates to a higher level of network services and greater peace of mind to the company. Author Biographies Leo A. Wrobel Leo A. Wrobel is president of Premier Network Services Inc., in Dallas.