Business Recovery Planning for Communications Leo A. Wrobel

Transcription

1 Business Recovery Planning for Communications Leo A. Wrobel Payoff Most businesses depend on their telephone networks to keep customer communications open. Lost service spells lost revenue. This article discusses the preliminary activities involved in getting a communications recovery planning project approved. Low-cost strategies to help prevent voice communications outages from occurring or to minimize their disruptive effect are emphasized. The information is intended to help IS and user managers who are assessing the risks in their organizations or are preparing to evaluate more advanced network protection measures. Problems Addressed Computer disaster recovery centers, which were merely a concept 10 years ago, currently represent a three-quarter-of-a-billion-dollar industry that serves to protect critical information systems. Surprisingly, however, of the 80 or more major computer outages that have led to the activation of a computer recovery facility since 1986, only a fraction resulted from an actual computer failure or from such incidents as flood, fire, or sabotage. In fact, the chief source of disaster has been telecommunications disruptions. In other words, organizations have had to mobilize expensive computer recovery facilities simply to use their telephones. All told, there have been no fewer than six major network failures in 1991 alone, affecting New York (three times), Washington DC, Atlanta, Los Angeles, and locations in New England. These failures do not include the scores of smaller, more regional disasters that happen frequently but do not make national news. One such incident involved a software failure in the Central Office of a Dallas telephone company that cut off phone service to a major hospital for more than three hours. In another incident, lightning struck a major long-distance carrier's facility, producing a day-long network outage. Hundreds of cable and fiber-optic cable disruptions also occur yearly that virtually isolate users, even though they may be unaware of the disruption. These disruptions take on a whole new dimension as the trend toward business automation and, in particular, online systems continues. Incoming 800 numbers and online point-of-sale terminals are ubiquitous in a modern economy. Companies have become highly dependent on the network for key elements of operations and customer service. When the telephone stops ringing, so does the cash register. This article discusses the preliminary work needed to get a communications recovery planning project approved. It then reviews the major threats to voice communications and recommends some low-cost strategies to eliminate avoidable risks and minimize the impact of risks that cannot be eliminated. Business's Reliance on Communications Many professionals in the disaster recovery and IS management fields have had to take a hard look at the communications network. Contingency planning is always timeconsuming and complicated, even when the manager's efforts are directed toward a more familiar IS environment. Compiling a contingency plan for the communications network is

2 especially complicated; still, the responsibility falls, for better or worse, on the business enterprise. A great deal of interest is centered on how to safeguard voice communications as part of an organization's overall disaster recovery plan. Failures of voice networks have plagued users over the past few years with ever-increasing frequency and severity. As a result, companies and disaster recovery service providers are designing more broad-based business recovery solutions in which voice communications plays an integral part. For example, in a service-oriented company, the average workstation serves as the front-line interface to the customer. Incoming calls are answered and some type of information processing takes place in real time an order is taken, a product is sold, or a service is rendered. These activities generate the cash flow that supports the business. Airline and transportation companies, bank card validation centers, or various inbound or outbound marketers rely heavily on workstation-based computing. In each case, the lowest common denominator is the person who calls in, requests a service, or purchases a product. In a service company, when the telephone is not answered, revenue is not produced. No amount of additional support personnel, money, or other resources can alter this fact. Assessing the Impact of Loss of Service A properly designed business recovery plan for such companies must require that not only individual workstations be restored but also incoming telephone service that turns customer calls into revenue for the company. Restoring computer function matters little if the customer cannot make use of it. On the surface, this example seems to reflect common sense; in practice, recovery of communications can be difficult to achieve without preplanning. The company may, for example, use a wide array of communications services, including incoming 800 numbers, outgoing Wide-Area Telecommunications Service lines, Direct Inward Dial circuits, custom circuits (e.g., private data and voice lines), and software-defined networks. It may also employ automated call-distributing (ACD) systems and other special call-handling equipment that resides on site. The risk to the company, then, includes not only internal threats to resident equipment from fire, flood, and sabotage (as in computer contingency planning) but threats from outside sources. Both can have an equally devastating effect on operations when disaster strikes. The major area of external exposure is undoubtedly the communications network. Preliminary Activities in Recovery Planning A persistent problem in addressing the need for network protection is the availability of funds and the required management support for these resources. Justifying money and people for disaster planning is difficult no matter how urgent the need. Developing a fullscale communications recovery plan is also a long-term project that has an average life of two or more years. One logical method of addressing the recovery planning project is to deal with it in three phases. Defining the Exposure in an Executive Summary Phase one examines what the business is really trying to protect and why. It involves a preliminary study of critical business systems. Senior managers in the major business units must be asked to estimate what the financial damage to the overall company would be

3 in the event of a communications outage affecting a critical system (e.g., a telemarketing center or order desk) for the duration of n days. The resulting figures are then crosschecked with other departments and with corporate financial managers to provide a reliable (although not exact) assessment of financial loss. The final report or presentation is made to senior management. This report must always concentrate on the business, not the technical, issues. Assigning a dollar value to an hour or day of downtime is an excellent strategy for obtaining management support, in terms of both money and personnel, to continue the project. Educating Personnel and Setting Audit Standards The IS department and internal audit function usually have defined and documented acceptable standards for auditing a data center's ability to recover from a disaster. A similar set of standards should be developed for the communications department and for the network itself. Phase two involves formulating these standards through a cooperative effort among departments. The result of this effort is a document everyone can claim ownership of and that ensures continuity so that all the involved departments IS, communications, operations, and internal audit are prepared for the third phase(implementation) of the project. Implementing the Network Disaster Recovery Plan It is not unusual for the implementation of a network recovery plan to take anywhere from 18 to 30 months. Whereas phase one involves only a high-level assessment of exposure, this phase deals with the specifics of the plan and requires detailed interviews with users and department managers and with personnel from virtually every internal department, from human resources to building security. The depth of study required accounts in part for the long time frame for implementing the plan. However, to be considered a true business recovery plan, it must include input from all departments. Contingency planners must tap the appropriate personnel to provide necessary information on market share, salaries (required to estimate losses in productivity in the event of a disaster), or legal issues. In addition, most of the organization's equipment and network vendors need to become involved and major network modifications may have to be made over time. During this phase, the company will remain exposed to many types of communications disasters; however, it can take a few proactive steps almost immediately to decrease the exposure. Much of the capital-intensive part of the task has already been paid for by the acquisition of the hardware set in place by the telephone companies. A local or long-distance carrier network can perform many disaster recovery functions; the organization must simply learn how to take advantage of them. By knowing how to avail itself of such technology and by taking a few rudimentary steps before disaster strikes an organization can improve its resistance to telecommunications disruptions immeasurably and at low cost. Identifying the Four Threats to Voice Communications One way to gain an understanding of the nature of the various threats to voice communications integrity is to divide the threats into four categories: The loss of the network's switching capability (e.g., a software glitch).

4 The loss of a network serving facility (e.g., a Central Office fire). Isolation from a network provider's serving facility (e.g., a cut cable). An internal disruption (i.e., a disruption inside the company's building). Each of these threats is discussed in the following sections along with an example and some suggestions for steps that can either prevent such outages from occurring or minimize their disruptive effect. The focus is on low-cost alternatives that can be undertaken with relative ease. This information is intended to help IS managers who are assessing similar exposure in their organizations or are preparing to evaluate more advanced network protection measures. Loss of Network Switching Capability The most notable example of this type of outage was the January 1990 failure of the AT&T network that resulted from a glitch in software that was, ironically, designed to increase reliability in the network. Approximately 50% of AT&T's regular traffic volume was blocked for most of a business day. Hardest hit were companies that depended on incoming 800 services or software-defined network services. The fact that Fortune 1000 companies had alternative carriers mattered little because incoming 800 numbers are not portable, or transferable, between carriers. Consequently, a user whose incoming AT&T 800 numbers were inoperative because of the outage could not have its other carriers process those calls, even if the connection to these carriers was available. Software-defined networks were similarly affected because they rely solely on the switching services of a single carrier, in this case AT&T. Long-Distance Access Codes During this type of outage, outbound services at least can be easily restored through the use of the long-established 10XXX dialing pattern. The surprise resulting from the AT&T incident was just how uninformed users were about the availability of 10XXX dialing or five-digit dialing codes. Because all major carriers can be dialed through an equal access end office, a disrupted long-distance user can in essence dial around a problem by selecting the access code for a long-distance carrier not affected by the outage (see Exhibit 1). Using 10XXX Dialing For example, an AT&T user who would usually dial 1 (214) could have instead dialed 10222, 1 or 0, then the regular number to complete the call on the MCI network. Similarly, the five-digit code would access US Sprint. It is ironic to think that the most widely available and useful and least expensive alternative for restoring long-distance calling capability was unknown to so many users. Companies would be prudent to keep a listing of such codes on hand at all times, because the long-distance company may not be able to provide them when an outage occurs. These codes are a costeffective insurance measure against long-distance outages.

5

6 Loss of a Network Serving Facility In any business, a total loss of a major facility is always a possibility, and communications providers are no exception. There have been dozens of cases of telephone Central Office being rendered inoperable over the past 10 years. Causes have included fires, floods, lightning, tornadoes, hurricanes, switch failure, and human error. The most far-reaching Central Office disaster occurred in May 1988 in Hinsdale IL when a fire in a suburban Chicago central office affected not only the 49,000 local telephone subscribers but more than half a million other users nationwide. Central office disasters are always messy; they affect a large area and the damage is not limited to basic dial-tone service. Other services that may be affected are data and voice private line circuits; cellular, paging, and long-distance access lines; and 911 emergency trunks. Given the magnitude of such disasters, however, a few measures can be taken at low cost by alert contingency planners who think ahead. Command and Control Emergency Response Cellular telephones operate from mobile telephone serving offices(mtsos) that are often diverse from an affected central office. Although only a limited number of cellular phones can be used in a given area because of limitations of the cellular network, they are indispensable for command and control. Other technologies not to be overlooked are radio telephones and two-way radios. Radio telephones can often be used for phone patches back to the public network by way of repeaters, whereas two-way radios are useful for communications in the immediate area. After police, fire, and other emergency services, the next class of service restored following a major central office disaster is usually pay telephones. Because Asynchronous Transfer Mode and other banking operations may be down for a significantly longer time, managers should keep some cash on hand, some of it in quarters, for incidentals. Isolation from a Serving Facility A cable cut is probably the most common cause of a communications disruption. A cut cable can affect any of the thousands of high-capacity fiber-optic lines that crisscross the United States. One well-publicized AT&T outage in New York was caused by a technician who unknowingly cut the wrong underground fiber-optic cable. Unfortunately, this particular fiber contained a 1.7G-bps carrier, or about 240,000 circuits (about 60% of New York's capacity, by some estimates). Often there is precious little that can be done about human error. People are fallible and accidents will happen, even to the best-prepared companies. Reasonable precautions include enforcing strict adherence to written policies as well as to procedures for maintenance and safety. Effective change control management of major software upgrades and revisions can also be an effective way to prevent disruption. Assessing Communications Rights-of-Way Telephone, electric, fiber-optic, gas, water, television, and other services are usually installed in the same public rights-of-way, mainly along streets and thoroughfares. In many cities, especially older ones like New York, Boston, and Philadelphia, these rights-of-way

7 date back decades. The result is that when contractors dig to repair or install facilities, they are never quite sure what they will hit on the way down. Although being invisible makes a telephone cable facility more secure, it makes it vulnerable in other ways. Record-keeping can be a problem, for example, when rights-ofway have been in continuous use for decades or more. Some companies have begun to use innovative schemes for installing communications facilities by laying fiber-optic cable in abandoned steam tunnels and subway tunnels. Although this eliminates one problem by making the facility more accessible, it can create another by making the facility visible and therefore vulnerable to disruptions caused by vandalism or tampering by unauthorized personnel. There are many methods to protect and diversify cables in a network. One inexpensive precautionary measure that addresses part of the problem is to meet with the account representative of the local operating company to assess the company's exposure in this area. Many times, alternate or diverse routing is available in the company's service area for little or no cost. Alternate and Diverse Routing. IS managers should understand the terminology and the difference between alternate and diverse routing. Many users, and the telephone companies for that matter, use the terms interchangeably. Generally, alternate routing refers to any cable facility that is different from the facility to which the company is currently assigned. Diverse routing refers to the use of not only a different facility but a completely different physical path. Most telephone companies can provide alternate routing with relative ease. Diverse routing is trickier, especially if the business is located far from its serving Central Office. An examination of the entrance facilities to the building is also recommended; it undermines the purpose of diverse routing if new facilities are simply brought in through the same building conduit. All construction activity in the vicinity of a business must also be taken into consideration. Pair Gain Units. Central offices and cables are not the only items to consider when the organization is assessing vulnerability in rights-of-way. Other components of the network that should be considered are telephone pedestals and pair gain units. The most common pair gain unit is called an SLC-96 pair gain unit (SLC stands for subscriber loop carrier). In areas where cable facilities are in short supply, distances are long, and laying new cable is expensive, these units are used to carry 544 simultaneous circuits on 16 pairs of copper cables. A fiber version also exists. Pair gain units work exceptionally well and have been broadly accepted by the local operating companies; however, pair gain units, pedestals, and fiber-optic repeaters are uniquely vulnerable to traffic hazards and can be suddenly disabled by an accident. For example, if a driver runs a car into a central office, a totaled car might be the only physical consequence. A similar collision with a pair gain unit would destroy the equipment, thereby isolating users. This is the same consequence as a total Central Office disaster, although probably for a shorter time because the equipment takes less time to replace.

8 Internal Disruptions Some of the least expensive insurance available to ensure network integrity involves the dedication of someone's time. It is a prudent idea to personally involve people from the IS department, because many contingency planning rules for the data center hold true for the communications network. IS Manager's Checklist A walkthrough of the site will enable IS managers to assess potential problems and to check that adequate precautions are being taken to protect the organization's communications equipment. A checklist of problems to look for would include the following items: Are all telephone closets and Private Branch exchange rooms locked and accessible only to authorized personnel? Do cable risers in multistory buildings pass through telephone closets belonging to other companies? Do they similarly restrict access, or do others have access to the organization's cables from other telephone closets? Are flammable materials stored in lower cable closets? Are the risers into the closets fire-stopped? Are backup copies of PBX software and assignments made regularly? Are they stored off site and included in the IS department's regular pick-up schedule? Are similar power conditioning schemes employed in the switch room as in the computer room? If not, can the switch room use computer room resources already in place, such as power conditioners and uninterruptible power supply (UPS) equipment? (The IS department's expertise in the area of power conditioning is valuable and should be shared.) Are sprinklers, air-conditioning pipes, and regular building plumbing located above the equipment? Do low sprinkler heads exist over pathways where equipment is rolled in? Are pipes in an area where they might freeze? Infrared Scanning The same technology that allows space satellites to look through clouds can be employed to identify electrical faults and shorts in walls before they cause fires. Infrared scanning is highly recommended for businesses residing in older buildings with dubious wiring or in new ones to reveal slipshod construction. AT&T and other companies provide infrared scanning (also called thermographics) service. Recommended Course of Action The interruption of critical business systems and communications networks is a risk that is often ignored until a disaster happens. Corporate management must shoulder the cost of a disaster recovery planning project. One way to influence corporate management is to initiate a study to quantify the extent of the company's dependence on the network and to quantify the impact loss of service would have on the company in terms of:

9 Lost sales (i.e., direct revenues). Lost market share. Lost customer confidence. Lost productivity (e.g., idle employees). Legal liability issues. These are terms that are understandable to executive management, and they will assist contingency planners in securing the necessary support and funding for the disaster recovery planning project. These figures must be quantified with the corporate controller, the vice-president of marketing, the sales vice-president, or corporate counsel, as appropriate. Consultants and outside resources can also be brought in when necessary. Establishing standards is another part of the disaster recovery planning project. Just as the organization has standards for the data center, standards for the network should be devised in connection with such physical security issues as fire protection and access control. Other issues directed toward the network itself for example, when backup circuits are appropriate must be included as well. All departments in the organization must have input to this process to ensure uniformity and compliance. After a formal network recovery plan has been implemented, all new services should be ordered with disaster recovery in mind. It is easier, for example, to require a switch, modem, or multiplexer vendor to provide guaranteed replacement times for equipment before the contract is signed. Otherwise, the organization could end up having to pay extra for the same service later. The task of disaster recovery planning for the communications network need not be put off because of budgetary constraints. There are many concrete actions the IS manager can and should be taking, both independently and with the aid of vendors, to protect the organization's critical network resources. Fault-tolerance in the network does not happen overnight. It evolves over time, through the collective impact of even the most minimal efforts, such as those described in this article. Any steps that can be taken, however small, result in a higher level of service to the company's customers and greater peace of mind to the organization. Author Biographies Leo A. Wrobel Leo A. Wrobel is president and CEO of Dallas-based Premiere Network Services, Inc., a telecommunications and disaster recovery consulting firm. He is a consultant in matters of telephone regulation and is also a city councilman. He is a member of the IEEE, the DPMA, and the Association of Contingency Planners.