Midcontinent Communications Disaster Recovery/ Business Continuity Plan

Transcription

1 Midcontinent Communications Disaster Recovery/ Business Continuity Plan Disaster Prevention and Recovery As a provider of critical communications services, whether commercial or life-line residential, Midcontinent Communications has an obligation to minimize the impact of adverse events up to and including disasters. While many companies have disaster recovery plans, our unique position warrants that Midcontinent s focus is on disaster prevention with recovery components only for very low probability events outside the scope of prevention. As such, Midcontinent disaster planning is intrinsic to network and infrastructure planning as outlined in the following sections: Northern Plains Network Technical Facilities Central Technical Facility Human Resources Standby Power Fiber optic networks require sites for periodic amplification or regeneration. These sites can be primary facilities where service is dropped off (see facilities below) or OTNs (optical termination nodes). OTNs can be small buildings, huts, pedestals or even pole-mounted enclosures. NPNet contains thousands of OTNs. Short duration commercial power failures are common on a widely-distributed network. All components of our fiber optic network are deployed with standby power in the form of battery backup. Longer duration power failures such as those resulting from adverse weather conditions or disasters are less common but tend to occur somewhere on a yearly basis. Because of this, key locations on the core network also have LP, natural gas or diesel generators in addition to batteries for sustainable power backup. Core locations that are small and remote are equipped with extended battery banks for multi-day continuity. Northern Plains Network The Midcontinent fiber optic network (NPNet) provides private business circuits, voice, Internet and video services across a three state region. NPNet is our primary asset for providing service to customers. Path Diversity and Owned Fiber Core network paths are arranged as a ring or mesh architecture using diverse paths, which means key network locations have two routes in and out. The switching equipment that makes up the network is capable of choosing either path within 50 milliseconds. In the event of a fiber cut or facility event, the network is routed to an alternate path without impacting service. Upon subsequent repair, traffic is re-routed automatically and path redundancy restored. Whenever possible, Midcontinent owns, installs and maintains the fiber optic cable of NPNet. In some situations it may be necessary to lease fiber to complete redundancy. In the case of recent acquisitions or smaller, remote locations, NPNet may contain non-diverse and/or leased circuits until expanding the redundant core is warranted. Standby power sites in metro edge locations such as those in residential neighborhoods are numerous. Covering each location with continuous power backup beyond batteries is not practical. For edge locations, portable generators are maintained to provide continuous coverage to 10% of all locations in addition to their battery banks. Most extended power failures are localized and 10% is sufficient coverage. For city-wide outages lasting several days, portable generators are pulled in from adjacent communities as every community maintains a reserve of portable generators, trucks and qualified technicians. Concentrated business parks are considered key areas and are equipped with extended standby power capability. In addition, all standby power; small or large, is monitored by the NOC. Network Operations Center The Midcontinent Network Operations Center (NOC) is staffed by trained professionals, 24 hours per day, 365 days per year. The NOC is tasked

2 with monitoring all network components, basic troubleshooting, escalation, dispatch and ticket tracking, and serving as a central communications and command center. All network components are deployed with status monitoring that reports back to the NOC. The NOC monitors several management consoles and acts upon events or critical trends as appropriate. The NOC also schedules and coordinates planned maintenance and service advisories. The NOC has access to dynamic on-call scheduling as well as static escalation lists for various critical events and disasters. In the event of a disaster, the NOC executes an established escalation and communication plan to ensure that the right resources are notified and dispatched to problem locations. Public Network Interconnections NPNet includes interconnections with many public networks for business, Internet, and voice traffic. These interconnections are considered critical nexus points and receive special consideration for recoverability. First, Midcontinent maintains connections with multiple providers each capable of providing full or partial level redundancy for each other. Should we lose an entire Internet provider for instance, we have the ability to maintain operations with the remaining providers. In addition, provider networks are always reviewed when selecting new circuits to ensure that provider paths don t overlap geographically. Second, all interconnections to primary facilities remote from our own are accomplished by diverse network paths consisting of at least 50% Midcoowned fiber. This ensures instantaneous recovery from fiber cuts or localized disasters. Lastly, all equipment serving these local interconnections or MANs (metro area networks) have redundant power, components, spares or a fourhour depot replacement plan. As indicated above, all interconnection points are monitored by the NOC. We also monitor these points in the long term for capacity prediction since the turn up of additional service can have a long lead time. Network Security Midcontinent maintains a security manager who is organizationally separate from the network staff. The security department is charged with maintaining and auditing network security and proactively preventing and testing for threats. Network security is multi-layer with different protective zones for different resources. Technical Facilities Throughout our footprint, Midcontinent maintains several primary technical facilities. These reside in larger metropolitan areas and serve a region of smaller communities. Each facility is partially responsible for service delivery and maintenance in the region and also participates as part of the larger NPNet network. These primary facilities are considered critical. Geographically Diverse and Redundant Equipment The most important characteristic of our facilities from a disaster prevention and recovery aspect is that they are geographically diverse. Since the upper Midwest is not prone to hurricanes or earthquakes it s very unlikely that a disasterlevel event will affect multiple locations. The loss of one site will have a less than catastrophic impact on the network as a whole. All critical equipment is deployed in a redundant fashion or with spares available. On a temporary and emergency basis, the spares or redundancy of one site can be moved to recover a damaged site. In effect, each of our technical facilities has a backup depot nearby, but not near enough to be affected by the same disaster.

3 Internetworked Each of our primary sites is internetworked with other sites. We can move equipment between facilities as mentioned above but if some of the facility network remains intact we also have the option in many cases to restore service by leveraging this internetworking and using the resources of a sister site. Physical Security Each of our primary facilities is considered a secure location. Customer interaction is conducted at locations separate and distant from the technical facility. Most buildings are manned by technical staff during working hours. Non-manned facilities are NOC monitored and alarmed. Doors are locked at all times and entry controlled by a proximity card and ID system. Photo IDs must be worn by all personnel at all times. Facilities with staff have an additional entry challenge to specific technical areas accessible only by approved technical and managerial personnel. All door entry is logged with records stored at the NOC. Contractors must verify with on-site personnel and wear ID badges at all times. In remote locations contractors must be escorted at all times. Fire Suppression Each of our primary technical facilities is protected by fire alarming. The main technical areas also utilize an FM200 fire suppression system that is maintained and inspected on an annual basis. FM200 is a gas-based fire suppression system triggered by heat or smoke. Once deployed, FM200 extinguishes all fires in six seconds and doesn t damage or otherwise adversely affect running equipment. Standby Power Each facility is equipped with continuous standby power. The usual configuration is batteries for brief outages in technical spaces and generator backup for the entire facility for sustained power. In all cases, the generator is sized to backup HVAC to maintain proper operating temperatures. Generators are exercised on a monthly basis and batteries tested annually if not NOC monitored. NOC As mentioned previously, the NOC monitors all facilities and will issue alters and escalations as required per established procedures. The NOC also handles security and in some cases provides surveillance and alarm monitoring. In addition to standard security and equipment monitoring, the NOC also provides environmental monitoring for each facility and will detect spikes in temperature, power, etc. Central Technical Facility (CTF) For cost and practicality reasons, Midcontinent maintains a central technical facility that contains equipment that is not readily distributed or has better availability and less vulnerability as a centralized asset. Our voice call server is the most notable example. In the case of the central technical facility (CTF), extreme measures are taken to ensure protection and redundancy. Staffing Staffed 24 x 7, the NOC is housed in the CTF and provides security for the entire facility backed up by standard alarming. All staff is security trained and aware. ID badge and contractor procedures are strictly enforced. NOC personnel and equipment are housed in an inner physical security zone. CTF Physical Security The CTF has a multi-layer security system consisting of two main perimeters. The outer perimeter doors are locked at all times. Entry requires a proximity photo ID badge and all windows and doors are secured afterhours with an intrusion detection system. The inner perimeter is also card restricted but in combination with personal identification for entry in the form of a PIN number. Egress is monitored with an antipassback system that requires a card swipe for exit or all security is revoked. The entire building is monitored by multiple full-motion color security cameras including all doorways, passage areas and critical technical equipment. All security video is recorded and time stamped for correlation with other events.

4 Structural Various physical security and anti-terrorist devices are employed including external bollards for vehicle protection, fencing and other non-disclosed measures. We are located in an urban area that is not on a flood plain. Our only adjacent building is a highly secured federal facility. The consulting firm of NCompass has estimated the CTF as being able to withstand an F2 Tornado. Fire Suppression and Flood All technical areas are protected by an FM200 fire suppression system, both above and below our raised floor. Technical equipment is not housed in low-lying areas and all such areas are equipped with NOC monitored water alarms. Power and HVAC The entire CTF is backed up by a series of battery banks (AC and DC) and a generator capable of providing continuous power to the entire facility. The generator is exercised monthly and the batteries are maintained by Midcontinent testing procedures and equipment. Cooling is N+1 redundant. We can lose any single unit and still maintain a normal operating temperature. Warm Site Backups Two additional Midcontinent facilities exist in reasonable proximity to the CTF interconnected on a ringed fiber network. In the event of a catastrophic disaster these sites will be used temporarily as warm sites to recover the functionality of the CTF and provide facilities for vendor replacement equipment, network and power access. Off-Site SAN and Backup Midcontinent maintains a SAN (storage area network) backup system for data that s capable of SAN storage. Other data is backed using traditional methods and stored off-site in a fireproof vault. The SAN is a shared, fully redundant hot-swap configuration. It is configured as two banks of disks that are geographically separated in different buildings miles away from each other. Redundant Equipment All equipment that supports redundancy is deployed as such. This means N+1 or 1+1 component redundancy. In certain cases redundancy also includes multiples of the entire unit. Specific, key resources are listed below: Call Server (phone switch) Complete N+1 component redundancy. Core network routers 1+1 processor and power, deployed on identical meshed routers. Internet routers 1+1 processor and power, deployed on identical meshed routers. DNS servers Deployed in load sharing pairs. DHCP servers Deployed in load sharing pairs. NOC servers and monitoring Deployed on VMware virtual redundant blades with 1+1 power. HUMAN RESOURCES Diverse Management and Succession Midcontinent maintains a diverse management structure with primary manager offices in multiple facilities in Sioux Falls, Minneapolis and several towns and cities in our region. In addition, we have documented succession plans that are refreshed yearly and stored at the office in Minneapolis. No single natural disaster or act of terrorism can entirely remove the management structure at Midcontinent. Travel Restrictions Midcontinent maintains a travel policy that limits the number of key personnel on charter flights. In addition, all such flights must be on a twin engine aircraft with two pilots.

5 Distributed Technicians and Engineers One of the strengths of NPNet is that we have general managers, technicians and engineers dispersed throughout our coverage area, each capable of responding to events or disasters in other areas. Flooding or an ice storm in one community will call in the resources of nearby communities within hours to implement disaster procedures and repairs. Pandemic Midcontinent maintains a pandemic team that meets regularly to discuss and plan actions in the unlikely event of a pandemic. Due to our extensive network that includes residential access, we have a unique capability to work from home en-masse to keep our services going without promoting the spread of disease by direct contact. In fact, we ve responded to inquiries of government agencies regarding this capability. Should a pandemic occur, Midcontinent expects to be an integral part of the solution for many businesses and government entities. Name: Mark Powell Signature: Title: Vice President of Business Solutions Company: Midcontinent Communications Address: 3901 N. Louise Ave., Sioux Falls, SD Telephone: Address: mark_powell@mmi.net