28 Jul Disaster Preparedness and Business Recovery Document (Sapulpa Location)

Transcription

1 28 Jul 2009 Disaster Preparedness and Business Recovery Document (Sapulpa Location) 1

2 Preface This Benefits Technology (BenTec) Disaster Preparedness and Business Recovery Plan describes the disaster recovery procedures planned for the safety, protection, and quick restoration of company computer resources and data assets in the event of an emergency or disaster and the process envisioned for implementation of these procedures. In addition, this document with attachments outlines the resources available to implement the plan. For additional technical information pertaining to this document, contact Allen Beaird ( ), Brad Armstrong ( ), or Maria Gee ( ) i

3 Table of Contents Preface... i Table of Contents... ii List of Illustrations... iii List of Attachments... iv Introduction Business Description and System Overview Emergency Response and Recovery Team Disaster Categories and Recovery Guidelines ii

4 List of Illustrations BenTec Business Process Telecommunications Configuration LAN/WAN Configuration Distributed Call Center iii

5 List of Attachments Emergency Contact List...Attachment 1 Telephone Equipment...Attachment 2 Network Server Equipment...Attachment 3 iv

6 Introduction Business Description and System Overview This document describes the BenTec Information Technology related Business Operations and Disaster Recovery procedures used to provide for the safety, protection, and restoration of company computer resources and data following disasters. This plan requires critical personnel to receive specialized training concerning emergency maintenance procedures, data backup and recovery procedures, damage control, and risk mitigation techniques. Emergency maintenance procedures, data backup and recovery procedures, damage control and risk mitigation techniques are required training for all system/network administrators and other individuals identified in this plan. Business Process Description BenTec provides employee benefit sales, service, and support to employees of its clients. The service and support provided by BenTec requires the extensive use of information technology including: Databases Web sites Electronic mail systems File servers Billing systems Scheduling systems FTP servers Telephony systems. BenTec sells insurance products to the employees of BenTec clients during employee benefit enrollment campaigns. Typically these enrollment campaigns last from one week to two months during which BenTec enrollment specialists assist employees with the selection and purchase of the employees' desired benefits. That assistance may come in the form of: 1. Field enrollers working one-on-one with employees and recording enrollment information via notebook computers, 2. Employee's calling the BenTec service center for voice only enrollment support, 3. Employee's calling the BenTec service center for cobrowsing support using BenTec developed systems and software. BenTec service center support may be provided by an agent located at the BenTec Sapulpa service center or via remote agents using VoIP phone technology. 1

7 Customer Location Customer s Public Telephone And Internet Service Providers Public Internet & Switched Phone System BenTec Public Telephone And Internet Service Providers Sapulpa Internet Firewall Agent s Internet Service Provider (Agent Phone is VoIP) Data Switch Sapulpa PBX Remote Agents Web Servers, Application Servers, Database, File Servers, etc. Phone Switch Sapulpa Agents Sapulpa Agents Sapulpa Agents Typical Enrollment Configuration Figure 2-1 Figure 2-1 illustrates the BenTec cobrowsing technology. To use the BenTec system, an employee accesses a web page via a computer connected to the internet. A message on the web page will instruct the client to contact an enrolment specialist (agent) by telephone. The enrollment specialist will greet the employee and ask the employee for the "key code" number displayed to the employee on the web page the employee is viewing. The agent will then use that key code to join the employee's computer session and view and control the information displayed to the employee using a process known as "cobrowsing." As the employee watches, the enrollment specialist takes over control of the display and explains each insurance product offering available to the employee. As each benefit is explained, the employee tells the enrollment specialist whether or not he wishes to order the product and the options and amounts desired. The enrollment specialist then completes the application forms as the employee watches. Digital enrollment data are stored in secure database servers located at BenTec's facilities in Sapulpa and the TulsaConnect collocation site in downtown Tulsa. The enrollment specialist may be located at either the Sapulpa service center or at a remote location serviced by VoIP voice links to the Sapulpa PBX system as shown above. System Description BenTec service center operations are currently supported by a 12 agent, expandable to 22 agent, service center operation located at 1200 Taft Street. Sapulpa, OK. The 2

8 TeleSynergy PBX system can be expanded via remote VoIP phone equipped agents by another 30 seats for a total expanded capability of 52 seats. The Sapulpa location data network is Win2003 based and telecom switching is provided by an advanced TeleSynergy PBX with ACD, auto-attendant, voice mail, a recording capability for 20 simultaneous extensions, FAX, and IVR capabilities. The primary PBX is backed up with a completely redundant failover PBX which, from a hardware and software perspective, is identical to the primary PBX. The service center desktop environment is primarily WinXP Pro. Power for the Sapulpa facility is provided via Oklahoma Natural Gas and Electric. Emergency power is provided by on-site UPS systems. Internet connectivity is provided from two sources, AT&T (DSL) and COX (DSL). Voice connectivity is provided via one standard business line (used for the BenTec central FAX and as the AT&T DSL number and by 1 AT&T PRI and a block of DIDs to (Additional lines, DIDs, and PRIs can be turned up as needed on approximately 14 to 30 days notice.) Additionally Sapulpa has two toll free outbound lines from CommPartners Connect ( and ) which are delivered VoIP to the BenTec PBX via PAP2 VoIP to analog adapters and a PBX analog trunk card. As currently configured, the Sapulpa PBX can support 30 local extensions and 30 remote VoIP extensions, 2 PRIs, and 12 analog trunks. BenTec services are provided by MS Exchange. Web servers are Windows 2003 IIS. Our enrollment system consists of a primary web server with a separate application server, and database server located at the secure TulsaConnect collocation facility. Our primary exchange server is also located at TulsaConnect. Our standby servers are virtualized Windows 2008 servers using VMware and are located at the Sapulpa office. Standby and production MS SQL servers are synchronized using transactional replication. Active and standby Exchange servers are synchronized using standard Microsoft Exchange Standby Continuous Replication. Development and file servers are located at the Sapulpa facility along with the production and standby PBX system. The TulsaConnect collocation facility provides primary power from OG&E, 40 hours of emergency battery time, and redundant diesel sources with 14 days of fuel on site. Additionally TulsaConnect provides redundant internet assess for BenTec servers via multiple carriers (Currently Time Warner Communications, Verizon Communications, Global Crossing, AT&T, and Level 3 Communications). Both Sapulpa and TulsaConnect location networks are continuously monitored by DELL Silver Streak software. Silver Streak provides the following functions: Firewall monitoring consolidates server logs into one view; reports on intrusion or attack activity; summarizes monthly activity, identifies whether or not the BenTec firewall is operating correctly. Intrusion monitoring identifies failed log-ins, account modifications, etc. Tracks "who did what to whom"-- this feature is commonly referred to as an audit trail. Also, documents whether or not someone is trying to access a resource he/she shouldn't to keep honest people honest after layoffs. 3

9 Real-time vulnerability scanning scans networks, systems, and applications for more than 2,000 known vulnerabilities, such as "backdoors", viruses, old software, configuration errors, etc. Also includes compliance reports, including "open ports"; and discovered windows users, services, and shares. Patch management identifies where patches are needed by device, server or desktop, by bulletin or by Microsoft product, such as Internet Explorer -- then remotely distributes and installs those patches. Multiple data backups are maintained by BenTec. Specifically, the BenTec central mail server maintains three backup copies of all mail folders. One located on a drive in the BenTec collocation facility server, one maintain off site by TulsaConnect, and one replicated copy maintained at the Sapulpa facility. Likewise the BenTec production database uses transactional replication to backup transactions from the TulsaConnect facility to the BenTec Sapulpa facility in real time. We also generate full database backup files (classic MS SQL ".Bak files") from the production server twice daily, once at noon and once a midnight. (Development database.bak files are generated weekly each weekend.) The Exchange server, file server and personal directories at Sapulpa are backed up each day using Backup Exec software to a series of tapes. These tapes are rotated each week with a complete weekly set of backups rotated each 10 weeks. Backup tapes are kept in a secure, fireproof safe located on site with one set maintained off site. Additionally, once a week a complete system backup is generated to a large external hard drive located in Sapulpa. PBX systems are configured with one primary and one secondary PBX server. The fully redundant servers are configured identically. The primary server is backed up weekly and then that configuration (along with all voice mails, message prompts, and configuration changes is restored to the failover PBX each weekend). Internally the Sapulpa network is protected via the use of Trend antivirus software. Procedurally, each morning we review system status event logs, PRI and VoIP line connectivity and identify and resolve any anomalies. Each month the Silver Streak system produces a detailed report and any anomalies or issues identified by Sliver Streak are resolved. 4

10 Emergency Response and Recovery Team Specific personnel are assigned to an IT Emergency Preparedness Team. This team is responsible for the implementation of the Disaster Recovery Plan should an incident occur. The members of the team and their roles are as follows. President The president (Dwight Pierce) is responsible for the overall resumption of business activities. His primary duties include making sure that adequate resources are provided and that all team members are kept abreast of the overall recovery process. The president will chair periodic meetings of the IT Emergency Response and Recovery team, where recovery plans will be reviewed. He may elect to test some or all of the Disaster Recovery Plan. He reviews and approves individual team members plans. He maintains an off-site emergency contact list including telephone and address information for key response team members as well as police, security system, phone system, ISP, and other emergency services. (Attachment 1) Vice President for Information Technology The VP for IT (Allen Beaird) is responsible for the overall coordination and implementation of the disaster recovery plan as well as setup of the Emergency Response and Recovery control center and resumption of Network, Telecom, and Desktop operations. His primary duties also include assessing the extent of the emergency, developing a list of resources needed to resume operations, overseeing recovery operations, and providing a recovery timeline to the president. The VP for IT may call on the IT resources of other BenTec divisions and Computer Point for additional support as required. The VP for IT will also be responsible for the resumption of service center enrollment/interviews and post processing operations. His duties in this area will include scheduling of customer service representatives, monitoring service center operations, and facilitating post processing operations. The VP, IT will attend periodic meetings of the IT Emergency Response and Recovery team. He will advise the president of alternate site suitability. He will perform a technical analysis of all plans, develop overall timelines, and recommend individual plans for approval. He will maintain a complete copy of this recovery plan both on and off site. Network Engineering The Network Engineer (Brad Armstrong) will assist the VP for IT with the setup of the Emergency Response and Recovery control center and resumption of Network, Telecom, and Desktop operations. His primary duties include technical support to other team 1

11 members and restoration of IT operations. He will work closely with the VP for IT and report and update recovery planning projects. The Network Engineer will also attend periodic meetings of the IT Emergency Response and Recovery team. The Network Engineer, in conjunction with the VP for IT, will develop a plan to quickly set up the Emergency Response and Recovery control center appropriate to the specifics of any disaster. This center will typically include emergency internet, , and telecom capabilities to support the Emergency Response and Recovery team. The Network Engineer will maintain off site sufficient equipment to implement this plan. The Network Engineer will also develop communication and network/server plans adequate to; restore minimal communication; LAN/WAN, router and firewall functionality -- and servers and desktops sufficient for service center operations. In addition to these duties, the Network Engineer must ensure that the latest upgrades, service packs, and hot patches be applied to all IT equipment. The Network Engineer must maintain system documentation and keep abreast of all applicable security advisories. Communications Manager The Communications Manager (Maria Gee) will assist the VP for IT with the resumption of service center PBX related and helpdesk related enrollment/interviews and post processing operations. Her primary focus will be on reestablishing Sapulpa provided services including Helpdesk services, , QuickBooks access, VoIP phone services, and enrollment platform support. 2

12 Disaster Categories Disaster Categories and Recovery Guidelines The following table categorizes disasters into four levels based on severity and duration of the problem. Level Description Examples Level-1 Catastrophic event requiring Earthquake, Fire, Storm, Ice, Flood, relocation of service center Extended Electrical, Internet or Level-2 Level-3 Level-4 Catastrophic incident adversely affecting service center operations Major incident adversely affecting service center operations Minor incident adversely affecting service center support Telecommunications interruption Earthquake, Fire, Storm, Ice, Flood, Extended Electrical, Internet or Telecommunications interruption Electrical Interruption, Multiple Hardware Malfunctions, Wide Spread Virus Infection, Security Breach of Firewall or Network, Denial of Service Attack, Loss of All Internet Connections or all Telecommunications Hardware Malfunction, Virus Infection, Denial of Service Attack, Loss of Internet Connection or Telecommunications Recalling the purpose of the plan is to provide for the safety, protection, and quick restoration of company computer resources and data assets, the first step in any response must be the determination of what systems have failed, the impact of those system failures to operations, the resources required to restore normal operations and the projected recovery time. The following chart describes critical BenTec systems and the equipment currently used to support those systems. System Equipment and System Dependencies Telecomm TeleSynergy PBX equipment, (PBX1, PBX2, TeleVNAT, TeleContact Servers) AT&T PRI, VoIP links from CommPartners or PAP2T Adapters, Internet connectivity for remote agents via COX and AT&T LAN Switches, MikroTik firewalls ( and ) Win Domain LAN, SRV-FS01, SRV-DC01, SRV-SQL01 Internet/VPN LAN, MicroTik firewalls ( ) ( ), Cox Coax to Ethernet modem, AT&T Modem, Cox & AT&T ISPs Production Web LAN, Internet, SRV-SQL01, SRV-WEB01, SRV-WS01, MicroTik LAN, Win Domain, Internet, SRV-EXCH01, MicroTik Figure 4-1 illustrates the current telecom configuration. Figure 4-2 illustrates the current LAN/WAN configuration. 1

13 AT&T Smart Jacks COX ISP Gateway INTERNET MicroTik Firewall PRIs AT&T ISP Gateway Catalyst 2950 Switch Remote VoIP Phone Equipped Agent PBX Server PBX Failover TeleContact TeleVNAT Servers Voice Recordings PBX TeleLEX Switch Remote VoIP Phone Equipped Agent Remote VoIP Phone Equipped Agent Remote VoIP Phone Equipped Agent Service Center Extensions and Agents Telecommunications Configuration Figure 4-1 The BenTec PBX system consists of a primary TeleSynergy PBX (TelePCX) server, a failover PBX serve, a TeleContact server (used to manage agent logins assignments and call statistics) and a TeleVNAT server used to support remote VoIP Extensions for remote agents. The BenTec PBX system currently has the ability to use up to 2 inbound PRI lines with failover -- or up to 3 PRI lines without failover. In addition to the primary PRI lines, BenTec currently uses 2 dedicated toll free outbound VoIP lines provided by CommPartners Connect and can increase the number of toll free outbound lines to a total of 12 with failover or 24 without failover. Additionally, in times of emergency we have a standard business line (currently used with our central FAX) that can be used for voice communications. 2

14 COX ISP Gateway MicroTik Firewall Sapulpa AT&T ISP Gateway Public Internet Sapulpa to Tulsa Connect Static VPN MicroTik Firewall Tulsa Connect Catalyst 2950 Switch Catalyst 2900 Switch SRV-SQL01 SRV-Web01 SRV-EXCH01 SRV-WEBVM01 SRV-WSVM01 SRV-FS01 SRV-NOC01 SRV-TS01 DEV-SQL01 SRV-SQLVM01 SRV-EXCHVM01 SRV-NOC02 SRV-DC01 SRV-WS01 TelePBX(prime) TelePBX(2) TeleVNAT TeleContact SRV-NOC03 Catalyst 2950 Switch Catalyst 2900 Switch Service Center PCs and Printers LAN/WAN Configuration Figure 4-2 BenTec maintains two server system sets. One set consists of a series of servers located at the remote TulsaConnect collocation facility and a virtualized set of servers located at the Sapulpa corporate office site. The primary production SQL server and the primary Exchange server are replicated to VMware virtualized servers located at the Sapulpa corporate office as shown in figure 4.2 above. Level-1 Recovery Guidelines - Relocation Finding a site for re-location during a disaster that has the necessary telecommunication connections can be the major limiting factor to quick resumption of business. In the event of a widespread emergency, a suitable site might be unavailable for weeks. BenTec will address this vulnerability by implementing an emergency distributed service center approach. This approach will allow enrollment specialist to work from any internet equipped location as long as the employee has a telephone and computer. BenTec web servers and PBX servers will need to be configured in an emergency location (such as the remote Tulsa Connection location or one of the remote BenTec Divisions). 3

15 CUSTOMER LOCATION EMERGENCY LOCATION RACK Telephone POTS Public Phone Providers MicroTik Firewall PBX Systems Computer Public ISP Providers Catalyst 2900 Switch Telephone VoIP SERVICE CENTER AGENT WORKS FROM HOME OR ANY LOCATION WITH INTERNET ACCESS SRV-SQL01 SRV-Web01 SRV-EXCH01 SRV-DC01 SRV-WS01 Computer Distributed Service Center Figure 4-3 Figure 4-3 illustrates the distributed service center approach envisioned for BenTec. Depending on the specifics of any emergency requiring the emergency operations we may use a combination of approaches for operation. If we are able to locate and distribute VoIP phones for our agents, then those agents could operate from their homes or any other suitable work site provided that that work site had Internet access. Each employee would simple take their IP Phones and login to the emergency rack PBX via VoIP in a fashion similar to normal day-to-day operations. However, if sufficient VoIP phones could not be located, we will update our enrollment web site to request a callers name and phone number. The web site would then notify an available CSR by to call the employee's number. Once communication between employee and CSR was established, the enrollment would continue as normal. The advantages to this distributed system are that: 1. Costly and perhaps unavailable telecommunications lines and PBX switches are not needed and 2. Enrollment Specialists can be located anyplace with adequate internet access -- even at home. The Enrollment Specialist only needs a computer, a phone line, and broadband Internet connection. Perhaps the biggest advantage is the speed at which this solution can be implemented. Switching to a distributed service center could be accomplished within hours with proper preplanning and pre-positioned equipment. Currently we have two sets of hardware. Physical servers located remotely at Tulsa Connect and virtualized servers located locally at Sapulpa. Local SQL servers and Exchange servers are continuously updated from the remote server via replication. 4

16 Level-2 Incident recovery guideline In Place Recovery A level-2 incident response is mainly one of minimizing damage done to BenTec systems and restoring operations as soon as possible. In order for recovery to occur, as a minimum for over-the-phone enrollments, telecommunications and the local area network must be made operational. If these tasks cannot be accomplished in a reasonable time, Management would implement the distributed service center approach as outlined in Level-1 Incident response guideline. During a level-2 incident we will take appropriate action to minimize damage to our facilities and systems. If appropriate, power will be shut off to equipment. Onsite backup tapes and disks will be removed and sent offsite immediately. Any equipment that might suffer water damage will be covered with plastic sheets as soon as power has been removed. Steps will be taken to ensure personnel safety. After a level-2 incident, vendors will be contacted and replacement systems located, priced, and delivery times determined as quickly as possible. Replacement equipment times and costs will be communicated to the IT Emergency Recovery Team as soon as cost and lead times are available and will be maintained and updated by the DNO during recovery operations. Plans for a level-2 incident will include complete documentation of all systems, copies of all installed software and regular backups. Copies of all documentation and backups will be maintained offsite in an easily accessible location (the homes of Dwight Pierce and Allen Beaird). Level-3 Incident recovery guideline In Place Recovery Level-3 incidents differ from level-2 and level-4 incidents in terms of damage done and expected recovery times. Level-3 incident recovery will always be done in place and completed in 72 hours. The following steps will be taken for level-3 incidents as appropriate: a. In the event of commercial power failure (system is on internal batteries only), systems will be shut down in an orderly fashion d as soon as possible to avoid data loss. (Recent Examples: NONE) b. In the event of equipment failure, the VP for IT in conjunction with the Network Engineer and outside consultants as required will ascertain the amount of replacement equipment required, costs, and lead times. The VP for IT will notify the president as soon as cost and lead times are available. (Recent Examples: PBX RAID failure) c. In the case of viral attacks, the network will be sub-divided and cleared section by section. Anti Virus vendors will be contacted and appropriate actions taken. (Recent Examples: NONE) 5

17 d. In the case of denial of service attacks, the Network Engineer will notify ISPs and failover to an alternative IP range. The Network Engineer will continue to monitor and review the situation. In extreme cases we may terminate internet access until the source of the attacks can be located and defeated. (Recent Examples: NONE) e. In the case of loss of internet or telecommunications lines, notify 3 rd party suppliers as quickly as possible. (Currently we have redundant ISP providers both at Sapulpa and at Tulsa Connect - We have Voice lines available to us from AT&T and via VoIP from CommPartners Connect. We also have the option of redirecting our phone numbers to one of the remote BenTec divisions in case of an extreme emergency.). Recovery plans for a level-3 incident will include reliable backup procedures antivirus programs and updates, maintaining OS and software updates, and staying abreast of security advisories pertaining to installed hardware and software. Level-4 Incident recovery guideline In Place Recovery Level-4 incidents are important for two reasons. First, careful analysis of the incident may provide means to avoid level-2 or level-3 incidents from occurring. Second, level-4 incidents may provide important vendor timelines that should be incorporated into the recovery procedures. Although level-4 incidents by definition do not stop production, they do pose a threat to the safety of company computer resources and data assets. A virus that is not automatically dealt with by the anti virus program, an attack on a web server, unauthorized disclosure of client data, disk driver errors, or hardware overheating are all examples of a potential level-4 incidents. Level-4 recovery procedures must include procedures to maintain good backups and up to date antivirus programs, keep all systems upgraded, and staying abreast of all security advisories pertaining to installed hardware and software. 6

18 ATTACHMENT 1 EMERGENCY CONTACT LIST Sr. Officer Contacts Work Cell Dwight Pierce Div. 1 - CCH - Bob Heffernan Div. 2 - BSG - Phyllis Stoll Div. 3 - WBS - Hunter Whittington Div. 4 - GAR - Marvin McKinney Div. 5 - BSC - Randy Wood Div. 6 - AWI - Jim Barrett Div. 7 - RCI - John Richter Div. 8 - VRG - Ryan Rich Div. 9 - PA - Josh Spivak IT Personnel Work Cell Allen Beaird Brad Armstrong Maria Gee Shawn Norton Briggs Hinton Sapulpa Police Dept Sapulpa Fire Dept Creek County Ambulance Creek County Sheriff St. John-Sapulpa Hospital